Reading #26: Picturephone: A Game for Sketch Data Capture (2009)

by Gabe Johnson

Comments: Chris

This paper talks about Picturephone, a sketching game that was mentioned in a previous paper. Picturephone is a game inspired by the telephone children's game in which a message is passed down a line of people by whispering one person at a time. The message usually changes drastically and in some cases can wind up being totally different from the original message.

For example, the original message might be, "Marty took a drink of water," and after one pass might change to "Marty drank some water." Eventually it might become "Marty took a drink of soda" and inevitably will become "Marty was arrested for arson while not wearing pants" after one or two more passes.

Picturephone has the first user sketch the story. The second person then creates a new story based on the sketch. A third person then sketches the new story, and a third player sketches this story. The sketches are then compared and graded somehow. The sketches are also labeled in the process.


This looks fun. I would like to play it. Who wants to play it with me? I would like to see what would happen if the sketch/story was repeated 20 times.

Reading #25: A descriptor for large scale image retrieval based on sketched feature lines (2009)

by mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, Marc Alexa

Comments: Francisco

This paper deals with sketched based image search, in which the images to be searched for are sketched by the user. The authors use a few asymmetric descriptors that match the main features of a stroke with objects in the images. They tested with a set of 1.5 million pictures of outdoor sceneries. They tested 27 sketches, and the results look similar to the queries. They illustrate several example sketches and top results.


I have thought about searching using sketch queries. We don't even have image search (where we input a normal image, not even a sketch) widely available. Hopefully that area and sketch searching will become widespread soon, as it is very useful.

Reading #24: Games For Sketch Data Collection (2009)

by Gabe Johnson and Ellen Yi-Luen Do

Comments: Chris

This paper discusses the use of games to collect data, specifically for sketch. The authors wish to understand "how people make and describe hand-made drawings." The paper describes two games: Picturephone (like the telephone game) and Stellasketch. Picturephone gives a description of a sketch for player 1 to draw, and player 2 must then describe the sketch that player 1 drew. More players can then draw the sketch based on player 2's text instead of the original text. This is fun. Stellasketch is like Pictionary. One player draws something based on a clue, and other players privately label the sketch. The point of using the games is to hopefully collect much more data for sketch research than the typical handful of users.


This is a cool idea. I actually want to play these games right now (I want to be in a user study). This is a very cool, free way to reward users for taking the study. Work is nice if it doesn't feel like work.

Reading #23: InkSeine: In Situ Search for Active Note Taking (2007)

by Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Ed Cutrell, Michael Shilman, and Desney Tan

Comments: George

This paper presents a note-taking application that helps the user create references by incorporating searching and gathering content. While taking notes, the user can perform searches by circling some previously written text. The actions are performed by pen gestures. They can add reference icons to a sketch, which appear as normal desktop icons and can link to files or URLs. 5 users tested the system.


For sketching to replace the mouse and keyboard, many unique applications such as this need to be invented and developed. Sketching introduces some interface navigation problems, which can be frustrating to the user, especially during sensitive applications like note taking. We need many novel solutions such as this.

Reading #22: Plushie: An Interactive Design System for Plush Toys

by Yuki Mori and Takeo Igarashi

Comments: Chris

This is a follow up of Teddy. This paper presents a system that can generate patterns that can be used to create plush toys. The program creates 3d models from 2d sketch inputs and finds a good pattern that can be printed and applied to fabric in such a way that the resulting plush looks like the 3d model. The program incorporates similar 3d conversion as Teddy, and it also includes some editing tools, such as cut, part creation, and seam insertion and deletion.


This is definitely unique. Once again, I like the conversion of the 2d stroke to a 3d shape. I am wondering how complex you can make the shape, since it seems like you have to make a big blob and carve away parts of it maybe. Also, I took a computer-aided sculpting course this semester, and I could have used this to make one of my sculptures (too bad I didn't read this paper when we were doing that project).

Reading #21: Teddy: A Sketching Interface for 3D Freeform Design (1999)

by Takeo Igarashi, Satoshi Matsuoka, Hidehiko Tanaka

Comments: Chris

This paper showcases a program that takes sketched objects and constructs 3d models from 2d sketches. Basically, it makes wider areas thicker. Once a sufficient stroke is drawn, a 3d model is generated and can be rotated in 3d space and drawn on in different orientations to create different 3d features. The interface supports cutting and erasing geometry..

This program is intended to open up new areas of 3d design and to contribute to the rapid prototyping stage of design. Some of the tools include create, bend, paint, extrude, and smooth. General positive feedback was recorded from some people.


I am always intrigued by the conversion of 2d drawings to 3d. I hadn't seen this paper before, and it is pretty interesting. This paper doesn't really do much recognition, and there are many possibilities for expansion by including sketch recognition techniques.

Reading #20: MathPad 2 : A System for the Creation and Exploration of Mathematical Sketches

by Joseph J LaViola Jr and Robert C Zeleznik

Comments: Marty

This paper presents a cool math sketch program that can do many things and simulate some math stuff... The cool thing about this is the interface. It is a big graph paper you can write lots of different equations, systems of equations, and diagrams. Gestures are used to help perform segmentation and identification. It can also generate graphs and plots. 12 people or so tested the system and gave some positive feedback. The interface is easy to use and the authors want to be able to include even more stuff and things... and bits.


This seems like a sketch interface for matlab or something. It can simulate many mathy things and is a general purpose math tool. I don't know what its current state is, since this is a fairly old paper.

Reading #19: Diagram Structure Recognition by Bayesian Conditional Random Fields

by Yuan Qi, Martin Szummer, Thomas P Minka

Comments: Sam

The authors use Bayesian Conditional random Fields (BCRFs) to analyze sketched diagrams to gather contextual information to better recognize complex diagrams.. which are complex. There are many equations which are boggling my mind at this time. 17 users drawer some diagrams, and the algorithms achieved high recognition rates in the low to high 90s.


This is a cool approach for recognizing large, context sensitive drawings, of which diagrams are excellent examples. The mathy approach works pretty well.

Reading #18: Spatial Recognition and Grouping of Text and Graphics (2004)

by Michael Shilman and Paul Viola

Comments: Marty

This paper discusses grouping and recognition of sketch diagrams. They take a big canvas and identify many different symbols in it. This is cool. You can draw the stuff in any order and it will segment out each symbol. This is hard and probably the main contribution of the paper, but I am sleepy. The grouping had 99% accuracy.. sweet man. Also, the recognition and the grouping were 97% accuracy.


I am dealing with the segmentation problem in hand gestures, and I can relate to this problem. It is nice to have this problem solved with a high accuracy. This makes more complicated sketch interaction possible.

Reading #17: Distinguishing Text from Graphics in On-line Handwritten Ink (2004)

by Christopher M Bishop, Groffrey E Hinton

Comments: Marty

This is an earlier text vs shape paper. It uses both stroke features, gaps, and time data to help separate text from other strokes. They used HMMs for recognition. They collected data from some dudes. The dudes drew some stuff, whatever they wanted, as long as the sketches contained some text elements and some non-text strokes. Recognition results were mixed, with some groups getting in the mid 90s and some in the mid 70s.


Shape v Text is a hard problem, and there are many solutions to solve it. I don't like this gaps and time solution, however. It just doesn't make sense to me... I think I would like entropy better or simply visual approaches. Also, I think we can combine gestures into the mix to denote text. blah blah blah

Reading #16: An Efficient Graph-Based Symbol Recognizer (2006)

by WeeSan Lee, Levent Burak Kara, Thomas F Stahovich

Comments: Ozgur

This paper takes a graph-based approach to sketch (symbol) recognition and explored several graph matching techniques. They compute many error metrics for matching graphs and represent symbols using graphs. They collected several types of symbols from some users and ran their 4 matching algorithms on the data, getting results in the mid- to high-90s for most algorithms.


Some symbols can naturally be represented as graphs. We have seen that graph matching can yield high accuracy for appropriate shapes. I think that this could be one component of a good general purpose recognizer.

Reading #15: An Image-Based, Trainable Symbol Recognizer for Hand-drawn Sketches (2005)

by Levent Burak Kara, Thomas F Stahovich

Comments: Jonathan

This paper takes an image-based approach to sketch recognition using an ensemble classifier consisting of four different classifiers. They want a system that can recognize sketches very fast (real time for interaction) and that is also rotation invariant (using a fast polar coordinate technique).

This paper really focuses on the sketch interface and making it an attractive alternative to paper. To be a viable alternative, interaction (and therefore recognition) must be able to occur in real-time with no interruptions to the user. They also want to be able to recognize many shapes as well as "sketchy" shapes.

They used 20 shapes collected from some users. They achieved recognition rates in the mid to high 90s.


This is a good paper for an introduction to image-based approaches. It is also useful for understanding sketch interfaces. Considering the year (2005), the sketches were recognized very quickly and would be recognized even faster on today's machines.

Reading #14: Using Entropy to Distinguish Shape Versus Text in Hand-Drawn Diagrams (2009)

by Akshay Bhat and Tracy Hammond

Comments: Ayden

The authors propose that entropy rates are higher for text strokes than for non-text strokes and attempt to separate shapes from text using this idea. They achieved a 92% recognition rate. They define entropy, calculate entropy for all letters of the alphabet, and perform classification on collected sketches.


I agree that text shapes have high entropy, and it is interesting to note that this approach has not been taken earlier in the history of sketch recognition. Obviously some primitive shapes, such as circle and rectangle, will have lower entropy than text, but what about helixes or more complex shapes? This might be good in some diagramming domains.

Reading #13: Ink Features for Diagram Recognition

by Rachel Patel, Beryl Plimmer, John Grundy, and Ross Ihaka

Comments: Jianjie

This paper aims to perform more accurate diagram recognition by performing a statistical analysis of features used for recognizing various diagram components from sketched samples. This is pretty much an introduction to some of the important concepts in sketch recognition and illustrates some general approaches to sketch recognition. The paper particularly focuses on shape vs. text.

The authors took 46 features grouped into 7 categories. They collected some sketches from 26 participants which contained some diagram elements and text. They used a statistical partitioning technique to find which features can best split the strokes into shape or text strokes and then constructed decision trees with significant features toward the start of the tree.

They tested their methods with some existing shape v text systems and found some interesting results...


Sketch recognition still remains in its infancy despite its age, and formal analyses like this are important to help us understand the processes and achieve greater recognition performance. This work seems kind of inconclusive, however, and I didn't understand the results very well.