Stuff.

Reading #26: Picturephone: A Game for Sketch Data Capture (2009)

2010-12-15T19:26:00.006-06:00

by Gabe Johnson

Comments: Chris

This paper talks about Picturephone, a sketching game that was mentioned in a previous paper. Picturephone is a game inspired by the telephone children's game in which a message is passed down a line of people by whispering one person at a time. The message usually changes drastically and in some cases can wind up being totally different from the original message.

For example, the original message might be, "Marty took a drink of water," and after one pass might change to "Marty drank some water." Eventually it might become "Marty took a drink of soda" and inevitably will become "Marty was arrested for arson while not wearing pants" after one or two more passes.

Picturephone has the first user sketch the story. The second person then creates a new story based on the sketch. A third person then sketches the new story, and a third player sketches this story. The sketches are then compared and graded somehow. The sketches are also labeled in the process.

_______

This looks fun. I would like to play it. Who wants to play it with me? I would like to see what would happen if the sketch/story was repeated 20 times.

Reading #25: A descriptor for large scale image retrieval based on sketched feature lines (2009)

2010-12-14T13:45:00.004-06:00

by mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, Marc Alexa

Comments: Francisco

This paper deals with sketched based image search, in which the images to be searched for are sketched by the user. The authors use a few asymmetric descriptors that match the main features of a stroke with objects in the images. They tested with a set of 1.5 million pictures of outdoor sceneries. They tested 27 sketches, and the results look similar to the queries. They illustrate several example sketches and top results.

_______

I have thought about searching using sketch queries. We don't even have image search (where we input a normal image, not even a sketch) widely available. Hopefully that area and sketch searching will become widespread soon, as it is very useful.

Reading #24: Games For Sketch Data Collection (2009)

2010-12-14T13:31:00.006-06:00

by Gabe Johnson and Ellen Yi-Luen Do

Comments: Chris

This paper discusses the use of games to collect data, specifically for sketch. The authors wish to understand "how people make and describe hand-made drawings." The paper describes two games: Picturephone (like the telephone game) and Stellasketch. Picturephone gives a description of a sketch for player 1 to draw, and player 2 must then describe the sketch that player 1 drew. More players can then draw the sketch based on player 2's text instead of the original text. This is fun. Stellasketch is like Pictionary. One player draws something based on a clue, and other players privately label the sketch. The point of using the games is to hopefully collect much more data for sketch research than the typical handful of users.

_______

This is a cool idea. I actually want to play these games right now (I want to be in a user study). This is a very cool, free way to reward users for taking the study. Work is nice if it doesn't feel like work.

Reading #23: InkSeine: In Situ Search for Active Note Taking (2007)

2010-12-14T13:19:00.006-06:00

by Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Ed Cutrell, Michael Shilman, and Desney Tan

Comments: George

This paper presents a note-taking application that helps the user create references by incorporating searching and gathering content. While taking notes, the user can perform searches by circling some previously written text. The actions are performed by pen gestures. They can add reference icons to a sketch, which appear as normal desktop icons and can link to files or URLs. 5 users tested the system.

_______

For sketching to replace the mouse and keyboard, many unique applications such as this need to be invented and developed. Sketching introduces some interface navigation problems, which can be frustrating to the user, especially during sensitive applications like note taking. We need many novel solutions such as this.

Reading #22: Plushie: An Interactive Design System for Plush Toys

2010-12-14T13:04:00.003-06:00

by Yuki Mori and Takeo Igarashi

Comments: Chris

This is a follow up of Teddy. This paper presents a system that can generate patterns that can be used to create plush toys. The program creates 3d models from 2d sketch inputs and finds a good pattern that can be printed and applied to fabric in such a way that the resulting plush looks like the 3d model. The program incorporates similar 3d conversion as Teddy, and it also includes some editing tools, such as cut, part creation, and seam insertion and deletion.

_______

This is definitely unique. Once again, I like the conversion of the 2d stroke to a 3d shape. I am wondering how complex you can make the shape, since it seems like you have to make a big blob and carve away parts of it maybe. Also, I took a computer-aided sculpting course this semester, and I could have used this to make one of my sculptures (too bad I didn't read this paper when we were doing that project).

Reading #21: Teddy: A Sketching Interface for 3D Freeform Design (1999)

2010-12-14T12:54:00.006-06:00

by Takeo Igarashi, Satoshi Matsuoka, Hidehiko Tanaka

Comments: Chris

This paper showcases a program that takes sketched objects and constructs 3d models from 2d sketches. Basically, it makes wider areas thicker. Once a sufficient stroke is drawn, a 3d model is generated and can be rotated in 3d space and drawn on in different orientations to create different 3d features. The interface supports cutting and erasing geometry..

This program is intended to open up new areas of 3d design and to contribute to the rapid prototyping stage of design. Some of the tools include create, bend, paint, extrude, and smooth. General positive feedback was recorded from some people.

_______

I am always intrigued by the conversion of 2d drawings to 3d. I hadn't seen this paper before, and it is pretty interesting. This paper doesn't really do much recognition, and there are many possibilities for expansion by including sketch recognition techniques.

Reading #20: MathPad 2 : A System for the Creation and Exploration of Mathematical Sketches

2010-12-14T06:14:00.004-06:00

by Joseph J LaViola Jr and Robert C Zeleznik

Comments: Marty

This paper presents a cool math sketch program that can do many things and simulate some math stuff... The cool thing about this is the interface. It is a big graph paper you can write lots of different equations, systems of equations, and diagrams. Gestures are used to help perform segmentation and identification. It can also generate graphs and plots. 12 people or so tested the system and gave some positive feedback. The interface is easy to use and the authors want to be able to include even more stuff and things... and bits.

_______

This seems like a sketch interface for matlab or something. It can simulate many mathy things and is a general purpose math tool. I don't know what its current state is, since this is a fairly old paper.

Reading #19: Diagram Structure Recognition by Bayesian Conditional Random Fields

2010-12-14T06:06:00.002-06:00

by Yuan Qi, Martin Szummer, Thomas P Minka

Comments: Sam

The authors use Bayesian Conditional random Fields (BCRFs) to analyze sketched diagrams to gather contextual information to better recognize complex diagrams.. which are complex. There are many equations which are boggling my mind at this time. 17 users drawer some diagrams, and the algorithms achieved high recognition rates in the low to high 90s.

_______

This is a cool approach for recognizing large, context sensitive drawings, of which diagrams are excellent examples. The mathy approach works pretty well.

Reading #18: Spatial Recognition and Grouping of Text and Graphics (2004)

2010-12-14T05:56:00.002-06:00

by Michael Shilman and Paul Viola

Comments: Marty

This paper discusses grouping and recognition of sketch diagrams. They take a big canvas and identify many different symbols in it. This is cool. You can draw the stuff in any order and it will segment out each symbol. This is hard and probably the main contribution of the paper, but I am sleepy. The grouping had 99% accuracy.. sweet man. Also, the recognition and the grouping were 97% accuracy.

_______

I am dealing with the segmentation problem in hand gestures, and I can relate to this problem. It is nice to have this problem solved with a high accuracy. This makes more complicated sketch interaction possible.

Reading #17: Distinguishing Text from Graphics in On-line Handwritten Ink (2004)

2010-12-14T05:48:00.003-06:00

by Christopher M Bishop, Groffrey E Hinton

Comments: Marty

This is an earlier text vs shape paper. It uses both stroke features, gaps, and time data to help separate text from other strokes. They used HMMs for recognition. They collected data from some dudes. The dudes drew some stuff, whatever they wanted, as long as the sketches contained some text elements and some non-text strokes. Recognition results were mixed, with some groups getting in the mid 90s and some in the mid 70s.

_______

Shape v Text is a hard problem, and there are many solutions to solve it. I don't like this gaps and time solution, however. It just doesn't make sense to me... I think I would like entropy better or simply visual approaches. Also, I think we can combine gestures into the mix to denote text. blah blah blah

Reading #16: An Efﬁcient Graph-Based Symbol Recognizer (2006)

2010-12-14T05:41:00.005-06:00

by WeeSan Lee, Levent Burak Kara, Thomas F Stahovich

Comments: Ozgur

This paper takes a graph-based approach to sketch (symbol) recognition and explored several graph matching techniques. They compute many error metrics for matching graphs and represent symbols using graphs. They collected several types of symbols from some users and ran their 4 matching algorithms on the data, getting results in the mid- to high-90s for most algorithms.

_______

Some symbols can naturally be represented as graphs. We have seen that graph matching can yield high accuracy for appropriate shapes. I think that this could be one component of a good general purpose recognizer.

Reading #15: An Image-Based, Trainable Symbol Recognizer for Hand-drawn Sketches (2005)

2010-12-14T05:28:00.008-06:00

by Levent Burak Kara, Thomas F Stahovich

Comments: Jonathan

This paper takes an image-based approach to sketch recognition using an ensemble classifier consisting of four different classifiers. They want a system that can recognize sketches very fast (real time for interaction) and that is also rotation invariant (using a fast polar coordinate technique).

This paper really focuses on the sketch interface and making it an attractive alternative to paper. To be a viable alternative, interaction (and therefore recognition) must be able to occur in real-time with no interruptions to the user. They also want to be able to recognize many shapes as well as "sketchy" shapes.

They used 20 shapes collected from some users. They achieved recognition rates in the mid to high 90s.

_______

This is a good paper for an introduction to image-based approaches. It is also useful for understanding sketch interfaces. Considering the year (2005), the sketches were recognized very quickly and would be recognized even faster on today's machines.

Reading #14: Using Entropy to Distinguish Shape Versus Text in Hand-Drawn Diagrams (2009)

2010-12-14T05:22:00.003-06:00

by Akshay Bhat and Tracy Hammond

Comments: Ayden

The authors propose that entropy rates are higher for text strokes than for non-text strokes and attempt to separate shapes from text using this idea. They achieved a 92% recognition rate. They define entropy, calculate entropy for all letters of the alphabet, and perform classification on collected sketches.

_______

I agree that text shapes have high entropy, and it is interesting to note that this approach has not been taken earlier in the history of sketch recognition. Obviously some primitive shapes, such as circle and rectangle, will have lower entropy than text, but what about helixes or more complex shapes? This might be good in some diagramming domains.

Reading #13: Ink Features for Diagram Recognition

2010-12-14T05:04:00.008-06:00

by Rachel Patel, Beryl Plimmer, John Grundy, and Ross Ihaka

Comments: Jianjie

This paper aims to perform more accurate diagram recognition by performing a statistical analysis of features used for recognizing various diagram components from sketched samples. This is pretty much an introduction to some of the important concepts in sketch recognition and illustrates some general approaches to sketch recognition. The paper particularly focuses on shape vs. text.

The authors took 46 features grouped into 7 categories. They collected some sketches from 26 participants which contained some diagram elements and text. They used a statistical partitioning technique to find which features can best split the strokes into shape or text strokes and then constructed decision trees with significant features toward the start of the tree.

They tested their methods with some existing shape v text systems and found some interesting results...

_______

Sketch recognition still remains in its infancy despite its age, and formal analyses like this are important to help us understand the processes and achieve greater recognition performance. This work seems kind of inconclusive, however, and I didn't understand the results very well.

Reading #12: Constellation Models for Sketch Recognition (2006)

2010-10-11T17:28:00.007-05:00

by D. Sharon and M. van de Panne (paper)

Comments: Francisco

A Constellation is a collection of objects arranged in a certain pattern, forming an image. Drawings can be imagined as constellations, especially with regards to common objects, such as the human face. Each face has the same features arranged in the same manner, with slight variances in size, shape, and location. However, several overall main relationships are always present: two eyes are on a horizontal line and are above the nose, for example.

The authors have used this concept for sketch recognition. Required drawing elements and their relative positions are set as features and classified using a maximum likelihood search. Many sketches were collected for training data, and common features were identified and labeled. 5 class values were defined, including faces, flowers, sailboats, airplanes, and characters.

_______

This is an interesting way to identify common objects within sketches. Since the authors use only 5 classes, I wonder if this technique will support many classes. Also, it would be interesting to see how detailed this can get to perform more intricate recognition, such as multiple types of faces, or even individuals.

Reading #11: LADDER, a sketching language for user interface developers (2007)

2010-10-11T16:53:00.015-05:00

by Tracy Hammond and Randall Davis (paper)

Comments: Jonathan

LADDER is a language to "describe how sketched diagrams in a domain are drawn, displayed, and edited." It is intended to help interface developers create sketch-based interfaces. LADDER is used to create shape descriptions. Shapes consist of components (such as lines), constraints (such as intersections), aliases, editing, and display properties. LADDER descriptions allow domain-specific definitions that can be used with domain-independent recognizers.

The paper gives many examples of shapes that can be modeled using LADDER. Such shapes include arrows and UML diagrams.

There are many predefined shapes (such as point, line, curve, ellipse), constraints (such as perpendicular, collinear, tangent, larger, acute), orientation-dependent constraints (such as horizontal, negative slope, above, centered below), editing methods (such as click, draw, encircle), and display methods (such as original strokes, ideal strokes, circle, rectangle, text, image).

Some shapes can be made up of certain numbers of segments, while some shapes can be made up of infinite segments.

When recognition occurs, primitive shapes are recognized first. Shapes are generated that contain the original strokes and their interpretations. Some shapes contain sub-shapes. Once primitive shapes are recognized, domain-specific shapes, using the domain descriptions, are recognized.

__________

LADDER is a nice tool for interface developers. I like that it allows complex shapes to be completely described using a programming language. However, I do see certain drawbacks, namely that each shape must be explicitly defined in LADDER. The paper mentions the capability to generate descriptions based on drawn examples, and I think this would be a great idea. It would be very tedious to define explicit shapes for large domains (COA...). I don't know if it has been implemented yet.

Reading #10: Graphical Input Through Machine Recognition of Sketches (1976)

2010-10-11T16:09:00.018-05:00

by Chrisopher F. Herot (paper)

Comments: Jonathan

This is an early sketch recognition system aiming to allow sketch input to computer programs. It contains a system called HUNCH that is used to recognize primitive sketches. It uses speed only to detect corners in a sketch. Curves were viewed "to be a special case of corners," and were modeled using b-splines. Speed was also used to determine how "careful" the user was, for instance faster strokes were less careful. This was used to help identify curves and decide whether to draw them as b-splines or just corners. The programs used to convert sketches to straight segments and curves were called STRAIT and CURVIT, respectively.

STRAIT and CURVIT did not always create the same interpretation as humans. They seemed to be user-dependent, as sketches were interpreted better for some users than for others. An improved method of straight segmenting was implemented, called STRAIN. It used a function of speed to determine which line endpoints to join (instead of a fixed distance, which is what STRAIT did).

Programs were also developed to detect overtraced lines and turn 2D sketches into 3D.

The paper discusses the importance of context in sketch recognition systems. The HUNCH system does not use context in its recognition schemes. For example, all its subroutines are always called in a fixed sequence and always perform recognition in the same way. However, similar strokes will probably be interpreted in different ways given their context.

The paper discusses an interactive system and examines the hierarchical structures of recognized sketches. It discusses various ways to tune the algorithms to work for a "truly interactive system."

__________

This is an early approach to sketch recognition, and it asks many questions as well as answers some. It can be rather dry and boring, but it does bring up some questions that are still relevant today and whose solutions can still be improved on.

For example, the latching problem: when should close endpoints be merged? It depends on the context, which is another contemporary issue. While I was working on my truss recognizer, I dealt with the latching problem when merging close line endpoints to form truss nodes. I used a distance computed based on stroke lengths, but that was obviously a poor choice.

This paper begins to explore machine learning techniques for interpreting sketches. It introduces many questions and proposes possible solutions to those questions by hypothesizing extensions to an existing primitive corner finding and beautification system. I believe the authors asked good and relevant questions, as we are still asking ourselves how to solve many of the same problems in better ways.

Reading #9: PaleoSketch: Accurate Primitive Sketch Recognition and Beautification (2008)

2010-10-11T15:39:00.013-05:00

by Brandon Paulson and Tracy Hammond (paper)

Comments: Jianjie

Paleosketch is a low-level sketch recognition system that can identify primitive shapes from single strokes. It is capable of recognizing Lines, Polylines, Circles, Ellipses, Arcs, Curves, Spirals, and Helixes.

Recognition occurs in three steps: pre-recognition, individual shape tests, and result ranking. Pre-recognition performs is essentially some pre-processing to make recognition easier. This includes removing duplicate points, feature graph generation (speed, curvature, etc) that can be used during recognition, tail removal, and two new features DCR and NDDE. NDDE is the normalized distance between direction extremes, and it is calculated by calculating the stroke length between the point with the highest direction value and the lowest direction value (direction value = change in y over change in x). DCR is the direction change ratio, and it is calculated by taking the "maximum change in direction divided by the average change in direction."

Individual tests are done for lines, polylines, circles, ellipses, arcs, curves, spirals, and helixes. Each test compares some stroke features and calculates the confidence that a stroke is that shape. When all tests are complete, the results are ordered using properties of the corner finding algorithm.

To test this system, data was collected from 10 users. This data was run on Paleosketch and compared with some features disabled as well as with Sezgin's algorithm. Paleosketch achieved a recognition rate of 98.56% across all shapes.

__________

Paleosketch is a good primitive recognition algorithm, essentially combining ideas from previous work and adding in some new stroke features and result ranking. It performs much better than other algorithms we have covered so far, including Rubine's and Sezgin's algorithms. I have used Paleosketch some, and I have found it to be fairly accurate in practice. Its biggest drawback in my opinion is its speed. It runs fairly quick for smaller strokes and collections of a few strokes, but when strokes become long or there are many strokes being analyzed Paleosketch slows down and can take up to a second or more to execute. This makes it a poor choice for online recognition systems. However, it is a great improvement over previous systems.

Reading #8: A Lightweight Multistroke Recognizer for User Interface Prototypes (2010)

2010-09-27T21:12:00.021-05:00

by Lisa Anthony and Jacob Wobbrock (paper)

Comments: Danielle

This paper presents a multi-stroke extension to Wobbrock's $1 single-stroke recognizer, called $N. This paper presents the same approach to incorporating recognition into any program, just as was done with $1. The paper contains the pseudo-code for the algorithm that fits on less than one page.

$N allows multiple strokes by connecting the strokes into one long stroke. The many possible permutations that could occur are generated when new templates are created. Resampling, scaling, and rotation all occur as in $1, with some adjustments to enhance the capabilities of $1.

In addition to providing support for multiple strokes, $N addresses some problems with the $1 algorithm. $N allows 1-dimensional gestures (such as lines) by calculating the ratio of the sides of the bounding box and using a threshold to determine if the gesture is 1D or not. $N allows rotation in the gestures as well. If rotation is desired, the gestures are rotated to the template angles instead of 0. Finally, $N provides optimizations to increase recognition speed. Templates are only compared if the starting angle of the stroke is "about the same." Also, the developer can choose whether to limit the number of strokes in gestures that will probably always have a set number of strokes (for example, + and = gestures).

The drawbacks to $N are scale invariance, using more strokes than in the template, collision of gestures, and large numbers of templates.

To test the algorithm, 40 middle and high school students used a sketch input program to input simple algebraic equations.

__________

This paper is nice in the same ways $1 was nice. It can allow any level of programmer to implement pen gesture interaction in any program. Because $N is overall better than $1 (despite having slightly less accuracy for 1-stroke gestures) it is good.

There are more possibilities with $N than with $1. There are more ways it can be used, and more ways it can be enhanced as well. It also seems that $N could easily be extended to 3D (of course with more computation required) to use for hand gestures or something similar.

Reading #7: Sketch Based Interfaces: Early Processing for Sketch Understanding (2001)

2010-09-15T20:04:00.023-05:00

by Tevfik Metin Sezgin, Thomas Stahovich, and Randall Davis (paper)

Comments: Marty

This paper describes a system that analyzes a sketch after it is drawn and analyzes what was drawn instead of how the sketch was drawn. It also allows multiple strokes in a sketch, something we have yet to discuss in this course.

One of the main features of Sezgin's system is its vertex detection, or corner finding, implementation. He uses a combination of speed and curvature to detect line corners. After segmentation, the straight edges of a sketch are stored as a polyline.

The second feature is curve handling. The system is able to model curves as Bézier curves by approximating the control points using a least squares method.

The system beautifies the drawn strokes "primarily to make it look as intended." Lines meant to be parallel are made parallel (also similarly with perpendicular lines), straight lines are made straight, and curves are rendered properly.

Finally, the system performs primitive object recognition. It uses simple geometric constraints to recognize ovals, circles, rectangles, and squares.

A user study was done to test the usability of the program and compare it to a tool-based drawing program. The participants found the author's system easier to use since any shape can be instantly drawn without having to select the corresponding tool. The author's report an accuracy of 96% when approximating drawn shapes from a set of 10 figures.

__________

This paper is an early beautification paper that turns sketched drawings into actual technical drawings such as schematics and diagrams. It does this by applying corner and curve finding to determine the user's intended sketch. I think this paper helps show that sketching can be a superior method of input than traditional menu and tool bar based drawing programs. Such interfaces were rare then, and still are now, and hopefully we can build upon this to help popularize sketch-based interfaces.

Reading #6: Protractor: A Fast and Accurate Gesture Recognizer (2010)

2010-09-14T10:27:00.007-05:00

by Yang Li (paper)

Comments: Wenzhe

Protractor is a modified $1 algorithm. The enhancements include support for up to 8 directions of rotation, scale invariance, and speed.

Protractor does the resampling as $1 does, and it uses N=16 for the number of points ($1 used 64 in its testing). Rotation invariance can be toggled on or off. If the gesture is to be rotation-independent, Protractor will rotate around the centroid until the indicative angle is 0, just as $1 does. If rotation is enabled, it rotates the indicative angle to one of 8 equidistant angles. Protractor does not scale the strokes as $1 does, so it is scale-invariant. The rotation adjustment step is also modified. Instead of taking an iterative approach to finding the optimal orientation, an angle is calculated that is close to the optimal angle.

Because of these modifications, Protractor performs significantly faster than $1 as the number of training examples increases. The recognition rates are not significantly different from $1. Because of the speed enhancements, Protractor is ideally suited for mobile device applications.

__________

I like this extension of the $1 algorithm. It sounds like it isn't much more difficult to implement that $1, and the speed enhancements without sacrificing accuracy are nice. It is nice to be able to specify orientation-dependent gestures. This, along with the scale-invariance, can help expand the limited 16-gesture set used by the $1 paper. The paper did show us a 26 gesture class, and Protractor did perform significantly better on that than $1 did in that case.

Reading #5: Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes (2007)

2010-09-14T07:46:00.005-05:00

by Jacob Wobbrock, Andrew Wilson, and Yang Li (paper)

Comments: George

This paper describes the $1 gesture recognizer. This sketch/gesture recognition algorithm is intended to be a simple, easy to program algorithm that can be implemented anywhere. This hopefully would allow gestures to be incorporated into rapid prototyped interfaces that otherwise might not have been able to use gesture input. This is because most user interface designers and programmers don't have the necessary knowledge or skills to be able to implement complex recognition algorithms, and current recognition toolkits are not everywhere in any language, especially in many environments human-computer interaction experts might use.

The authors describe the algorithm in 4 parts: point resampling, indicative angle rotation, scaling and translation, and finding the optimal angle for best score. These transformations applied to each input stroke allow them to easily match up to a few template strokes for each gesture. The recognition result is the template gesture with the smallest Euclidean distance to the input stroke.

The $1 algorithm is compared to the DTW and Rubine algorithms, and it is found to compete well against them, achieving high recognition rates and recognition speed. The $1 algorithm pseudo code is given as well to aid programmers.

__________

This paper is very clearly written and the $1 algorithm is indeed very simple. I find it interesting that such a simple, almost naive, approach can perform very well if executed intelligently. It is easy to imagine improvements and how to add more recognition capabilities to this algorithm, such as rotation-dependent or time-dependent gestures.

Reading #4: Sketchpad: A Man-Machine Graphical Communication System (1963)

2010-09-08T21:07:00.011-05:00

by Ivan E. Sutherland (paper)

Comments: Jonathan

This paper presents the initial sketch-based interaction work of Ivan Sutherland. This was one of the first systems to use a pen to draw on a screen, ushering in a new form of human-computer interaction.

To use the system, the user has a set of buttons and switches to activate certain modes and tools, such as a line tool or a delete mode. When the desired settings are set, interaction using the pen accomplishes the desired task. It is important to note that the pen does not perform any free-form drawings, but rather creates geometry using only pre-defined tools or performs commands using pointing or dragging. This makes it more like a CAD system that uses a pen for input (note that the mouse did not exist at the time of this work).

The paper shows its age by emphasizing things like the data structures and memory usage as well as generic representations of sketch elements. A "light pen" is used as the input device.

Most of the paper details the various constraints and tools and how they were implemented using non-procedural object oriented methods, all of which were new ideas (as discussed in this video).

__________

This paper introduced many new ideas about human-computer interaction, graphical displays, and programming. It was the first of its kind in almost every aspect. It is hard to appreciate it now without reading comments from many years ago. Much of the paper seems trivial to implement using our current software development languages and tools. I found it interesting that many ideas were introduced back in 1962 that are still active, and hard, research problems today (such as recognizing artistic drawings and electrical schematics).

Reading #3: "Those Look Similar!" Issues in Automating Gesture Design Advice (2001)

2010-09-07T10:34:00.011-05:00

by Long, Landay, and Rowe

Comments: Sam

This paper presents the quill gesture design tool that is aimed at helping developers create pen gesture-based interfaces. The quill software gives advice to the developers if multiple gestures might be ambiguous to the computer or visually similar to people.

The authors conducted some experiments to determine what kinds of gestures are perceived as similar by people by having a few hundred participants judge a large number of gestures and pick the most complex ones. They then developed an algorithm for predicting gesture similarity.

Interface designers use quill to input gestures for their interfaces. quill uses the similarity algorithm and Rubine's methods to give feedback to the users and train and recognize the gestures. The paper talks in detail about challenges related to giving advice such as how, when, and where advice is displayed in addition to what advice is displayed.

The authors conclude that the quill system, while it could use many refinements and improvements, is a good start and can possible inspire other advice-giving systems for gesture-based interfaces.

__________

I can appreciate the assistance given to developers relating to gesture definitions. There still are not many systems that can do this, especially with 3D hand gestures. I have run into issues in my own research where two gestures I didn't think were similar actually were, and it can be a pain to re-define gestures, especially if you discover the similarity after a large gesture set has been defined, which can make it difficult to think of a new unique gesture. I would really appreciate more development of these tools for 2D and 3D gestures.

Reading #2: Specifying Gestures by Example (1991)

2010-09-05T20:40:00.017-05:00

by Dean Rubine (paper)

Comments: Danielle

This paper presents Rubine's gesture-recognition algorithm and his implementation of a program that doesn't require a hand-coded recognizer. His goal is to increase the adoption of sketch-based gesture recognition in user interfaces by making it easier to integrate recognition by providing example gestures fed into a learning algorithm rather than hand-coding the recognizer.

Rubine has implemented a gestural drawing program in which simple single-stroke gestures are used to create and manipulate a drawing. Example gestures include rectangle creation, ellipse creation, copy, rotate-scale, and delete. The user of the program is able to add new gesture examples to aid recognition as well as modify the structure of each gesture.

He presents his simple gesture recognition algorithm, which assumes stroke segmentation is already taken care of. For the stroke drawn as the gesture, 13 features are computed. Rubine states that these 13 features are capable of recognizing many gestures, but fail in some cases. Once the features are calculated, they are input to a linear classifier that gives the class name of the stroke. He discusses how the classifier is trained, which is basically the standard method of training a linear classifier.

The classifier always gives one of the gestures from all gesture classes. A probability function is used to determine the probability that the gesture was classified correctly, and if that value falls below a threshold, the classification is rejected, as the gesture is ambiguous. He also rejects gestures based on the number of standard deviations of the gesture from the mean of the classification gesture class.

Rubine says his methods perform well in practice using 10 different gesture sets. He reports recognition rates in the mid to high 90s for varying numbers of examples per class, gesture classes per set, and test gestures per class.

__________

This paper seems to be on the cutting edge of sketch recognition technology for its time. Indeed, the concepts presented in this field are still widely used and studied today. Very little work and very few non-hand-coded recognition applications existed in 1991. I was impressed by the high accuracy achieved on the gesture sets using the linear classifier, though the accuracy reporting didn't seem complete. I have seen other systems, such as in our lab, that can recognize much larger classes of data, and I am particularly interested in 3D extensions of this method as well as other classification algorithms I have been brainstorming, which I look forward to implementing.

Reading #1: Gesture Recognition (2010)

2010-09-02T11:16:00.013-05:00

comments: Chris

Tracy Hammond (paper)

This paper is a summary of some well-known gesture recognition techniques in sketch recognition. It begins with a presentation of Rubine's feature-based algorithm. It describes the 13 features and gives some examples of how they are used as well as some illustrations to help understand the features. It briefly touches on the training and recognition system Rubine used, but doesn't go into much detail. Long's 22-feature extension of Rubine's algorithm is then presented in the same way. It presents Long's extra 11 features. Finally, Wobbrock's $1 algorithm is described.

__________

This is a pretty good summary of these well-known sketch recognition methods. I think it summarizes the key points of each approach and would allow the implementation of each method without much trouble. It could probably make up part of a handbook/reference guide. There were many grammatical errors/typos and some missing figures and details, however.

Homework #1: CSCE 624 Introduction

2010-09-01T15:19:00.009-05:00

picture of myself

dalogsdon gmail

2nd year Master of Computer Science

I am taking sketch recognition to gain a deeper knowledge of current work in the field. I have a creative and artistic background in addition to computer science, so I feel I might have a unique viewpoint to the issues we will discuss in the class.

Ten years from now, I expect to know what the next big technological advancement in computer science was. As an undergraduate, I rather enjoyed my computer-human interaction, software engineering, figure drawing, painting, and photography courses. My favorite movies are action/adventures and comedies. If I could travel back in time, I would not meet anyone for fear of destroying my existence, but I might go to my parents house and peek in the windows to see myself as a baby. It is perhaps interesting to note that 30 of my spinal vertebrae are fused.

Recent Developments and Applications of Haptic Devices

2010-05-13T16:41:00.002-05:00

paper

Comments: Manoj, Franck

This paper basically collects all the major haptic devices and technologies for comparison. It separates various input devices by degrees of freedom. It also presents some glove based devices and discusses vibration and hydraulics, among other things.

EyeDraw: enabling children with severe motor impairments to draw with their eyes

2010-05-13T13:00:00.005-05:00

paper

Comments: Franck, Murat

EyeDraw is a system which runs using an Eyegaze system to allow drawing with the eyes. Many disabled people have used eye-tracking systems for years to use a computer and to communicate. The EyeDraw system was developed and tested with 4 users. With feedback from the users, a 2nd version was developed and some improvements were made to enable easier use. Users gave positive feedback, though drawing was still difficult. The paper mentions a 3rd version under development.

----------

The EyeDraw system is the most successful application I have seen for drawing with the eyes. It seems to solve the Midas touch problem somewhat, and the users can actually draw with it.

Coming to Grips with the Objects We Grasp: Detecting Interactions with Efficient Wrist-Worn Sensors

2010-05-13T04:14:00.005-05:00

paper

This paper aims to recognize gestures by attaching an accelerometer and an RFID transmitter to a wrist worn device. The idea is that we know what device is being used due to RFID tags placed on devices, and the accelerometer data paired with the device ID can be used to classify the current gesture. A "box test" was performed in which various items were placed into and removed from the box using different RFID antenna types, different objects, and with different subjects. A long term study was also conducted in which the bracelet was worn for an entire day to test real world applications.

----------

I think the idea of this paper is nice, and if RFID tags were present in all our everyday items, much information could be obtained about how users interact on a day to day basis. I think it would only work well with a high number of tagged items, so if that doesn't happen, I don't know how useful this will be.

User-Defined Gestures for Surface Computing

2010-05-13T03:56:00.005-05:00

paper

This paper is a study of gestures for multitouch display interaction. The gestures were user defined for 27 commands. In all, 1080 gestures were observed. The purpose of this paper is to "help designers create better gesture sets informed by user behavior." The resulting gesture set was completely user defined. It turned out that the combined gestures predicted by the authors only covered 43.5% of the final gesture set. The authors also found that "users rarely care about the number of fingers they employ, that one hand is preferred to two, that desktop

idioms strongly influence users’ mental models, and that some commands elicit little gestural agreement."

----------

This paper illustrates the necessity of user input when designing s system. Often, surprising results can be obtained by studying users in this way. This step is crucial when designing a new type of system in which little is known about the input or little input is formally defined.

Whack Gestures: Inexact and Inattentive Interaction with Mobile Devices

2010-05-12T15:40:00.002-05:00

paper

This paper explores using simple hitting gestures to interact with a device. For example, if the device is attached to the belt, a simple smack or wiggle can provide interaction. A custom device was developed and tested for this project. Users found that the whack gestures were simple, easy to remember. They experimented with multiple whacks and combinations of whacks and wiggles.

------

This type of interaction could be useful for cell phones, ipods, and other common pocket devices. For example, if the cell phone starts ringing, it could be silenced with a whack. Some ipods also use a shake, or wiggle, to advance songs. I personally don't use that feature very often.

Device Agnostic 3D Gesture Recognition using Hidden Markov Models

2010-05-12T15:32:00.003-05:00

paper

This paper attempts to determine how to effectively use HMMs to classify 3d gestures "regardless of the sensor device being used." They use a decomposition technique which successfully works for any sensor combination.

------

I am currently beginning learning HMMs, and this could be helpful as I design my gesture recognition algorithms for the glove.

Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes

2010-05-12T15:27:00.002-05:00

paper

This paper is a well known sketch recognition paper that can recognize 16 shapes with a simple algorithm.

---------

This is a good paper for beginning sketch recognition. It is easy to implement, and can recognize a fair set of shapes. Higher level systems can be built on top of this to create simple yet rich sketch interfaces.

The $3 Recognizer: Simple 3D Gesture Recognition on Mobile Devices

2010-05-12T15:24:00.002-05:00

paper

This is an extension of the $1 to 3D. It does not support all the original $1 shapes, but it does add a few new shapes, such as tennis serve and hacksaw.

--------

This is cool, and I would like to see a video or a user study of how well this works for 3D sketching.

An Empirical Evaluation of Touch and Tangible Interfaces for Tabletop Displays

2010-05-12T15:18:00.002-05:00

paper

This paper compares several different methods for interacting on a table type display. They have implemented a touch based interface as well as a model-based interface on a single surface. They compared the touch and model based interaction for a variety of computing tasks and found that touch was better for some tasks and models were better for other tasks.

-------

I personally have never played around with any model based interfaces. I can see the usefulness of such an interface, though I am not sure that many small pieces would be such a good thing.

FreeDrawer: a free-form sketchingsystem on the responsive workbench

2010-05-12T15:13:00.002-05:00

paper

These dudes use a pen to sketch in 3D. They employ curve and surface smoothing to create smooth drawings.

----------

This paper was not so interesting. Drawing in 3D is nice, but I have seen better methods. Also no tests were done, so we don't know how well it actually works.

That one there! Pointing to establish device identity (2002)

2010-04-27T11:17:00.005-05:00

Colin Swindells, John C. Dill, Melanie Tory

Simon Fraser University

Kori M. Inkpen

Dalhousie University

paper

Comments: ...

This paper deals with the issue of human-computer identification. As the number of computing devices increase per person, the number of entries in wireless network lists increase. This makes it difficult for a person to select another computer or device to connect to in order to send information to that device. Traditionally, the name of the device is selected from a list of all visible devices on the network. As more and more devices are added to that list (with non-descriptive names much of the time), it becomes hard for humans to select the correct device to connect to. This trend contrasts with the increasing ease of computers to automatically enter, exit, and identify previously connected computers and devices. A solution to the human-computer identification problem is pointing. The device which the user wants to connect to is simply pointed at, and the computer can identify the target device and connect. This paper presents a device, called the gesturePen which sends an IR signal to tags installed on the target devices. By pointing the pen at the device, the device ID can be acquired and easily connected.

The paper illustrates some similar point-to-identify solutions, and points out that all others use a system which constantly broadcasts the device ids, which can still overwhelm the user. The gesturePen system tags "are only activated when ‘pinged’ by the gesturePen."

Liquids, Smoke, and Soap Bubbles – Reflections on Materials for Ephemeral User Interfaces (2010)

2010-04-22T00:35:00.020-05:00

Axel Sylvester

University of Hamburg

Tanja Döring, Albrecht Schmidt

University of Duisburg-Essen

paper

Comments: ..

This is a short paper intended to "provoke thoughts about durability, control, and materiality

of tangible user interfaces" by introducing the concept of an "ephemeral user interface" composed of transient materials, liquid, smoke and soap bubbles, that eludes complete user control by demanding that the inputs be treated delicately, as the bubbles inevitably will burst. The user interacts with a computer system by generating and then manipulating soap bubbles which can be empty or filled with smoke. The interaction surface is composed of a dark liquid on which the bubbles land after being generated.

This work is motivated by the increasing presence of computing in our everyday tasks and the lack of research in the area of materials used for interaction, despite studies illustrating "the importance of materials and materiality for humans." By using such unusual and transient materials such as smoke and soap bubbles, this work easily provokes thought about the possibilities of materials and "handles" used for interaction through its unusualness and "contradiction to ordinary technical and durable materials of computer technology."

Soap bubbles are highly symbolic and therefore are relevant to many fields including science, art, and entertainment. A fascination with soap bubbles occurs when viewing them as "'in-between' spaces - spaces that are neither real nor fully virtual." This is easily applied and understood from a computing interface perspective.

The system consists of a small, dark, round pool of liquid with camera beneath tracking the bubbles, which are blown onto the surface of the liquid from above. Either empty or smoke filled bubbles can be generated, and an overhead projector can illuminate the bubbles.

Once on the surface of the liquid, the bubbles can then be moved either by blowing or gently touching. In one application, the size of the bubbles determines the brightness of the ambient light in the room, and the x and y coordinates control red and blue hues of the ambient light in the room.

The researchers see this as a playful, entertaining, yet useful interaction mechanism as computing is further integrated into our everyday lives. For example, the paper suggests "a growing demand for user interfaces for services where specific and accurate control is not necessary and playful interaction with diverse materials suits the situation well." To illustrate this concept, the paper also suggests the concept of "buttons on demand" which could use these ephemeral materials or simple ambient displays.

----------

I like the ideas that this paper provokes. I hadn't thought of such approaches to tangible user interfaces. I tend to view currently available hardware and think of ideas of how to use those for interfaces. This paper inspires me to think of alternate materials and input methods and devices.

Toward Natural Gesture/Speech HCI: A Case Study of Weather Narration

2010-04-08T00:58:00.002-05:00

Indrajit Poddar, Yogesh Sethi, Ercan Ozyildiz, Rajeev Sharma

Pennsylvania State University

paper

Comments: ...

This paper discusses the limitations of current gesture recognition, claiming that all the restrictions imposed in most work violate the naturalness of the HCI involved. Therefore, they have decided to impose no restraints on the user by analyzing videos of weathermen, which is a domain they claim to be analogous to HCI. They employ some vision techniques to identify the person's head and hands and extract 5 features for each hand (distances, angles, velocities).

They use an HMM to recognize the gestures and have defined possible causal models. The speech was also analyzed in conjunction with the hand gestures to try to improve correctness and accuracy in recognizing the gestures.

To begin with, three main types of gestures were imagined: "here," which refers to a specific point, "direction," which can be something like east(ern) or north(ern), and "location," which is a proper noun form of "here." Three classes of gestures were named: contour, area, and point. The speech was analyzed in conjunction with the gesture to determine at what time some keywords were spoken: before, during, or after the gesture.

Analysis of speech and gestures shows that relevant keywords are spoken during the gesture the majority of the time, and sometimes after the gesture. Therefore, the speech can be used both as classification and verification of the gesture. Separate analysis of just the video vs the video and speech shows higher correctness and accuracy when speech is included.

Though the accuracy of this system is considerable lower than other gesture recognition systems, the authors claim this is much more natural, as the subjects analyzed were not participating in any user study at all. They were just naturally speaking and making gestures. The authors state that this study can "serve as a basis for a statistical approach for more robust gesture/speech recognition for natural HCI."

----------

As someone who is currently working on a hand gesture recognition project (using the data glove), I am thinking about the implications of this work in my own project. We are currently imagining a very limited gesture set, though we have been thinking about the differences of gestures among different users. We have been imagining a user study to determine what specific gestures to use, but this paper makes me think of eventually extending my current work in a much more natural direction where users can perform the gestures they want to and the system will respond in a unique way to each user while letting the user perform his own natural gesture, undefined by the game. This could make for an interesting system considering the domains we are targeting.

The Wiimote with multiple sensor bars: creating an affordable, virtual reality controller (2009)

2010-04-08T00:15:00.005-05:00

Torben Sko, Henry Gardner

Australian National Univeristy

paper

Comments: ...

This paper discusses using a Wii remote as a viable method to control a virtual reality system by using multiple sensor bars to define a much larger field of view for the Wii remote. This way, the Wii remote can be used across a surrounding display.

Five sensor bars were arranged in front of the user in a vertical position, as illustrated by the image above and by the video. Software allows the Wii remote to "bunny hop" from one sensor bar to another, since the Wii remote can only see 4 IR sources at one time, and one sensor bar contains 2 IR LEDs.

The researchers modified the Half Life 2 engine to create a game suitable for testing. The Wii remote was able to successfully track across the whole screen, allowing the user to play the game as normal.

The biggest limitation of this system comes from the "bunny hopping" feature of the Wii remote. Because it knows where it is based on the currently visible IR sources, it must be constantly pointed at the screen, which causes fatigue for the user.

----------

I was impressed at the effectiveness of this method. The video clearly shows that the Wii remotes are very adequate for precise aiming across the large 2-walled display. I think the limitation imposed by the Wii remote technology is not a big issue, since more specialized systems can create a controller that the user can set down and not keep aimed at the IR sources. The main contribution of this paper is that a Wii remote style or gun-style free-hand aiming system can be implemented for surround screen usage using inexpensive parts.

That being said, I would like to see them improve on this by allowing the user to rest and lower the controller away from the screen, whether they are able to do this with the Wii remote or some other hardware. Naturally, this demo makes me think of glove applications on this type of screen, though I don't have any specific ideas yet...

The Peppermill: A Human-Powered User Interface Device (2010)

2010-04-06T02:12:00.006-05:00

Nicolas Villar and Steve Hodges

Microsoft Research, Cambridge, UK

paper

Comments: ...

This paper presents the Peppermill, which is a wireless and batteryless interaction device. The device is powered by the user and momentarily sends out a digital signal.

The paper gives some background on user-powered devices, beginning with the Zenith Space Commander developed in 1955. The authors also mention MIT's user-powered button. Both these devices have the limitation that power is generated and therefore interaction only happens on the down-press of a button. The authors aim to improve on this idea by providing a method for richer interaction.

The method they came up with is a rotary control that is powered when the user twists the device. The user can twist the knob in 2 directions with varying speeds. A simple circuit detects the direction and speed along with a set of modifiers consisting of three buttons. This simple device is thus capable of very rich interaction.

The authors give an example of usage for this device by using it to control a video browsing application. When no buttons are pressed, a set of videos is cycled through, much like changing the channels on a tv. The speed of rotation controls the speed of video cycling, and the direction controls the direction of cycling. When the green button is held down and the control is rotated, the volume is adjusted. Once again, the speed of rotation controls the speed by which the volume is adjusted and the direction determines if the volume is adjusted up or down.

The authors talked a bit about future work, most notably a method of providing haptic feedback to the user while the knob is being turned.

----------

I was intrigued by this control, not only because it is human-powered, but also because of its unique interaction style. When I first looked at how this device is used, I actually didn't know it was human-powered. I am impressed with the versatility of a device that has no batteries or cord.

This method of interaction, especially without batteries, got me thinking about our own projects. I wonder if we can come up with a glove that is somehow human-powered. That would allow for greater motion than the wired gloves, and would not need batteries, like the wireless gloves need.

Gameplay Issues in the Design of 3D Gestures for Video Games (2006)

2010-03-23T11:48:00.007-05:00

John Payne, Paul Keir, Jocelyn Elgoyhen, Mairghread McLundie, Martin Naef, Martyn Horner, Paul Anderson

Digital Design Studio, Glasgow School of Art

paper

Comments:

This paper talks about the importance of designing 3D gestures and the relationship between gestures and gameplay. It discusses some important concepts such as affordance, mapping, and feedback. It stresses the importance of simplicity and the mapping of gestures to actions.

The team developed a 3D gesture capturing device which they call the 3motion. It uses a combination of accelerometers similar to a Wiimote to perform 3D gestures. It should be noted that this work was done almost a year before the Wii was released.

To test their device, several simple games were used. These included a tilt-ball game, an alarm game, the classic helicopter game, and a spell-casting game. Unique 3D gestures were defined for each game.

Two users were employed to test out each game and its gestures. Some games and their corresponding gestures had more success than others, which the researchers attribute to varying degrees of "informative tutorials, single word instructional phrases, effective semiotics and appropriate user feedback" among the games. These principles, they say, are very important to ensuring that "the gesture based interaction is intuitive, fun and rapidly understood."

While not initially an important factor, they came to realize that the gestures and the type of gameplay were tightly coupled and must be evaluated together.

----------

I was very interested in this paper particularly because of the final project for this class. We are also using 3D gestures, though with a glove instead of a handheld device.

We are also faced with the problem of designing the gestures for our system, and we plan on doing a preliminary study to help guide us to the correct gestures. We can use the insight of this paper to help guide us in out design.

I also wonder what influence the upcoming Wii and its controller had on this research, if any? I do not remember when the Wii was announced, unfortunately, though I doubt it was announced as early as this research.

An Architecture for Gesture-Based Control of Mobile Robots

2010-03-11T05:44:00.016-06:00

Soshi Iba, J. Michael Vande Weghe, Christiaan J. J. Paredis, and Pradeep K. Khosla

Carnegie Mellon University

paper

Comments: ...........

This paper presents a system for controlling mobile robots using hand gestures.

Previous work has been done with controlling robots in various ways, but most of those have used a keyboard and mouse to control the robots. This is deemed inappropriate for novice or unfamiliar users, so a more intuitive interface is necessary for these kinds of users.

The goal of this project is to work toward an intuitive, multi-modal system for controlling mobile robots. This project introduces hand gestures as a means to control mobile robots by waving in the desired direction the robots should move or pointing at the intended location for the robots to move.

The system uses a CyberGlove, a Polhemus 6DOF sensor, and a GPS unit in the robot itself.

6 gestures were used to control the robot:

OPENING: Moving from a closed fi st to a flat open hand
OPENED: Flat open hand
CLOSING: Moving from a flat open hand to a closed fist
POINTING: Moving from a flat open hand to index fi nger pointing, or from a closed fi st to index fi nger pointing
WAVING LEFT: Fingers extended, waving to the left, as if directing someone to the left
WAVING RIGHT: Fingers extended, waving to the right

They also incporated a "wait state," which simply was a gesture other than those above.

Robot control can occur in two modes: local and global. In local mode, the gestures are interpreted as if from the point of view of the robot. In global mode, they are interpreted in world coordinates to control the robot from the user's view. The reason for having the local control mode is to operate the robot remotely, in which video signals from the robot are viewed. The global control mode is used if the robot is in sight of the user.

The gestures work like this for Local Control:

CLOSING: decelerates and eventually stops the robot
OPENING, OPENED: maintains the current state of the robot
POINTING: accelerates the robot
WAVING LEFT/RIGHT: increase the rotational velocity to move left/right

The gestures work like this for Global Control:

CLOSING: decelerates and eventually stops the robot (cancels the destination if one exists)
OPENING, OPENED: maintains the current state of the robot
POINTING: "go there"
WAVING LEFT/RIGHT: directs the robot towards the direction in which the hand is waving.

A Hidden Markov Model algorithm was used to detect and recognize the gestures with an accuracy of 96%. The wait state feature helps the recognition significantly compared to systems without a wait state.

----------

I like that these researchers are trying to get a more intuitive interface for controlling robots so that novice users can use the system. I always like this approach to projects where appropriate. It is also interesting to see a new use of the data glove that I have not thought of before.

I wonder how feedback is given from the robot/system to the user. It would be very important to know exactly how your actions are affecting the robot, so that you don't over-steer or over-accelerate the robot, for example. This problem would be magnified if there is any sort of delay between a gesture and the robot's response as perceived by the user or if the user is using local mode, which definitely would introduce some lag.

I would like to see a usability study, obviously, to sort out issues like the one I have described, especially if the research is aimed at the general public.

I am also interested in the high-level multi-robot control to come...

Feeling the Beat Where it Counts: Fostering Multi-Limb Rhythm Skills with the Haptic Drum Kit (2010)

2010-03-09T10:11:00.023-06:00

Simon Holland, Anders J. Bouwer, Mathew Dalgleish, Topi M. Hurtig

The Open University, UK

paper

Comments: Murat and

This paper presents a "haptic drum kit," which adds vibrotactile actuators to the wrists and ankles to teach rhythms to people, specifically novice or unexperienced drummers.

The paper gives a background on the "human innate capacity for rhythm." Basically, all people involuntarily respond to natural rhythms and "periodic phenomena in the environment." Our brains might even have dedicated neurons for rhythmic processing. Specific instruction in rhythmic techniques may help people overcome specific physical challenges or limitations.

The experience of creating rhythm depends on the prior exposure, or "feeling," of various rhythms. Dalcroze, a famous music instructor, noted that students were better able to work with technical and written elements of music if they had the previous experience of "feeling" musical and rhythmic examples. He invented a system of feeling rhythm in his teachings by having students perform activities, such as walking, in specific rhythmic ways.

There is a theory known as sensory motor contingency theory that suggests that in order to learn some physical skill in some domain, the learning must be able to manipulate the domain physically. This has applications in musical rhythm because people can use arms, legs, and other things to create and modify rhythm. This theory provides support for the haptic drum kit that these researchers have devised.

Finally, the paper talks about the concept of entrainment, which is the tendency for two two connected processes to "connect" in some common rhythm. This is important to this research because the "students" or users will be playing a drum beat along with an audio and haptic beat. The convergence of the user to the presented rhythm is necessary to learn the rhythms. All these theoretical views on rhythmic learning helped to inspire the Haptic Drum Kit.

The Haptic Drum Kit consists of four vibrotactile actuators attached to the wrists and ankles with wristbands. The actuators are connected to a circuit board which is connected to a computer running the drum kit software and controlling audio playback.

A user study was conducted with 5 people. The goal of the study was to determine if rhythms can be taught using a combination of audio and haptic feedback. 20 rhythms were chosen from different classes of rhythms to represent a complete range of rhythms that typical drummers might learn.

When each rhythm was presented to a user, it was done with audio only, with haptics only, and with audio + haptic feedback. The user was to play along with each rhythm. It is unclear how exactly the rhythm is sent to each limb, though it seems that whatever limb is to play the current note is vibrated, while the audio lets the user know which drum to actually hit.

Probably because of this, all users preferred the mixture of audio and haptic feedback for playing back the drum patterns.

A few issues were revealed with the vibrotactile actuators. They involved things like being too "quiet" or "soft" or being slightly delayed, thus "blurring" the rhythm on fast rhythms.

Using results of the study, the paper discusses the state of the hardware and possible upgrades and future work.

------------

Being someone who is interested in rhythms and drumming but lacks the training or experience, I am interested in the approach to rhythmic training using vibrotactile feedback. We have already seen a suit that employs haptic feedback to teach specific movements. This uses a similar concept to teach a different kind of "movement," that which creates rhythm.

This system also explores multi-modal learning, presenting audio tracks coupled with vibration. The researchers seemed to be pretty successful at teaching the drum patterns despite some hardware limitations.

I think this work is more promising than TIKL, perhaps due to its better mappings of vibration to action and fewer vibrotactile actuators.

Office Activity Recognition using Hand Posture Cues (2007)

2010-03-02T00:50:00.008-06:00

Brandon Paulson, Tracy Hammond
paper

Comments: Manoj and eventually others...

This paper shares the results of using simple machine learning algorithms to see if hand postures can be used to help identify tasks. Specifically, the 1-nearest-neighbor algorithm was used on a set of 12 office-related gestures.

Gestures were collected from 8 users with a CyberGlove. The users performed each of the 12 tasks 5 times each. The sensor readings were captured at 10 frames per second and averaged together across the whole gesture. The algorithm was trained on both single-user and multi-user gestures. Per-user training yielded much higher accuracy across all gestures (94%). Only a few of the gestures were confused. In these cases, a simple examination of the gestures shows that the gestures are very similar, such as holding a mug and stapling a paper.

------------

This work is similar to what we did with the RPS-15 data in this class. One of the class members shared a 95% accuracy across all users using the 1-nearest-neighbor algorithm. Our RPS gestures were more rigidly defined using a picture of the gesture. In contrast, the users in this paper were told to perform an activity such as dialing a phone. These can be done in many different ways.

I thought of some things that could be added to this work to help clear up some of the gesture confusions. Most simply, including more sensors up the arm could help disambiguate certain gestures. For example, the elbow and shoulder are probably going to be different when drinking from a mug than when stapling a paper, even though the hand gesture is very similar (this is similar to the proposed 3D sensor being attached to the hand that is given in the paper). This method would be easier than incorporating a 4th dimension to the data, namely time. However, analyzing gestures over time might be a very valuable addition, though the analysis would be more complex.

I am also interested in seeing the results of these same gestures with other, better learning algorithms.

Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound (2008)

2010-02-23T11:54:00.005-06:00

Takayuki Iwamoto, Mari Tatezono, and Hiroyuki Shinoda

The University of Tokyo

Paper

Comments: Manoj and others...

This paper is an introduction to the idea that ultrasonic transducers can be used for airborne interaction. The idea is that a person can interact with a virtual 3D object by manipulating it with the bare hand. The paper gives examples of hand-tracking systems that create 3D models of the hand, though the authors have not yet done work in that area.

This paper mostly discusses the feasibility of using ultrasonic vibrations to give tactile feedback to the user. Most of the paper gives technical details about the hardware and some calculations of the sound pressure the user is likely to feel.

The paper gives results of a setup with an electronic balance and a microphone to determine the force and resolution of a prototype the authors have created. They also report that people can feel the pulses at 250-300 mm above the device.

Since the ultrasonic device is deemed a success, the next step of this research is to create a 3D interaction system that uses the device to give tactile feedback.

This video shows some of the more recent work, including hand tracking and holographic display.

------------

I think this system is an interesting idea. I personally have been thinking about methods of interaction with 3D objects using direct manipulation with the hand and providing some form of tactile feedback. The most obvious solution is the CyberTouch glove, which the authors of this paper denounce since it is always in contact with the hand even when not vibrating. I am interested in reading more papers in this area, and I hope we get to some more in this course.

Computer Vision-Based Gesture Recognition for an Augmented Reality Interface (2004)

2010-02-18T11:21:00.010-06:00

Storring, et al. paper

This paper talks about gesture recognition using vision-based techniques to recognize 6 hand gestures for use in augmented reality.

Because the gestures are recognized using a vision based approach, the only hardware needed is a standard camera. The user does not need to wear a glove.

The hand gestures are recognized by manipulating the image of the hand's pixels in such a way that the hand becomes white and the background becomes solid black, even if there are other objects in the background.

To count the number of fingers that are outstretched, concentric circles about the hand are examined.

One extended finger is recognized as a pointing gesture, and a transition between three states is used for a click gesture, in which the thumb is used to "click."

The generality of this algorithm means people with varying hands can use it. Also, it works for both left and right hands.

The author claims that this gesturing method was easily understood and used in a simple AR application, which I don't feel like describing...

------------

I like the vision-based approach used in this method. The CyberGlove is not easy to wear for a long period of time, so I like any solution that works with a bare hand as input. I have not studied any computer-vision methods, so this is new to me. I was wondering what the resolution was of the camera they use, because it seems that doing per-pixel calculations would take a long time.

I also wonder how sensitive the glove is to different skin tones. They mentioned an initialization step, but I don't remember if they specified if that included setting the skin color.

Motion Editing with Data Glove (2004)

2010-02-16T11:36:00.009-06:00

Lam, Zou, and Komura paper

Comments: Franck and Sashi also Manoj... Murat too

I'm sure we have all made hand puppets in which we use two fingers as legs. Lam, et al. have used this idea to map this motion to 3D models to control animation. Specifically, they use the finger-leg motions to manipulate previously-captured motion data. This can potentially turn an existing walking animation into a running, dancing, or hopping animation.

The finger motion data was captured using a P5 glove. This glove does not provide as good or as much data as our CyberGloves, but are adequate for this research, especially when considering the limitations of human fingers themselves when compared to legs.

The human body has many more degrees of motion than the fingers, which the researchers acknowledge and try to compensate for in their mapping function. The mapping function they have implemented converts the finger-leg motion into full body motion. To control things like arms, simple, common rules are used, such as the opposite movement of the arms to the legs when walking or running. As expected, this method produces unnatural animations. The researchers propose some methods of remedying this for future work.

A small experiment was described in which an animator manipulated the motion-capture data by performing walking, running, and hopping motions with the hand. A complete analysis was not presented, nor was a complete description of the users.

------------

I think this is an interesting concept that can be expanded on. I like the fact that Lam tries to tackle the problem of key-frame, static animation by introducing the dynamic, real-time finger controlled animation. I think this method would be good for establishing timing and rhythm, but I think the animations could be more defined by physics-based calculations rather than a direct mapping of the hand to the body.

EyePoint: Practical Pointing and Selection Using Gaze and Keyboard (2007)

2010-02-15T22:04:00.035-06:00

Kumar, et al. Paper

Comments: Frank and Murat also Manoj

EyePoint™ is a gaze-based system for mouse cursor control that aims to provide a worthwhile alternative to the mouse to the degree that the average computer user might choose this system over a mouse, depending on the task at hand. Kumar wants to move eye tracking systems out of the domain of disabled users and into everyday computing with normal people.

Design of the system

EyePoint's design incorporates three principles:

1. They do not want to "overload the visual channel for pointing," i.e. don't map interaction directly to eye movement.

2. They aim to increase selection accuracy by including zooming/magnification in such a way that it enhances accuracy but does not inhibit performance. The solution to this is to magnify a square around the gaze area and overlay a grid of orange dots, called "focus points," to enhance fine selection. The magnification avoids distortion of the interface that some techniques, such as fish-eye magnification, introduce. The magnified area is partially transparent as well, to avoid obscuring the interface, as some zooming techniques do.

3. They want to reduce jitter, which they do with "fixation detection and [a] smoothing algorithm."

4. The selection method must be "fluid" and simple in order to make fast selections while maintaining usability by both disabled and normal functioning people. The solution to this is to perform selections with some "hotkeys" (corresponding to click, double click, right click, mouse over, start click-and-drag, and end click-and-drag) on a standard keyboard. The thought is that pressing keys is faster and better than dwell and zooming selection methods.

This system uses a Tobii 1750 eye tracking system.

Usage of the EyePoint software

1. The user looks at the target on the screen.

2. The user presses down one of the hotkeys (click, double click, right click, mouse over, start click-and-drag, and end click-and-drag)

3. A square around the estimated gaze point is magnified in the manner described above.

4. The user looks at the target in the magnified area (this helps enhance the selection)

5. The user releases the hotkey

6. The action of the hotkey is performed

The researchers think that this system can equally be used by able-bodied and disabled users. They propose changing the hotkeys to some other input device depending on the situations of the users. No testing was done in this area, however.

User Study and evaluation

A user study was conducted with 20 normal, able-bodied, participants who were very experienced computer users.

A quantitative evaluation was performed which had the users perform specific tasks, including navigating through web pages and clicking targets. Data including timing and error rates were collected. The tasks were varied to account for any learning curves.

A qualitative evaluation was also performed using a questionnaire to get users opinions of the EyePoint system compared to a standard mouse input.

Results

Web browsing:

The "focus points" did not help with timing or accuracy.

The average time to click a target with EyePoint was about 400 ms slower than a mouse.

The mouse had a lower error rate by about 10% in the normal EyePoint tests.

User's were divided on which system they thought was faster and easier to use (mouse or EyePoint).

75% of the users said they might choose EyePoint in certain situations.

Most users preferred the focus points.

Target selection:

EyePoint is only marginally slower than the mouse for selection.

EyePoint has much higher error rates than the mouse.

Users generally preferred EyePoint for its ease of use despite its lower performance.

Analysis

It was found that EyePoint is similar in speed to the mouse. The error rates for EyePoint vary from user to user. The study participants preferred EyePoint to the mouse.

The researchers conclude that it is possible and practical for an eye tracking system to be used by the common computer user in place of the mouse.

Watch the video if you want:

------------

I like this eye tracking system the best of the papers we have read so far this semester. I have thought of using eye tracking to augment the mouse in some way, which would mean a system is mostly mouse-based with some eye help. This system is the opposite. It is a an eye-based system with some hand click help. This is pretty cool, and the speed and accuracy results look a lot better than some of the other work we looked at.

I am most intrigued by the magnification solution to enhance accuracy and reduce error by introducing a second focusing saccade at the cost of speed. I do think emphasizing accuracy over speed was a good move by Kumar, et al. since the mouse is already pretty darn fast and the time to actually focus on a target doesn't seem that it could be any faster than the mouse, at least for experienced computer users.

The EyePoint system might increase selection speed for computer novices that might not be used to using a mouse and therefore use it slower than us experienced users. After all, not all people are experience computer users, but nearly all people can see.

Brainstorming

2010-02-11T01:09:00.009-06:00

I have a few ideas about how we can aid the instruction of salsa dancing using the tools we have available.

We can use the HMDs to overlay some sort of 3d model, whether it be in first person or just showing a 3d model floating before the user. This way, the user can get real-time visual feedback as he naturally moves his head. Because the head is probably not going to remain still while dancing, this allows the user to constantly receive feedback without interrupting the dance. If feedback was given on a separate monitor, it would interrupt the dancing for the user to look at the screen.

We could use the eye tracker to determine which direction the user is looking to give feedback or keep the user from being distracted.

The vibrating gloves can be used in a variety of ways. They could vibrate when the user needs to do something, like grab his partner or perform a specific move.

---------- ---------- ----------

I also have a few ethnography suggestions for the undergrad CHI class.

First of all, they could scan through a list if student groups on campus and pick one that looks interesting. They could then attend a meeting (with permission if required, but hopefully anonymously) and observe the group.

Another possibility lies in the on-campus parking lots. The students could find a good vantage point to see a large portion of a particular lot (such as lot 50 since it is large and full during the days) and observe drivers' behaviors.

Similarly, they could watch the crosswalks to observe the students' behaviors in that situation.

VARK Questionnaire

2010-02-08T22:21:00.002-06:00

So I filled out the VARK questionnaire that Josh e-mailed our class about. This questionnaire is supposed to give an idea of how you learn (Visual, Aural, Read/Write, Kinesthetic). Here are my results:

Visual: 6

Aural: 8

Read/Write: 8

Kinesthetic: 8

I was surprised at my results, as I thought I was mostly a visual learner. According to this, I am pretty equal across the board, but visual is slightly lower. The first thing I thought of to perhaps explain this is that I like to learn from demonstration, which I suppose can be a combination of these, and I might think of that as visual. However, this questionnaire is short, and obviously isn't going to be as accurate as a professional evaluation. This makes me wonder what a thorough examination of myself would say... perhaps I don't know as much about myself as I think I do, or perhaps I perceive myself differently from reality...

Lab Day!

2010-02-02T15:20:00.010-06:00

So today we went into the lab to see some of the devices we have available for this class.

3D Display Glasses.

These glasses have 2 cameras in front and 2 displays, one for each eye. A control box is attached that has video outputs and inputs so a computer can process and alter/augment the user's vision.

This device was overall pretty easy to use. I was able to instantly function normally in an office or lab setting, though I may be impaired in other settings.

I can immediately see the benefits of these glasses, and I have a few ideas of applications. The first thing I thought of when I put them on was 3D object manipulation using augmented reality techniques. I'm sure many of us have seen the demos with the papers with markers on them that show a 3D model on the paper when viewed with a webcam on a computer. This could make those kinds of things feel more real, since it would be as if the 3D model is actually in the user's hand instead of on a computer monitor.

On a more whimsical note, I think these glasses could be coupled with facial recognition software and Facebook to allow the user to identify people they know on Facebook. Information about that person could be displayed around the head or body so the user can remember things about that person, such as a favorite food, color, or the person's name. This could help people get to know each other.

I did notice three possible problems. First, depth perception is altered while wearing the device. Everything seems farther away. While I was fine and adjusted to this quickly, other people may not be able to adapt or might become impaired by this. The long-term effects of this altered depth perception might be detrimental as well, but that would require more testing. Second, the resolution of the device limits the details the user can see. The biggest problem I noticed with this was the difficulty of reading small or far away text. Third, peripheral vision is blocked by the glasses. This, coupled with the extended depth perception, gives the user a kind of tunnel vision which makes it difficult to walk around. The biggest problem with this is the difficulty of seeing your feet while wearing it, so it is easy to trip over things. I also experienced problems going around corners, as my shoulder hit a few times.

Josh had us use the 3D glasses and fill out a questionnaire. We went through some actions to get a feel for the device and help Josh understand the device's capabilities.

Eye tracking glasses

These glasses have a camera attached that feeds data to a software application that recognizes the human eye and can determine the location the user is looking.

My impression of the device was medium. I was impressed with the eye recognition software and that it worked. However, the accuracy was not great, and I had trouble moving the cursor with my eyes. I also had some difficulty positioning the camera so my eye was completely in the viewing window, but I don't know if this is a problem or not. The device also requires that the user's head remain still, which is an impossible task. This device would be awesome if we can calibrate it better and be able to move our heads.

In its current state, I can see this device augmenting other input devices, such as a mouse or glove. I don't think it can currently be used as a standalone input device. For this class, I think we could implement some sort of eye gesture system similar to the EOG glasses we read about. The glasses seem to work fine for coarse input like that, though I had some difficulty with the up-left gesture.

CyberTouch gloves

I have been working with the gloves since last semester. My initial thoughts on the glove were very positive once I got one connected to a computer. The 3D hand in the configuration utility really illustrates what this glove can do, and the vibrotactile feedback adds a new layer of functionality and opens up a new realm of possibilities.

I have thought of a few ways this glove can control the mouse. First of all, it could be combined with a 3D location sensor, such as the Flock of Birds, to create a system for controlling the cursor on a large display (similar to Vogel et al.'s work). The vibrotactile actuators could add the feedback that Vogel was missing from his system. We could also just use the angles of one or more fingers to control the mouse. Finally, we could combine this glove with Josh's 3D glasses and the Flock of Birds to create a 3D interaction augmented reality application, in which 3D objects can be grabbed by the user and manipulated.

I have noticed a few problems with the glove. First, the calibration is not perfect, especially with the thumb. Second, the vibrotactile actuators have 5 discrete levels of vibration intensity instead of adjusting continuously. The lowest level of intensity is still pretty intense as well. I would like to have some gloves with a finer intensity akin to a cell phone vibrotactile actuator.

These are my initial thoughts and ideas for the devices we played with in class today.

Comments: Manoj, Paul

3DM: A Three Dimensional Modeler Using a Head-Mounted Display (Butterworth, et al.) (1992)

2010-01-28T03:09:00.009-06:00

This paper is about creating a 3D modeling environment using a head mounted display (with a small screen for each eye) to immerse the user in a 3D world. The user can fly through the 3D geometry and scale himself up or down to get different views of the world.

A floating toolbar is always present in the user's view. It contains functions for creating, editing, and viewing geometry as well as some other basic functions.

The user uses a 6D mouse to more accurately navigate and work with the 3D models.

It is reported that this system is useful for rapid prototyping, especially with organic shapes such as trees and rocks. The 3DM system is not very good at creating geometry with specific constraints (such as those in CAD programs), as it does not support them.

------------

I like this work, as it seems to be pioneering the virtual reality 3D navigation and modeling that we have seen in later work, such as HoloSketch. This system seems to be aimed at novices who want to rapidly create some 3D model to prove a point or illustrate something. As mentioned in the paper, it does not have any built-in constraints, though a grid that can be snapped to was added due to user feedback on this issue.

I think this work is a good base and can (and has been) expanded on in future work.

No specific user studies were discussed. Instead, the author reports some of the feedback of "users," whoever they may be.

Comments: Paul, Kevin

TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning (2007)

2010-01-26T11:18:00.031-06:00

by Jeff Lieberman and Cynthia Breazeal of the Department of Media Arts and Sciences, Massachusetts Institute of Technology

http://robotic.media.mit.edu/projects/robots/tikl/tikl.html

When learning a motor skill, such as when in rehabilitation or dance class, people learn through several means: through sight, sound, and touch. The most difficult method of instruction is through touch, since the instructor also needs to be performing the action and also cannot guide all joints of the student. This research aims to create a suit that is embedded with vibrotactile actuators at each joint to give corrective feedback to help teach the motor skill to the student.

The idea is that the instructor will wear a suit with motion sensors and will be tracked with a tracking system. The student also wears a suit equipped with motion sensors and vibrotactile actuators. When the student mimic's the instructor's movements, the suit will vibrate the joints that are incorrect proportional to the error in posture.

The suit they have created covers the wrist, elbow, shoulder, and chest. Motion sensors are used and tracked by a Vicon system to model the subject's arm. The vibrotactile actuators are placed around the wrist and elbow joints. Four actuators are placed around each joint. When the control software detects that the subject is not positioned correctly, the actuators vibrate in a specific way. For rotational correction, the actuators vibrate clockwise or counterclockwise around the joints to simulate torque on the joint. For joint angle correction, two of the sensors (top and bottom of the wrist) vibrate. One of them will vibrate with more intensity, indicating the direction the hand must move. It is intended that there is a vibrational "force field" around the proper join angles.

The suit is connected to a computer, and control software processes the motion capture data from the teacher and the student (the user) to determine the student's errors. The error data is sent to a custom hardware controller which transforms the error data into the proper vibrations.

A user study was performed with 40 subjects. Half the subjects received visual correction while the other 20 received visual correction plus vibrotactile feedback. Each person in each group was shown the same videos of an instructor posing in a specific position. The videos were shown in a random order and were only a few seconds long to ensure that no subject would be able to perfectly mimic every pose.

In a questionnaire given immediately after the test, subjects who had the vibrotactile feedback felt that it didn't help much at the beginning but might help over time if they kept using the system. The vibrotactile group also reported more fatigue than the non-vibrotactile group.

The studies were video recorded, and analysis of the video shows enhanced performance with vibrotactile feedback. The error rates computed by the control software show consistently lower errors in all trials.

Hinge joint corrections were found to be easier to understand than rotational joint corrections. The error rates for hinge joint correction are much lower for the vibrotactile feedback group, while error rates for rotational joint correction are not significantly different for the vibrotactile feedback group.

Overall, the study shows a significant gain in performance (up to 27%) and accelerated learning (up to 23%). It is pointed out that the positions of the vibrotactile actuators and the methods of feedback were not optimized, so it is expected that these number will improve even more as the technology is optimized and a full-body suit is created.

For future work on the technology itself, Lieberman Breazeal plan on finding a lower cost relative positioning system to aid in the adoption of this technology. They also want to develop or find smaller, more powerful tactile actuators. They want to increase the performance and learning gains with rotational joints. They wonder if this system can be successfully scaled up to the full human body, since there would be around 100 actuators and the body might not be able to handle that. They want to learn the long-term effects of this system as well.

In other future work, they want to test this system with less or no visual feedback, so this can be used for visually impaired or blind people. They also want to explore neurological rehabilitation and posture correction.

------------

I thought this work was very interesting and presents a great use of vibrotactile technology. I have been thinking of other ways to use this technology than the project Manoj and I are currently working on with our vibrotactile gloves. I especially liked the possibility of using the suit to help blind people perform certain tasks or learn certain motor skills, since that is related to my own work, which is to help blind people see drawings and images and also draw using vibrotactile feedback to the fingers.

I was impressed that the system worked well in its initial unoptimized, simplified form (consisting of just one arm). I hope this suit will perform on par with or better than this initial study.

I was wondering why they used video and images of the instructor in the user study rather than a live instructor. I would be interested to see how the suit performs with a human instructor. This would be a different form of visual stimulus, and might be superior.

Comments: Kevin, Josh, Paul

HoloSketch: a virtual reality sketching/animation tool

2010-01-25T21:17:00.012-06:00

by Michael F. Deering

Sun Microsystems Computer Corporation

Holosketch is an attempt at creating a 3D modeling and animation interface that provides a more direct form of control over the 3D models. It uses stereo shutter glasses with a CRT display refreshing at 112.9 Hz. The glasses alternatively block each eye so each eye gets its own display, so to speak. This way, the user perceives the 3D models in 3D. A six-axis mouse is used to try to provide a direct method of manipulation in an attempt to improve upon the 2D mouse and its approximation of 3D manipulation. The head position is also tracked, so that moving around the 3D object, to some degree, gives an alternate view of the object, similar to the real world.

The Holosketch system allows creation of simple 3D primitives and incorporates simple animation gadgets to allow novice users to create models and animations easily. A keyboard is used in conjunction with the 3D mouse, or "wand," as it is referred to. In some modes, the conventional mouse is used in conjunction with the 3D mouse.

The menu system was reworked for the Holosketch project. Since the interface is 3-dimensional, a conventional menu system would be visually intrusive and take away valuable rendering time for the 3D geometry, since it has to be rendered in real time and the computer's processing power was much more limited back in the mid 1990s. The new 3D menu system is similar to a context menu. It is a radial menu containing many options which are selected with the wand.

Instead of a traditional user study involving multiple participants performing specific tasks, an artist was employed to use the system for a month and give feedback. She responded positively, only complaining about a few minor interaction issues. It reportedly took a few days to learn how to use the system well. It was determined that novices would be able to pick up the system quickly.

------------

I like the attempt at a direct 3D manipulation. Using the six-axis mouse looks like it would perform better than the 2D mouse. Sometimes unexpected things happen when manipulating 3D models with a conventional mouse. I also like the head movement to alter the models.

It is really interesting that this work was done so long ago. I would like to see a modern instantiation of this research in combination with some of the other work we have seen. If we could incorporate Holosketch with Vogel's work with the freehand pointing and MIT's BiDi screen that allows are hand gestures, we could have a 3D modeling system in which you can reach out and grab the 3D model with your bare hand and manipulate it (someone please tell me if this exists).

Comments: Kevin, Josh, Franck, Paul

Wearable EOG Goggles: Eye-Based Interaction in Everyday Environments

2010-01-24T23:16:00.005-06:00

This work takes another approach to the technology of eye tracking. It uses some goggles with attached electrodes to sense changes in the electric field of the eye. It describes the eye as a dipole with the cornea and the retina as the endpoints. When the eye moves, the offset in the electric field is sensed by the goggles. Their demo shows the goggles recognizing 8 directions of movement: up, down, left, right, and diagonal combinations.

Their hardware is completely worn on the user with no wires going to a computer or some other device. The sensors are attached to the goggles, and a DSP and some other hardware is attached to both the gloves and another wearable unit. The data is transferred to the computer using Bluetooth.

------------

I like the approach these people are taking by using other technologies to sense eye movements. I would like to know how accurate these sensors are. Can these goggles be used for pointing, for example?

I would also like to see an expanded user study to really show what these goggles can do. The study they give seems to be just a confirmation that the device kind of works. I would like to see a study that really puts the device through its paces and discovers how accurate it is and what its effects are on users. If we know the accuracy, maybe more people can use this technology for different areas of research.

Comments: Kevin, Josh, Franck, Paul

Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays

2010-01-21T01:35:00.005-06:00

This research aims to provide an interaction mechanism for large displays that allow users to interact with the display both from a distance and up close. They have come up with a method to use the hand to control the cursor by attaching reflective markers to a couple fingers and points on the hand and wrist and tracking those points.

There are several "modes" of interaction. There is a clicking mode in which a click can be performed by either pushing the index finger down in the air or bringing the thumb up to the side of the hand. Feedback is introduced with some on-screen animation and sound since there is no tactile feedback on the index finger and little feedback with the thumb.

There are a couple different pointing modes they have come up with. First, they use a ray-casting mode that basically acts as if a pointer is coming out of the index finger. There is also a relative mode of movement as well as a combination of pointing and relative movement.

They found that their methods worked pretty well and had high accuracy, though the clicking and pointing modes cannot be used together.

------------

I think this type of interaction is really interesting and cool. I really like touch interaction, and this extension of touch to control from a distance is interesting, especially because the interaction at a distance works up close when the display is touched. No other systems allow an easy transition from far away to up close.

This research made me think of the CyberTouch gloves (link) I am currently working on with Manoj. We have the ability to get fingertip and wrist locations already, and we have the added benefit of vibrotactile feedback with this particular glove. We are currently working with 2D and 3D tracking, so we have the ability to do things like this here in our lab, which is exciting to me.

Comments: Kevin, Josh, Manoj, Franck, Paul, Sashi

Noise Tolerant Selection by Gaze-Controlled Pan and Zoom in 3D

2010-01-20T23:19:00.011-06:00

This work involves using eye trackers to provide a gaze interface that uses panning and zooming to navigate a 3D-like interface. Their work is tolerant of noise introduced to the system. Unlike other gaze systems, this one does not rely on dwell time to make selections.

An application called StarGazer is used to provide the 3D interface used for the gaze tests. This program consists of a circular keyboard that the user pans and zooms to select letters and thus type using only the eyes.

It was found that this system works better than those based on dwell time. Users were able to type at a relatively high word per minute using the StarGazer program, 8.16 wpm, with a low error rate of 1.23%. Also, users remained in control even when some noise was introduced to the system, which was noteworthy. Only efficiency was decreased, not accuracy or the number of errors produced. Additionally, users only needed about 5 minutes to understand how to use the system.

------------

I think this work is interesting because it helps pave the way to eliminate the mouse and keyboard, which I feel are barriers to a natural computer-human interaction. This system still has a way to go before it can truly replace the keyboard in terms of typing speed. It looks like it is faster than the mouse, however. I would like to see this system used in some other areas besides typing. I think the quick learning curve and use of cheaper, off the shelf gear will also help this system gain popularity.

Comments: Josh, Manoj, Franck, Murat, Paul

Introduction

2010-01-20T01:03:00.010-06:00

Hey everyone! My name is Drew.

dalogsdon@gmail.com

Originally from Midland, TX, I am a first year MS student in computer science here at Texas A&M. I am taking this class because I am interested in computer-human interaction. I have seen many people's research in the undergrad CHI class, and most of it is really interesting to me. I think this sight and touch class will give me another glimpse of current research and give me a great opportunity to get more hands-on experience and do more research in this area.

In ten years, I expect to be working full time somewhere on earth. I would also like to become a somewhat known artist on the side. That's about as specific as I can get, because I am like Josh Peschel, opportunities just seem to come to me with little input from myself.

I can't decide what I think the next big technological advancement in computer science will be, though I really hope we can get some better desktop interfaces. I want something new to come along and revolutionize computer interfaces in the way windows and GUIs revolutionized computing. Perhaps mind control or something... I think we can expect something from Apple pretty soon though, seeing what the iPods and iPhones have done for technology and CHI.

I find multi-touch very interesting, as it can not only allow for a more natural, intuitive computing experience, but it can also allow collaborative computing on one computer with one large display. The Microsoft Surface computer and James Bond movies have illustrated this, and I think it can be a valuable tool for both business and recreation.

I don't care who I have lunch with, as long as he or she pays.

I think I will say my favorite movie is The Waterboy. I grew up in Louisiana, and I feel that this is one of the few movies that my whole family can sit down and enjoy when I visit home. I also find it consistently entertaining.

I think people would like to know that I am an artist in addition to a computer scientist. I enjoy the traditional arts, mainly drawing and painting. I don't like doing digital art, perhaps because I feel like my creativity and talents are inhibited by drawing hardware and software. Maybe I can do something in the area of computer-human interaction to help give digital artists a more creative experience.

Fly: A Tool For Authoring Planar Presentations

2009-04-30T15:10:00.002-05:00

I read the paper "Fly: A Tool For Authoring Planar Presentations" by Leonhard Lichtschlag, Thorsten Karrer, and Jan Borchers at RWTH Aachen University. Their website is here.

RubberEdge - UIST 2007

2009-04-21T15:02:00.003-05:00

I read the paper RubberEdge: Reducing Clutching by Combining Position and Rate Control with Elastic Feedback from UIST 2007. It was written by Géry Casiez, Qing Pan and Christophe Chaillou of the University of Lille, France and Daniel Vogel of the University of Toronto, Canada. You can read the paper here.

The paper give descriptions of three types of input device:

Isotonic: This type of device uses position control to manipulate selection. Examples of this are the mouse, touch pads found on laptops, and pen inputs/touch displays.

Isometric: This type of device is stationary and responds to force or pressure applied to it, increasing the rate of movement based on the pressure applied. Examples of this are joysticks and the "nubs" found in older laptops.

Elastic: This type of device combines isotonic and isometric controls into a single input device. Part of the device uses position control while another part uses rate control.

The main problem the researchers are trying to solve is that of "clutching," which happens in isotonic devices when the limits of the position control interface are reached and the device must be moved or "re-calibrated" to continue input in a particular direction. An example of clutching occurs when a mouse reaches the edge of a mouse pad and must be picked up and moved back to the center of the pad to continue moving in one direction. A similar issue occurs in laptop touch pads.

Elastic devices solve the clutching problem by providing some sort of isometric control at the edge of the position control area. The paper goes on to describe some previous implementations of elastic devices that fail to work intuitively or have certain mathematical flaws.

The team's solution to the clutching problem is "RubberEdge," a device which basically is a touch pad with a ring around it that is attached to elastic bands and can be pushed outward a bit to control mouse movement at the edge of the touch pad.

The paper describes some trajectory problems that arise when transitioning from the touch pad to the elastic boundary and the methods and mathematics used to solve them.

The team claims that a small circular RubberEdge input device is 20% more efficient that a standard laptop touch pad. They designed and implemented several tests to determine the efficiency of this input.

Ninja Cursors

2009-03-24T15:12:00.010-05:00

I read Ninja cursors: using multiple cursors to assist target acquisition on large screens by Masatomo Kobayashi and Takeo Igarashi at the University of Tokyo.

Some displays can be very large, especially if comprised of multiple displays. The goal of this research is to reduce the amount of time it takes to move the mouse cursor to select something on a large screen.

The approach these researchers have taken is to place several cursors on the screen at the same time. They have shown that 2-8 cursors provides the best results, significantly reducing selection time but not overpowering the user with too many cursors.

The main problem with this approach was pointing to multiple items at the same time. This was solved by putting the selections in a queue and only allowing one item to be selected at a time. The item to be selected is highlighted, while any other items in the queue have a red arc around them. A shorter arc means the item is closer to the front of the queue.

They also encountered the problem of the lasso tool. They set in place several restrictions, such as "the lasso cannot intersect with an object" and "the lasso cannot be empty" to help the lasso tool work properly.

I personally do not like the "ninja cursors." While they do improve usability a bit, the many cursors distract me more than help me. Maybe this is different for different people.

A demo used to be available here, but that site is no longer accessible.

Kinematic Templates

2009-02-26T14:24:00.009-06:00

I read the paper entitled Kinematic Templates: End-User Tools for Content-Relative Cursor Manipulations by Richard Fung, Edward Lank, Michael Terry at the University of Waterloo in Ontario, Canada.

Click here to read the paper.

Kinematic templates represent an area of drawing somewhere between completely freehand and rigidly defined drawing tools. Basically, kinematic templates actively adjust cursor movement in specific ways to aid the drawing of regular shapes, such as circles and lines, while retaining the human element that is lost by using shape tools.

Kinematic templates are especially useful when drawing with an input device that is not designed for drawing, such as a mouse. However, they are also useful for cleaning up drawings done with a drawing tablet. These tools can also be used by people learning how to draw or who just want to create a simple, clear drawing. Of course, they can be used by professional artists as well, if desired.

The research team has developed several different templates to aid drawing. These include templates to guide movement parallel to an axis, along orthogonal axes, concentrically about a point, through a point, around a point, and many others which are all listed in Table 1 in the paper.

To use the kinematic templates, the user defines regions in the composition in which the templates will take effect. This is illustrated very well in the video above. Different templates can be layered to help create more complex shapes.

The following image gives an example of using kinematic templates to draw a sun:

The research team has created two types of kinematic templates: passive templates and active templates. Passive templates only affect the cursor when it is moving. Active templates can move the cursor when it is held in place. It should be noted that the templates only take affect when the mouse button is held down, so "users cannot completely lose control of the cursor when using kinematic templates" (p4).

In addition to the pre-defined templates, users can create their own templates using Python scripts.

The research team is considering several areas of future research, including automatically generating templates based on existing compositions or imported images.

Please read their paper to discover more details and how the kinematic templates work.

Elevator Ethnography

2009-02-24T13:09:00.007-06:00

George Lucchese, Eric Scott, and I performed a study of people's elevator usage at the West Campus Garage on campus at Texas A&M.

We initially wanted to observe people using elevators and see if we could find some interesting information about their usage. Since it is pretty difficult to observe all the floors at the same time, we had to figure out an interesting way to do this. Our solution was to observe the elevators in the West Campus Parking Garage, since its set of elevators is behind a glass wall and each elevator has a large windows. This enabled us to observe all the elevators on all the floors at the same time. We could easily see how many people were in an elevator and which floors they got on and off on. As an added bonus, the stairwell was right beside the elevators and also was behind a huge glass wall, so we decided to monitor the stair usage as well and compare it to the elevator usage.

This is a view of the elevators and stairs as seen from our observation point.

We recorded the gender, transportation method, number of floors traveled, and direction traveled for each person that used the elevator or stairs. We performed our observations at three times of the day: 9:00am, 12:00pm, and 5:30pm.

We gathered a large amount of data and immediately entered it into the computer and started making charts and graphs. In the end, we ended up with 11 interesting charts, most of which illustrated the obvious.

Here are some observations we made from our charts:

People travel more floors by stairs earlier in the day and fewer floors as the day goes on.

People travel about the same number of floors by elevator all day long.

People walk up stairs more floors than they walk downstairs.

People take the elevator the same number of floors whether they are going up or down.

Females tend to travel more floors than males do.

Males travel a lower number of floors than females do.

Males travel to the roof twice as much as females do. (Possible data error due to small sample size)

The same number of people travel only one floor.

Many more people travel down in the morning, and many more people travel up in the evening.

A few more people travel down at noon.

More people take the elevator than take the stairs.

People usually travel one or two floors on the stairs.  Few people travel more floors on the stairs.  The exact opposite is true for elevators.

Fery few people park on the first floor or roof of the garage.

People travel about the same number of floors in the morning (except the first floor and the roof).

People travel more floors at noon.

People travel fewer floors in the evening.

People travel up a few floors more than many floors.

People travel down many floors more than a few floors.

Males tend to use the elevator more often than females use the elevator for any number of floors.

As you can see, most of this data seems normal. In fact, it is.

We did come up with a couple interesting observations. First of all, we observed that most people will take the stairs if traveling two or less floors, i.e. going from the first floor to the third floor. In this case, about 75% of people will take the stairs. If traveling more than two floors, the number of people taking the stairs dramatically decreases to about 20%. We also observed that males tend to use the elevator more than females do, no matter how many floors they are traveling.

While our data does not give any immediately useful information, we believe it can be used as a basis for many other studies that can be done for both elevator/stair usage and parking issues on campus.

The Mole People

2009-02-24T12:59:00.004-06:00

The Mole People is an interesting study of homeless people who live in the tunnels beneath New York City. It is an extensive ethnography that reveals many details about tunnel dwellers and attempts to lower the negative perception of these people.

It is written by a woman who frequently went into the tunnels and talked with and made friends with many of the homeless who live there. She wants everyone to have respect for these people, as most people hate the tunnel dwellers and view them as the "outcasts of the outcasts."

This book shows that there are many different kinds of people who live in the tunnels. Some seek shelter from the danger of the streets, some enjoy the darkness and cave-like feel, some are gangs hiding from the law, and some are intelligent people with educations who just prefer to be homeless. Some are single, some are in relationships, some have families, and some reject relationships and society altogether. There are also people of all ethnic and racial backgrounds. Some people have always been homeless, some ran away from home, some lost a home, and some purposefully left a home. Some want to return to the surface, and some want to stay in the tunnels.

This book describes the lives and personalities of all these types of people, illustrating the diverse nature of these people who all live in various types of tunnels at varying depths underground.

The Mole People is as entertaining as it is educational, and I enjoyed this book and found it fascinating. The author tells the stories of the various people as stories, which educate while simultaneously entertaining. Some of the stories are funny, and some are sad. They all reveal important details about the people who live in the tunnels. I would recommend this book.

The Media Equation

2009-02-24T12:43:00.004-06:00

The Media Equation is a book which illustrates that people treat all media, including images, television, and computers, as they would treat real people. Many studies are done on groups which explore specific attributes people apply to media. The authors explore several specific topics within broad categories such as manners, personality, emotion, social roles, and form. Each chapter presents a study done in a particular area of the categories mentioned above. The hypotheses of the authors are all confirmed in each chapter, which goes to show how much people do treat media as fellow humans.

I like this book, in particular because it showed just how deep people's relationship with different media really is. I know that people treat computers like other people, but prior to reading this book I did not know just how deep that relationship goes, or that is applies to many types of media. The structure of the book makes it non-surprising, but when I pull back and think about the things it discusses, I truly am surprised that people behave in such ways. As such, I would recommend this book to others.

This book was pretty easy to read for a few reasons. First of all, the language used is aimed at a general audience. No overly technical terms are used, and any that are used are explained fully. The authors made sure everybody can understand this book. Second, every chapter comes to a positive conclusion. This makes it easy to predict the outcome of the chapter and to review the book, since the reader does not have to check to see if some experiments were failures. Finally, each chapter is organized in the same way. First, the hypothesis is presented with an explanation why the authors thought this. Second, the studies they perform are described in detail, with a few "rules" that basically summarize the hypotheses. Finally, the conclusions are presented with a discussion of why the particular results probably occurred.

A good interface

2009-01-29T00:12:00.002-06:00

Hey there, I'm here to tell you about a good user interface I have been experiencing for the last few months: my iPod touch.

I know the iPod touch/iPhone is famous for its excellent interface, but I want to tell you a little bit about it anyway.

With all the varied features (music, video, photos, wifi, applications, e-mail, etc.), the interface for such a complex device is easy to get wrong. There are several reasons why Apple seems to have got this one right. Here is my list of a few design elements that make the iPod touch easy to use.

1. Few buttons: The iPod touch has 2 physical buttons and a volume rocker. This limits the confusion that can come with many buttons. Furthermore, each button only does 2 things. The power button on top puts the iPod in standby mode when pressed and turns it off when held for a few seconds. The home button goes home (duh) and switches to music controls when double-clicked. Because of the many features of the iPod touch, it would not make sense to have a few physical buttons to control the wide array of features. This would give too many functions to one button, which would be confusing to use.

2. "Natural" navigation: The iPod touch contains several navigation features that add to the usability of the device. For example, when looking through lists, web pages, e-mails, documents, etc., you just "flick" the screen and the document scrolls. This gives the user more precise control over the scroll speed and feels pretty natural. The icons on the home screen scroll in a similar fasion if you have more icons than the screen can display. The screen on the iPod touch is a "multi-touch" screen. The screen can detect 3 different points of contact instead of the usual one, which eliminates even more buttons and zoom bars, among other things. The iPod also contains an accelerometer, so different modes can be accessed if the iPod touch is held sideways. All these things help eliminate clumsy, tiny buttons. These all add up to greater ease of use.

3. Automation: The iPod touch automates many tasks and remembers many settings so the user doesn't have to press as many buttons. For example, when you come in range of a wireless network you have accessed before, the iPod automatically connects to it, and remembers the password if there is one. It also automatically scans for updates for any "apps" you have downloaded and notifies you if any are found. When opening the e-mail app, it automatically checks for new mail and downloads it. If you remove the headphones, the music automatically pauses. This all helps the usability of the device.

These are just a few things I have noticed about my iPod. I really like this device, and it is just as useful as it is entertaining. It is easy to use all the features and get the most out of it.

The Design of Everyday Things

2009-01-28T23:26:00.003-06:00

This is a pretty cool book. Basically, the author, Donald A. Norman, discusses various pitfalls in the designs of many everyday objects. Norman, a psychologist, discusses the ways that humans respond to new devices and how to design them so they are immediately usable. Many examples are given illustrating bad designs and common problems people have with them. He mentions many common, everyday errors all people make and how simple mistakes in the use of badly designed objects can lead to catastrophe. This book is a good introduction to understand the necessity of good design and user interfaces.

This book was very easy to read. Norman uses plain English in a conversational tone to explain his research and discoveries. No overly technical jargon is used in presenting the material. In addition to this, he gives many simple, real-world examples that perfectly illustrate the concepts he discusses.

A few topics in the book piqued my interest. First of all, Norman mentions many common household items, such as light switches and doors, that millions of people have problems with every day. This strikes me that such simple things should be so hard to use. What is even more striking to me, though, is that little or no effort seems to have been made to change standard items like these on a large scale. Light switches today are functionally the same as light switches fifty years ago.

Another topic that I found fascinating was the tendency of people to blame themselves when having trouble with what should be a simple object. In fact, the improper designs of these objects do not give some sort of natural mapping or clearly illustrate the proper use. The result of such poor designs is really to blame for the troubles people have with telephones, air conditioners, refrigerators, and many more seemingly simple items.

Finally, the last thing I enjoyed from this book was the presentation of many common slips and errors that people make on a daily basis. This may be partially because of the humor in the examples, but also due to the ability to personally relate to the topic. For example, people frequently start a sentence over, stutter, or pause many times during speech, implying that many small errors are being made. People are so used to such errors that many go unnoticed. Norman presented several common types of errors people make all the time. For example, many people start performing a frequently-done activity when trying to do something else, like driving to work when intending to drive to the store or throwing a dirty shirt into the toilet instead of the laundry bin. I found the somewhat formal description of several different, specific types of error fascinating, as I could relate to all the types he presented. Perhaps this chapter helps people feel less clumsy, since it shows that all humans have many slips.

This book was definitely a good read, and it is definitely one of the most interesting books I have read for any class. I think it was a good introduction to Human-Computer Interaction, even though it doesn't deal with computer interaction. It presents the fundamentals of good design for anything.

2009-01-22T00:38:00.005-06:00

Howdy!
I just wanted to type something.