Gesture Keyboard - User centered design of a unique input device for Indic Scripts???

Ashish Krishna1, Rahul Ajmera1, Sandesh Halarnkar, Prashant Pandit

HP Laboratories India

HPL-2005-56

March 18, 2005*

* Internal Accession Date Only

???HCI International 2005, 22-27 July 2005, Las Vegas, Nevada, USA

1Human Factors International, 4th Floor, Chemtex House, Hiranandani Gardens, Mumbai ??? 400 076, India

Approved for External Publication

??? Copyright 2005 Hewlett-Packard Development Company, L.P.

Gesture Keyboard - User centered design of a unique input device for Indic Scripts

Abstract

Indic text input presents a unique challenge in the field of keyboard design because of the number of standalone characters, conjuncts, matras, and symbols. It poses a huge challenge to the HCI practitioners to accommodate the humongous array of Devnagari characters in a keyboard. Solutions for input in Devnagari scripts using a physical keyboard have been around for a while but none of them have emerged as a standard mechanism due to several usability related concerns. In this paper we have described the design of a new gesture based keyboard, which has a dual input mode 1. Gesturing 2. Tapping. The ???gesture keyboard??? concept is based on partial hand writing recognition as well as touch-typing paradigms. Conceptually, it is a stylus sensitive keypad that supports tapping for getting the base consonants and some symbols, while it also recognizes handwritten matras as per handwriting recognition methods. This paper also reports the findings of the user study, which was carried out to find the optimum sizes of the keys and layouts.

1 Keywords

Devnagari text entry, Gesture Keyboard (GKB), Matras, Modifier, Syllable, IMEs

2 Introduction

Given the well-understood challenges of achieving high accuracy with recognition of natural handwriting and dealing with the great variability in writing styles, a number of alternative schemes (generically called Input Method Environments or IMEs) have been developed for English and some Oriental scripts. Some of these are based on development of simplified gestures (e.g. Uniscript and Graffiti for English), others are based on interactively arriving at the intended character using partial handwriting input or other cues from the user.

This new keyboard introduces a paradigm shift in text inputting methods. We have developed a prototype for Devnagari, which takes best of both worlds; it takes cues from handwriting recognition (matras formation) and touch-typing (base characters). The new design has all the base consonant keys and the user is required to make a Matra (a gesture) on top of it, to make a syllable. Thus significantly reducing the need to hunt and peck and leading to quick learnability, retention and reduced hunting load.

The Gesture Keyboard is not presented as an alternative to the conventional English keyboard but has been proposed as a peripheral input device. The user would have an additional device (GKB) connected to his workstation along with the regular ASCII English keyboard and a mouse as shown in Figure below.

Figure 1: typical computer setup with a gesture keyboard

2.1Need for a Devnagari input device

Computer usage in India is almost entirely in English and is restricted to the English-speaking, upper and upper- middle class people. This stands in direct contrast with the demand for communication in Indian languages. Census data for the year 1991shows that English is the first language for only 0.02% of Indians and second and third language for 8% and 3% respectively. Hindi TV programs claim all the top 10 slots in television program ratings. Circulation of daily English newspapers had a market share of 15% in the year 2002. (Chand, Ganu, Joshi, Mathur & Parmar, 2004) 2001 census indicates that almost 70% of the Indian population specks 5 major languages i.e. Hindi, Bengali, Telugu, Marathi and Tamil with Hindi being the most spoken language at 40.22%. (http://www.censusindia.net/language.html)

2.2Difference with roman script

Using a standard keyboard for Devnagari scripts creates usability issues, as the Devnagari Script possesses some conceptual differences from the Roman script.

Some of these are summarized below:

???The concept of matras (character modifiers) has no parallels in the Roman script.

???Each character has a shiro rekha (the top bounding line on Devnagari characters) on top of it.

???Character modifiers, basically matras and other special characters, can occur before, on top, below or after the main character that they modify. (Deb and Deshwal, 2003)

???Since in the written form there is no clear specification of the sequence in which modifiers should be added to the consonant, sequential entry modes like a keyboard would not fit the users mental models

???More than one modifier can be attached to one consonant

2.3Issues with using an Inscript Keyboard

Current Indian language typing solutions have a steep learning curve. According to Arjun Mahanto, the Hindi Officer in IIT Bombay, learning to type requires approximately fifty hours of training and practice for a person to reach speeds of 25 words per minute. This is too much of a barrier for ordinary people and only professional typists are willing to make this investment. (Chand et al., 2004)

Figure 2: Inscript keyboard (with multiple characters on a single key)

Commercially the demand for Devnagari input has been patched by the Inscript keyboard. This is essentially a QWERTY keyboard with Devnagari characters mapped onto the keys. The minimum number of characters that are required even representing Devnagari is 54, base consonants (34), vowels (19), and a halant key. This excludes Devnagari numerals and various Vedic symbols and modifiers i.e. Ref, Ru, Half Ra. This issue manifests itself on this keyboard, which is essentially meant for entering just 26 alphabets. The aftereffect is that each key has multiple characters (two to three characters per key) mapped onto them. These additional symbols on the keys are inputted using combinations of toggle keys like shift, alt and control. This keyboard has a very high learning curve due to the multiple shift modes as is suited for professional typewriters who undergo rigorous training to learn this keyboard.

3 Introduction to the GKB

Computing devices demand text input schemes that can be quickly learnt and retained to achieve a fair speed and easy to use rather than ???hunt and peck???. Roman keyboards are not particularly amenable to accommodate the phonetic non-alphabetic script like Indic. Many alternate layouts exist which are mapped on the Roman alphabets of QWERTY keyboards. However, no such known work exists for Indic scripts. (Joshi & Rathod, 2002)

Keyboards support a ten fingered typing and also have two entry modes namely expert and novice. While the novice uses ???hunting and pecking??? to type, experts do a head up touch-typing. So, there is always a trade off between the learning and typing speed achieved in the keyboard.

There is an analogy between keyboards and styli. Keyboards can be used with no training: the letters can be tapped out one-by-one using hunt-and-peck. This is similar to what is currently done with styli. No new training is required, and letters are printed one-by-one. However, unlike styli, keyboards have a ???growth path.??? With practice, hunt-and- peck with two fingers can become faster than handwriting. If even higher speeds are desired, then keyboard users can learn touch-typing. Touch-typing not only achieves high speeds, it also enables ???eyes-free??? operation, that is, the ability to type without having to look at your hands. This suggests that the solution to the problem of stylus text entry requires developing an analogue of touch-typing. An electronic pen (or stylus) would be more attractive as an input device if it supported expert users with some analogue of touch-typing. (Goldberg & Richardson, 1993)

Working on this analogy we tried to develop a gesture-based keyboard which had gestures enabled on each key. So that user neither has to write fully as in the case of pen based interfaces instead they can just make gestures to get a matra along with the base consonant nor touch type fully to get a syllable. In this case, the design of gesture became an important consideration for us. They have to be very intuitive and also very easy to learn.

3.1Gestures

Few major criterions for designing gesture (matras) were, 1.easy to learn, 2.fast to write and 3.easy to retain matras. Instead of developing new gestures for each Devnagari matra (as in the case of graffiti and unistrokes), we implemented these matras as they appear in the Varnmala. The only difference being the breakage of multi stroke matras into a combination of different single stroke matras. Also for a half character we designed a horizontal stroke, which was derived from the mental model of cutting a character in to half.

All these aforementioned criterions for matras were satisfied as there was no learning involved; users had to write them they way they normally write on paper. Also, the retention was easy as the users had to learn few matras namely horizontal strokes and oblique scoring for getting secondary glyphs from the keypad.

3.2Spatial mapping of gestures

In order to limit the technological complexity of recognition and thereby trying to reduce instances of errors, we have mapped the gestures on the coordinates of the writing space. This means that the number of elements that needs to be recognized has been reduced. The same gestures made in different areas of the pad produce different combinations wherein the consonant is selected using the spatial mapping and the modifiers are assigned after recognition of the gestures.

3.3Layout

The Varnmala structure was useful for people to locate keys and to reduce the cognitive load. Even users who had ???forgotten??? the alphabetical order could remember the local sequence of the letters they were searching for. For example, if they were looking for the (n) key, they would mutter an entire line of consonants ???(t) (th) (d) (dh) (n)??? before locating the key. (Chand et al. 2004)

Figure 3: Varnmala layout

We tried to design the gesture keypad as cognitively easy for a novice user as possible, so that with minute training and instructions they are able to achieve a decent writing speed. We kept the layout very basic, as it appears in the Devnagari varnmala books in the block.

3.4Feedback

Performance data with keyboards, where the auditory feedback can be switched on and off suggest that typing is significantly faster and more accurate with auditory feedback on than with off. (Birdwell, Monty & Snyder, 1983; Blake, Muto & Roe, 1984) Based on these prior studies we added a click sound on every pen down. This can be again toggled on and off based on user???s preference. The timings of the feedback and the frequency of sound are still to be investigated.

In addition visual feedback has been provided so that the user can see his handwritten input on the tablet real-time. We believe that expert users of the system will not use this functionality much but it should be provided to help the novice users. With some errors always built in with this keyboard due to handwriting recognition and pen tilt errors users may be able to catch and correct their errors when they see their handwriting on the computer screen.

3.5Deleting using the GKB

The Devnagari script is very differently from roman scripts because of the conjuncts it forms when Consonant- Consonant-Vowel modifier, Consonant-Vowel-Vowel combination happens. So, in the case of normal ???backspace??? where 1 unicode is deleted at a time and it becomes very irrational to display the left over characters from conjuncts, which the user never actually wrote.

Figure 4: Editing problems eradicated using a horizontal score on the backspace

To overcome this problem we added one more gesture along with a normal ???backspace??? which was horizontal scoring on the backspace key. It deleted the whole syllable in a go, so that user can start afresh.

3.6Technology used

The technology behind GKB can be broadly categorized into three major components:

???Graphics tablet with electronic pen (hardware): This device is used to capture handwritten gestures as digital ink

???Handwritten gesture recognition (software): This algorithm recognizes the gestures and maps them to corresponding matras

???GKB controller (software): This software component maps the pen position to corresponding base character (consonant or vowel)

The tablet is used to provide a keypad layout on which user makes gestures using the pen. The digital ink is then processed & recognized. Position of the pen and recognizer output is used to form the final syllable, which is then transmitted to the application in focus.

3.7A scalable solution for multi lingual environment

One of the biggest advantages of the solution is its ability to accommodate inputs in other scripts. The tablet does not have labeled keys instead it had a layout print stuck on the pad surface, this layout can be changed easily thus enabling flexibility of multi lingual input.

4 Finalizing keys and layout size

Prior research (e.g. Clare, 1976) tells us about the design and size of keys which are used for touch-typing using fingers. We found out it???s worth investigating the natural writing sizes and style for Devnagari before coming up with an optimal key size and finally layout size. This study was crucial in terms of knowing the average size and styles in which user writes the matras and base consonants. The size has to be taken in account to make the keyboard much more robust and comfortable to write for a user.

We had to come up with a key size and in turn layout size, which has the following characteristics:

???The users should be able to see the key label

???The key top size should be optimized for gesturing as well as tapping using a pen

???They keys should prevent the erroneous mapping of the alphabets

???As the key size and the layout size are directly proportionate, it should minimize the hand movement on the keypad

4.1User study

4.1.1Objective

The objective of this exploration was to find the average size users take to write consonants and modifiers (matras) and to find relationship between handwriting styles with different pen grip diameter and tip thickness.

4.1.2Method

Each user was given a Hindi paragraph (76 words) and they had to copy this Hindi corpus in their own handwriting on the provided paper. They were given basic instructions explaining the task and the goal of the experiment.

Participants were provided with two different size blank papers (A4 = 21 cm x 29.8 cm and A5 = 14.8 cm x 21 cm). They were first asked to write on the A4 sized paper and then on A5 paper with 0.5mm pen and 9 mm finger grip diameter pen. Same activity was then repeated with 0.1 mm tip and 12mm finger grip diameter pen.

4.1.3Subjects

21 voluntary subjects took part in this study. All of them, 1 left-handed and 7 right-handed subjects had normal vision. All subjects were experienced computer users.

???7 HP labs Employee (who know Hindi and occasionally write in Hindi)

???11 Students from 10th standard, from a local high school in Bangalore (who have Hindi as second language and write Hindi regularly)

???3 Teachers from the same high school-Bangalore (who write in Hindi almost once everyday)

Figure 5: Participants for user study

4.1.4Apparatus

4.1.4.1 Pen sizes:

???0.5 tip and 9mm diameter grip pen

???0.1 tip and 12mm diameter pen

4.1.4.2 Paper sizes:

???A4 = 21 cms x 29.8 cms

???A5 = 11.5 cms x 29.8 cms

Figure 6: Pens used for study (middle one is the stylus)

4.1.4.3 Hindi corpus used:

The Hindi paragraph used for this user study was taken from the BBC???s Hindi news portal (http://www.bbc.co.uk/hindi/news/020419_vajpayee_ac.shtml) and was modified a bit so that we could get all the nuances of Hindi matras in the corpus.

Figure 7: Hindi Corpus

4.1.5Evaluation method

Subjects??? hands written samples were collected and analyzed by measuring the base consonants and the modifiers with help of digital calipers. Each subject???s consonants and Modifiers size were noted down and divided according to the core strip, bottom strip and top strip.

Figure 8: Character study

4.1.6Analysis

Out of the four data sets that were collected, the most appropriate pen size and paper size that resembled the Gesture pad was 0.1 tip and 12mm diameter pen and A5 sized paper. So we analyzed this dataset and came out with the recommendation for the key size. We took the final data set and divided it in three groups, namely Top, Core and

5 Conclusions

This keyboard has been accepted quite well with the novice users who have never used Devnagari keyboard before. In our small sessions of user studies they all seem to be satisfied and were able to achieve a speed of 10-12 syllables per minute. Also after a dedicated usage of an hour everyday for 2-3 days they were able to go up to 25-30 syllables per minute.

The keyboard has a high satisfaction factor associated with it because writing with pen on paper (in our case the keyboard) is very intuitive, powerful and efficient. It aptly exhibits the same characteristics of simplicity, naturalness and straightforwardness like handwriting.

6 Future Directions and discussions

We are currently planning to do a constrained usability study to measure the efficiency of the gesture keyboard against the other ???standard??? Devnagari input mechanism namely Inscript keyboard with novice as well as expert users. The aim of this proposed study is to see whether the satisfaction ratings change with these different user groups and also to compare the words per minute that we are able to achieve with this keyboard.

We are also studying various layouts (Varnmala, frequency and common letter-pair) using paper mock-ups and analyzing which one suits the mental model of the users the best. User studies are also underway to determine which layout is the fastest to learn and reduces the pen travel while writing.

7 Acknowledgements

The authors would like to thank Dr. Shekhar Borgaonkar, Dr. Sriganesh Madhvanath, and Dr. Girish Prabhu for their valuable inputs and support during the project.

8 References

Chand Aditya, Ganu Ashish, Joshi Anirudha, Mathur Gaurav & Parmar Vikram (2004). ???Keylekh: A Keyboard for Text Entry in Indic Scripts???, CHI 2004, April 24???29, 2004, Vienna, Austria. ACM

Clare C.R. (1976). ???Human Factors: a most important ingredient in keyboard designs???, Handbook of Human computer Interaction 2nd edition, page no pp 1296-1297

Goldberg David & Richardson Cate (1993). ???Touch-Typing With a Stylus??? InterCHI 1993. ACM

Deb Dr. Kalyanmoy & Deshwal Priyendra Singh (2003). ???An Optimal Character-Key Mapping for the Devnagari???

Joshi Anirudha & Rathod Amit (2002). ???A Dynamic Text Input scheme for phonetic scripts like Devanagari??? Development by Design, Bangalore 2002

Monty, Snyder and Birdwell, (1983) & Roe, Muto and Blake, (1984). Handbook of Human computer Interaction 2nd edition, page no pp 1299-1301