AirDraw: Leveraging Smart Watch Motion Sensors for Mobile Human Computer Interactions

AirD raw: Leveraging Smar t Watch Motion S ensors for Mobile Hu man Computer Inter actions Danial Moaze n Seyed A Sajjad i Ani Nahape tian California State U niversit y, Northridge Calif ornia State University, N orthridge California State Universit y, Northri dge Northridge, USA Northridge, USA Northrid ge, USA gdanialq@gmail.c om seyed.sajjadi.94 7@m y .cs un.edu ani@csun.edu Abstract — Wearable computing is one of the fastest gr owing technologies today. Smart watches are poised to t ake over at least o f half t he wearable devices m arket in t he near f uture. Smart watch screen size, however, is a limiting fact or for growth, as i t restricts practical text input. On the other hand, wear able devices have some features, such as consistent user interaction and hands-free, h eads-up o perations, which pave the way for gesture recognition methods of t ext entry. This paper proposes a new text input method for smart watches, whi ch utilizes motion sensor data a nd machine learning approaches to detect letters written in the a ir by a user . This method is less computationally intensive and less ex pensive when c ompared t o com puter vision approaches. It is also not aff ected by light ing factors, w hich limit computer vision solutions. The AirDraw system prototype developed to test this approach is presented. A dditionally, experimental results close to 71% accuracy are presented. Index Terms — Wearable C omputing, Mobile Text Input, Smart Watches, Dynamic Time W arping (DTW). I. I NTRODUCTI ON Wearable computing is one o f the fastest growing technologies, today, wit h forecasts suggestin g that sm art watches, alone , wi ll take over half t he we arable c omputing market by 2018 [11]. A new approa ch t o mobi le user interfac e design for wearable t echnolog y is im portant t o unl ock the potential o f t hese systems. A limitin g f actor in smart w atch's marke t growth is their small scre en and inter face difficulties [ 9]. This paper offers a new mobile te xt input met hod for smart w atches, wh ich utilizes the motion se nsors on the wat ch a s the resource f or arm moveme nt inf ormation in 3 D. Usi ng d ynamic ti me warping (DT W) for letter cl assification, the s y stem ai ms to extrapolate the letters the user is writi ng i n t he ai r. A prototype, c alled A irDraw, is presented and used f or the system evaluation. The strong clas sification accura cy is presented in t he experimental results. In t he area of gest ure de tection and recogniti on, m ost of the existin g s ystems re ly on computer vision [8] . The n eed for multiple camera s to make up a 3D mod el, the depen dence on optimal lightin g c ondition s, and c omputationa lly intensi ve image proc essing demands are some of the drawbacks o f computer vision based s y stem s f or gesture recogniti on [8]. These shortfalls call f or al ternative a pproaches, as in our alternative levera ging motions sensors onboar d we arable systems. II. R ELATED W ORK Wearable c omputin g is one of the suc cessful a ttempts in the field of ubiquitous com puting, w ith smart w atches gaining fast popularit y. Major t ech c ompanies such a s Sam sung, LG, Sony a nd rec ently Apple pr oduce their ow n s mart watche s. Additionall y, t here are wri st based activi ty trackers suc h as those fr om Fitbit equippe d wi th moti on sen sors a ble t o dete ct, recognize, a nd measure se veral dai l y act ivities i ncluding s teps and sleep patter ns [12]. The use r i nteracti on diffi culties, due to smart watches’ small scre ens, have encouraged the new ideas f or U I approaches. O ne of these appr oaches, propo sed b y Komninos and Dunlop [9], relies on taps o n the t ouch scree n as t he basic text i nput a ppr oach. They propose d a ne w la yout f or ke yboard with only 6 b uttons. Some ha nd writi ng recogniti on approaches like Unis trokes [10] and Graffiti [ 6] also tried to make text input easier for u sers. N one of t hese ap proaches a re hands-free or hea ds-up. Gesture recogniti on s ystems have been examined in the literature. A grawal e t al. le v era ge the a cceler ometer sens or in mobile ph ones to c apture the inf ormation written on the air [1]. They treat each written ch aracter as a collection of stroke s and classif y letters accord ing t o the detected strokes. Goldbe rg and Ric hardson pr oposed a u nistrokes a pproach [ 10], with a defined stroke f or each letter. The approac h requires the us er to lear n the shorthand f or t he letters to be able to w ith the system. O ur appr oach uses the natural f orm of letter, limiti ng the knowledge based t he u ser needs to interact with t he system. The use of hidden Mar kov Model for hand ges ture recognition has been c onsidered as well [ 3] [ 4]. Abhinav P arate e t al. use a wrist device t o diffe rentiate a smoking ges ture fr om other gestures, s ome of whic h are very similar t o sm oking s uch as fo od inta ke. The project utilizes a low-power 9-a xes inertial measurement unit ( IMU) o n a wristband. IMU pr ovides 3D orie ntation of the wrist b y using the data fr om accel erometer, gyroscope a nd c ompass [2]. Yujie D ong e t al . al so made a wrist based wearable devi ce using a n e xpensi ve sens or ( $2,000 US) to meas ure t he food intake [ 5]. III. A PPROACH A. AirDraw Syste m Prototy pe A prototy pe, called AirDraw, with hardwa re and so ftware com ponents was a ssembled and develo ped t o validate the propo sed appr oach. AirDraw c ons ists of a smart watch and a handheld device, with communicatio n facilitated by a Bluetoo th c onnectio n betwee n the devices. Android Wear applicatio ns were developed for the smart watch (Wear) and the handheld (Mo bile). The gravity and linear acceleratio n of the smart watch in thre e a xe s, x , y and z, i s collected and transmitted to the more resource-available handheld f o r processi ng. The hardware s chema is p rovided in Figure III .1. Figure III.2 illustrates the software overv iew of the sy stem. The wear applicatio n housed on the smart watch filters the data to sm ooth the sign al and to save on the data transmission cost. It also ca lculatio ns t he angle of the use r’s hand and sends that informatio n t o the handheld for manipulating the frame o f reference of the x-axis and y -axis data. The mobile application housed on the hand held processes the ac celer ation and a ngle in formation to obtain the ro tated acceleratio n data that independ ent of the user’s arm positio n. That data is then passed to a cl assifier fo r letter pr edication . Figure III.1: Hardware schema; h andheld d evice, we arable device a nd t he Bluetooth connect ion. Figure III .2: Software Overview. Wear Application and Mo bile Application B. Data Filteri ng A weighted moving a verage filter i s used to smooth the sign als fr om sensors. Figure III.3 shows the improvem ent obtained fro m applyi ng this smoothing a lgorithm. Th e blue line is the o riginal accel eration s ignal along the z-axis o btained while drawing a c ircle without an y sm oothing f ilter applied. The red line shows the same signal for the ver y same movement with the smoo thing algorit hm applied. Figure III.3: Comparing the signal with and without weighted moving average filter over t ime. The b lue line shows t he a cceleration signal on the z-axis while writing the letter a without any filter app lied. Th e red line sh ows t he same signal fo r the very same movement with the filter applied. C. Arm Angle Calculation The angle that user’s arm make w ith the horizon line is needed to cancel the effect o f the user’s arm orientatio n on the linear acceleration data w hile d rawing or writing in the a ir. Implem ented in the Wear ap plicatio n, the smoothed g ravity signa l is us ed to calculate the angle between that the smart watch’s x-a xis (or user’s arm) w ith the horizon line. Figure III.4 shows the smart watch’s fixed frame o f reference and how the x-axis coincides wi th user’ s arm. Figure III.3: Smart watch’s fi xed frame of refer ence. The arm’s orientation aff ects the accelerati on data alon g the x- axis a nd t he y - axis. To canc el its effect, the arm’s a ngle is used to rotate the fra me of reference accordi ngly. As a result of the rotation, up a nd d own m ovements a re represent ed in y -axi s accelerat ion cha nges. Bac k an d forth m ovements a re represented as x-a xis accelerat ion c hanges. Inde pendent of t he arm angle, left a nd right movements are extrap olated f rom t he accelerat ion along the z-ax is. In order to calculate the a ngle, first w e need to calculat e the norm of the gra vity vect or. In 3D e nvironme nts this is done as sh own in Equati on III.1. Equatio n III.1: With t he nor m, we can to normalize t he gravity along each axis. Normalize d gravit y for eac h axis is cal culated b y dividing the gravit y along ea ch a xis by the norm. The calculation for x is show n in Equation III. 2. Equation II I.2: The angle that x makes with h orizon can be calc ulated by taking the arctan gent of g x-normal over g y -no rmal . Figure I II.4 sho ws the effect of the r otation o n signals x and y. Both Figure III.4.a and Fi gure III.4.b show the acceleration data whe n the w earable dev ice is m oving up and down repeatedl y w hile the device is rotati ng around z-axis 90 degrees. Figure III.4.a show s the data before r otation and Figure III.4. b shows the data a fter r otation. Note the effect of the de vice’s orientation being cancele d out along t he x-a xis and added t o the y - axis. This addition and deleti on is in the reverse direction when the de vice is m oving bac k and forth. Figure III.4: Comp aring the acceler ation data before and after applying t he rotation of the frame of reference while the de vice is moving up and down repeatedly an d rotating around z-axi s 90 d egrees. 3.5.a , on the t op, shows the acceleration without applying the rotation on the frame of reference an d 3.5.b, on the bottom, s hows the acce leratio n a fter applying the rotation on the frame of reference. D. Session Detecti on Communication between the devices is expensiv e in terms of power. To minimize that c ommuni cation an d the data sy nchronization costs, session d etection is carried out by the Wear applicat ion, so that sy nchronization is o nly c arried out during an a ctive session. Th e t ime window that the user is writing a letter is referred to as session. The start of a session is sign aled b y the acceleratio n signals being greater than 1 m/s2. Once t he accele ration remains less t han 1 m/s2 for more than 400 millis econds, the sessio n will be consid ered complete. Table III.2 is compares the average number of data transfers between de vices t o write t he same f ive words, with a contin uous connection and a connection during active sessions. The data f or each word is the average of five time samplings. These w ords ar e the 5 top m ost searched f ood rela ted words in the U.S. in 2014 acco rding to Google Trend [7 ]. Table III.2: Compa ring the amount of data (averaged across 5 measurements) updated and transferred between devices with a continuous connection and with a connection only during a ctive sessions. The words are the 5 top Googl e searched foo d related words in the U.S. in 2014. Connection Type Air Written Wor ds Pizza Chicken Cake Wine Coffee Continuo us 281.4 305.4 228.8 201.2 297 During Active Sessions 186 198.8 118.2 118.2 155.8 Data Transfer Savings 33.9 % 34.9 % 48.3 % 41.5 % 47.5 % E. Letter Cla ssification Letter cl assificati on is carr ied out using a supervised learning algorithm call ed dynamic time warpi ng ( DTW), with only one i nstance of t he data ( writing each letter once ) for training. An y data that ca n be converted into a linear se quence can b e compare d with DTW, wit h t he algorit hm returning the distance betwee n signals. In o ur appr oach, a compari son along each a xis is carried out separatel y. Then the total distance i s calculated b y a dding the distances for the x-axis, y-axis a nd z-axis. W hen t he user writes a le tter, t he acceler ation da ta of the newl y written letter is c ompared with the data saved in t he training phase for all the letter s. The lett er with the mi nimum total distance is determined t o be the inten ded letter. IV. E XPERIMENTAL R ESULTS This section presents e xperimen tal re sults using the AirDraw implementatio n. Al l the experimentatio n was carried out w ith a single use r, with the smart watch is worn on t he user’s dominant hand (r ight hand). The t est for e ach letter was com pleted before starting the next letter. On av erage, each took 1.5 secon d to write. All of the presented confu sion matrices below provide the actual lette r along the y -axis and the predicted let ter alon g the x axis. Table IV.1 illustrates the results f or 5 non-sim ilar letters (a, b, j, w and z) using the DTW a lgorithm . These letters were chose n as they are distinct from each other i n terms of stroke shape and count. The size of t he lette r written on air i s 12 inches b y 12 inches. Each letter is written 100 times and the results a re averag ed. When w e decrease t he ra nge of the ar m mo vement (size of the letter) t o 6 inches by 6 inche s, it is seen on table IV.2 t hat for some letter s uch as b, z, and j, the accu rac y dr ops. Table IV.1: Detecti on accuracy of DWT algorithm for 5 non -similar letters. The size of the lett ers is 12 inches by 1 2 inches. The test is d one 100 times for each letter. Predicted a b J W z Actual a 100% 0% 0% 0% 0% b 5% 95% 0% 0% 0% j 0% 4% 84% 10% 2% w 0% 10% 16% 74% 0% z 0% 0% 0% 4% 96% Table IV.2: Detecti on accuracy of DTW algorithm for 5 non -similar letters. The siz e of t he letters is 6 inc hes by 6 i nches. T he test is done 100 times for each letter. Pre dicted a b J W z Actual a 96% 4% 0% 0% 0% b 22% 77% 0% 0% 1% j 0% 12% 70% 18% 0% w 0% 6% 0% 94% 0% z 0% 0% 0% 14% 86% A set o f 5 le tters is chosen to te st t he perf ormance of the DWT algorith m f or the simil ar letters. The le tters ch osen are a, d, g, q a nd u. A gain the te sts w ere r un 100 times f or eac h letter and the aver age of t he results a re pre sented i n Ta ble IV.3. Table IV.3: Det ection accuracy o f DTW a lgorithm for 5 similar letters. The size of t he letters is 12 i nches b y 12 i nches. The test is done 100 times for each letter. Predicted a d g q u Actual a 54% 0% 19% 6% 11% d 0% 100% 0% 0% 0% g 0% 17% 18% 65% 0% q 3% 0% 16% 81% 0% u 0% 0% 0% 0% 100% As ex pect, diffe rentiatin g simil ar letter is more challe nging than differentia ting non-simila r letters. Table IV. 4 provi des the results f or t he entire E nglish alphabet. The tests a re run 20 time s for e ach letter. The average acc uracy is 71%. Wit h the he lp of a spellchec ker application, t his can be drama ticall y improved. V. C ONCLUSION We presented a new approach to m obile text entry for wrist worn weara ble systems, such a s t he p opul ar smart wat ch. Motion sens ors on the de vices are u sed to extrapolate air- written lette rs. The data filtering a nd classificati on approaches are cons idered and evaluated on the AirDraw sy stem p r oto t ype . An average 71% ac curacy rate is f oun d cl assify ing all the letters of the English alphabet, with half the letters achieving at least 80% a ccuracy . I n conclusion, air writing with a smart watch is shown to be a feasible and prom ising mobile user interface option fo r wrist worn wear able sy stems. R EFERENCES [1] S. A grawal, I. Constanda che, S. Ga onkar, R. Roy Choudhur y, K. Cav es, an d F. D eRuyter. Using mobile phones to wri te in air. In M obiS ys , 2011. [2] Parate, M.-C. Chiu, C . Chad owitz, D. G anesan, an d E . Kalogerakis. Risq: Rec ognizing sm oking gestures with inertial se nsors on a wri stband . In P roceedings of the 12th Annual I nternational Confere nce on Mobile S ystems, Applications, and Services, MobiS ys ’14, pa ges 14 9–161, New York, N Y, USA, 2 014. ACM. [3] Shuai Yan g; P remarat ne, P.; Vial, P., "Han d gesture recognition: An overview," Broadband Netw ork & Multimedia Tec hnolog y (IC-B NMT), 2013 5t h IEEE International C onference, vol., no., pp.63, 69, 17-19 N ov. 2013. [4] B. Min, H. Yoon, J. S oh, Y. Y ang, T.E jima,"Hand gesture recognition using hidden Markov m odels," S ystems, Man, and C y bernet ics, 1997. Co mputational C yberneti cs and Simulation., 1997 IEEE International Confe rence on , vol.5, no., pp.4 232,4235 vol.5, 12-15 Oct 1997. [5] Dong Y, Hoover A, Scisco J, Muth E. A new met hod for measuring m eal i ntake in hu mans via automated wrist motion trac king. Appl Psychoph ysiol Biofeedback. 2 012,vol.3 7, no.3, pp205, 215, 2012. [6] C. H. Blic k ensto rfer, “Graff iti: W ow!!!! ,” P en Comput. Mag., vol. 1, pp. 30,31, 1995. [7] http://www. google.com/tr ends/topchart s#vm=chart&ci d= foods&geo=US &date=201 4&cat=, 3/3 1/2015. [8] Gouthaman, S.; P andya, A.; K arande, O.; Kal bande, D.R., "Gesture detection system using smart watch based motion sens o r s," C ircuits, S ystems, C ommunicati on a nd Information Techn ology A pplicati ons ( CSCITA), 2 014 International C onference on , vol., no., pp.3 11,316, 4-5 Apr 2014. [9] Komninos, A.; D unlop, M., "Te xt Input on a Smart Watch," P ervasive Computing, IEEE , vol.13, no. 4, pp.50,58, Oct .-Dec. 201 4 [10] D. Goldberg and C. Ric hardson, “Touch-t yping w ith a stylus,” in Proc. INTE RACT Hu man Factors Comp ut. Syst., 1993, pp. 80–87. [11] http://www.ccsi nsight.com/pr ess/c ompany-news/1 944- smartwatches -and-smart-ba nds-dominate-fa st-gr owing- wearables-mar ket, 4/2/20 15. [12] https://www.fit bit.com/s urge, 4/2/20 15. Predicted a b c d e f g h i j k l m n O p q r s t u v w x y z Actual a 90 10 b 80 20 10 c 60 5 35 d 60 15 5 5 5 10 e 45 10 15 15 15 f 85 15 g 20 80 h 80 5 10 15 i 95 5 j 50 45 5 k 5 35 60 l 85 15 m 85 15 n 60 10 30 o 5 5 80 5 5 p 5 5 85 5 q 15 5 35 45 r 5 80 10 5 s 5 25 5 5 5 55 t 10 5 5 80 u 15 5 35 45 v 20 35 55 w 10 0 x 40 60 y 15 10 5 5 5 10 50 z 15 85 Table V.7: Accuracy percentage of DTW algori thm for the detection of lowercase English a lphabet. The test is r epeated 20 ti mes for each letter. The size of the letters is 12 inch es by 12 inches.