«2 Patent Application Publication «) United States Jobs et al. (S41) TOUCH SCREEN DEVICE, METHOD, AND RINTEREACE FOR JRMINING COMMANDS BY APPLYING EURISTICS, Inventors: Steven P. Jobs, Palo Alto, CA ( Scott Forstall, Mountain View, CA (US); Greg Christi, San Jose, CA (US): Stephen O. Lemay, Sen Francisco, CA (US); Seott He Santa Clara, CA (US); Marcel Van (Ox, San Francisco, CA (US; Bas ‘Ording, San Francisco, CA (US) gory Noviek, Santa Clara, CA (US): Wayne C, Westerman, San Pranciseo, CA (US); Imran ‘Chaudh, San Francisco, CA (U Patrick Lee Coffman, Menlo Park, CA (US); Kenneth Kocienda, Sunnyvale, CA (US): Nitin K. Ganatra, San Jose, CA (US); Freddy Allen Anzures, San Francisco, CA (US); Jeremy A, Wyld, San Jose, CA (US); Jeffrey 6) Paul D, Mateos, Los Altos, CA (US); Charles J. Pisu CA US): View Seott Mountain View. CA ( Blumenberg, San Franciseo, (US); Franeiseo Ryan Tolmasky. Cupertino, CA (US); Richard Williamson, Los Gatos, CA (US) Andre M.J. Boule, Sunnyvale, CA (US); Henri C. Lamiraus, S Carlos, CA (US) US 2008012279641 (10) Pub, No.: US 2008/0122796 AI (43) Pub, Date: May 29, 2008 Correspondence Adres: MORGAN LEWIS & BOCKIUS LLPY APPLE, COMPUTERINC. 2 PALO ALTO SQUARE, 3000 EL, CAMINO REAL. PALO ALTO, CA 94306 11/880,635 Sep. $, 2007 Related Us ay @y Appl. No. Fite: Application Data (60) Provisional application No, 601937,99, fied on Jun, 29, 2007, provisional application No. 60/937,993, fled on jun. 29, 2007, provisional application No. 601879 469, fled on Jan. 8, 2007 provisional aplica- tion No. 60'879.253 file on Jan. 7, 2007, provisional application No, 6Q¥824,769, filed on Sep. 6, 2006, (1) IatCl GO6E 3041 (2006.01) US.Cl as/I73 6 ABSTRACT Acomputerimplemented method for usin conjunction with ‘computing device witha touch seeen display comprises: ‘detecting one or more finger contacts withthe touch sereen clspay, applying one or more heuristics to the one oF more Finger contacs to determine command forthe device, and processing the commang, The one of more heuristics eom- frse: a heuristic for determining that the one ar moe finger contacts comespond to @ one-dimensional vertical screen serlling command, aheuritc for determining thatthe oneor ‘more fugercontetscomespond tea two-dimensional screen ‘tanslation command, anda heuristic for determining thatthe ‘one oF more finger contacts correspond to a command 10 ‘transition fom displaying a respective item inasetofitemsto spaying a next tem inthe se of items. === ——4—— =a 9a SePatent Application Publication May 29, 2008 Sheet 1 of 302 US 2008/0122796 AI wine Portable Multifunction Device = 136 [Operating System | 126 [applications (continued) 149 [Communication Module [x 128 | [Calendar Module ais [Contact/Motion Module Lis |Widget Modules et [Graphics Module va [Weather Widget ~~ 434 149-2 [Text Input Module 1 135 [Stocks Widget laa [GPS Module L436 Calculator widget 1 145 4 [Applications 37 [Alarm Clock Widget | 149 5 [Contacts Module 438 [Dictionary Widget 1 Telephone Module L- : 139 149.6 |Video Conference Module" 54, User-Created Widgets) [1.5 [E-mail Client Module L s4q | [Widget Creator Module fei instant Messaging Module |" ,,. | [Search Module [Blogging Module Vows |? [Camera Module i 144 Lr 162 Image Management Module |~ Rowet 145 | System |Video Player Module }~ fae [Music Player Module tar co| External L- 124 1 [Browsing Module a bo 118 103_[ RF Circuitry A fou \ 408 Speaker 104 A tN ne (103 Kato i 122] ontroller Circuitry | 440 103 |] Peripherals | | 493 WaTphone Interface Proximity] 113 Sensor 166 103 120] Processor(s) ‘Accelerometer(s) 168 WO Subsystem ~~ 158 106 +] Display sores) Other Input 160 || Controtier perdi Controller(s) 158—| 7 7 r $103 $103 yo 103 Optical Other Input Touch-Sensitive L116 12] Sensor(s) Control Display System fa pois Figure 1APatent Application Publication May 29, 2008 Sheet 2 of 302 US 2008/0122796 AI Memory 102 Portable Multitunction Device 100. (Operating System |} 126 (apprications (continued) [449 [Communication Module + 128 | [Calendar Module aie [Contact/Motion Module te Widget Modules Lito [Graphics Module b- Weather Widgets) 149.5 Text input Module L- be [Stocks Widget ties IGPS Module 1136 [Calculator Widget — fae [Applications i an [Alarm Clock Widget | s4q 5 [Contacts Module on [Dictionary Widget | [Telephone Module i 138 1496 [Video Conference Module |" 15 [User-created Widget(s) 1 1. [E-mail Client Module }~ 141 | [Widget Creator Module ea Ft Instant Messaging Module |" ,,. | [Search Module 1188 [Blogging Module LX 443 | [Online Video Module i iCamera Module as LE image Management Module [~ 4. eo nd Music PayerModue [7172 | Power |r 162 Notes Module L- = Map Module [~ 184 | External |_- 124 [Browsing Module “ar Bor 118 103 _[RF Circuitry fetes PS es Speaker 104 +] 103; afr 11 +) Controller jee 122—| Circuitry] 103 L110 Peripherals | | 493 Werphone Interface | |{" '[Proximity] 118 Sensor W166 403 120] Processor(s) z bd ‘Accelerometer(s) 168 y103 VO Subsystem _~ 158 106 7] — Optical Display Other Input 160 ‘sensor(s) Wo || Controller coaroae Controller(s) 156—| Tt r 7 103 103 7103 a Optical Other Input Touch-Sensitive 116 a Sensor(s) Control = | 112 ~] Display System fa pois Figure 1BPatent Application Publication May 29, 2008 Sheet 3 of 302 US 2008/0122796 AI Portable Multifunction Device 100 206 210 Speaker 114 Touch Screen 4 External Port 124 Figure 2Patent Application Publication May 29, 2008 Sheet 4 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ 300A SS 208 Current Time ~» 308 310“ Day and Date~ 312 208 Wallpaper image 314 Touch Screen 112Patent Application Publication May 29, 2008 Sheet 5 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 x 3008 $$ Sw 406 208 Current Time ~» 208 3to~ Day and Datem 312 208 318 (7 316 Slide to unlock 302 306 Touch Screen 112 Home \ 204 Figure 3BPatent Application Publication May 29, 2008 Sheet 6 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 X 402 Current Time 404 © 406 q Enter Passcode r \ —— 1 2 | 3 \ J ABC | DEF 4 ) 5 6 GH JKL- MNO, 7 8 | 9 \ PQRS ) Tuv WXYZ 6 ~ Emergency 0 | Cancel Call |, : )\—* Figure 3CPatent Application Publication May 29, 2008 Sheet 7 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 208 | | 208 At 144 443 445, 149-4 149-2 M42 148 \ \ 6 >) (User- [+ Ke | ' ABC | Creates \ / \ / \ Widget _/ Calculator Alarm Dictionary Widget 1493 e405 1498 cf \ | \ Music 446 / Touch Screen 11, Figure 4APatent Application Publication Pe May 29, 2008 Sheet 8 of 302 ortable Multifunction Device 100 US 2008/0122796 A1 4008 Ey a 208 Online Video Clock 149-4 149-3 laa, 4\ 402 — Current Time 404 153 Browser 147 Figure 4B Ip ow Weather 149-1 iPod 152 J Touch Screen 112Patent Application Publication May 29, 2008 Sheet 9 208 | | 208 of 302 US 2008/0122796 A1 Portable Muttifunction Device 100 \ 500 ZA 402 Current Time 404 © 406) pole CEdit sz Instant Messages = 502 (> Jane Doe ~ 504-1 510-1 Hola » 506-1 Today 3 John Smith 504-2 596.2 508-2 510-2 Are you going tonight? January 3s, 2006 3 408-123-4567~ 504-3 es 510-3 Bye ~ 506-3 January 27, 2006 $ Bob Adams ~> 504-4 508-4 510-4 CU later + 06-4 January 35, 2008 $ 516 Touch Screen 112Patent Application Publication May 29, 2008 Sheet 10 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ Z 402 Current Time 404 © 406) gq Instant Messages | 602 Jane Doe~ 504-1 604-41 January 29, 2006 Figure 6APatent Application Publication May 29, 2008 Sheet 11 of 302 US 2008/0122796 AI Portable Multifunction Device 206 600B g Ss 21 492 Current Time 404 3 (Instant Messages \. ¢92 Jane Doe~ 504-1 | |(R U there? } a ye 604-1 January 30, 2008 604-2 8:30 608-2 (How was your vacat| XN © 406 Figure 6BPatent Application Publication May 29, 2008 Sheet 12 of 302 US 2008/0122796 AI Portable Multifunction Device 208 4 402 Current Time 404 © 406) <(instant Messages |~¢02 Jane Doe 504-1 606-1 208 (RU there? | i ~ 604-1 (IM is the best. ) _s ~ << (g29 January 30, 2008, ° 62 7 604-2 8:30 AM. ( >) obo2 Hola) ( end low was your vacat| “614 (vacant) (vacate ) (vacation) sa (vacat ) Figure 6CPatent Application Publication — May 29, 2008 Sheet 13 of 302 US 2008/0122796 AI Portable Multifunction Device 206 400 \ 600D a 4) 402 Current Time 404 we 406 208 | «(Instant Messages gg) Jane Doe soa-1 (RU there? } ved e 604-4 (IM is the best. ) S920 } January 30, 2008 oe ~ 604-2 830 AM y - 208 | Figure 6DPatent Application Publication May 29, 2008 Sheet 14 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 Ww 600E NS A a2 Current Time 404 © 406 | | < Instant Messages » 602 ~~ Jane Doe~ 504-1 92a — 604-2 January 30, 2008 8:30 AM 608-2 606-3 $ en was your vacation? ‘How was your vacation? ‘Vv BON iw a AAS WY Figure 6EPatent Application Publication May 29, 2008 Sheet 15 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 600F 2 402 Current Time 404 © 406 4 New Message 636. ~ Cancel | To: 932 634 +) (Text box 612 XQ risyY YY a|wlE Rit Y 2123, Space SB J || Retum Jo | Figure 6FPatent Application Publication May 29, 2008 Sheet 16 of 302 600G | Aw To:vl 532 Mike Van Os Venu Vick Tom Vivan 5 gee > RR j Z X C V/B NM) |] Sy AAAKNAAALY 2123 | Space Il Retum a) \ sat US 2008/0122796 A1 Portable Multifunction Device Current Time 404 © 406 New Message 636~(Cancel) et + ) Wb) 638, 650-123-2332 A 712-666-3466 It u 642 872-739-2322 v 712-342-3455 DF GH AAA Figure 6GPatent Application Publication May 29, 2008 Sheet 17 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ 600H a fs ZA 492 Current Time 404 © 408 q New Message «xs Cancel fs SS To: Mike Van Os 632 cunt + dinner +644 cn you make dinl~«s6 7 YoY nanan AL r WON v KEK dinner | Return or Figure 6HPatent Application Publication May 29, 2008 Sheet 18 of 302 US 2008/0122796 A1 Portable Muttitunction Device 206 100 ~ To: Mike Van Os Current Time 404 © 406 New Message 636~{ Cancel ) ez 656 (ean you make innit will ‘pick Dan from the day care. & alelolrlelelalel) oA Space Figure 61 8 ’ Return J 20Patent Application Publication May 29, 2008 Sheet 19 of 302 US 2008/0122796 AI Portable Multifunction Device 2) 402 Current Time 404 © 408 New Message 636~{ Cancel 208 | To: Mike Van Os 32 ox +) fom wo (nerd? |) you ma... day of" Dan frie sing \ _fom the day care. wa|| Return 4 S20 |Patent Application Publication May 29, 2008 Sheet 20 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 Ss 4) se Current Time 404 © 406) f S 3 New Message 6364 Cancel j To: Mike Van Os 632 634 +) (ean you make dinner? | will \pick Danfirom the day care, 612 2) (Send Figure 6KPatent Application Publication — May 29, 2008 Sheet 21 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 700 i a Z\ 492 Current Time 404 © 405) a 208 | Done 295 Instant Messages» 502 7" Jane Doe ~ 504-1 ria / Hola ~ 506-1 Today CO Le ? John Smith - 504-2 or 506-2 ‘e Are you going tonight?January, 28, 2006 12 408-123-4567 513 SP Bye ~ 506-3 January 27, 2006 x ("Bob Adams ~ seas 564 Cu later January 25, 2006 co ~ 6 | Delete | - 734 ‘Touch Screen 112 Figure 7Patent Application Publication 208 208 © Groups Se Bob Adams Darin Adler Darin Amon (8 ) Kim Brook 870 Cancel May 29, 2008 Sheet 22 of 302 US 2008/0122796 A1 Portable Multifunction Device 100 Current Time 404 © 406) Figure 8A, All Contacts ‘he 804 1 = NV ——— (First Name \ Last Name } 806 Br N a2 Other Number Touch Screen 112Patent Application Publication — May 29, 2008 Sheet 23 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 a 4 M2 Current Time 404 © 406 A) |( Groups) all Contacts “| 802 204 806 c ¢ (First Name x Last Name ) BU N Bt Aaron Jones Arlene Bascom 4 (B 814 : t wi Ay : Bob Adams v Bruce Walker ajo 82 Cancel Other Number Touch Screen 112 Figure 8BPatent Application Publication — May 29, 2008 Sheet 24 of 302 US 2008/0122796 AI Portable Multifunction Device 900 Z\ 402 Current Time 404 © 406) | <{ Instant Messages |~go2 Jane Doe» 504-1 Other Phone f Save) 64> ~Patent Application Publication May 29, 2008 Sheet 25 of 302 US 2008/0122796 AI Portable Multifunction Device 206 400 ~ 1000 201 Camera roll 1004 Figure 10Patent Application Publication May 29, 2008 Sheet 26 of 302 Portable Multifunction Device US 2008/0122796 A1 206 400 1100 4 Me Current Time 404 © 405 11104 ~\ Camera Roll (32) Done | X 4 Thumbnail Thumbnail Thumbnail Thumbnail 1102-01 1102-02 1102-03 1102-04 ‘Thumbnail Thumbnail Thumbnail Thumbnail 1102-05, 1102-06 A 4102-07 1102-08 ' 1112: Thumbnail Thumbnail Thumbnail 1102-09 1102-11 1102-12 108) ‘Thumbnail Thumbnail Thumbnail Thumbnail 102-14 1102-15 1102-16 Figure 11Patent Application Publication — May 29, 2008 Sheet 27 of 302 Portable Multifunction Device 206 100 R Z 402 Current Time 404 lel | ‘1202 Rl Camera Roll | 11 of 29 Figure 124, 1110) Done } US 2008/0122796 A1 > 406Patent Application Publication — May 29, 2008 Sheet 28 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ i ZA 402 Current Time 404 © 406 a Ot — S| | Camera Roll | 11 of 29 1110 Done | Delete ~7° Cancel ~? \ Camera Figure 128Patent Application Publication — May 29, 2008 Sheet 29 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 Ns Current Time 404 © 406) ‘al | 1202 / \ 2g Camera Roll | 11 of 29 1110" Done | Figure 12¢Patent Application Publication May 29, 2008 Sheet 30 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 3 a Ff 4 492 Current Time 404 © 405) 8 Photo Albums 1308} Graphic i rae Album Name 1306-1 Graphi ‘aoez Album Name 12062 | > Graphic pra 1304-3 Album Name 13063} > ey 1314 Graphic v ‘sora Album Name 12064 9 > Graphic 04.5 Album Name 13065 = > ii Album Name 2306-6 > / [ )~ 1310 Figure 13APatent Application Publication May 29, 2008 Sheet 31 of 302 US 2008/0122796 AI Portable Multifunction Device 206, 100 oT 208 2 M02 Current Time 404 © 408 208 Photo Albums sare Photo Library 1306-7 "> OQ Camera Roll 13068 | > 1 Gaze Last 12 Months 1206-9 ara Wallpaper 1306-10 Graphic Beach Fun 1306-11 > 1304-11 Graphi ij ta0t-12 Camping 1306-12 > Graphic i 4204-43 Christmas 1306-13 > Home \ \ 204 a, Figure 13BPatent Application Publication May 29, 2008 Sheet 32 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 \ Figure 14 2) 402 Current Time 404 © 406) Settings 1420 4 Music:1402 . Now Playing ts Re It fepeata 1404 > Shuffle: uffle:~ 1406. > Time Per Slide 1408 2 seconds a Transition-~1410 Wipe across a a Sta > TV signal-a4414 NTSC > Auto Rotatexs1416 On > Donew1418 Home \ 204Patent Application Publication May 29, 2008 Sheet 33 of 302 US 2008/0122796 AI Portable Multifunction Device 208 400 \ 1500 4) 402 Current Time 404 © 406 | | ( Photo Albums ) Album Name (32) XN AM502 ‘Thumbnail Thumbnail Thumbnail Thumbnail 1506-01, 1506-02 1506-03 1506-04 ‘Thumbnail Thumbnail Thumbnail ‘Thumbnail 1506-05 1506-06 A 1506-07 1506-08 1514 ‘Thumbnail Thumbnail | Thumtna ‘Thumbnail 1506-09 1508-10 1506-11 1506-12 ‘Thumbnail Thumbnail Thumbnail ‘Thumbnail 1506-13 1508-14 1508-15 1506-16Patent Application Publication May 29, 2008 Sheet 34 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 1600A a 41 402 Current Time 404 © 406) q ( Album Name ) 11 of 87 ~ * 1602 1620)Patent Appli ion Publication May 29, 2008 Sheet 35 of 302 US 2008/0122796 A1 woe \, @WOH aw, cay (sajawosajaooy, eT auoydoroiwy Figure 16B 2091 vy S ZBjOLL | swen wngiy Oz } oe For AW, JUANG OF 7 BOE Bar10suag) (For sues ‘Aywcorg 29}49C UOHOUNJNINW B1GeHOdPatent Application Publication May 29, 2008 Sheet 36 of 302 US 2008/0122796 AI Portable Multifunction Device 206 400 ~ 1700 gp eee is 4 M2 Current Time 404 © 406 | { AlbumName ) 41 of 87 NX A” 1602 Image 1606 Email photo 2 \ z Figure 17Patent Application Publication May 29, 2008 Sheet 37 of 302 US 2008/0122796 AI Portable Multifunction Device 100 8 18008 402 Current Time 404 © 406) Figure 18APatent Application Publication May 29, 2008 Sheet 38 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 N 18008 — 42 A ( Cancel ) To: Ce: Subject: Current Time 404 A 1 Great Photo \_ Image / \ 1608 Jn / Figure 18B Send) \ / © 406) APatent Application Publication May 29, 2008 Sheet 39 of 302 Portable Multifunction Device Z M02 Current Time 404 — & | | Cancel | 1 Great Photo Ce: Subject: 1 Great Photo Figure 18C. US 2008/0122796 A1Patent Application Publication May 29, 2008 Sheet 40 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 Y 18000 208 02 Current Time 404 © 406) 208 1 Great Photo ( Send : 18221 + ) a, ons, ‘y f 1824 Ce: aon Se Subject: | 1 Great Photo Figure 18DPatent Application Publication May 29, 2008 Sheet 41 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 1800E 208 AA 492 Current Time 404 © 406 208 | Z S 7 (Cancel | 1GreatPhoto { Send | : To: | 1822-4 Cc: eid 024 Subject: 1 Great Photo . <> < / ) (zy x wl alm) 2123 ) (— ) ‘Return Kets J saa) Figure 18EPatent Application Publication May 29, 2008 Sheet 42 of 302 US 2008/0122796 AI Portable Multifunction Device oe 400 \ Zi 402 Current Time 404 © 406) F (Cancel ) — 1Great Photo C Send_) To: B Wd Arlene Bascom — 1826 arlene@def.com Bob Adams A badams@abc.com Kim Brook hie kim@apple.com'* Bruce Walker Y bwalker@gmail.com cV BN JX, A NILA ‘M) (a Figure 18FPatent Application Publication 18006 7 S\ | Cancel | \ J © 492 Current Time 404 1 Great Photo To: Bob Adams Cc: Darin Adler Subject: 1 Great Photo Bob, took 1832 This is a picture | toom| Image 1606 May 29, 2008 Sheet 43 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 ‘ > 406 > Send - [alw/e/ R|T/Y u! 1 ]o| P aaa aN Figure 18G yoPatent Application Publication May 29, 2008 Sheet 44 of 302 US 2008/0122796 AI Portable Multifunction Device 206 400 \ 1800H ee 2 402 Current Time 404 © 406 [ 208 | 208 > . 7 | Cancel | 1GreatPhoto { Send \ / Xu To: Bob Adams Cc: Darin Adler Subject: 1 Great Photo Bob, 1836 This is a picture | took while visiting New York City. ONO ajwle RT YU ARAA AS |p F| GH J) Je 6000 M Ya NAA DS NS F2ggs I Space ( » Return Figure 18HPatent Application Publication May 29, 2008 Sheet 45 of 302 US 2008/0122796 AI Portable Multifunction Device 100 18001 208 A) 492 Current Time 404 © 406 / ~ 51808 1814, - \ R| |. Cancel 1 Great Photo y Send | To: Bob Adams Cc: Darin Adler Subject: 1 Great Photo Bob, This is a picture | took while visiting New York City. SN Image 1606 4) \ Z sso Save Draft 1s Delete Message | sion Edit Message Figure 181Patent Application Publication May 29, 2008 Sheet 46 of 302 US 2008/0122796 AI Portable Multifunction Device AA 492 Current Time 404 © 406 Al | Groups) ali Contacts (First Name \ Last Name ) on 7 Bob Adams ‘isté) Darin Adler e20) Darin Amon Kim Brook N Cancel Other Email Touch Screen 112 Figure 18JPatent Application Publication May 29, 2008 Sheet 47 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 \ 208 > 406) Current Time 404 208 Groups All Contacts CA Darin Adler Darin Amon Cancel (First Name Y Last Name ) Bob Adams L Other Number Touch Screer Figure 190Patent Application Publication May 29, 2008 Sheet 48 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N 19008 ey i Current Time 404 © 406) 3 1902 This is what it will look like when “Bob Adams” calls. Move and scale to adjust. Figure 19BPatent Application Publication May 29, 2008 Sheet 49 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ™ | 2000 Current Time 404 © 406 002 208 This is what the wallpaper will look like. Move and scale to adjust. Set Cancel Wallpaper 2004 2006 : / \ 4 Figure 20Patent Application Publication May 29, 2008 Sheet 50 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 Nx a Videos Playlists ? Music Videos a Podcasts a TV Shows a Movies a Figure 241APatent Application Publication May 29, 2008 Sheet 51 of 302 Portable Multifunction Device US 2008/0122796 A1 Figure 21B Z 402 Current Time 404 > 406 , Playlists My Movies @*% > Recently Added “sPatent Application Publication May 29, 2008 Sheet 52 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 2100 208 A) 492 Current Time 404 © 405) 208 - ~\ 2106 (Playlists My Movies Graphic King Kong 3h 7m Graphic Spiderman 2h 15m, Graphic Star Wars 1h 55m. Figure 21CPatent Application Publication May 29, 2008 Sheet 53 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 208 2) 402 Current Time 404 © 406 g Settings Scale: Scale to fit a Figure 22A\Patent Application Publication May 29, 2008 Sheet 54 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ “| 402 Current Time 404 © 408) 2204 Scale q ( Settings ‘ 2206 2208 (Scale to fit | Scale to full) 2210 2212 $ $ Cancel Done So Home \ Figure 228Patent Application Publication May 29, 2008 Sheet 55 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N a g Nl st] 3 D8 a8 a x iS OF Sul] JUSIND Figure 23APatent Application Publication May 29, 2008 Sheet 56 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 208 | | 208 o1ez ORz OBpIA aez WF owl juaUND Figure 23BPatent Application Publication May 29, 2008 Sheet 57 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 208 208 FOR owl] JUBIND a oe Home \, Figure 23CPatent Application Publication May 29, 2008 Sheet 58 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 23000 208, 208 | yor Oui) JUeLND oF cm Figure 23DPatent Application Publication May 29, 2008 Sheet 59 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 2400, “402 Current Time 404 © 405) a { Sunny Santa Cruz, CA\ ; 66° H: 76° L: 58° Monday 75°58? Tuesday xe 72°55" Wednesday Xt oses7° Thursday 75°58" | | Friday xe 77°60" ‘Saturday x 78° 62° : Figure 248Patent Application Publication May 29, 2008 Sheet 60 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 24008 a a Al Zi 402 Current Time 404 © 406) | | City, State or Zip Code ee Ne San Francisco, CA a2 Cupertino, CA Santa Cruz, CA New York, NY ‘Fahrenheit Celsius ) Remove Done 2408 2410, Figure 248Patent Application Publication May 29, 2008 Sheet 61 of 302 US 2008/0122796 AI Portable Multifunction Device 100 2400¢ Z| 402 Current Time 404 © 408 a City, State or Zip Code Rutl| San Francisco, CA Cupertino, CA (Rut!) Rutland, VT) Rutland, MA ay Cc |) E[R } thy D AA AN AA S\N. / SWS OG Figure 24CPatent Application Publication — May 29, 2008 Sheet 62 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 4 M2 Current Time 404 © 405 20 iF Weather | Done San Francisco, CA ana oF } °C 1e \ 204) Figure 24DPatent Application Publication May 29, 2008 Sheet 63 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 \ 208 qi 76° H:76° L:58° (Monday xt 78° 68° Tuesday x TT° 67° | Wednesday Xf 79°64 Thursday x 78°68? Friday xe 79°66? [Saturday xt 80°67" | 2402, : Figure 24EPatent Application Publication | 208 ‘Speaker 111, 2500A At May 29, 2008 Sheet 64 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \y Optical Proximity Sensor 164) | Sensor 166, Current Time 404 © 406: (mma (ae) ‘ome \ 204 4) Figure 25A INDU = 11240.35—-2. Bho } AAPL 68.30 -1.29 DIS 29.90 -.14 GOOG 373.78 -1.61 INTC —- 17.49 +.16 1d 1w 1m 3m 6m ‘y 2y WS fe (em)Patent Application Publication May 29, 2008 Sheet 65 of 302 US 2008/0122796 AI Portable Multifunction Device ae OE ee SEN Optical Proximity 25008 — Current Time 404 > 406 | 208 INDU) = 11240.35 = -.01% AAPL 68.30 -1.85% DIS fash 8) --46% GOOG 373.78 -42% INTC 17.49 +.92% 1d 1w 1m 3m 6m ‘y 2y foo™~ Microphone) / Home ‘Accelerometer(s) 13) \ 204 | 168 VS Figure 25BPatent Application Publication May 29, 2008 Sheet 66 of 302 US 2008/0122796 AI Portable Multifunction Device 100 ~ 208 ‘Speaker 111 2500C, 3 im 402 Current Time 404 2 — IS 0508" X ) INDU Dow Jones Ind. Avg. 2510 AAPL Apple Computer DIS Walt Disney Company GOOG _ Google, Inc. INTC Intel Figure 25CPatent Application Publication May 29, 2008 Sheet 67 of 302 US 2008/0122796 A1 Portable Multifunction Device 7206) 100 ‘ 208. 208 ~< as Current Time 404 Dow Jones Ind. Avg. i Apple Computer Walt Disney CompanyPatent Application Publication May 29, 2008 Sheet 68 of 302 US 2008/0122796 AI Portable Multifunction Device 2500E 21 402 Current Time 404 © 406 al | ( Done \2514 Quote: INDU Dow Jones Ind. Avg. 2510 AAPL Apple Computer DIS Walt Disney Company GOOG _ Google, Inc. INTC Intel Figure 25EPatent Application Publication May 29, 2008 Sheet 69 of 302 US 2008/0122796 AI Portable Multifunction Device _I 400 \ 26008, ag fs “| 402 Current Time 404 © 406 f All Contacts 2604-~efn (A) Bob Adams Darin Adler Darin Amon NxxE 2000 0000 0000 Dial Voice Mail Figure 26APatent Application Publication May 29, 2008 Sheet 70 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 21 492 Current Time 404 © 406) 5 Enter a name for this contact oS oN | Cancel | New Contact Save | L / Narey, - ~ 2606 First Name | a. Last Name | J (~ ~\ Company i Teleliio aa CeeeuduE oP Figure 268Patent Application Publication May 29, 2008 Sheet 71 of 302 US 2008/0122796 AI Portable Multifunction Device § 2) 402 Current Time 404 © 406) ( Enter a name for this contact 3 0 > i _ Cancel | New Contact |Save \ / 4 ~ = = a 8 |Ron \ J (smith } X |Company “ Figure 26CPatent Application Publication May 29, 2008 Sheet 72 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 2600D q < 402 Current Time 404 © 406 a (All Contacts } Info ‘Done } (a Add if 1 ‘ Photo Ron Smith > 2607 ) ) |Add New Phone > 261 GN > “sb Add New Email > ey \ 26124 . - ~ +) (Add New Address > ( Delete Card _) |Add NewURL > QO Voice Mail Figure 26DPatent Application Publication May 29, 2008 Sheet 73 of 302 US 2008/0122796 AI Portable Muttifunction Device 208 100 \ 2600E _ q AA 402 Current Time 404 © 406 af | (All Contacts) Info (Done } ( ~ Add *, | Ron Smith > Photd |. 7 2607 (A) Add New Phone > 2610, +) Add NewEmail > Take Photo 22) (Choose Existing Photo” a ~ L Cancel 2014 Ww 2 © ggg, OO Favorites Recent Dial Voice Mail Figure 26EPatent Application Publication — May 29, 2008 Sheet 74 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 al al ‘Phone ) Home 2620 Figure 26FPatent Application Publication — May 29, 2008 Sheet 75 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 oT 4, 492 Current Time 404 > 405) _ Edit Phone | Label 2620 Edit | {- mobile : home 4 work main Home fax Work fax pager other preferred Add Custom Label~ 2626 oo { Home \ \ Figure 266Patent Application Publication May 29, 2008 Sheet 76 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 2600H 208 2 402 Current Time 404 © 406 {Edit Phone | Label 2638 Done| \ ) one) 208 mobile home f work main Home fax Work fax pager other 7) preferred CB Add Custom Label Figure 26HPatent Application Publication May 29, 2008 Sheet 77 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N 26001 2 402 Current Time 404 © 406 / ~ = 4 Edit Phone Label 26384 Done | S J Nae, mobile home 4 work main Home fax Work fax pager other = ( Delete \ ) Figure 261Patent Application Publication — May 29, 2008 Sheet 78 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ a 402 Current Time 404 © 406 3] | | Cancel Edit Email 2048 Save | NX S XX S Email r 5 Home > 2646 Paes Q wie[R{T ly u;1/o|P CK A . A K Figure 26)Patent Application Publication — May 29, 2008 Sheet 79 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 41 492 Current Time 404 © 406) {Cancel | Edit URL 2640 Save) ‘URLPatent Application Publication — May 29, 2008 Sheet 80 of 302 US 2008/0122796 AI Portable Multifunction Device 26001 a im 2 402 Current Time 404 © 406 ll | S) aa | | Cancel | Edit Address 2658 Save ee) (street Street Figure 26LPatent Application Publication — May 29, 2008 Sheet 81 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ 4 2 Me Current Time 404 > 406) 81 | | All Contacts | Info ( My. fi. : Bruce Walker Graphic Work (408) 123-4567 Home (408) 974-1234 Work — bwalker@company.com \ / Home — brucew@aol.com la Nor ~ | Instant Message Add to Favorites | QO Voice Mail Figure 26MPatent Application Publication May 29, 2008 Sheet 82 of 302 US 2008/0122796 AI Portable Multifunction Device 100 2600N a 2) 402 Current Time 404 © 408) al “All Contacts | Info ( Edit | Graphic Bruce Walker J (Work — (408) 123-4567 ) Send an instant message zee Work (408) 123-4567) aseo{ Home (408) 974-1234) ( Cancel ) Figure 26NPatent Application Publication May 29, 2008 Sheet 83 of 302 Portable Multifunction Device 26000 | (allcontacts) Info | Graphic (Work (408) 123-4567 : (408) 974-1234 brucew@aol.com ‘Add new Address Figure 260 Zi 492 Current Time 404 Bruce Walker : Apple Computer > bwalker@company.com 7 Delete Card) Dial ze) > US 2008/0122796 A1 QO Voice MailPatent Application Publication May 29, 2008 Sheet 84 of 302 US 2008/0122796 AI Portable Multifunction Device 206 400 ‘ ‘a Ss ZA 492 Current Time 404 > 406 lel | \ | i 8) | Cancel Edit Phone Save | \ ) aay, 2626 Phone 2676 Figure 26PPatent Application Publication — May 29, 2008 Sheet 85 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 27004, gp fs My Zi 402 Current Time 404 © 405) S| Edit bor Favorites 2r0e~ NX S 2704 é Kim Brook-7=: Mobile“? >) Arlene Bascom Mobile > Bob Adams Mobile > Bruce Walker Work Fo Aaron Jones Other > Darin Adler Work > © a3 YW 2 000 000 000 Dial Voice Mail Figure 27APatent Application Publication May 29, 2008 Sheet 86 of 302 US 2008/0122796 AI Portable Multifunction Device 5 Zi 402 Current Time 404 © 406) i 8] | | Done +2718 Favorites Mobile = (-) Arlene Bascom Mobile | = -_ — (=) Bob Adams Mobile *=} (=) Bruce Walker Work ) Aaron Jones Other TT =) Darin Adler Work O 000 TY 2 000 900 000 Favorites] Recent Contacts Dial Voice Mail Figure 278Patent Application Publication May 29, 2008 Sheet 87 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 2700€ el g ql ZI 402 Current Time 404 © 406 Done 2718 Favorites 208 (-) Kim Brook Mobile = Arlene Bascom Mobile = Bob Adams Mobile = ‘Bruce Walker mer Remove | (=) Aaron Jones Other = =) Darin Adler Wok = ooo WW : 000 000 000 Favorites} Recent Contacts Dial Voice Mail Figure 27CPatent Application Publication May 29, 2008 Sheet 88 of 302 US 2008/0122796 AI Portable Muttifunction Device 206. 100 X (a 8 2) 402 Current Time 404 > 406 = al | ‘Done wae Favorites (=) Kim Brook Mobile = Arlene Bascom Mobile = (=) Bob Adams Mobile} Other (=) Darin Adler Work © e38 : 000 000 000 Favorites| Recent Contacts Dial Voice Mail Figure 27DPatent Application Publication — May 29, 2008 Sheet 89 of 302 US 2008/0122796 AI Portable Multifunction Device 2700E Z 402 Current Time 404 © 406] 0 \ 3] | | Done vzris Favorites Kim Brook Mobile lene Bascom Mobile () Bob Adam Mobile?) NS = =e = (=) Aaron Jones Othe = (=) Darin Adler Wok = © 000 Ww : 000 000 000 Recent Contacts Dial Voice Mail Figure 27EPatent Application Publication May 29, 2008 Sheet 90 of 302 US 2008/0122796 A1 Portable Multifunction Device 208 100 N Figure 27F 2700F § 4 492 Current Time 404 (> 408 2 | | Done Lore Favorites Mobile = Mobile = (-) Bruce Walker Work = ») 727e Bob Adams Mobile” 7 = (-) Aaron Jones Othe = (=) Darin Adler Work = 800 Ww © aa 33g, OO Favorites Recent Contacts Dial Voice Mail ‘ome \ 204Patent Application Publication May 29, 2008 Sheet 91 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ Ql ql ZA 492 Current Time 404 © 406) 2810 2612 33% al (Missed) ( Clear ) 4 2802 2604 2808-808 Kim Brook (3) 2pu >) <>" Bruce Walker (2) Jan 26 10:53 am ( Bob Adams Jan 26 10:30 AM (408) 654-3210 (5) Jan 26 9:16 am > “Darin Adler A Jan 26 9:09 AM 12814 Aaron Jones! Jan25 10:37 PM ' 2 Bruce Walker v Jan 25 9:32 PM >) John Smith Jan 2sa27em (> 000 000 000 se OO Dial Voice Mail Figure 28APatent Application Publication May 29, 2008 Sheet 92 of 302 US 2008/0122796 AI Portable Multifunction Device 208 1400 oi [| 28008 Current Time 404 © 406) 2812 | 432. a “cE (Missed) © Clear Bruce Walker (2) Jan 26 10:53. am > >’ Darin Adler Jan 26 9:09 AM ©) Bob Adams van 256:43PM |, . te 2614 Kim Brook (3) § Jan255:30 pm (> (408) 234-5678 W ven 25448 Pm Arlene Bascom Janzs4cspm (>) Bruce Walker Jan2sas0pm (>>) Favorites Figure 28BPatent Application Publication May 29, 2008 Sheet 93 of 302 US 2008/0122796 A1 Portable Multifunction Device 208 100 X 2800C fs 4) 402 Current Time 404 © 405 §| | Recent Calls} Bruce Walker Graphic Bruce Walker | 2 Missed Calls 421072006 «10:53 AM; 10:58 AM Work (408) 123-4567 2816 Home (408) 974-1234 Figure 28C (Work — bwalker@company.com } (| Home brucew@aol.com 2822 | Instant Message Add to Favorites kd 800 seo OO Favorites Dial Voice MailPatent Application Publication May 29, 2008 Sheet 94 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 X 2800D a q Z 402 Current Time 404 © 406) ~ am 4 (Recent Calls} Outgoing Call { Clear ) (408) 654-3210 San Jose, CA Five missed calls 01/29/06 1:27 pm, 3:02 pm, 4:15 pm, 4:47 pm, and 6:29 pm Call (408) 654-3210 ( Instant Message Create New Contact 1826 Add to Existing Contact 2828 J Favorites Voice Mail Figure 28DPatent Application Publication May 29, 2008 Sheet 95 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N 2900 208 Z| 402 Current Time 404 © 408) (650) 132-22} am 208 x © kh Favorites Recent Contacts Voice Mail Figure 29Patent Application Publication May 29, 2008 Sheet 96 of 302 US 2008/0122796 AI Portable Multifunction Device 5 Z 402 Current Time 404 © 406 4 (650) 132-2234 Calling . . . 200 000 000 000 xp 000 Mute Keypad Speaker 3006 3016 3010 + Wo Ly Add Call} Hold Contacts 3018 3008 3020 a= EndCall 2 w © Favorites Recent Contacts QO Voice Mail Figure 30APatent Application Publication May 29, 2008 Sheet 97 of 302 US 2008/0122796 AI Portable Multifunction Device Is 8 “| 402 Current Time 404 © 406] 4 (650) 132-2234 sou 00:55 oooo oooo ooo0oo p Mute Keypad Speaker 3006 3016 2010 + | fy Add Call Hold Contacts 3018 3008 3020 | © EndCall au x © & Favorites Recent Contacts Figure 30BPatent Application Publication May 29, 2008 Sheet 98 of 302 US 2008/0122796 AI Portable Multifunction Device f] 3000c Sa a 2) 402 Current Time 404 © 406) a | Graphic for Arione Arlene Bascom Bascom me m2 | Ignore a8 (Hold Call + Answers) End Call + Answers | x © a Favorites Recent Contacts QO Voice Mail 204 / Figure 30CPatent Application Publication May 29, 2008 Sheet 99 of 302 US 2008/0122796 AI Portable Multifunction Device 208 2 402 Current Time 404 © 406 Arlene Bascom [ 208 3031 00:01 ( 000 ) ooo 000 xD 000 Mute Speaker 3006 3010 + ps Add Call Contacts 3018 ) am EndCall sou + O de Favorites Recent Contacts Voice Mail Figure 30DPatent Application Publication May 29, 2008 Sheet 100 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 x 208 B Current Time 404 © 406) 208 van (650) 132-2234 28 Hold su Arlene Bascom & 00:05 ooo \ ooo 000 xp ooo Mute Keypad | Speaker 3006 3016 3010 = Merge Swap Contacts mm EndCall xu & O d Favorites Recent Contacts Figure 30EPatent Application Publication May 29, 2008 Sheet 101 of 302 US 2008/0122796 AI Portable Multifunction Device 100 \ Speaker 111, 3000F ZI 402 Current Time 404 © 406 [ 208 208 aus (650) 132-2234, 01:07 305 Arlene Bascom On Hold 000 ) 000 900 Kp ooo Mute Keypad | Speaker 3006 3016 3010 rv a Merge Swap Contacts 3038 3009 3020 / @= EndCall x4 se © da Favorites Recent Contacts QO Voice Mail Figure 30FPatent Application Publication — May 29, 2008 Sheet 102 of 302 US 2008/0122796 AI 208 | | 208 | Mute Keypad Speaker 3006 + Add Call} — Hold Contacts (3018 x Favorites Portable Multifunction Device Current Time 404 © 406) Conference 7S) 01:05 ‘S 000 | 883) 000 3016 3010 | 3008 3020 @ EndCall wu | © a Recent Contacts Figure 30GPatent Application Publication May 29, 2008 Sheet 103 of 302 US 2008/0122796 AI Portable Multifunction Device 208 400 ~\ 3000H (208 ZI 492 Current Time 404 © 406. Va 8048 a] | (Back 7 Conference 3058 (650) 132-2234 ms _( Péveh 3050-1 3056 “FagvArlene Bascom 20 { Private ) 3086-2 E 3056-4 ea Figure 30HPatent Application Publication May 29, 2008 Sheet 104 of 302 US 2008/0122796 AI Portable Multifunction Device 208 \ B Current Time 404 (© 406 ** Conference F (650) 132-2234 it, Private (End Call) Arlene Bascom 2% Private ) : J 3062 208 la oD 2 2 x Figure 301Patent Application Publication 208 [208 Portable Multifunction Device 100 Z 402 Current Time 404 © 406 3033 (650) 132-2234 & 01:45 90258037 2033 Arlene Bascom‘__2 Hold 000 7 seo |X) o0o°o Mute Keypad | Speaker 3006 3016 3010 A no Merge Swap Contacts 3009 3020 = EndCall su t+ O da Favorites Recent Contacts QO Voice Mail Figure 30J May 29, 2008 Sheet 105 of 302 US 2008/0122796 ALPatent Application Publication May 29, 2008 Sheet 106 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ (987) 654-3210 3066 Ignore 028 _ Hold Call + Answers) | End Call + Answer 20%) w Ok Favorites Recent Contacts QO Voice Mail Figure 30KPatent Application Publication May 29, 2008 Sheet 107 of 302 US 2008/0122796 AI 208 | | 208 206 3000L, 402 Mute 3006 + 3018 Favorites Add Call —_ Hold Contacts @ EndCall su x © & Portable Multifunction Device Current Time 404 © 406 (987) 654-3120 3 00:01 4 Speaker 3010 | oo0o°o oo0oo oo0o ~ ko D So o Qa 3008 3020 Recent Contacts Figure 30LPatent Application Publication May 29, 2008 Sheet 108 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ (208 | 402 Current Time 404 © 406) 208 sos Conference S38Hold 308 (987) 654-3210 {, 00:10 ae noone , ooo 000 Kp ooo Mute | Keypad | Speaker 3006 3016 3010 A uv & Merge Swap | Contacts 3038 3009 3020 } ( @e EndCall ws | * © Favorites Recent Contacts Figure 30MPatent Application Publication May 29, 2008 Sheet 109 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 i al 1 20 | 3 \ ) ABC) | DEF L432) U8 GH» | JKL) ~\-MNO 7 | 8 | g | ™ (PQRS, | TUV) \_WXYZ_) [* [ol a \ Je J | - ~ 3071 ss8074) | End Call | Hide Keypad \ JX J w © da|# joo Favorites Recent. Contacts Voice Mail Figure 30NPatent (208 208 Application Publication May 29, 2008 Sheet 110 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 402 Current Time 404 > 406) All Contacts (First Name \ Last Name ) Bob Adams Darin Adler A Darin Amon N Cancel Other Number ‘Touch Screen 112 Figure 300Patent Application Publication — May 29, 2008 Sheet 111 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 3000P_ S 4) 492 Current Time 404 © 405) ‘a ~ ( \ @ | | All Contacts Info Edit | XY \ J a . Bruce Walker Graphic (Work — (408) 123-4567 ,) Home (408) 974-1234 ) \ Work — bwalker@company.com ) / Home brucew@aol.com ) - > Add to Favorites 338 so OO Dial Voice Mail Figure 30PPatent Application Publication May 29, 2008 Sheet 112 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 oi al 7 al Calculator 149-3 Calendar us Clock 149-4 408 Photos. 4 Map 154, Settings 412 Browser war Figure 300 sogoa 3 Current Time 404 © 406 Camera 43 Weather 149-1 © iPod 182 Touch Screen 112Patent Application Publication May 29, 2008 Sheet 113 of 302 US 2008/0122796 AI Portable Multifunction Device 100 208 Z 402 Current Time 404 © 406 Touch to return to call. 3078 [208 ( Cancel ) 1GreatPhoto { Send X / XY / To: Ce: Subject: 1 Great Photo Figure 30RPatent Application Publication Portable Multifunction Device 31008, 2 402 Current Time 404 © 405) Arlene Bascom mobile 3102 Graphic for Arlene Bascom 3104 r \ or ~ | Decline Answer wv © da| Bi lao Favorites Recent Contacts Voice Mail Figure 310, May 29, 2008 Sheet 114 of 302 US 2008/0122796 A1Patent Application Publication May 29, 2008 Sheet 115 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 fe ZZ 402 Current Time 404 © 406) & (987) 654-3210 3110 Decline 306 w © de| Joo Favorites Recent Contacts Voice Mail Figure 318Patent Application Publication May 29, 2008 Sheet 116 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 32008, a s ZA 402 Current Time 404 © 406) a Voicemail $2002 3204 0 208% 3208 . «@ ’ 2x } SY) aa» ' ) a, weet OY 3214-ty 210 : 3 - ‘Aaron Jones “ois. ’3 (408) 246-8101 ony 0 b210- ‘ S24 “Bob Adams}? ‘o1s108 >) 3210- 3214-4, ~ “Bruce Walker “O1/40/06 >) 3218 3220 S 000 Ww 000 000 ) 000 Favorites Recent Contacts Dial | Voice Mail Speaker Options 3216 3218 Figure 32APatent Application Publication May 29, 2008 Sheet 117 of 302 US 2008/0122796 AI Portable Multifunction Device 7] 32008 3g A Z. 402 Current Time 404 > 408 3 Voicemail oS 5 8202_g3204 = 3210-4 3212-3214 Aaron Jones 011106 (>) Delete voicemail from Bob Adams Cancel Delete (3228 J cz ee Speaker Options OQ 3216 3218 3230 3222, $80 Ww © aa ggg | QO Favorites Recent Contacts Dial | Voice Mail Figure 328Patent Application Publication May 29, 2008 Sheet 118 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 3200 “| 402 Current Time 404 © 406 — 7 S 3 | | Greeting | Voicemail (4) { Speaker | Not heard 3232 3214-h ~ @ aaron Jones — 01/3106" > eb. 2214-2 3& (408) 246-8101 ors108?? (> Download 3234 ~ 3214 Il, Bob Adams }. o13106*?? (>) Pause 3220 "3204 oO : aotna > Bruce Walkery 01/30/06" ) Play 3230 ~ 304 6° S208. 1:22 7 ) 2:43 NY a“ alt \ss20 ( Delete | \ J) \ } vO ky Favorites Recent Contacts Figure 326May 29, 2008 Sheet 119 of 302 US 2008/0122796 AI Patent Application Publication Portable Multifunction Device 206 100 \ 3200D ee 402 Current Time 404 © 408 4 a 4 Greeting | Voicemail (4) [ Speaker | 210 at 3214 @ Aaron Jones — 0131/08 (>) 3214-4 \ >) } 3210-2 y > (408) 246-8101 o1ati0¢ Delete voicemail from Bob Adams Cancel Delete | y 3228 3226 ) 2:43 Delete 52? QO Contacts Voice Mail Favorites Figure 32DPatent Application Publication May 29, 2008 Sheet 120 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ Zi 492 Current Time 404 © 405) a Voicemail 3242 To retrieve your voicemail, please first set a password and greeting. ( Set up now | w © dm Boo Favorites Recent Contacts Voice Mail Figure 32EPatent Application Publication 208 | 208 | Portable Multifunction Device 206 100 \ 2) 402 Current Time 404 3 406 Voicemail 3246 3248 sg Password Greeting | fo ‘ [Input Field 3249 | xX J © Favorites Recent Contacts Voice Mall so™ { Home \ 204 Figure 32F May 29, 2008 Sheet 121 of 302 US 2008/0122796 A1Patent Application Publication May 29, 2008 Sheet 122 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ Ww Favorites 1) 402 Current Time 404 © 406 Voicemail 246 ange ~ Password Greeting ) O > Record Play 3250 3282 <“) e Speaker Reset 3254 3256 © dm Blo Recent Contacts Voice Mail Figure 326Patent Application Publication May 29, 2008 Sheet 123 of 302 US 2008/0122796 AI Portable Multifunction Device 100 \ Speaker 111, 3200H 208 402 Current Time 404 © 406 g Voicemail 3246 3248 a ft _ Password Greeting ) O Record Stop 3250 3258 h Sent ye > v Trash > Personal > Work > Figure 33Patent Application Publication May 29, 2008 Sheet 125 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 \ 208 ( Cancel \ J \ 3402 34044, \ pe" New Message” ¥ Send | Current Time 404 © 406) 7 WEY +) (Home \ \, 204 \ A Figure 34APatent Application Publication May 29, 2008 Sheet 126 of 302 US 2008/0122796 AI Portable Multifunction Device _[ 208 100 X (als 3400B 4 492 Current Time 404 © 406) 4 New Message (Cancel ) To:B +) Bart Simpson bart@simpson.com Arlene Bascom arlene@def.com Bob Adams 4 3426 badams@abe.com Kim Brook 3428 kim@apple.com Bruce Walker t bwalker@amail.com fa] w 1GOG jytu" {ofr} NY SZ + \ AY _ AN a 7 DiF GH) yy z|x/olv) 8) n/m] (@ SO) OE NY Figure 348Patent Application Publication May 29, 2008 Sheet 127 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 2 402 Current Time 404 © 406 ~ 53402 34044 \ q ( ‘Cancel ) Barbeque on Saturday? Send | To: Bruce Walker Cc: Janet Walker Subject: Barbeque on Saturday? (Attach Rs410 Bruce, Do you and Janet want to have| py a|wle Figure 34CPatent 208 | | 208 Application Publication May 29, 2008 Sheet 128 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 3500A, Z 402 Current Time 404 © 406) ae 53502 (or » (3504 Mailboxes {Inbox ‘6% ron yone: January 30, 200¢ ” A J 8805-1 eo Re: Project Orion"°*" 3124 2 é Bruce Walkep®""4anuary 30, 2006" Lunch meeting today? 7 5 3510-3} Kim Brook” "A January 30, 2006" Draft agreement bast Bob Adams! January 29, 2006 Project Orion 3512, . é Darin Adler January 29, 2006 fantasy football Aaron Jones —_ January 29, 2006 Fwd: status report 4578) Figure 35,Patent Application Publication May 29, 2008 Sheet 129 of 302 US 2008/0122796 AI Portable Multifunction Device 3500B E Z 402 Current Time 404 © 406 ., > 3502 (pr 53504 3 || Mailboxes "~~ Inbox 6/ 3510-1 ~ Jarary 30, Aaron Jones**" J A 30, 2006" : Biz 3506-2 by 3514 @ Bruce Walker’ ey 30, 2006" Kim Brook Janvary 30, 2006 — \V_ Assis 5 3530 Draft Agreement A Details January 30, 2006 9:50 AM a John, 49882 The draft agreement for the Redstone Project is attached. Please look it over and give me your comments. Thanks. PHO? Figure 368Patent Application Publication May 29, 2008 Sheet 130 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N 2) 402 Current Time 404 © 406) + ~ So 4 |{ Mailboxes ¥ inbox 6° Aaron Jones?" sary 30, 20067" 3514 ea 3510-4, Bruce Walker’ Jant fiary 30, 2006" 7 Z 10- Kim Brook” Jatlary 30, 2006" 3628 Draft Agreement danvary30, 2006 9s0am_ A | From: Kim Brook > a To: jsmith@company.com > oa eee |Cc: Aaron Jones > 43a, | v \___bwright@job.comt>” ei, 3534-1 John, / BS Figure 35CPatent Application Publication May 29, 2008 Sheet 131 of 302 US 2008/0122796 AI Portable Multifunction Device 3500D ZI 402 Current Time 404 © 408 53504 6 208 208 ( . ~\ 3502 Mailboxes |" Inbox ~ January 30, a Aaron Jones**" J 30, 2006" ‘3512-1 506-2 e Bruce Walker” January 30, 2006" . _53506-3, 3510-9, Kim Brook January 30, 2008 3539 or 3539 a 13538} a ‘oe 3528 vn LV 4 K 3599 LV dois 3539 Draft Agreement January 30, 2006 9:50 AM From: Kim Brook > To: jsmith@company.com > Cc: Aaron Jones > bwright@job.com > Figure 36DPatent Application Publication May 29, 2008 Sheet 132 of 302 US 2008/0122796 AI Portable Multifunction Device 3500E 21 402 Current Time 404 208 (Mailboxes f°” Inbox Aaron Jones?" Janay 30, 2008" 12-1 © Bruce Walker” ‘enkay 30, 2006" Kim Brook” Jalary 30,2006 w Rss18 Delete Email from Kim Brook r \ | Cancel Delete \ 3540 yy iS 3542 yy Y comments. Thanks. Figure 35EPatent Application Publication May 29, 2008 Sheet 133 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 3500F 4 Z 492 Current Time 404 © 408 i, >\ 53502 53504 4 Mailboxes * Inbox 6° \ J a) 5. 1510-1 Aaron Jones*™ Janktary 30, 2006" prt 3506-2 4 © Bruce Walker’ Janbiayy,30, 2006"°"") " 4 Kim Brook** Janary 30, 2006 w 23518 Reply Reply Forward = comments. Thanks. Figure 38FPatent Application Publication May 29, 2008 Sheet 134 of 302 US 2008/0122796 AI Portable Multifunction Device Is 35006 LZ 402 Current Time 404 © 406 4 ‘Mailboxes FP nbox 6) OO 3850 3552 3510-1 ~ January 30, Aaron Jones" 30, 2006" Re: Project Orion" 35124 )- é Bruce Walker® "January 30, 200677" Lunch meeting today” a . 5. 510-3] Kim Brook’"""A January 30, 2006" Draft agreemeht 10514 Bob Adams = January 29, 2006 Project Orion v 3 "2 . é arin Adler January 29, 2006 fantasy football Aaron Jones January 29, 2006 Fwd: status report Figure 356Patent Application Publication May 29, 2008 Sheet 135 of 302 US 2008/0122796 AI Portable Multifunction Device 3500H 2 al | 402 Current Time 404 © 406) Lay =e »\ 53502 al ‘Mailboxes | Inbox (6) 3950 3552 3510-1] Aaron Jones“ Jarfdary 30, 20065 @ Bruce Walker™” "Janka 30, 208 Kim Brook* salary 30, 2006" 3528 vs| Draft Agreement 5 3530 January 30, 2006 9:50 AM Details a > John, 5 3532 The draft agreement for the Redstone Project is attached. Please look it over and give me your comments. Thanks. 3558 Ov 6a v Figure 35HPatent Application Publication May 29, 2008 Sheet 136 of 302 US 2008/0122796 AI 8 Portable Multifunction Device 100 35001 208 Z 492 Current Time 404 © 406) —— ~\ 53502 Mailboxes | _ Inbox (6) - - 3550 3552 10-1 Aaron Jones” January 30, 2006" e e esbe2 3510-2) Bruce Walker 1 January 30, 2006" i Kim Brook*°V January 30, 20064 208 | Reply | Reply All Forward Cancel } Figure 351Patent J Application Publication May 29, 2008 Sheet 137 of 302 US 2008/0122796 A1 Portable Multifunction Device 208 400 \ 3500J “402 Current Time 404 © 406 [ Mailboxes "inbox “(edit ) Aaron Jones**** Yanuary 30, 2006 Re: Project Orion 78? > Here is the user manual. Please remember this is probably outdated. ~~3564-1 3506-2 Bruce Walker’ 4 January 30, 2006 Lunch meeting today ~-3508-2 > Hi, any time for today? | wfant to show you some pictures. -3564-2 : Bob Adams V January 29, 2006 Project Orion > Anyone have the user manual? Darin Adler January 29, 2006 fantasy football > Time to come up your teams now! U 7 Figure 35)Patent Application Publication May 29, 2008 Sheet 138 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ~ 3500K “ 2. 402 Current Time 404 © 405 4 S i A 3566 From: |Bruce Walker > g 3530 Lunch meeting today January 30, 2006 9:30am Hi, anytime for today? | want to show you some pictures. 3570-4 Nice jpg 2MB 7 9572-2 Figure 35KPatent Application Publication May 29, 2008 Sheet 139 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 3500L, 4 402 Current Time 404 © 405) 4 | inbox | 206, (a Whe From: (Bruce Walker > To: (Jen Taylor > }52 3504 Lunch meeting today January 30, 2006 9:30am @ Mark as Unread Hi, anytime for today? | want to show you some pictures. Nice 3572-1 XM o c wr 3572-2 Image 2 Figure 35LPatent Application Publication May 29, 2008 Sheet 140 of 302 US 2008/0122796 AI Portable Multifunction Device 100 206 3500M 3 fS 2 402 Current Time 404 © 406 {| tox) 1086, (Wy From: (in Joss >) 3530 Re: Project Orion January 30, 2006 7:30am Here is the user manual. Please remember this is probably outdated ~Aaron 3570-3 User Manual 14MB > Figure 35MPatent Application Publication May 29, 2008 Sheet 141 of 302 US 2008/0122796 AI Portable Multifunction Device 206, 100 X 3500N, As q ZI 492 Current Time 404 © 400) dis yr 3874 4 (Message) User Manual ... 3512.3 Page 1 Page 2 \ Home \ 204 Figure 35NPatent Application Publication Portable Multifunction Device 206 100 \ 35000 So 2 402 Current Time 404 © 406) — . (Mailboxes Inbox Aaron Jones —_ January 30, 2006 Re: Project Orion > Here is the user manual. Please remember this is probably outdated. Bruce Walker — January 30, 2006 Lunch meeting today > Hi, Any time for today? | want to show you some. pictures. Bob Adams January 29, 2006 Project Orion Anyone has the user manual? Darin Adler January 29, 2006 fantasy football > Time to come up your teams now! 3560 3310 Figure 350 May 29, 2008 Sheet 142 of 302 US 2008/0122796 A1Patent Application Publication May 29, 2008 Sheet 143 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 400 \ 3600 § 4) 402 Current Time 404 © 406 3 Settings “”Y Done | S| LL J 3616- Accounts sx XS Message List Displays: 2608 3616- Sender > Display Newest Messages: 3608 Top S616 Message Display Locations: 3610 Preview Pane Es Preferred Message Format: 3612 HTML ma, Rules a L™ { Home Figure 36Patent Application Publication May 29, 2008 Sheet 144 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ~ 3700A a Zi 402 Current Time 404 © 406) 3 | Settings Rules \ 3102") My spouse” 7°25 Project Pluto#? °’> Invoices™* 08S 5 Spam 537044 STO Done” S70 i ( Home \ 204 | Figure 37APatent Application Publication — May 29, 2008 Sheet 145 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 37008 'g g MS ZA 402 Current Time 404 > 406 8 || Cancel | My spouse Save) Rule Name: My spouse If To | Subject contains myspouse@mac.com [VJ move to Personal mark as read Figure 37BPatent Application Publication 208 | | 208 Portable Multifunction Device 206 400 ~ Z 402 Current Time 404 © 406) Move Message Kim Brook January 30, 2006 Draft agreement* Drafts Sent Trash Personal Figure 38A May 29, 2008 Sheet 146 of 302 US 2008/0122796 A1Patent Application Publication May 29, 2008 Sheet 147 of 302 US 2008/0122796 A1 Portable Multifunction Device 208 | | 208 Move Message Kim Brook January 30, 2006 Draft agreement Inka (rn) im Brook January 30,2008 «| Draft agreement Current Time 404 © 408 \ Drafts Tengen 3.208 ent Sent Trash Personal vy Work Figure 383Patent Application Publication May 29, 2008 Sheet 148 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ Current Time 404 Welcome 3904 http:/Awww.company.com/start 3908 Cj Web page Block 3 =_ Block 1 3914-1 Block 7 Block 4 ser 3014-4 Block 5 oe 3914-5 Block 2 30234 Block 8 3914.2 a 3914-8 Block 6 Figure 39,Patent Application Publication (| 39008 s ZL 402 Current Time 404 © 406 a 3924 Welcome 3904 al May 29, 2008 Sheet 149 of 302 US 2008/0122796 A1 Portable Multifunction Device 100 http:/www.company.com| 3926 Clear 3928 Search 3930 Go to URL 3932 oO OY Figure 398Patent Application Publication — May 29, 2008 Sheet 150 of 302 US 2008/0122796 AI Portable Multifunction Device 100 208 Z 402 Current Time 404 > 406 Welcome 3904 3908 [> 208 http://www.company.com/start 3908 pe Block 4 3914-4 228) Block 5 a 3914-5 Figure 39CPatent Application Publication — May 29, 2008 Sheet 151 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 8 B 208 N Ww 402 Current Time 404 208 BOSE }e}s/Wo0o' Auedwoo mMMMy//:d})4 EA e < EV ° =~ E+ & Home 204 NU Figure 39DPatent Application Publication — May 29, 2008 Sheet 152 of 302 US 2008/0122796 AI [woo Auedwos mamy/:d}y, Beer FUSE BWIODIO/\\ DesE YOUeAS weer THN 99D Figure 39EPatent Application Publication May 29, 2008 Sheet 153 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 X q Z 402 Current Time 404 © 406) q O) » 3 WD a 9 3) (2) (zx) a) f \o Oo} go i: Bu a m oH 2 Oo!) > 3G => |v a oO fF Zi)a | oO oO “| /S)/8 a : ° x ® 3 3 gs ee x Q 3 3 om Vv Vv Vv n a Oe o < oO Figure 39FPatent Application Publication May 29, 2008 Sheet 154 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ Optical ‘Sensor 164 ‘urrent Time 404 © 406) 208 ° 208 > GeM 7 OECE /mopul\ MON, dqe~, o6ed Beer 9u0q Figure 396Patent Application Publication May 29, 2008 Sheet 155 of 302 US 2008/0122796 AI Portable Multifunction Device is (| 3900H a L402 Current Time 404 © 406) g + |http://www.company.com/start © 3920 3910 Web page 3912 Block 3 a 3014-3 Block 1 “ 391d Block 7 Block 4 a 3914-4 Block 8 Block 5 | 3914-8 28) Block 2 aa Figure 39HPatent Application Publication May 29, 2008 Sheet 156 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 39001 208 Z M2 Current Time 404 ( Share 68 Start 208 ‘http ‘/www.nyt] — 3908 Q Search 3972 www.nytimes.com www. nyt.ulib.org 3974 Figure 391Patent Application Publication May 29, 2008 Sheet 157 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 oi f] 3900 Ns Z) 492 Current Time 404 © 406) cr ~ 7 _ a] | | Share \.30966 Start | Cancel al X 7 XS 7 Z ‘http :/www.company.com| 3908 7 Q Search se 2123 space search| Figure 39JPatent Application Publication — May 29, 2008 Sheet 158 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘\ 3900K if Zi 402 Current Time 404 © 406 7 5) 7 \ a Share \-zoee Start | Cancel St} \ J \ ) C > {htin-/Awww comnany com) la \ Email Link oss la \ Email Content — xs SMS Link F { G y HJ y K y L ) Figure 39KPatent Application Publication May 29, 2008 Sheet 159 of 302 US 2008/0122796 AI Portable Multifunction Device 206 400 \ 3900L q Zi 402 Current Time 404 © 406) 4A + f=] at g 8 E\7 3 5 ee Pay Bs 8)| 5 ES 2 8 hn z 3 'B | § g Figure 39LPatent Application Publication May 29, 2008 Sheet 160 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N a a 9 a f s 3 TG 2 s - Q ° 3 2 iy a (x) QY n oO oy 3 Oo S| o 3 oO 8 Figure 39MPatent Application Publication May 29, 2008 Sheet 161 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 4000A, a Z 402 Current Time 404 © 406) al + http:/;www.company.com/start oO 3220 3910 a Web page nie Content ae Multimedia goss Inline ponent Multimedia a Content 4002-7 ¥ Content «4028 ¢ 4002-4 Inline Content = Multimedia 4002-5 Inline Content Multimedia 4002-8 Content 4002-8 Content - 4002-6 3902 3022 Figure 40Patent Application Publication May 29, 2008 Sheet 162 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ a eal \ S 8 al Oo ans / $8 4 8 ° e E. a Es 7 35 ~ a3 3 5 5 oO B ah / \ 8 \ 8 \ B Figure 40BPatent Application Publication May 29, 2008 Sheet 163 of 302 US 2008/0122796 AI Portable Multifunction Device 208 400 \ | 208 g00v gtoy gov 2 Ze:0 zzov ator Lor 2 ° S| 3 2 a 3 oO BE zor ¢ 9€:0- Figure 40CPatent Application Publication May 29, 2008 Sheet 164 of 302 US 2008/0122796 AI Portable Multifunction Device 2068 400 \ olor vor Su] JUSIIND zLoy ¢ 0- Figure 40DPatent Application Publication May 29, 2008 Sheet 165 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 oi 208 9007 egor y zor olor ? vor Sul, JUaLIND \ 372 \ OR) BR o (Home \ 204 Figure 40EPatent Application Publication May 29, 2008 Sheet 166 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 201 q )=1:08 401 0 & 2. Inline c Multimedia Content 4002-1 Figure 40FPatent Application Publication 208 208 Portable Multifunction Device 206 100 ~ 100A Z 402 Current Time 404 © 408) + http://www.company.com/start © 3920 10 Web page /« ' 108-1 3a. Accounts’ Menu + Content 1 4112-1, Content 3 4112-3 Area ls amr Sign in 4102-2 4106 Sick Hore “tran ketone Content 4 Content 2 4112-4 4412-2 J pag / Home 204 Figure 414 May 29, 2008 Sheet 167 of 302 US 2008/0122796 A1Patent Application Publication May 29, 2008 Sheet 168 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 4100B q 1 492 Current Time 404 © 406 4 + http://www.company.com/start G Sy 3920 3910 (oY 4114-4 Cancel +41" Accounts a Accounts 4118-4 Portfolios 4118-2 Stock Trade 4118-3 Bank Accounts 4118-4 | Figure 418Patent Application Publication May 29, 2008 Sheet 169 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 X 4100C q AA 492 Current Time 404 © 406) a + http:/www.company.com/start G 3920 10. Webpage “a, ~ wa (‘Sign In’ Button 1 Content 1 4412-4 Content 3 4112-3 4102-3- lick Here 4102-45 0% Nane Content 4 Content 2 4112-4 4112-2 Figure 41CPatent Application Publication — May 29, 2008 Sheet 170 of 302 US 2008/0122796 AI Portable Multifunction Device 4100D 4 4) 402 Current Time 404 © 406) q + http:/www.company.com/start © ey 392 3910. Web page f 4108-4 3912 ( Password 7 Content 1 Atte-t Content 3 4112-3 Cecounts Bp rst0e2 4102-36"! lick Here 102d Content 44" re He Figure 41DPatent Application Publication May 29, 2008 Sheet 171 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 X 4100E pS Ss ZL 402 Current Time 404 © 406 ea + http:/Awww.company.com/start © i“ 3920 3910 4114.2 — Done }*'* 428 Submit ) / \ “ User Name | ) 4102-3 NX / Password ) 4102-4 \ / Space | Return J sa) 626 (tome \ NZ Figure 41EPatent Application Publication May 29, 2008 Sheet 172 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 208 2) 402 Current Time 404 © 406 http:/Awww.company.com/start 3920 3910 208 Web page ‘ >) sou | URL Www.aaa.org’ }41s0 Content 1 Content 3 Attet 4112-3 ecounts IP -4102-41 (Signin 4102-2 24 4102-3. an” 4102-45 oN Content 2 Content 4 4112-2 4112-4 Figure 41FPatent Application Publication May 29, 2008 Sheet 173 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 4200A M (Aw Current Time 404 © 406 gj + http://www.company.com/start © 3920 3910 1208 4 am 5 (Home \, (ay Figure 428Patent Application Publication May 29, 2008 Sheet 174 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 42008 ZI 402 Current Time 404 > 406 4 + http://www.company.com/start 3920 1 4208 4210 Hee --e--- & © DN , a Figure 428Patent Application Publication May 29, 2008 Sheet 175 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ gq “| (2 ao2 Current Time 404 © 405 g + [http:swww.company.comistart | © “| |_ 3920 3910 v Figure 42¢Patent Application Publication May 29, 2008 Sheet 176 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ x 4 402 Current Time 404 © 406 3 i. 4302, Now \, al Playlists NY Playing , 90's Music te 4318 Top 25 Most Played 8) “> On-The Go > ayo 4312 4314 a 4340 Bi eee Artists ‘Songs Videos More Figure 43APatent Application Publication May 29, 2008 Sheet 177 of 302 US 2008/0122796 AI Portable Multifunction Device 100 43008 is 21 402 Current Time 404 © 406 7 ~ 43021 No \ | | (Playlists) Top 25 Most Played “Ming Shuffle IC 4320. 1 Track Name Veer 2 Track Name 4 4:12 > 3 Track Name «a 2:25 > 4 Track Name ’ 3:40 > 5 Track Name 5:05 > 6 Track Name 3:20 > Figure 43BPatent Application Publication 208 | | 208 May 29, 2008 Sheet 178 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N Z 402 Shuffle — 43 (Playlists) Top 25 Most Played Current Time 404 © 406 Now. Playing / a | Background 4326-1 1 Track Name 434 3:20 > 2 Track Name 4:12 > 3 Track Name 2:25 > Playlists Figure 43CPatent Application Publication May 29, 2008 Sheet 179 of 302 US 2008/0122796 AI Portable Multifunction Device 206 § 4) a2 Current Time 404 © 406 a) ooo 30. Now sj | (Playlists) Top 25 Most Played Playing 7 23 Track Name 5:33 > 24 Track Name 4:27 > 25 Track Name 4:20 > Background 4326-2 4320-2 | 4310 42 4314 ol Artists ‘Songs Videos More Figure 43DPatent Application Publication May 29, 2008 Sheet 180 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ 208 As Academy of St. Martin In: Andrew Lloyd Webber ne Current Time 404 © 406 4302./Now Artists Playing > A304 A B c D E E 4330-2 G BY 4 1 Barry White i The Beatles Mi Bee Gees 4 P Ben Harper Q . s T = u v . w x Y 4308 4310 4312 4314 Zz 5 5 5 eee Playlists | Artists Songs Videos. «= More Figure 43EPatent Application Publication 208 2 al May 29, 2008 Sheet 181 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 ~\ 402 Current Time 404 (> 406 4302. \ , "u Now. \ Artists Paying > 1330-1 4336 rn AS B i c Andrew Lloyd Webber’? 5 Set E B 5 4330-2 4334. F = s ; wo Barry White 1 J The Beatles kK 5 Bee Gees M N Ben Harper ¢ . Q R . s . . uU Vv w x y 4312 4314 4316 Z 3 ‘ $ Bay Q* Playlists: Artists ‘Songs Videos More Figure 43FPatent Application Publication May 29, 2008 Sheet 182 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ~ fl Z| 402 Current Time 404 © 406 a Songs A Shuffle Ce A E After All 6 All Because of You i All | Want is You i M N = Oo P . Q R 2 s 2 v # Ww 25 or 6 to4 3° Movement: Rondo ... # a 0 Playlists: Artists: ‘Songs MorePatent Application Publication May 29, 2008 Sheet 183 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ 208 208 Movie: 583903 Music Videos © 406 Current Time 404 4302, Now Playing / Videos Incredibles Unknown 1:27 A King Kong } Unknown 2:30 4330-4 Y Playlists @ A Artists U2 | still haven't found what ... 4:31 U2 Beautiful Day 4:09 The Beatles Vantaedan: Songs Videos Figure 43HPatent Application Publication May 29, 2008 Sheet 184 of 302 US 2008/0122796 AI Portable Muttifunction Device 206 100 ~\ 43001 4) 492 Current Time 404 © 406) 43024 Now Videos Playing Movies.““°° J Music Videos" 1:27 King Kong Unknown 2:30 Playlists a U2 | still haven't found what ... 4:31 U2 Beautiful Day 4:09 The Beatles Yesterday 4:09 a a More Figure 431Patent Application Publication May 29, 2008 Sheet 185 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 X 4300) sy 21 492 Current Time 404 © 406 3 4302, Now Mors * Playing , (Edit Hp) Albums > Audiobooks > Compilations > Composers > Genres > Podcasts > 4312 4398 4390 ‘ Playlists Artists Songs 4340 Videos Figure 43JPatent Application Publication May 29, 2008 Sheet 186 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ~ 4300K a iS | 402 Current Time 404 © 406 a 4302, Now al More . —— 1362 Playing / (Done ) 5 4348-1 — ‘Albums 4380-1) ee (=) Audiobooks = } Compilations — (=) Composers = = P 4360-5) = (=) Genre 4350 ees : — Podcasts = § v 4312 < 4314 4316 Playlists Artists Figure 43KPatent Application Publication May 29, 2008 Sheet 187 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 208 | 2) 402 Current Time 404 © 406 4302, Now \ q oe Playing / (Done) (=) Albums = =) Audiobooks ADA UU Figure 43LPatent Application Publication May 29, 2008 Sheet 188 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 X 4 AA 492 Current Time 404 © 406 43024, Now {NOW q More Playing Artists 4310 300.2, = Audiobooks = C ") Compilations = (7) Composers — Podcasts = 4308 4350 4312 4314 4316 5 5 5 Playlists \, 204 Ne Figure 43MPatent Application Publication — May 29, 2008 Sheet 189 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 1 2 492 Current Time 404 © 406) 4302; Now q re 4 Playing (Done) (=) Playlists = (a) act — 7) Artists — © Songs = 4 = ey —_ = @) Con lations ©) Composers Playlists Artists Figure 43NPatent Application Publication May 29, 2008 Sheet 190 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 N 4) 402 Current Time 404 © 405 4302, Now \, 4 We Playing / (Done) Playlists Genres Nu C) Videos = . *) Albums Playlists Artists Songs 449 Videos Figure 430Patent Application Publication May 29, 2008 Sheet 191 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 i 4300P, 3 - ia 2) 402 Current Time 404 © 406) sol 4302: sl More YNow 4362 (Dane (Done } (a 4364 = 7) Playlists a — Genres 4350-1 aaa) = —_ \7) Songs = Videos = 4352 4366 Albums Artists 4310 ) Compilations N (7) Composers 4555 Playlists Genres Figure 43PPatent Application Publication May 29, 2008 Sheet 192 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 NX 43000 gp i$ 4 2 402 Current Time 404 © 405) 3 4373 4302, Now q ead ete * Playing , All Songs 43708 A 8 4 5 € Abbey Road ie The Beatles er I All that | am ei Santana a M 7 am N All that you can’t leave beh... ‘i that y wna? Q R f Anthology s J Bryan Adams sare i 4375-2 Bs w ae A 7 Y | Barry White's Greatest Hits 4,775 Z Figure 43QPatent Application Publication May 29, 2008 Sheet 193 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 2) 402 Current Time 404 © 406 208 | | 208 ‘Albums + Abbey Road“ t Now Playing Shuffle Track 1 Come together 4378 Track 2 Something 4372-2 Track 3 Maxwell’s Silver Hammer 4372-3 Track 4 Oh! Darling 4372-4 Track 5 Octopus's Garden 4372-5 Track 6 | Want You 4372-6 Track 7 Here Comes The Sun aa722 4308 4310 4316 Ag AO Playlists Artists Songs gs4p Videos Figure 43RPatent Application Publication May 29, 2008 Sheet 194 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 4300S a /\ 492 Current Time 404 (> 406) a f \ 4980-1 The Beatles ( j Come Together 4380-3 Abbey Road S23 a 40807 2803 Figure 438Patent Application Publication May 29, 2008 Sheet 195 of 302 US 2008/0122796 AI Portable Multifunction Device 206 400 ~ 4300T 8 Z 402 Current Time 404 © 406) 4 / The Beatles oN —_— f ‘co ) Come Together = \ / Abbey Road ss O12 a 227 DS 4380-6 Enlarged Album Cover gad Playlists Artists ‘Songs Videos Figure 43TPatent Application Publication May 29, 2008 Sheet 196 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 oi 2) 402 Current Time 404 © 406 The Beatles /Nbum q Come together | Cover | Abbey Road Thumbnail @ 0 @ OO vx 41 > Come together 2 Something 3 Maxwell's Silver Hammer 4 Oh! Darling 5 Octopus's Garden 6 | Want You 7 Here Comes The Sun Playlists Artists ‘Songs Figure 43UPatent Application Publication May 29, 2008 Sheet 197 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ a Z 402 Current Time 404 © 406 fa | - The Beatles ‘Album >) a (com) Come together = |_ Cover Abbey Road * * &: oO © 43622 1 > Come together 2 Something 3 Maxwell's Silver Hammer 4 Oh! Darling 5 Octopus’s Garden 6 | Want You 7 Here Comes The Sun Figure 43VPatent Application Publication May 29, 2008 Sheet 198 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 208 402 Current Time 494 © 406 4302, Now Playing 208 “ More Genres Alternative & Pink Blues Classical Country Electronics & Dance Jazz Pop R&B Reggae Rock > > > > > > > > > > > Soundtrack @ A Playlists Artists ‘Songs Videos Figure 43WPatent Application Publication May 29, 2008 Sheet 199 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 430% 208 4) 402 Current Time 404 © 406) 4302, Now Playing / 208 The Beatles Bon Jovi Bryan Adams Chicago David Bowie Joe Satriani VvvVvVVV Vv John Fogerty Playlists ArtistsPatent Application Publication May 29, 2008 Sheet 200 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ 4300Y Z 402 Current Time 404 © 406 4302, Now Playing / © More Composers A A. P, Carter Aaron Copland Adam Clayton, Bono, Larry ... Adam Topol/Zach Gill Andrew Lloyd Webber Anthony Kirdz, Michael ... Antuan Pattor, Nsilo Red B Barry White Barry Gibb Beethoven @ A Artists ‘Songs N Playlists Figure 43YPatent Application Publication May 29, 2008 Sheet 201 of 302 US 2008/0122796 AI [ 208 | | 208 Portable Multifunction Device 206 100 \ 43002 21 492 Current Time 404 © 408 aa. 4302, Now Compilations Y Playing / All Songs a All that | am bs Santana H L__) j ) Antholog Fs L , Bryan A aie i c 8 rc 5 P Curious George a Jack Johnson S —— T G v w x Y Zz Playlists The Righteous Brothers +. } BAS Artists ‘Songs Videos Figure 43ZPatent Application Publication May 29, 2008 Sheet 202 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 ‘ 402 Current Time 404 4308 $ Playlists I Compitations| SO! x NOW \ The Righteous Brothers Playing, Shuffle 2S | Little Latin Lupe Lu Hot Tamales Gotta Tell You How | Feel My Baby 4310 4312 4314 4316 $ 5 $ Artists Songs 449 Videos Figure 430APatent Application Publication May 29, 2008 Sheet 203 of 302 US 2008/0122796 AI 4392 Portable Multifunction Device Ze Current Times © 45) a| - . The Beatles © ) Come Together ‘ Abbey Road OS 133: a) 22 DC Album Cover KK | DO Playlists Artists Songs Videos o—~ Microphone Home | 204 j Figure 43BB 4392Patent Application Publication 2 al 208 206 ago0cc May 29, 2008 Sheet 204 of 302 US 2008/0122796 A1 Portable Multifunction Device 100 JaA0D windy Figure 43CCPatent Application Publication May 29, 2008 Sheet 205 of 302 US 2008/0122796 AI Portable Muttifunction Device 100 430000 208 L402 Current Time 404 © 406) [ 208 z L Z OBL booed ewen windy vez eg Figure 43DDPatent Application Publication May 29, 2008 Sheet 206 of 302 US 2008/0122796 AI ===) Figure 440 sss Figure 44B 6 Figure 44C @ oO)atent Application Publication — May 29, 2008 Sheet 207 of 302 US 2008/0122796 A1Patent Application Publication May 29, 2008 Sheet 208 of 302 US 2008/0122796 AI Figure 44H Figure 441 (=== Oee Figure 44JPatent Application Publication May 29, 2008 Sheet 209 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ (| 45004 | Zi 402 Current Time 404 © 406 3 Now 5 Videos (4) Playing / Movies ~«510 King Kong | Unknown 2:30 -——— The Shawshank Redemption | Unknown + 2:22 Music Videos-~4s15 | All that lam Santana | [still haven't found what ... 4:31 Playlists a a S Figure 450,Patent Application Publication May 29, 2008 Sheet 210 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 X Current Time 404 © 406) ie Vi Now \\ Videos Music Videos (30) Playing ” Anthology Bryan Adams [| Allthat lam Santana The Beatles (10) ~ +20 U2 (1 8) 4525 Playlists a a a More Figure 458Patent Application Publication May 29, 2008 Sheet 211 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 45000 NM 402 Current Time 404 © 405 [208 / Music Now \ Videos U2 (18) Playing / | | still haven't found what ... 4:31 | Beautiful Day 4:09 Videos MorePatent Application Publication Portable Multifunction Device 45000 a ZI 402 Current Time 404 © 406 Videos (50) Neng Movies (20) TV Shows (0) +42 Music Videos (30) Podcasts (0) 4595 om AS Playlists Artists ‘Songs More ~\ | Home 204 , Figure 45D May 29, 2008 Sheet 212 of 302 US 2008/0122796 AIPatent Application Publication May 29, 2008 Sheet 213 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ 4500E i Z 402 Current Time 404 © 408 i Now qi Videos (50) Baying » Movies (20) — -4540 Music Videos (30) ~+° ge A Artists Songs Playlists Figure 45EPatent Application Publication May 29, 2008 Sheet 214 of 302 US 2008/0122796 AI Portable Muttitunction Device 206 100 ‘ 208 Zi 402 Current Time 404 © 406) 208 (Videos Movies (20) Bering The Incredibles Unknown ~ 1:55 ) King Kong Unknown 2:30 The Departed Unknown 2:30 Cartoon (6) ~ +550 Foreign (11) 4555 Be A/S)- Playlists Ariists Songs More Figure 45FPatent Application Publication May 29, 2008 Sheet 215 of 302 US 2008/0122796 AI 203 208 Portable Multifunction Device Current Time 404 © 406 Videos (4) being Movies «4510 Ile Srawsnalik ReGernpuon Unknown ~ 2:22 Videos 515 All that | am Santana = Artists Songs om Bla): | still haven't found what ... 4:31 Figure 456Patent Application Publication May 29, 2008 Sheet 216 of 302 US 2008/0122796 AI Rap Album Template 4600 \ RAP ARTIST NAME 10. : & Generated Artwork 4608 4608 Figure 46APatent Application Publication May 29, 2008 Sheet 217 of 302. US 2008/0122796 AI Ye YY) Yy Whe UY WY i ee ee in am Se aja EB: Ba 3 ee SERS SSO NOES AN ARR So Sia antes Renae OOR ES SOS sh OREN Peper meueuaaenenererr ney 4628 Figure 46BPatent Application Publication May 29, 2008 Sheet 218 of 302 US 2008/0122796 AI Electronica Album Template 4640 \ 4642 ELECTRONICA ARTIST NAME Electronica Album Title 4644 (4 4648 Figure 46CPatent Application Publication May 29, 2008 Sheet 219 of 302 US 2008/0122796 AI 47044708 4702 Figure 47A a>] A706 dik 0:63 Figure 47B5 ary \ 4746 dae, ad | DOG Figure 47E Figure 47¢ Figure 47D TV 96L77 10/8007 SA. ZOE JO OZZ AIS 007 “67 KEINPatent Application Publication May 29, 2008 Sheet 221 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ 4800A a 4 21 492 Current Time 404 © 406 : Notes (3) 4802 481Q 481241 4814-1 Where Dec. 7 ae 4810-2 4812-2 4814-2 “Packing List “Dec.7 = 48193 er BIS 4814-3 Christmas Shopping List} “Dec. 17 >” 22-016 Touch Screen 112, Figure 48APatent Application Publication May 29, 2008 Sheet 222 of 302 US 2008/0122796 Al Portable Multifunction Device 208 100 N 48008 al Current Time 404 By) | < Notes Christmas Shopping List 4820 4810-3 Figure 488Patent Application Publication — May 29, 2008 Sheet 223 of 302 US 2008/0122796 AI Portable Multifunction Device cy 48000 (a Ss 4) 402 Current Time 404 © 406 q a Christmas Shopping List (+) 4820 4810-3 Podlfor == an PC Gi [4830 a Macy’ Fyn std for Emily 20 -2123 Space Return Figure 48CPatent Application Publication — May 29, 2008 Sheet 224 of 302 US 2008/0122796 AI Portable Multifunction Device Current Time 404 List Day Month } 7 Thursday Dec. 7, 2006 { Friday Dec. 8, 2006 12PM_ Go to the gym 2PM _— Have lunch 5PM Meeting Doug 12PM Meet Richard and Andy Monday Dec. 11, 2006 9AM _ Demo new model Touch Screen 112 Figure 49Patent Application Publication May 29, 2008 Sheet 225 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 ‘ i 4. 402 Current Time 404 © 406 8] | | Today | _ list Day Month q Thursday Dec. 7, 2006 > 12 AM 1AM 2AM 3 AM 4AM 5AM Done x 706 ‘Touch Screen 112 Figure 498Patent Application Publication — May 29, 2008 Sheet 226 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 4900c eee 21 492 Current Time 404 © 406 7 \ or ‘DD Q| | (Today | { List Day Month | (+) < January, 2007 > Sun Mon Tue Wed Thu Fri Sat 31 1 2 3 4 5 6 14 | 15) 16 47 | 18 19 20 21 22 | 23 24 25 26 27 28 29 | 30 31 4 2 3 12 PM Lunch with Eddie 3PM Pick up Jack 5PM_ Pick up Lady Touch Screen 112 Figure 49¢Patent Application Publication May 29, 2008 Sheet 227 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ~ 49000, ql 4 42 Current Time 404 © 406 | 7 \ \ al Today List Day Month SS aa | Starts: ) Ends: All Day: OFF Repeat: None Alert: OFF Notes: > ) Touch Screen 112 Figure 49EPatent Application Publication May 29, 2008 Sheet 229 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 Ny ol Title: Location: > Starts: Ends: 4986 4966 4960 462.4964. (Enter ‘ome \ 204 Figure 49FPatent Application Publication May 29, 2008 Sheet 230 of 302 US 2008/0122796 AI Portable Muttifunction Device 206 100 \ 49006 4) 402 Current Time 404 © 406) |‘ cancel | Add Event (Done | Name: Location: > Starts: Ends: Saturday, December#0, 2006 Sunday, Decembe! Monday, January 1, 2007 ( Enter Home | \, 204 Figure 496Patent Application Publication May 29, 2008 Sheet 231 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ ZA 402 Current Time 404 © 406) Repeat Event Every Two Weeks Every Month Every Year Custom Repeat > (eg. every 2™ Fri. & 37 Thu., until Dec. 10, 2007, etc.) Touch Screen 41 Figure 49HPatent Application Publication 208 | | 208 Portable Multifunction Device 206 100 \ 49001 Z 402 Current Time 404 © 406) Va > i _ Cancel Add Event | Done | Title: Location: > "Starts: Mon., Dec. 18, 12:35 PM Ends: Mon., Dec. 18, 2:10 PM All Day: OFF ( Repeat: Every week > Alert: OFF Notes: > ] Touch Screen 112 f™ Home 204 7 \ \ / Figure 491 May 29, 2008 Sheet 232 of 302 US 2008/0122796 A1Patent Application Publication — May 29, 2008 Sheet 233 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 ~ 4900 a A 42 Current Time 404 208 208 Cancel ) Add Event Title: Location: > Starts: Ends: All Day: 4964 4990 4960 49624964 Figure 49)Patent Application Publication — May 29, 2008 Sheet 234 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 X 490K eee q A) a2 Current Time 404 © 405) a a aa | Today | © List Day Month A) / Qe q Monday June 4, 2007 > 12 PM 1PM 2PM 3PM ————— Soccer Summer Camp Stanford Stadium 5PM — “Daily Exercise Stanford Track Field Figure 49KPatent Application Publication May 29, 2008 Sheet 235 of 302 US 2008/0122796 A1 Portable Multifunction Device 208 100 ~ 4900L 21 402 } | <{_ Mon Jun a ep (Daily Exercise Stanford Track Field Monday, June 4, 2007 From 4:50 PM to 6:45 PM. Current Time 404 © 406 Event (edit )Patent Application Publication May 29, 2008 Sheet 236 of 302 US 2008/0122796 AI Portable Multifunction Device 206 208, Z 402 Current Time 404 © 406) 3 Set the details for this event. Edit (Done Daily Exercise > Stanford Track Field Starts — Mon June 4, 2007 4:50 PM Ends 6:45 PM | Repeat Never > “Alert None > “Notes > Delete Event Figure 49MPatent Application Publication — May 29, 2008 Sheet 237 of 302 US 2008/0122796 AI Portable Multifunction Device __ [206 100 \ 4900N 5 g is 7 2 Current Time 404 & 48 Set the details for this event. (Cancel ) Start & End 208 Save | Starts Mon June 4, 2007 4:50 PM | Ends 6:45 PMPatent Application Publication May 29, 2008 Sheet 238 of 302 US 2008/0122796 AI Portable Multifunction Device 100 5000A 208 | 402 Current Time 404 208 | Edit World Clock 5010-1 5012-4 50141 4 i 4:00 pm Cupertino p f QQ Today lso10-2 50122 50142 New York 7:00 pm Today 50123 5014-3 7:00 am Tomorrow 5010-3 , ~” Manila lek 50144 or ondon 42:00 am Tomorrow 5022 5024 5026 Alarm Stopwatch Timer Figure 50APatent Application Publication May 29, 2008 Sheet 239 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N 5000B 8 7 = 402 Current Time 404 © 406 3 5034, | “Cancel No32 Alarm at Done | | Everyday M+ F Other ‘5036 ‘S38 “Boao ‘Boaz Sound Beep > oy c 5044 546 5048 550 Tim 5062 ©5064 5066 ‘ 5082 Enter 5060 Touch Screen 112 Figure 50BPatent Application Publication Portable Multifunction Device 206 100 oT 208 Lh May 29, 2008 Sheet 240 of 302 US 2008/0122796 A1 Current Time 404 OBE 6 208 Alarm 5020 5022 5024 5026 World Alarm Stopwatch Timer Figure 50CPatent Application Publication May 29, 2008 Sheet 241 of 302 US 2008/0122796 A1 Portable Multifunction Device 208 100 \ 5000D a ( Cancel ) : Repeat Sound Snooze ; Label Current Time 404 Add Alarm Never > | Marima > Alarm > Figure 50DPatent Application Publication May 29, 2008 Sheet 242 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 X ‘5000E 3 a 21 402 Current Time 404 © 406 a I 00:00.0 - Start ‘sso Reset | 5020 5022 5024 5026 World Alarm Stopwatch} Timer (Home 208) ay) Figure 50EPatent Application Publication May 29, 2008 Sheet 243 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 20 Current Time 404 |} 00:02.1-~ | Stop | Lap ose: 208 Figure 50FPatent Application Publication May 29, 2008 Sheet 244 of 302 US 2008/0122796 AI Portable Multifunction Device Current Time 404 © 406) 00:12.1-~ _ Stop : ( Lap ate lap 3 00:10.1-- 5007-1 lap 2 00:07.3- 50072 lap 1 00:03.4--5007s 5028 [eon ae Sle ee wat) Ballet Alarm 9 Figure 506Patent Application Publication May 29, 2008 Sheet 245 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 5000H 208 Current Time 404 © 408 208 When Timer Ends Start se Figure 50HPatent Application Publication May 29, 2008 Sheet 246 of 302 US 2008/0122796 AI Portable Multifunction Device 100 4 “| 402 Current Time 404 04:14:12-" When Timer Ends Marimba > Cancel = Figure 501Patent Application Publication May 29, 2008 Sheet 247 of 302 US 2008/0122796 AI Portable Multifunction Device 206, 100 Y ‘51004, Ms Current Time 404 ‘9 Welcome 3004 3906 “| 210 http://www.company.com/start 3908 Cj" Web page Block 3 od Block 1 a Block 7 Block 4 Soler 3014-4 G22) Block 5 aad 3914-5 Block 2 Bors) Block 8 3914-2 aaa 3014-8 Block 6 3014-6 | Figure 514.Patent Application Publication May 29, 2008 Sheet 248 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ~ 208 208 3950 This is what the web clip widget will look like. Move, scale, and/or rotate to adjust. Block 4 3914-4 Block 5 3944-5 2028) | Cancel | Create wz | Widget sxx) + O 3923 Figure 51BPatent Application Publication May 29, 2008 Sheet 249 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘\ 52008 208 ZI 402 Current Time 404 © 406) [208 (Q Textbox 202 8212 5206 5208 5210 5 $ $ (Q |<) (Map | Satelite | List )Patent Application Publication May 29, 2008 Sheet 250 of 302 US 2008/0122796 AI 208 208 3 \\ Portable Multifunction Device Figure 2BPatent Application Publication May 29, 2008 Sheet 251 of 302 US 2008/0122796 AI Portable Multifunction Device Map | Satellite | List jome (Acceleromeler(s) 204 } Figure 52¢Patent Application Publication [208 208 52000 May 29, 2008 Sheet 252 of 302 US 2008/0122796 AI Portable Multifunction Device \, 204 / Figure 52DPatent Application Publication May 29, 2008 Sheet 253 of 302 US 2008/0122796 AI Portable Multifunction Device 5200E om 8 Current Time 404 © 406 208. Info Starbucks Phone (408) 765-4321 5222 ) Address 1291 S. Mary Ave Sunnyvale, CA Figure 52EPatent Application Publication — May 29, 2008 Sheet 254 of 302 US 2008/0122796 AI 208 | [208 402 Current Time 404 © 406 Map ) Info ) Local Map 5224 J Starbucks, 1291 S. Mary Ave Sunnyvale, CA Phone (408) 765-4321 22 ~~?) - ¢ 5250 ( URL http:/www.starbuckscupertino.com _ ) Portable Multifunction Device Figure 62FPatent Application Publication May 29, 2008 Sheet 255 of 302 US 2008/0122796 AI 52006 Zi 402 Portable Multifunction Device [206 100 Current Time 404 © 406) Pal Stai 4 Q Starbucks si “hoe, 5228, starbucks, 1291 S. Mat Ate Sunnyvale, CA “see? S) Starbucks, 201 S. Mary Ave. #1, Sunnyvale, CA Starbucks, 639 S. Bernadino Ave., e) Sunnyvale, CA Starbucks, 296 Main St, C) Los Altos, CA Starbucks, 1037 El Monte Ave, (6) Mountain View, CA Stai Santa Clara, CA bucks, 2000 El Camino Real, lo Alto, CA (6) bucks, 3113 Mission College Blvd, Map Satellite Figure 526Patent Application Publication May 29, 2008 Sheet 256 of 302 US 2008/0122796 AI Portable Multifunction Devi 100 \ Current Time 404 Choose a location to view on the map Bookmarks Moscone West 5230 Washington Monument Colosseum Eiffel Tower Sydney Opera House Figure 52HPatent Application Publication May 29, 2008 Sheet 257 of 302 US 2008/0122796 AI Portable Multifunction Device [206 400 \ 53004, ——EEE————SSlS—— al 3 Current Time ~ 308 310 Day and Date 12 Terry called at 12:00 pm, yesterday | James sent SMS at 9:00 am, today | Percy left voicemail left at 4:23 pm, today | Percy sent SMS at 7:23 pm, today Wallpaper image 314 302 306 Touch Screen 112 Figure 3APatent Application Publication May 29, 2008 Sheet 258 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 mi 53008 af al Current Time » 308 310“ Day and Date~ 312 Percy | Missed Calls (5) Missed SMS (4) Henry | Missed Calls (3) Missed Voicemail (7) Wallpaper image 314 302 306 Touch Screen 111 Figure 53Patent Application Publication May 29, 2008 Sheet 259 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ s300¢ . tp a g Current Time ~» 308 310 Day and Date~ 312 Calls [ Thomas (5) } SMS ( Eddie (4) ] ( Judy (3) ] Voicemail | 408-765-4321 (3) Wallpaper image 314 Touch Screen 112 Figure 63CPatent Application Publication — May 29, 2008 Sheet 260 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 53000 a 208 | Current Time »» 308 310“ Day and Date 312 208 40 Calls | 2 IMs | 2 Voicemail | 12 Email | Wallpaper image 314 Slide to unlock 302 306 Touch Screen 112 So Home \ Figure 53DPatent Application Publication May 29, 2008 Sheet 261 of 302 US 2008/0122796 AI Portable Multifunction Device 100 5400 444 Sensor 164) \Sensor 166 Touch Screen 12 Figure 54 Side Side (5442Patent Application Publication Device 5500 5510 Device 5500 Touch screen 5514 Incoming call from: John Doe mobile 5606 592 950: Figure 554 Touch screen 5514 nin Incoming call from: John Doe mobile aca} 550: Decline? sso) ’ 5502: poe Figure 55B May 29, 2008 Sheet 262 of 302 US 2008/0122796 AI Movement 5512 \Patent Application Publication May 29, 2008 Sheet 263 of 302 US 2008/0122796 AI Touch screen 5514 5506 Incoming call from: a John Doe 5500 mobile —~ Beating) iid} 5508 Movement v 5512 5510 5504 5502 Figure 55C Touch screen 5514 a’ Incoming call from: ry John Doe Eso mobile i CD} 09 Figure 55DPatent Application Publication — May 29, 2008 Sheet 264 of 302 US 2008/0122796 AI 5608 Figure a Figure 56B Ws. 5606 Figure 56C Figure 56D and W;" are two numbers with opposite signs 5602 Figure 56E Figure 56FPatent Application Publication — May 29, 2008 Sheet 265 of 302 US 2008/0122796 AI Figure 56G Figure 56H Figure 561 Ww | { W, 9602 ‘ 604 (We) Py [We f | f | 5608 5608 5608 5606 Figure 56K Figure 56L.Patent Application Publication May 29, 2008 Sheet 266 of 302 US 2008/0122796 AI Input Field 5620 ) oO yoy c \vyal ABABA SAAS) Figure 56M Figure 56N a Goal 5620 ) Figure 560Patent Application Publication May 29, 2008 Sheet 267 of 302 US 2008/0122796 AI 5704 Figure 57A Figure 57CPatent Application Publication — May 29, 2008 Sheet 268 of 302 US 2008/0122796 AI Figure 58Dtet. t=t t=ts t=t, he gy Y y Device 100 Device 100 Device 100 Davies 100 Figure 594 Figure 59B Figure 59C Figure 59D wonvoyqng uopeayddy juaped TV 96L7710/8007 SA ZOE JO 697 AUS. 8007 “6z KEINDistance a an “In Range In Contact Figure 59E woneonqng woeoyddy: usyeg TV 96477 10/8007 SA ZOE JO OLZ US 8007 “67 AUIWPatent Application Publication May 29, 2008 Sheet 271 of 302 US 2008/0122796 A1 Figure 59FPatent Application Publication May 29, 2008 Sheet 272 of 302 US 2008/0122796 AI Distance t In Range) i In Contact : : ¢ ¥ ¥ ¥ ts t ts Time Figure 59G Distance In Range In Contact Figure 59HPatent Application Publication May 29, 2008 Sheet 273 of 302 US 2008/0122796 AI Figure 60A an _In Range _ \ 7 N Cc In Contact v {| di af 4 5 / t + 4 Time Figure 60BPatent Application Publication May 29, 2008 Sheet 274 of 302 US 2008/0122796 AI Figure 60C Distance __ a \ a y In Range , ¢ dy J ennal In Contact ds ¥ Y tu ts Figure 60DPatent Application Publication May 29, 2008 Sheet 275 of 302 US 2008/0122796 AI Distance In Range + a nGonlact - Selection \ l/ SN ro “t d, d” dd” ro Yo¥ > tte fe tho tte Figure 60FPatent Application Publication May 29, 2008 Sheet 276 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ‘ Z 402 Current Time 404 © 406 q Start Cancel ) Home 204 EY Figure 606Patent Application Publication — May 29, 2008 Sheet 277 of 302 US 2008/0122796 AI 208 208 Portable Multifunction Device 206 100 \ To: -PQRST | UVWXY 2? Zi 402 Current Time 404 © 406) New Message C Cancel ) 7 S (Send } 7 aad) } Nf Nf ? Il 23168 Space) a JN 7 Figure 60HPatent Application Publication May 29, 2008 Sheet 278 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ sogor 208 Zi 402 Current Time 404 © 406) 3 New Message ( Cancel ) To: (U 019 ) ( Send ~ ‘2310 | ‘Space | a (Home \ — (Rcoslerometeris) 204 ‘68 Figure 601Patent Application Publication — May 29, 2008 Sheet 279 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ gl A Z 402 Current Time 404 © 405) fol 7 al New Message ( Cancel To: A \ 7 ~ (Text box ( Send ) L \ \ 12345, % Figure 60JPatent Application Publication May 29, 2008 Sheet 280 of 302 US 2008/0122796 A1 Portable Multifunction Device 206 100 X 6000K q Z ae Current Time 404 © 406 a New Message ( Cancel } To: (Text box ON 6 | back \ J oF > \ J 7 \/ \r 7 \ 4h ABC Space | &] as JN. / Figure 60KPatent Application Publication May 29, 2008 Sheet 281 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 N 6000L, a A 402 Current Time 404 © 406 a New Message C Cancel To: ABCDE| FGHIJ KLMNO | PQRST | UVWXY | Z.,7! i \ A) (1238@ | <> Space| Return \ . Figure 60LPatent Application Publication — May 29, 2008 Sheet 282 of 302 US 2008/0122796 AI Portable Multifunction Device 208 400 \ s Z 402 Current Time 404 © 406) | New Message To: 12345 | 67890 | @#$%* ~&"() ‘= J\_ so) A) ; ‘ Figure 60MPatent Application Publication May 29, 2008 Sheet 283 of 302 US 2008/0122796 A1 Portable Multifunction Device 208 100 ~ 402 Current Time 404 > (RU tiene?) 604-1 123 | <([instant Messages bn 02 (How was your vacat| Jane Doe~ 504-1 606-1 — ‘IM is the best. January 30, 2006 606-2 8:30 (Hola| 608.2 A (vacation st Figure 61Patent Application Publication May 29, 2008 Sheet 284 of 302 US 2008/0122796 AI Portable Multifunction Device 206, 100 ‘ 6200A, 2) 492 Current Time 404 > 406 Settings 6204 oF Airplane Mode Ga) xD Sounds > s Wallpaper > a [| General > Calendar > (de | Contacts > iPod > mai > ‘Touch Screen 112 Home 204 | Figure 62A,Patent Application Publication — May 29, 2008 Sheet 285 of 302 US 2008/0122796 AI Portable Multifunction Device 208 » 6208 Current Time 404 ip 406 Settings 208 6206 (ES) mail > Touch Screen 112 204, NX Figure 62BPatent Application Publication May 29, 2008 Sheet 286 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 \ Current Time 404 © 406) 20 B Sounds 6212 6214 Ringtone Voicemail Instant Messages v F v v Mail Keyboard Clicks Touch Screen 112 Figure 62CPatent Application Publication May 29, 2008 Sheet 287 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 ~ 4. 402 Current Time 404 © 406 q Settings Wallpaper Wallpaper > ( Photo Library > () Camera Roll > (] Mail > () Last 12 Months > (] Beach Fun > LJ Hawaii Vacation > Touch Screen 112 Microphone) // 113 204 Figure 62DPatent Application Publication May 29, 2008 Sheet 288 of 302 US 2008/0122796 AI Portable Multifunction Device 208 100 ‘ 6200E ql ZA 402 Current Time 404 © 406 3 Gettings| General |] mot Backlight > Date & Time > Keyboard > Network > Touch > Legal > Reset > Touch Screen 112 Figure 62EPatent Application Publication May 29, 2008 Sheet 289 of 302 US 2008/0122796 AI Portable Multifunction Device 206 100 \ 6200F “402 Current Time 404 © 408 | Shuffle ea OFF) Compilation Sound Check OFF Audiobook Speed NORMAL > EQ OFF > Videos Scale FIT > TV Out OFF > Compilation NTSC > Start Movies and TV Shows > Touch Screen 112 Figure 626Patent Application Publication May 29, 2008 Sheet 291 of 302 US 2008/0122796 AI Portable Multifunction Device 100 63008, 414 Sensor 164, ‘Application 1 2221 Touch Screen 12 Application 2 Toximity Sensor 166, * Figure 630, Dien Timer / B0-1 3:00 | (Sut Ap. Dim Timer)Patent Application Publication — May 29, 2008 Sheet 292 of 302 US 2008/0122796 AI Portable Multifunction Device 100 63008, 441 Sensor 1 Sensor 166 Application 1 222-1 Touch Screen A112 Figure 63B Dim Timer / B04 a 00:00 | J (App. | dim timer expired)Patent Application Publication — May 29, 2008 Sheet 293 of 302 US 2008/0122796 A1 6300C Portable Multifunction Device 100 speaker tical ey ena Application 1 222-4 Touch Screen i2 Figure 63C _- Touch Timer {Br 00:10 (App. | timer expired)Patent Application Publication — May 29, 2008 Sheet 294 of 302 US 2008/0122796 AI Portable Multifunction Device 100 ‘Application 1 222-1 Touch Screen 112 Figure 63DPatent Application Publication — May 29, 2008 Sheet 295 of 302 Portable Multifunction Device 100 ™ 6300E 411 Sensor 164) Sensor 166, ‘Application 1 222-4 US 2008/0122796 A1 Dim Timer Yo 230-1 16:00 / Touch Screen 112 Application 2 222-2 Figure 63E (App. 1 timer extended)Patent Application Publication May 29, 2008 Sheet 296 of 302 US 2008/0122796 AI Portable Multifunction Device 100 6300F 441 Sensor 164 Sensor 166 ‘Application 1 2224 Figure 63F Dim Timer {230-1 Dim Timer /~ 230-2 (App.2 dim timer expired)Patent Application Publication May 29, 2008 Sheet 297 of 302 US 2008/0122796 AI Portable Multifunction Device 100 rerraeeuthl Sensor 166, Global Dim Timer /o 20 ‘Application 1 05:00 222-1 AV : 6 | J start Gtobat Dim Timer) as Touch Screen ~ 112 2 ‘Application 2 222-2 Figure 636Patent Application Publication Portable Multifunction Device May 29, 2008 Sheet 298 of 302 US 2008/0122796 A1 100 6300H Att Sensor 164) (Sensor 166 Application 1 222-1 Touch Screen 42 Application 2 222-2 Figure 63H Global Dim Timer oO 00:00 (Global Dim Time Exceeded)Patent Application Publication — May 29, 2008 Sheet 299 of 302 US 2008/0122796 AI Portable Multifunction Device 100 za 63001 414 sensor 164) \Sensor 166 ai 232-1 ay 00:04 ) ‘Application 4 Touch Screen 112 ‘Application 2 2222 Figure 631Patent Application Publication May 29, 2008 Sheet 300 of 302 US 2008/0122796 AI Portable Multifunction Device 100 \ ical Toy 300) 114 Sensor 164) (Sensor 186 Global Dim /~ Timer 232 ‘Application 1 , 10:00 Touch Screen 112 Application 2 Figure 63JPatent Application Publication May 29, 2008 Sheet 301 of 302 US 2008/0122796 AI e402 ‘At a computing device with a touch screen display, detect one or more finger contacts with the touch screen display. | 6404 ‘Apply one or more heuristics to the one or more finger contacts to determine a command for the device. ‘Apply a heuristic for determining that the one or more finger contacts | | ~g40g ‘correspond to a one-dimensional vertical screen scrolling command, ‘Apply a heuristic for determining that the one or more finger contacts | | ~g4og correspond to a two-dimensional screen translation command. ‘Apply a heuristic for determining that the one or more finger contacts correspond to a command to transition from displaying a first iter in a, }~6410 set of items to displaying a next item in the set of items. t e412 Process the command, Figure 648Patent Application Publication May 29, 2008 Sheet 302 of 302 US 2008/0122796 AI | fe Display a web browser application at a computing device with a touch screen display t oat Detect one or more fist inger contacts withthe touch sereen display. t oes ‘Apply a fist set of heuristics forthe web browser application tothe one or mare fist finger contacts to detarmine a first command for the device. ‘Apply a heuristic for determining thatthe one or more first finger contacts - corespondto 2 ane-dmensional vical screen sroling command 6438 “Appi a euristic for detemining that the one or more fist ger contacts coraspondte a two-dimensional sereen translation commands yoo “Apply a heuristic for determining tha the one or more fist ngerconiacls | ‘correspond to one-cimensional horizontal secsen scaling command uaz T ous Process the fist command, | eas Display a photo album application on the touch screen display. 1 - ee DDetect one or more second finger cantacs wth the touch screen display. | oa ‘Apply a second set of heuristic fr the photo album application to the one or more ‘secand finger contacts to determine a second command forthe deviee. ‘Apply a heuristic for determining thatthe one or more second finger contacts correspond to a command to transition from displaying a frst image in a set of 6452 images to displaying a next image inthe set of images, ‘Apply a heuristic for determining that tho one or more second fingor contacts ‘correspond to a command t transition from displaying the firs image inthe set of ._ “6454 images to cisplaying a previous image in the set of images. T ous Process the second command, Figure 648US 2008/0122796 AI ‘TOUCH SCREEN DEVICE, METHOD, AND. ‘GRAPHICAL USER INTERFACE FOR DETERMINING COMMANDS BY APPLYING HEURISTICS RELATED APPLICATIONS [0001] This pplication claims peiovty to US. Provisional Patent Application Nos. 60/987.901, “Touch Sereen Device, ‘Method, and Graphical User Interface for Determining Com- ‘mands by Applying Heuristics” filed Jun, 29, 2007; 60937, ‘993, “Portable Multifunction Device” fled un. 29, 2007, 601879.469, “Portable Multifunction Device,” filed Jan. 8, 2047; 601879,253, “Portable Multifunction Device” filed Jan..7, 2007; and 60/824,769, “Portable Molifnetion Device, filed Sep. 6, 2006, All of these applications are ‘incorporated by referenced herein in their entirety. [0002] This pplication is related othe following applica- tions:(1) US. patent application Ser. No. 10/188,182, "Touch Pad For Handheld Device.” led Ju. 1,2002; (2) US. patent application Ser. No. 107722048, “Touch Pod For Handheld ‘Device filed Nov. 25,2005; (3) US. patent application Ser. 'No, 1043,256, "Movable Touch Pad With Added Function- ality ila Aug. 18, 2008;(4) U.S. patent application Ser. No. 10(654,108, “Ambidextrous Mouse.” filed Sep. 2, 2008; (5) US. patent application Ser. No, 10/840.862, “Multipoint ‘Touelsereen, fled May 6, 2004; (6) U.S. patent application Ser. No. 10/903,964, “Gestures For Touch Sensitive Inpot Devices.” filed Jul. 30, 2004; (7) U.S. patent application Sex ‘No. 11/038,590,“Mode-Based Graphical Use Interfaces For ‘Touch Sensitive Input Devices” filed lan. 18,2005; (8) US. patent application Ser. No. 11/057,050, “Display Actuator” filed Feb, 11,2005; (0) U.S. Provisional Patent Application 'No, (4¥658,777,"Multi-Functional Hand-Held Devic," filed ‘Mat. 4, 2005; (10) U.S patent application Ser. No. 11/367, 749, “Muli-Fusctional Hand-Held Device” filed Mar. 3, 2004; and (11) US. patent application Ser No. 29/281,695, “Toons, Graphical Use Interfaces, ancl Animated Graphical User Interfaces For a Display Sereen or Portion Thereot” fled Jun. 28,2007. Al ofthese applications are incorporated by reference herein, TECHNICAL FIELD, 10003]. Thedisclosed embodiments relate generally to elee- ‘wonic devices with touch screen displays, and more partion lay, to electronic devices that apply heuristics to detected ‘user gestures on a touch sereen display to determine com- mands. BACKGROUND 0004] As portable electronic devices bocome more com pect, and the mumber of functions performed by a given Gevive increase, it has become a significant challenge to design user interface that allows users to easly interact with ‘a mulifanetion device This challenge is partculr signii- cant forhandbeld portable devices, which have much smaller screens than desktop of laptop computers. This situation is ‘unfortunate because the use interface isthe gateway through ‘which users receive not only content but also responses to user actions or behaviors, including use attempts fo socess a device's features, tools, and functions. Some portable com- ‘munication devices (eg, mobile telephones, sometimes called mobile phoues, cell phones, cellular telephones, and the like) have resorted to adding more pushbutton, increas May 29, 2008 ‘ng the density ofpush buttons, overloading the funtions of pushbutton, orusng complex menu systems to allo a user toaccess, store and manipulate data, These conventional ser interfaces often result in complicated key sequences and ‘mer hierarchies that must be memorize by the wser [00S] Many conventional user interfaces, such as those that include physical pushbutton, are also ifleible. This ‘may prevent a user interface from being configured andor adapted by either an applicstion running on the portable device or by users. When coupled with the time consuming requiremen! to memorize muliple key sequences and met hierarchies, and the difficulty in activating a desired pushbut- ton, such inflexibility s frustrating to most users. 006} To avoid problems associated with pushbutton and ‘complex menu systems, portable electronic devices may use touch seroen displays that detect user gestures onthe teh sereen and translate deteted gestures into commands to be perfomned, However, user gestures may be imprecise a par- ticular gesture may only roughly correspond t0 a desited ‘command. Other devices with touchscreen displays, suchas, desktop computers with touch screen displays also may have dlficulies wanslating imprecise gestures into desired com- mans 007] Accordingly, there isa need for touch-screen play electonic deviees with more transparent and intuitive ser interfaces for traslating imprecise user gestures into precise, intended commands that are easy to use, configure, andlor adapt. Such interfaces increase the effectiveness, eli- ciency and wser satisiction with portale multifunction devices SUMMARY 10008] The above deficiencies and other problems assoc ate ith userinterfaes forporabledvicesand touch sercen devices ar redcedor eliminate by the dislosed muta tion device n some embodiment, the device is portable. ln some embodiments, the device has a oueh-seasitve display {ako known asa “Youch sree) with graphical user ine. face (GUD, oneor more processors, memory andone er mare modules, programs or ses of instructions stored inthe memory fr performing mpl anton. In some emboli rents, the User iaterats with the GUL primarily through finger contacts and gestures onthe toh sensitive dispey In some embodiments, the fanetions may ince telephoning, video conferencing, e-mailing. instant messoping, Hogeing, Agia photographing, digi vidoing, web browsing, digi tal music playing, andor digital video playing, Instructions for performing these functions may benched ina computer readable storage medium or other computer progr prot configured for exeetion by one or more proceso. {0009} In an aspect of the invention, a compote-imple rented mod for use in conjuction with 3 computing device with touch sersen display comprises: detecting one cor more finger cones with the touch sree display, apply {ng one or more heuristics to the one or more ingr contacts to detemine a command for he device, an processing the command. The ane or more heuristics comprise: a ewstic for determining thatthe on or more finger contacts coe spond 10a one-dimensional vertical sereen semling com: ‘mand, heuristic for determining thatthe one or mre finger conte coespond to a two-dimensional sree tansltion command, and a heute for detenining that the oneUS 2008/0122796 AI ‘more finger contacts correspond to @ eommand to transition from displaying a first item ina set of items to displaying @ ext item in the set of tems, [0010] In another aspect of the invention, a computer- implemented method is performed at a computing device ‘with touch sereen display. While displaying a web browser application, one or more frst finger contacts with the touch screen display are detected first set heures forthe we ‘browser application i applied tothe oe oF more fst finger contacts to determine a first command Jor the device; andthe first command is processed. The fist set of heuristies com- prises: a heuristic for determining thatthe one or more first finger contats correspond to a one-dimensional vertical screen scroling commands a hewisti for determining that the one or more fist finger contacts comrespond to a two trol, power mangement) and fies communication emeen vero harvard sofware omponens (O117] Thecommunication medule 128 facilitates commn- nication wth oer devises over one or more extra ports 124 and la inches vere software component for han- lings recive the RF ier HR anothers ont 124, The external ort 124 (eg, Universal Serial Bas (USB), FIREWIRE, cc) alae for coupting directly 0 csher deve rine vera network (ie Itemet, ‘wireless LAN, et), Insome embodiment th extra poet ‘Ba malin eg 30-pa) connector ht sth ame a, or similar ond compatible wth the Spin connector sed oniPod (adem of Apple Computer, lc) devices. [0118] The conatotion medl 130 may det contact with the touch screen 112 (in conjunction with the display ontrler 186) and other touch sensitive devices (8, 8 touchpad or physical click whee), The contact nsion od te 13 nels various svar companentforperorming ‘atious operations rtd o detection of cont suchas determining if contact has occured, determining ihre is movement ofthe conte! and cking the movement ass thetoochscren 112 and detemining ifthe conc has een broken (if the contact has ceed). Determining move- sent ofthe pint of contact may ince determining speed (ogni) velocity (gto and diction), nde an acceleration a change in magaitde andlor detion of the point of contact, These operations may be applied to single contacts (e.g., one finger contacts) or to multiple simulta- ‘neous contacts, mlitouchnuipl inger contacts). May 29, 2008 In some embodiments, the contacVmetion module 130 and the display controller 156 so detects contact ona tovehpad In some embodiments, the contac motion module 130 and ‘the controller 160 detects eontat on a cick whee! 0119] The graphics module 132 includes various known software components for rendering and displaying graphics ‘onthe touch sereen 112, including eomponens for changing the intensity of graphies that are displayed. As used herein, the term “graphics” includes any object that canbe displayed toa use, including without imitation text, web pags, icons (och as userinterface objects including soft keys), digital inages, videos, animations and the like [0120] |The text input module 134, which may bea compo- ‘ent of graphies module 132, provides soft keyboards for centering text in various applications (eg, contacts 137, cesmail 140, IM 141, blogging 142, browser 147, and any other application that needs text inp). [0121] The GPS module 135 determines the location ofthe tem controler 156, coatact module 130, sraphies madile 132, audio creitry 110, spaker 111, RF cette 108 text input module 134, e-mail client module 140, and browser ‘module 17, the online veo module 188 allows th sero acces, browse, receive (ea by streaming andor dowload), play back (e., on the ouch sereen or on an extemal, con nosted display’ via extra port 124), send an e-ilwith a Tinka pticuaronlne video, andothervxe manege online Vileos in one or mor ile formats, sch as 1.264 Tn some embodiments, instant messaging module 14, rater than email cient mule 140, used to send link toa partcuar onlin video. Adtonldseriptionoftheonine video ap cation canbe found in U.S, Provisional Patent Appition ‘No, 600536.562, “Pore Matifunction Device, Method, and Graphical User Interface for Playing Online Videos,’ fled Jun 20, 207, the coutent of which is hereby incopo- raed by reference May 29, 2008 [0161] Boch ofthe above identified movtles and applica- tions correspond toast of instructions for performing oneor ‘more functions desribed above. These modules (i. sets of instructions) need not be implemented as separate sofware prograus, procedures or modules, and hus various subsets these modules may be combined or otherwise re-arranged in various embodiments. Forexaple, video player module 148. ‘maybe combined with musi player module 146intoa single ‘module (eg, video and music player module 152, FIG. 1B). Insome embodiments, memory 102 may sorea subset ofthe ‘modules and data stractures identified shove. Furthermore, ‘memory 102 may store additonal modules and data struc: tures act deseribed above [0162] In some embodiments, the device 100 is a device ‘where operation of predefined set of functions on the device is performed exciusively though a touch screen 112 and/or touchpad. By using a touch sereen andor a touchpad a8 the Primary inputcontol device far operation ofthe device 100, the numberof physical inputiconimot devices (such as push ‘buttons, dials andthe like) onthe device 100 may be reduced [0163] The predefined! set of functions that may be pet= formed exclusively though a touch seen andlor a touchpad include navigation between user interfaces. In some embodi- ‘ments, the touchpad, when touched by the wser, navigates the device 100 t a main, home, oF root ment from any user interface that may be displayed oa the device 100. In such embodiments, the touchpad may be refered to as “men ‘button lasomeother embodiments, the menu button may be ‘physical pus button or ater physical iputlcontol device instead ofa touchpad, [0164] FIG. 2 ilustrates « portable multifunction devioe 100 having a touch screen 112 in accordance with some embodiments. The touch sereen may display one of more graphics within user inerice (UT) 200, In this embodiment, as well as others described below, a user may seloct one of ‘more of the graphies by making contact of touching the sraphies, for example, with one oF more fingers 202 (not drawn to scale inthe figure). In some embodiments, selection ‘of oneor more graphics occurs when the user beaks contact ‘ith the one of moze graphics. In some embodiineas, the contact may include a gesture, such a8 one oF more taps, one ‘ormore swipes rom left orig, right to lett upward andor downard) andor a rolling ofa finger (irom right left, left {oright,upwardandor downward) that hasmade contact with, the device 100, In some embodiments, inadvertent contact witha graphic may not select the graphic. For example, a ‘wipe gesture that sweeps over an application icon may not solet the corresponding application when the gesture coer sponding to selection is tap, [0165] "Thedevice 100 may also include one or more physi- «al uttons such as“home" ormenu bution 204, As described previously, the menu button 204 may be used to navigate to any application 136 in a set of applications that may be executed on the device 100, Alternatively, in some embod ‘mets, the menu buttons implemented a6 a soft key ina GUL in ouch sroon 112 [0166] Inoneembodiment, the device 100 includesa touch screen 112, menu button 204, 2 push butloa 206 for posser- ‘ng the device on/off and locking the devie, volume adjust- ‘ment button(s) 208, a Subscriber Identity Module (SIM) card slot 210, ahead set jack 212, anda docking/charging external port 124, The push button 206 may be wsed to turn the power ‘onvoff onthe device by depressing the burton and holding the Dutton in the depressed state fora predefined time interval; toUS 2008/0122796 AI Jock the device by depressing the button and releasing the button before the predefined tie interval has elapsed andor to unlock the device or initiate an unlock process. Ia an alternative embodiment, the device 100 also may accept ver- bal input for activation or deactivation of some functions ‘through the microphone 113, [0167] Attention is now directed towards embodiments of user interfaces (“UI”) and associated processes that may be ‘plemented on a portable multifunction device 100 [0168] FIGS. 34.3¢ illustrate exemplary user interfaces forunlockinga portable electronic device in aecordance with some embodiments. In some embodiments, user interface 3004 includes the following elements, ora subset or superset thereof: [0169] Unlock image 302 that is moved with a finger gesture to unlock the device; 10170] Aeros 304 that providesa visual eet the unlock geste; [0171] Cleannel 306 dat provides additional cues to the unlock gesture [0172] Time 308 [0173] Day 310 [174] Date 312: and [0175] Wallpaper image 344 [0176] Insome embodiments in ation to or in place of ‘wallpaper image 314, an unlock user imerface may include a device charging status icon 316 and aheadset charging status icon’318(eg,,U1300B, FIG. 38). The device charging status ‘con 316 indicates the battery stats while the device 100 is. being recharged (eg. in adock), Silly, headset charging status icon 318 indicate the battery status ofa neadset asso ciated with device 100 (eg., a Bluetooth headset while the Deadset is being recharged (e., in another portion ofthe dock. [0177] In some embodiments, the device detects contact ‘ith the touch-sensitive display (ea. user's finger making contact on or near the unlock image 302) while te devie is. in a userinterface lock state, The device moves the unlock ‘mage 302 in accordance with the contac. The devie tans tions to a user-interface unlock stat if the detected contact corresponds to a predefined gesture, such as moving the unlock image aemss channel 306, Conversely, the device ‘maintains the user-interface lock state if the detected contact doesnot correspond to the predefined gesture. This process saves battery power by ensuring thatthe device i not sei dentally awakened. This process is easy for users to perform, Jn part because ofthe visual eve(s) provided on the touch [0178] In some embodiments, ater detecting an unlock gesture, the device displays a passeoe (or password) inter face (ex, UL 300C, FIG. 3C) for entering a passcode t0 complete the unlock process. The addition of a passcode protects agsinst unauthorized use of the device. In some ‘embodiments, the passcode interface includes an emergency call icon that permits an emergency call (eg, 9911) withoot entering the passcode. In some embodiments, the use of passcode is a user-selectable option (eg. part of settings 412), 0179] As noted above, processes that use gestures on the touch sereen to unlock thedeviee are deseribedin U.S. patent pplication Ser Nos. 11/322,S49, “Unlocking A Device By Performing Gestures On An Unlock Image,” file! Dev. 23, 2005, and 11/322,550, “Indication Of Progress Towards Sat- May 29, 2008 isfation OF A User Int Condition” filed Dee. 28, 2005, ‘which are ereby incorporated by reference [0180] FIGS. 44 and 4 stustate exemplary wer inter Faces fora menv of applications ona portable multifunction device in accordance with some embodiments. In some embodiments, user interface 400 inclodes the following elements, or subse or superset herent: [0181] Signalstegthindcatorts} 402 for wirelesseom- smunicatn(s), suchas cella and WiFi sinals, {0182} Time 44; (0183) Blociooth nicator 405; [018s] Borer stam indicator 406; [0185] Try 408 with Sons for raquenly wed applica tions, such as [0186] Phone 138, wich may include an indicator Aldo the numberof missed cll oe voicemal mes sages [0187] "E-mu client 140, which may inelude an indi- aioe 410 ofthe number of unread e-mails, [0188] Browser 147; and [0189] _ Music player 146: and [0190] _Jeons for ater applications, such a: (o191) IM 141; [o192] Image management 144 [0193] Camere 13 [0194] _ Video player 145; [0195] Weather 149-1 [0196] Stocks 149-2; [0197] Blog 142; [0198] Calendar 148; [0199] Calculator 149-3 [0200] Alann clock 149-4 [0201] Dictionary 149.5; 2nd [0202] User-created widget 149-6 [0203]. Insome embodiments user interlace 400B includes the fllowing elements, ora subset or superset there: [204] 402, 404, 406, 141,148, 144,143, 149-3, 149-2, 149-1, 149-4, 410,414, 138,140, and 147,08 described hove! [0205] Map 18 [0206] Notes 183 [0207] Settings 412, which provides access to setings Tr the device 100 and its various applications 136, as eserbed futher bolow; [0208] _ Video and music player module 152, also referred to a Pod (tademark of Apple Compute, lc.) module 152: and [0209] |Online video module 155, also refered to asYou- ‘Tube (trademark of Google, Ine.) modole 156. [0210] -Insomeembodiments, 14004 or 008 displaysall ‘ofthe available applications 1360none screens that heres ‘no aced 10 scroll through a list of applications (eg, via a sorll bar). In some embodiments, asthe numberof applica tions inerase, the icons corresponding to the applications ‘may deerease in size so that al applications may be displayed ‘on. single screen without seroling In some embodiments, hhaving all applications on one sereen and a menu button ‘enables user to access any desced pplication with at most ‘wo inpols, such as setivating the menu butlon 204 and then activating the desired application (eg, bya apor ober finger gesture on te eon corresponding othe application). In some embodiments, a predefined gesture on the menu button 204 (¢_, a double tap or a double click) acts a a shortcut tat initiates display of « particular user interoce in a particular