USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING

PRACTICAL SIKULI
Tsung-Hsiang (Sean) Chang MIT CSAIL

Collaborators: Tom Yeh, Rob Miller

1

WHAT IS SIKULI?
2

WHAT IS SIKULI?
2

2 3 1
3

Low-level API? Macs system design?

EMPTY THE TRASH CAN

5

OUTLINE
What How Sikuli Sikuli

Sikuli is

Sikuli works for GUI automation for GUI testing

Conclusions

How Sikuli Works

TEMPLATE MATCHING
click( )

TEMPLATE MATCHING
click( )

SYSTEM DESIGN

DEMO 1: EMPTY TRASH CAN

2 3 1
10

REGION AND MATCH

Region

match = region.nd(
11

or text pattern)

REGION

basic

attributes: x, y, width, height is a Region is also a Region methods: Screen(0).nd(W).click(X)

12

Match Screen

chained

KEY METHODS OF REGION

nd, ndAll, exists, wait, waitVanish Mouse Actions click, doubleClick, rightClick, hover, dragDrop mouseDown, mouseUp

Keyboard Actions type(text), paste(text) keyDown, keyUp

13

EX2: FACEBOOK STALKER

14

EX2: FACEBOOK STALKER

14

RESTRICT SEARCH REGION

15

But the Region constant is bad

16

APP CLASS

refox

= App.open(Firefox)

refox.focus(), refox.close() refox.window([n]) returns

the Region of Firefoxs n-th window

17

FIND IN CERTAIN APP

18

EX3: MUTE SPEAKERS

19

EX3: MUTE SPEAKERS

19

SPATIAL OPERATORS

20

SPATIAL OPERATORS

20

SPATIAL OPERATORS (2)

21

TEXT MATCHING AND RECOGNITION

click(OK), nd(Cancel) region.text()

- read the text in a region

22

VISUAL-EVENT-DRIVEN PROGRAMMING
observes

visual events in any Region

Region.onAppear(Pattern, handler) Region.onVanish(Pattern, handler) Region.onChange(Pattern, handler)

23

VISUAL-EVENT-DRIVEN PROGRAMMING

24

VISUAL-EVENT-DRIVEN PROGRAMMING

24

VISUAL DICTIONARY

25

GUI TESTING

26

VISUAL ASSERTION
make

sure the appearance of visual feedback

27

VISUAL ASSERTION
make

sure the appearance of visual feedback

27

JUNIT INTEGRATION

28

SIKULI IN COMMAND LINE

usage: Sikuli-IDE [--args <arguments>] [-h] [-r <sikuli-file>] [-s] [-t <sikuli-test-case>] --args <arguments> specify the arguments passed to Jython's sys.argv -h,--help print this help message -r,--run <sikuli-file> run .sikuli or .skl file -s,--stderr print runtime errors to stderr instead of popping up a message box -t,--test <sikuli-test-case> run .sikuli as a unit test case with junit's text UI runner

29

CONCLUSIONS

30

PLATFORM INDEPENDENCE
Works

on ANY GUI that can be displayed on Windows/Linux/Mac

Virtual

machines desktop

Remote Mobile

simulators: Android, iPhone

Web: Flash, HTML+Javascript

31

PROGRAM AGAINST UI
Sikuli programs are written against the user interface instead of an API

UI: visible, familiar, always exists API: faster, probably more stable
32

FUTURE WORK

fast

and accurate OCR on screen integration

Accessibility API

33

Thanks!

http://sikuli.org or, just google sikuli

New Sikuli X 1.0-rc2 released. Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
34