Hudson

SF JUG

Kohsuke Kawaguchi
Sun Microsystems, Inc.
Rise of Continuous
Integration
 Offload
from people, push to
$ computers
co m p u te rs

p e o p le

tim
2
Spend more CPU power to
help you
 First
on your laptops and
workstations
 IDEs are at the forefront
 And then to the servers
 a.k.a. “Continuous Integration”
 More frequent build/test executions
 Static code analysis tools
 And more to come

3
Hudson
h ttp s:// h u d so n -cio
. rg /

 Open-source CI server
 Emphasis on ease of installation and
use
 “java -jar hudson.war” execution
 Or your OS-specific package
 Configure everything from browsers
 Extensibility
 230+ community-developed public
plugins
 By 230+ contributors

 Estimated 13,000 installations 4
Adoption in all kinds of
businesses

5
Support Subscriptions

 Community releases every week

 Over 340 releases to date
 Rapid new features & bug fixes
delivery

 “Sun Continuous Integration Server”

 For enterprise that needs stability
 Bug fix only patch releases for 18
months

6
Basic Feature Set

 Hudson notices that there is a change

 Check out the source code
 CVS, Subversion, ClearCase, Mercurial,
Accurev, Perforce, Git, StarTeam, …
 Do builds
 Ant, Maven, shell script, NAnt, …
 Record and publish results
 Close the feedback loop
 RSS feeds
 E-mail/IM notification
 Tray application to notify developers
 Tool integration
 Findbugs, cobertura, …
7
Demo
Why do I care?

 Life before Hudson

 Dev forgets to commit a file
 RE nightly build breaks. It blocks SQE
 People waste time
 Life after Hudson
 Hudson will tell you in 5 mins if a build
broke
 No one else notices that you forgot to
commit a file
9
Why do I care?

 Life before Hudson

 Dev makes a change Monday AM
 SQE nightly finds a bug Monday night
 Dev fixes it Tuesday
 Lot of wasted time
 Life after Hudson
 Dev makes a change Monday AM
 Hudson finds a regression 30 mins
later
 Dev can fix it before lunch
10
Why do I care?

 Life before Hudson

 SQE test runs every night, results sent
out in e-mail
 After the enthusiasm of the 1st week is
gone, nobody looks at them anymore
 Regressions go unnoticed until it’s too
late
 Life after Hudson
 Tests run Hudson after every commit
 E-mail sent out only when tests start
failing
11
▪ So it manages to keep people’s attention
Matrix projects

 Run the same

thing on
different
environments
 such as different
JDKs, different
databases, and
different OSes
 Aggregate the
results

12
FindBugs integration

13
Tracking changes

14
… and more

 Browse workspaces  Build time trend report

15
Inter-Project Support

 Youcan host many projects on 1

Hudson
 And when you do, 1+1 becomes more
than 2
 You can…
 Execute tests separately and correlate
them with builds
 Mark builds as “promoted” based on
test outcome
 Track which build of libraries are used
where 16
Test/Build Separation
 Tests often dominate the build time
 Multiple tests are run in parallel
 Improves turn-around time time

JAX-WS #10 JAX-WS #11 JAX-WS #12 …

Unit test with GF

Unit test with Tomcat

SQE test

Test w/o container3OS x 2JDK = 6 tests

17
Test/Build Separation
Technique
 Copy bits from the build to test via
wget by permalinks
 Ant has the equivalent <get> task

$ wget –O foo.zip

http://server/hudson/job/foo/lastSuccessfulBuild/
artifact/foo.zip

 “Fingerprinting” to keep track of
where binaries are used

18
Build Promotion

 CIproduces a lot of “successful”

builds
 Often overwhelming to downstream
consumers


 Run tests as fast as you can

 If a build passes tests, promote it


19
Got the idea?

 Automation
 Reduce turn-around time
 Make things transparent
 Remove people from the loop

 Save people’s time
 Push jobs to servers, keep
workstations idle for you

20
Doing Distributed Builds

21
Why?

 There’sonly so much 1 computer

can do


 But lots of PCs = tedious

 What does Hudson do to help you?

22
Installing new slaves

 For
first 20 or so slaves, we did it
manually
 Insert CD, click, type, click, type, click,
…
 But that doesn’t scale

 Then we automated
 Available as “Hudson PXE Plugin”

23
Automated System
Installations
H u d so n + P X E plugin
>

lIS O im a g e s o f O S

l
l

 Slaves
 Power on, hit F12
 PC boots from network
(PXE)


24
Automated System
Installations
H u d so n + P X E plugin
>

lIS O im a g e s o f O S

l Your corporate IT
guy & his DHCP
l server

 Slaves
 Power on, hit F12
 PC boots from network
(PXE)
 Choose OS from menu
 Installs non-interactively
25
Distributed builds with
Hudson
 Master
 Serves HTTP
requests
 Stores all important
info
 Slaves
 170KB single JAR
 Assumed to be
unreliable
 Scale to at least 100
 Link
 Single bi-di byte 26
How master and slaves
start talking
 For Unix slaves, via SSH
 Only need SSHD and JRE on slaves
 We just need a host name

27
How master and slaves
start talking
 For Windows, DCOM
 We just need admin user name and
password
 No manual intervention
 Works even from Unix masters

28
How master and slaves
start talking
 Via Java Web Start
 When master cannot see slaves
 A separate socket connection is made




29
Automating JNLP launch

 Once
started, can be installed as
Windows service

30
Automating JNLP launch

 Emulate the JNLP client headless

$ java -jar slave.jar -jnlpUrl URL

31
Automated Tool
Installation - JDK
 JDK from http://java.sun.com/
 Hudson automatically chooses the
right bundle
 Always up to date with new releases

32
Automated Tool
Installation - Apache
 Ant and Maven from Apache






33
Heterogeneous Cluster
Challenge
 Your builds/tests need to run in specific
environment
 Dependency on individual nodes hurts
utilization
jobs slaves

Wombat
Windows test Windows #1

GlassFish
Windows test Windows #2

Hudson Windows test

Solaris
#1

Hudson Solaris test

34
Labels to rescue

 Label is a group of slaves

 Tie jobs to labels

jobs slaves

Wombat
Windows test Windows #1
Windows
GlassFish
Windows test Windows #2

Hudson Windows test

Solaris
#1
Solaris

Hudson Solaris test Windows #3

35
Concurrent Builds

Hudson #10

Hudson #11

Hudson #12

Hudson #13

Hudson #14

Hudson #15
time

36
Making builds sticky

 We want jobs to be mostly on the same

slave
 Faster check out
 Consistent results
 Minimizes disk consumption
 But does it softly


 Hudson uses consistent hash* to

achieve this
 More schedule controls become
possible:
* Use faster machines more frequently 37
Forecasting failures

 Hudson monitors key health metrics

of slaves
 Low disk space, insufficient swap
 Clock out of sync
 Extensible
 Slaves go offline automatically

38
Clean up mess after
builds
 Kill run-away processes
 Daemons, background processes left
by your build
 Works on Windows, Linux, Mac, and
Solaris



39
Load Statistics
Monitoring

40
When it’s time to add more
slaves

41
Hudson EC2 plugin

 Automatically
provisions slaves on
EC2 on demand
 Based on the current load
 Picks the right AMI depending on
demand
 Starts slave agent
 Shuts down unused instances


42
 Putting it all together

capacit
y
# of
executors

queue length usage

time
43
Hudson “Appliance” on
EC2
 Run
the master in the cloud too, if
you like
 Hudson on stock OpenSolaris AMI
 Data stored persistently in Elastic
Block Storage
▪ Dynamically expandable thanks to ZFS
▪ Online, too
 Packaged as a wizard

44
And a lot more I couldn’t
show
 IDE plugins
 iPhone/Android apps
 REST API and CLI
 Deployment Automation
 Hadoop, Selenium
…

45
Conclusions

 CI is here to stay
 Using lots of PCs is a key

 Hudson is very easy to get started

 Once started, there are a lot you can
do


46
Q&A

Kohsuke Kawaguchi
Sun Microsystems, Inc.
Compartmentalization
full Metro build #10 full Metro build #11

JAXB #10 JAXB #11 JAXB #12

JAX-WS #20 JAX-WS #21 JAX-WS #22

time

 Faster feedback
 Good engineering practice anyway
 Isolation of problems

48
Automated System
Installations
 SupportsOpenSolaris, Ubuntu,
CentOS, Fedora
 Trivial with most Linux
 Cooperate with Windows, too


 Quite useful outside Hudson, too

 No more broken CD drives
 No more CD-Rs

49