Framework and

Application
Benchmarking
Paul M. Jones
php|works atlanta
September 12-14, 2007
“Statistical Histories for
Fun and Profit”
• Benchmarking as economic predictor
• Benchmarking as engineering analysis
• Methodologies, results, and interpretation
• Application
• Single frameworks
• Framework comparisons
Read These
 About Paul

• Savant template system

• DB_Table, Text_Wiki
• PEAR Group
representative
• http://paul-m-jones.com
 Full Disclosure
• Solar framework
• Zend framework
• Zend_Db,
Zend_View
• “Rigor and integrity”
• Public project
 Profiling vs
Benchmarking
• Profiling applies to sub-systems
(component design)
• Benchmarking applies to entire
systems (architecture and
interaction)
• Profile is to unit test, as
benchmark is to system test
Benchmarking Subjects
• CPU • Programmer
productivity
• RAM
• Disk access • Time to initial
implementation
• Database access • Time to add new
• Network access major feature
• Requests/second • Time to fix bugs
-- controlling for variables is key --
-- must be numerically measurable --
What You Tell The Boss

• New site linked by

TechCrunch
• Could we handle
the traffic spike?
• “Sure thing, boss.”
 What You Tell The Team

• “...(time to load-test.)”
• Ran “ab”, optimized,
handled spike
• But how many req/sec
were possible?
• Where were the limits?
 Relative Limits

• There is a top limit

imposed by each stack
component
• When is effort
warranted?
• “Fast” and “slow” are
relative (need context
or requirements)
 Benchmarking Tools

• Use “ab”, “http_load”, “httperf”, “flood”, etc

• Emulate concurrent users hitting a URI
• Reports req/sec, latency, min/max times, etc

ab -c 10 -t 60 http://localhost/...
 Outer Limits
• Hardware & OS

• Amazon EC2, 1.7 GHz, 1.75

GB

• Fedora

• Apache 2.2

• ~2100 req/sec

• mod_php (5.2.1) & Xcache

• ~1160 req/sec

• Thanks Clay “Killersoft” Loveless

 Application
Benchmarking
• Run “ab” against a test installation
• As similar to production as possible
• Hardware, OS, server, PHP installation,
database, data, application code, etc.
• Network latency (use localhost)
• Server processes (turn everything off)
 Controlling for Variables

• Run “ab” multiple times

• Sessions (esp. file-based) will
cause exponentially declining
returns
• session_id(‘constant_value’)
 Benchmarking as
Economics
• About resource planning, not
speed (per se)
• With supply of “R” requests/
second, you need “S” servers
for demand of “T” traffic
• What stack components can
you modify to increase “R” so
you can handle more “T” with
same or fewer “S”?
 Framework
Benchmarking
Web Server + PHP
Framework
Boot Front Page Action View

• Dynamic dispatch cycle as the limiting factor

• Improvements off critical path will not
increase responsiveness
• Page cache? Action cache?
 Single-System
Methodology
• Use “ab” tool on a minimal application
• Sane config
• No action code
• No view layout or helpers
• Static view text
• No caching
• No database
 Single-System Analysis
• Now you know
how much you
need to improve
• Work on the
critical path
• Iterative
benchmarking
 Benchmarking as
Engineering Aid
• Every architecture
decision has a cost
• Compare benchmark
stats between versions
• Are added features are
worth req/sec cost?
• (Thanks, Laura
Thomson.)
 Multiple-System
Comparison
• Examine responsiveness in
context of other systems
• Compare architecture costs
between systems
• Use the same minimal app to
remove variability in
application code
 Compare Like With Like
• Frameworks to be compared
should be reasonably similar in
“style” or “class”

• PHP5, design patterns, front &

page controllers, controller &
view separation, plugins or
extensions, no globals

• Cake, Solar, Symfony, Zend

• Code Igniter? Prado?

 Multiple-System
Methodology (1/2)
• “Minimal” application
• Reduced or optimized config
• Especially Symfony
• Production mode
• No debugging or logging
• No database connection
• No layouts or view helpers
 Multiple-System
Methodology (2/2)
• A user session
• Xcache to reduce filesystem access
• Restart web server between runs (Xcache)
• 5 runs of “ab -c 10 -t 60”, averaged
• Calculate req/sec as percent of PHP (thanks,
Richard “Cyberlot” Thomas)
 Multiple-System Table
pct-php avg 1 2 3 4 5
baseline-html 1.7993 2082.8 2057 2156.7 1968 2112.2 2120.2
baseline-php 1.0000 1157.6 1264.4 1107.1 1131.7 1162.5 1122.1
cake-1.1.17 0.0850 98.38 99.95 96.98 97.69 97.66 99.6
solar-0.28.0 0.1234 142.82 142.5 142.79 141.26 143.92 143.62
symfony-1.0.6 0.0572 66.17 65.4 65.95 66.01 66.79 66.71
symfony-1.0.6-dw 0.0613 70.93 71.29 71.52 70.5 69.85 71.51
zend-1.0.1 0.0833 96.45 93.98 98.58 95.44 98.16 96.11
 Multiple-System Graph
cake-1.1.17

solar-0.28.0

symfony-1.0.6

symfony-1.0.6-dw

zend-1.0.1

0 0.0325 0.0650 0.0975 0.1300

What Does This Mean?
• Both more and less than you think it might
• Speed is not the only consideration
• Features
• Intangibles
• Even the slowest is still pretty fast
• Each AJAX call is one request
• Given comparable or equivalent features,
faster (probably) ~= better
 Code Igniter, Prado
pct-php avg 1 2 3 4 5
baseline-html 1.9364 2330.53 2677.95 2209.94 2316.75 2250.23 2197.80
baseline-php 1.000 1203.53 1446.63 1112.10 1144.40 1171.62 1142.88
ci-1.5.4 0.1830 220.30 222.44 217.22 220.43 221.05 220.38
prado-3.1.0 0.0289 34.78 34.42 34.56 34.50 34.36 36.06

ci-1.5.4

prado-3.1.0

0 0.05 0.10 0.15 0.20

 Failure Modes
• Apache/Lighty, Xcache/APC, mod_php/fastcgi
• EC2 load fluctuation between/during runs
• Order of testing
• Different values of -c, -t
• Memory and CPU usage
• Added features and app code will reduce
dynamic response ... but curve is unknown.
• Differences might be too great to minimize
 Links
• Google: “web framework benchmarks code”

• http://code.google.com/p/web-framework-
benchmarks/

• “New Year’s Benchmarks”

• http://paul-m-jones/blog/?p=238

• “Do All Frameworks Suck?”

• http://www.laurathomson.com/2007/07/do-all-
framewor.html

• Google: "cargo cult science"

• Questions?
• Comments?
• Criticism?
 Avoid Frameworks?
• If framework code is only at 7-12% of max
PHP, should we not use frameworks?
• “Slow” relative to what? Must compare like
with like, and proposed systems to existing
systems.
• You can’t avoid them, unless you write one-
off applications.