Scaling the drupal.

org
Infrastructure
Keeping up with exponential growth
Gerhard Killesreiter and Kieran Lal
gerhard@killesreiter.de

Drupal vzw

Drupal infrastructure – p. 1
History of the drupal.org infra-
structure
• Until autumn 2005: single server for Apache and
MySQL, managed by Kjartan Mannes.

Drupal infrastructure – p. 2
History of the drupal.org infra-
structure
• Until autumn 2005: single server for Apache and
MySQL, managed by Kjartan Mannes.
• Since then: SUN Fire v20z as MySQL-Server,
two Dell as Apache server, hosted at Open Source
Lab of the Oregon State University (OSUOSL).

Drupal infrastructure – p. 2
History of the drupal.org infra-
structure
• Until autumn 2005: single server for Apache and
MySQL, managed by Kjartan Mannes.
• Since then: SUN Fire v20z as MySQL-Server,
two Dell as Apache server, hosted at Open Source
Lab of the Oregon State University (OSUOSL).
• Since early summer 2007: two load balancers on
loan from OSUOSL

Drupal infrastructure – p. 2
History of the drupal.org infra-
structure
• Until autumn 2005: single server for Apache and
MySQL, managed by Kjartan Mannes.
• Since then: SUN Fire v20z as MySQL-Server,
two Dell as Apache server, hosted at Open Source
Lab of the Oregon State University (OSUOSL).
• Since early summer 2007: two load balancers on
loan from OSUOSL
• Since summer 2007: a MySQL-Slave on loan
from OSUOSL.

Drupal infrastructure – p. 2
History of the drupal.org infra-
structure
• Until autumn 2005: single server for Apache and
MySQL, managed by Kjartan Mannes.
• Since then: SUN Fire v20z as MySQL-Server,
two Dell as Apache server, hosted at Open Source
Lab of the Oregon State University (OSUOSL).
• Since early summer 2007: two load balancers on
loan from OSUOSL
• Since summer 2007: a MySQL-Slave on loan
from OSUOSL.
• Since autumn 2007: new MySQL-Master bought
by the Drupal association (not yet deployed).

Drupal infrastructure – p. 2
Why? Exponential growth
• Number of pages viewed by people grown
exponentially from 6 to 16 Million per month.

Drupal infrastructure – p. 3
Why? Exponential growth

15,00 M
pages in millions/month

10,00 M

5,00 M
2005-11-26 2006-03-06 2006-06-14 2006-09-22 2006-12-31 2007-04-10 2007-07-19

Drupal infrastructure – p. 4
Why? Exponential growth
• Number of pages viewed by people grown
exponentially from 6 to 16 Million per month.
• Other indicators also grow exponentially.

Drupal infrastructure – p. 5
Why? Exponential growth
300000 2000
Nodes Projects
Users Developers
Comments Helpers
250000
Service providers
CVS accounts 1500
200000

150000 1000

100000
500
50000

20 020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 0
01 01 01 02 02 02 02 03 03 03 03 04 04 04 04 05 05 05 05 06 06 06 06 07 07
-0 -0 -1 -0 -0 -0 -1 -0 -0 -0 -1 -0 -0 -0 -1 -0 -0 -0 -1 -0 -0 -0 -1 -0 -0
7- 9- 2- 3- 6- 9- 2- 3- 6- 9- 2- 3- 6- 9- 2- 3- 6- 9- 2- 3- 6- 9- 2- 3- 6-
02 30 29 29 27 25 24 24 22 20 19 18 16 14 13 13 11 09 08 08 06 04 03 03 01

Drupal infrastructure – p. 6
Main problem
As soon as you improve the efficiency of the web
server, the DB server will get all the extra load from
the extra pages you are able to serve.

Drupal infrastructure – p. 7
What did we try?
• PHP Opcode Cache (APC)

Drupal infrastructure – p. 8
What did we try?
• PHP Opcode Cache (APC)
• Drupal improvements (more caching, less
locking)

Drupal infrastructure – p. 8
What did we try?
• PHP Opcode Cache (APC)
• Drupal improvements (more caching, less
locking)
• Switched off statistics module.

Drupal infrastructure – p. 8
What did we try?
• PHP Opcode Cache (APC)
• Drupal improvements (more caching, less
locking)
• Switched off statistics module.
• change from MyISAM to InnoDB (and back for
the search tables)

Drupal infrastructure – p. 8
What did we try?
• PHP Opcode Cache (APC)
• Drupal improvements (more caching, less
locking)
• Switched off statistics module.
• change from MyISAM to InnoDB (and back for
the search tables)
• Evaluated „INSERT DELAYED“RR and friends

Drupal infrastructure – p. 8
What did we try?
• PHP Opcode Cache (APC)
• Drupal improvements (more caching, less
locking)
• Switched off statistics module.
• change from MyISAM to InnoDB (and back for
the search tables)
• Evaluated „INSERT DELAYED“RR and friends
• Squid as reverse proxy

Drupal infrastructure – p. 8
What did we try?
• PHP Opcode Cache (APC)
• Drupal improvements (more caching, less
locking)
• Switched off statistics module.
• change from MyISAM to InnoDB (and back for
the search tables)
• Evaluated „INSERT DELAYED“RR and friends
• Squid as reverse proxy
• More RAM: from 2 to 4 GB (webserver), 4 to 8
GB (DB server)

Drupal infrastructure – p. 8
What did we try? II
• memcached

Drupal infrastructure – p. 9
What did we try? II
• memcached
• Master/Slave DB-setup

Drupal infrastructure – p. 9
What did we try? II
• memcached
• Master/Slave DB-setup
• Use http:bl blocklist to lock out illegitimate
harvesters

Drupal infrastructure – p. 9
What did we try? II
• memcached
• Master/Slave DB-setup
• Use http:bl blocklist to lock out illegitimate
harvesters
• Extra caching for forum block and download
pages

Drupal infrastructure – p. 9
What did we try? II
• memcached
• Master/Slave DB-setup
• Use http:bl blocklist to lock out illegitimate
harvesters
• Extra caching for forum block and download
pages
• tune google down (100-150k pages/day vs. 10k
pages/day)

Drupal infrastructure – p. 9
What did we try? II
• memcached
• Master/Slave DB-setup
• Use http:bl blocklist to lock out illegitimate
harvesters
• Extra caching for forum block and download
pages
• tune google down (100-150k pages/day vs. 10k
pages/day)
• Buy better server (done, is beeing burned in).

Drupal infrastructure – p. 9
Things that didn’t work so well
• Optimizing search failed (auto-fixed by
upgrading to D5 at Jan 7th).

Drupal infrastructure – p. 10
Things that didn’t work so well
• Optimizing search failed (auto-fixed by
upgrading to D5 at Jan 7th).
• Use a Sun T2000 server from OSUOSL
(processor too slow).

Drupal infrastructure – p. 10
Things that didn’t work so well
• Optimizing search failed (auto-fixed by
upgrading to D5 at Jan 7th).
• Use a Sun T2000 server from OSUOSL
(processor too slow).
• memcached (page cache).

Drupal infrastructure – p. 10
Things that didn’t work so well
• Optimizing search failed (auto-fixed by
upgrading to D5 at Jan 7th).
• Use a Sun T2000 server from OSUOSL
(processor too slow).
• memcached (page cache).
• Fixing groups.drupal.org (much less traffic than
drupal.org).

Drupal infrastructure – p. 10
Things that didn’t work so well
• Optimizing search failed (auto-fixed by
upgrading to D5 at Jan 7th).
• Use a Sun T2000 server from OSUOSL
(processor too slow).
• memcached (page cache).
• Fixing groups.drupal.org (much less traffic than
drupal.org).
• http:bl blocklist only blocks about 2500 page
requests/day.

Drupal infrastructure – p. 10
Things that didn’t work so well
• Optimizing search failed (auto-fixed by
upgrading to D5 at Jan 7th).
• Use a Sun T2000 server from OSUOSL
(processor too slow).
• memcached (page cache).
• Fixing groups.drupal.org (much less traffic than
drupal.org).
• http:bl blocklist only blocks about 2500 page
requests/day.
• Using fast-cgi (unexpected crashes, moved back
to mod_php starting June 2007)

Drupal infrastructure – p. 10
Things that worked well
• APC

Drupal infrastructure – p. 11
Things that worked well
• APC
• Master/Slave setup (after initial problems)

Drupal infrastructure – p. 11
Things that worked well
• APC
• Master/Slave setup (after initial problems)
• tuning google down

Drupal infrastructure – p. 11
Things that worked well
• APC
• Master/Slave setup (after initial problems)
• tuning google down

Drupal infrastructure – p. 11
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)
• Setting up Master/Slave

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)
• Setting up Master/Slave
• Moved servers to better datacentre.

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)
• Setting up Master/Slave
• Moved servers to better datacentre.
• Setting up Cacti to allow us monitor the servers

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)
• Setting up Master/Slave
• Moved servers to better datacentre.
• Setting up Cacti to allow us monitor the servers
• Put servers into cfengine to let them easier run
them

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)
• Setting up Master/Slave
• Moved servers to better datacentre.
• Setting up Cacti to allow us monitor the servers
• Put servers into cfengine to let them easier run
them
• Provide backup facilities

Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)
• Setting up Master/Slave
• Moved servers to better datacentre.
• Setting up Cacti to allow us monitor the servers
• Put servers into cfengine to let them easier run
them
• Provide backup facilities
• Provide FTP-Server for faster downloads
Drupal infrastructure – p. 12
Things that OSUOSL helped
with
• Fixing PHP/Apache/APC setup
• Setting up memcached
• Tweaking MySQL (including sending patches to
MySQL AB)
• Setting up Master/Slave
• Moved servers to better datacentre.
• Setting up Cacti to allow us monitor the servers
• Put servers into cfengine to let them easier run
them
• Provide backup facilities
• Provide FTP-Server for faster downloads
• Lots of other stuff I forgot Drupal infrastructure – p. 12
Plans for the future
• Get more webnodes

Drupal infrastructure – p. 13
Plans for the future
• Get more webnodes
• If neccessary, buy more slave server

Drupal infrastructure – p. 13
Plans for the future
• Get more webnodes
• If neccessary, buy more slave server
• Fix Drupal to be able to run master/slave setup
without hacks.

Drupal infrastructure – p. 13
Plans for the future
• Get more webnodes
• If neccessary, buy more slave server
• Fix Drupal to be able to run master/slave setup
without hacks.
• Monitor system resource usage to get early hints
on possible bottlenecks.

Drupal infrastructure – p. 13