S

heduled Maintenan e Windows (What,

Why and How)

Christine Hogan and Tom Limon elli

This paper is based on Chapter 12 of The Pra ti e of System and Network Ad-
ministration by Thomas A. Limon elli and Christine Hogan [LH01℄. Copyright
2002 by Lumeta Corporation and Christine Hogan. All rights reserved. No part
of this publi ation may be reprodu ed without the prior onsent of the publisher,
Addison-Wesley.

1 Introdu tion
Computers and networks need maintenan e. They need hardware and software
upgrades, pat hes, and periodi re-ar hite ting. This paper looks at a stru tured
way to perform those tasks. All system administrators (SAs) need to do mainte-
nan e, and need to minimize the impa t the maintenan e has on their ustomers.
These days people rely heavily on the omputer and network infrastru ture
to perform their daily work. They are not tolerant of ad-ho approa hes to main-
tenan e. Bringing a server down in the middle of the day in order to repla e
a broken tape drive, or to add more disk spa e, is no longer a eptable. SAs
are expe ted to perform these tasks outside of normal working hours. What we
re ommend is a stru tured approa h to maintenan e. S hedule periodi main-
tenan e windows, delay intrusive tasks until the next maintenan e window, and
plan upgrades in advan e of the need for them.
S heduled maintenan e windows will look dierent for dierent ompanies.
Small ompanies typi ally have less equipment and therefore only need short
maintenan e windows. Fast-growing ompanies need frequent maintenan e win-
dows. High-availability ompanies need maintenan e windows that use redundant
systems to maintain servi e availability during the window. Mid-sized ompa-
nies need infrequent, but long maintenan e windows. Large ompanies need a
ombination of lo ation-spe i maintenan e windows, and \high-availability"
maintenan e windows for shared infrastru ture.
This paper will on entrate on the long, infrequent maintenan e windows,
and will in lude some spe i s for high-availability sites. Other sites use a subset
of these te hniques. We all the te hnique des ribed here the \ ight dire tor"
te hnique, named after the role of the ight dire tor in NASA spa e laun hes.1
The ight dire tor te hnique guides the a tivities before the window, during
exe ution, and after. It is summarized in Table 1.
Preparation S hedule the window
Pi k a ight dire tor
Prepare hange proposals
Build a master plan
Exe ution Disable a ess
Shutdown sequen e
Exe ute plan
Perform testing
Resolution Announ e ompletion
Enable a ess
Have a visible presen e
Be prepared for problems

Fig. 1: The Three Stages of a Maintenan e Window

2 Motivation
Some ompanies are willing to s hedule regular maintenan e windows for major
systems and networking work in return for better availability during normal oper-
ations. Depending on the size of the site, this ould be one evening and night per
month, or perhaps an entire weekend, from Friday evening to Monday morning,
on e a quarter.
System administrators often like to have a maintenan e window during whi h
they an take down any and all systems, and stop all servi es, be ause it redu es
omplexity and makes testing easier. For example, in utting email servi es over
to a new system, you need to transfer existing mailboxes as well as swit hing the
in oming mail feed to the new system. Trying to transfer the existing mailboxes
without utting the feed and the read a ess for those mailboxes, and yet ensure
onsisten y is a very tri ky problem. But, if you an bring email servi es down
while you do the transfer, it be omes a lot easier. In addition, it is a lot easier
to he k that the system is working orre tly before you turn the mail feed and
the read a ess ba k on again than it is to deal with having dropped or boun ed
mail if something didn't work quite right with the live utover.
However, you will have to sell the on ept to the ompany at large in terms of
a benet to them, not in terms of it making the SA's life easier. That means that
1The origin of this terminology, and a number of the approa hes used, was with Paul Evans.
Paul is an avid observer of the spa e program.
you need to be able to promise better servi e availability the rest of the time. In
other words, you need to plan in advan e: if you have one maintenan e window
per quarter, you need to make sure that the work you do this quarter will hold
you through the end of the next quarter, so that you won't need to bring the
system down again. You should also be prepared to provide metri s to ba k up
your laims of higher availability2 from before and after you have su eeded in
getting s heduled maintenan e windows. A single large outage also an be mu h
less annoying to users than many little outages [LRNL97℄.

3 S heduling
A maintenan e window is by denition a short period of time in whi h a lot of
systems work needs to be performed. It is disruptive to the rest of the ompany,
and so the s heduling must be done in ooperation with the ustomers. You need
to nd out what dates do and don't work for them. In parti ular, you will almost
ertainly need to avoid the end-of-month, end-of-quarter and end-of-s al-year
dates so that the sales team an get all of the rushed orders entered, and the
a ounting group an produ e the nan ials for that period. You will also need
to avoid produ t release dates, if that is relevant to your business. There may
be major trade shows or other demos that must be avoided. Universities have
dierent onstraints around the a ademi year. Some businesses, su h as toy and
greeting ard manufa turers, may have seasonal onstraints.

4 Planning
As with all work on important systems, the tasks need to be planned by the indi-
viduals performing them so that no original thought, or problem solving, should
be involved in performing the task on the day. There should be no unforeseen
events, only planned ontingen ies.
Planning for a maintenan e window also has another dimension, however.
Sin e they o ur only o asionally, the system administrators need to plan far
enough in advan e so that they an get quotes, submit pur hase orders, get them
approved, and have any new equipment arrive a week or so before the maintenan e
window. Sin e the lead-time on some equipment an be 6 weeks or more, this
means starting to plan for the next maintenan e window almost immediately
after the pre eding one has just ended.
2 The most ee tive way of doing this is to use an existing monitoring system that an
produ e histori al graphs. The most popular ones at the time of writing are MRTG [Oet98℄
and Cri ket [All99℄.
5 Flight dire tor
The ight dire tor is the one who is responsible for rafting the announ ement
noti es and making sure that they go out on time. She is also responsible for
s heduling the submitted work proposals based on the intera tions between them
and the sta required, de iding whi h ones (if any) don't make the ut for that
maintenan e window, monitoring the progress of the tasks during the mainte-
nan e window, ensuring that the testing o urs orre tly, and ommuni ating
status to the rest of the ompany at the end of the maintenan e window.
The person who lls the role of ight dire tor must be a senior system admin-
istrator who is apable of assessing work proposals from other members of the
SA team, and spotting dependen ies and ee ts that may have been overlooked.
The ight dire tor must also be apable of making judgment alls on the level of
risk versus need for some of the more riti al tasks that impa t the infrastru ture.
She must have a good overview of the site and the impli ations of all the work.
In addition, the ight dire tor annot perform any te hni al work of her own
during that maintenan e window. This means that typi ally the ight dire tor
is a member of a multi-person team, and the other members of the team take
on the work that would normally have been the responsibility of that individual.
The ight dire tor is not normally a manager, unless the manager was re ently
promoted from a senior SA position, be ause of the skill requirements des ribed
above.

6 Change proposals
One week prior to the maintenan e window, all hange proposals should have
been submitted. A good way of managing the hange proposal pro ess is to
have all the hange proposals online in a revision- ontrolled area. Ea h SA edits
do uments in a dire tory with his name on it. The do uments supply all the
required information. One week before the hange, this revision- ontrolled area
is frozen and all subsequent requests to make hanges to the do uments have to
be made through the ight dire tor.
A hange proposal form should answer at least these questions:
 What hanges are going to be made?
 What ma hines will you be working on?
 What are the pre-maintenan e window dependen ies, and due dates?
 What needs to be up for the hange to happen?
 What will be ae ted by the hange?
 Who is performing the work?
  How long will the hange take in a tive time and elapsed time, in luding
testing, and how many additional helpers will be needed?
 What are the test pro edures? What equipment do they require?
 What is the ba k-out pro edure and how long will it take?
7 The master plan
One week before the maintenan e window, the ight dire tor freezes the hange
proposals, and starts working on a master plan. The master plan takes into
a ount all the dependen ies, elapsed and a tive times for the hange proposals.
The end result is a series of tables, one for ea h person, showing them what
task they will perform during what time interval and who the oordinator for
that task is. There is also a master hart that shows all the tasks that are
being performed over the entire time, who is performing them, the team lead,
and what the dependen ies are. The master plan must also take into a ount
omplete system-wide testing after all work has been ompleted.
If there are too many hange proposals, the ight dire tor will nd that
s heduling all of them produ es too many on i ts, either in terms of ma hine
availability, or in terms of the people required. There needs to be sla k in the
s hedule to allow for things to go wrong. The diÆ ult de isions about whi h
proje ts should go ahead and whi h ones have to wait should be made before-
hand, rather than in the heat of the moment when something is taking too long
and blowing the s hedule, and everyone is tired and stressed. The ight dire tor
makes the all on when some hange proposals need to be ut out, and gets all the
parties involved in the s heduling on i t to dis uss it and pi k the best ourse
for the ompany.

7.1 Building a Master Plan Template

On e we had run a few maintenan e windows, we dis overed that there was
a formula that worked well for us. The systems on whi h most people were
dependent for their work were operated on during the Friday evening. The very
rst thing to be upgraded or hanged was the network. Next on the list was
onsole servi e, then the authenti ation servers. While these were in progress,
all the other SAs helped out with hardware tasks like memory, disk or CPU
upgrades, repla ing broken hardware, or moving equipment within, or between,
data enters. Last thing on Friday night, large data moves were started so that
they ould run overnight.
The remaining tasks were then s heduled into Saturday, with some people
being s heduled to help others in between their own tasks. Sunday was reserved
for omprehensive system-wide testing and debugging be ause of the high impor-
tan e pla ed on testing.

8 Disabling a ess
The very rst task in the maintenan e window is to disable or dis ourage system
a ess, and provide reminders that it is a maintenan e window. Depending on
what the site looks like, and what fa ilities are available, this pro ess may involve:

 Pla ing noti es on all doors into the ampus buildings with the maintenan e
window times learly visible.
 Disabling all remote a ess to the site, whether by dial-in, dedi ated lines,
wireless or over a publi network.
 Making an announ ement over the publi address system in the ampus
buildings to remind everyone that systems are about to go down.
 Changing the helpdesk voi email message to announ e that this is a main-
tenan e window, and stating when normal servi e should be restored.

These steps redu e the han e that someone will try to use the systems during
the maintenan e window, whi h ould ause in onsisten ies, or a idental loss or
damage of their work. It also redu es the han e that the person arrying the
on- all pager will have to respond to urgent helpdesk voi emails saying that the
network is down.

9 Me hani s and oordination

There are some key pie es of te hnology that enable the maintenan e window
pro ess des ribed here to pro eed smoothly. These are not solely useful for main-
tenan e windows, but they are riti al to their su ess.

9.1 Shutdown/boot sequen e

In most sites there are some systems, or sets of systems, that are required to be
available in order for other systems to shutdown or boot leanly. If a ma hines
tries to boot when ma hines and servi es that it relies on are not available, it
will fail to boot properly. Typi ally, the ma hine will boot, but it will fail to run
some of the programs that it usually runs on start-up. These programs might
be servi es that others rely on, or just programs that run lo ally on someone's
desktop. In either ase, the ma hine will not work properly, and it may not
be apparent why. When shutting down a ma hine, it may need to onta t le
servers, li ense servers or database servers that are in use in order to properly
terminate the link. If it annot onta t those servers it may hang for a long time,
or indenitely, trying to onta t those servers before ompleting the shutdown
pro ess. It is important to understand and tra k the dependen ies that ma hines
have on ea h other during boot up and shutdown. You do not want to have to
gure it out for the rst time when a ma hine room unexpe tedly loses power.
The most riti al systems, su h as onsole servers, authenti ation servers,
name-servi e ma hines, li ense servers, appli ation servers and data servers, typ-
i ally need to be booted before other ma hines su h as ompute servers and
desktops. There will also be dependen ies between the riti al servers. It is vital
to maintain a boot sequen e list for all data enter ma hines, with one or more
ma hines at ea h stage, as appropriate. Typi ally the rst ouple of stages will
have few ma hines, maybe only one ma hine in them, but later stages will have
many ma hines. All data enter ma hines should be booted before any non-data
enter ma hines, sin e no ma hine in a data enter should rely on any ma hine
outside a data enter.
The shutdown sequen e is typi ally very lose to, if not exa tly the same as,
the reverse of the boot sequen e. There may be one or two minor dieren es.
The shutdown sequen e is a vital omponent to starting work at the beginning
of the maintenan e window. The ma hines that are operated on at the start of
the maintenan e window are typi ally those with the most dependen ies on them,
so any ma hine that needs to be shut down for hardware maintenan e/upgrades,
or moving, has to be shut down before the work on the riti al ma hines starts,
and it is important to get the ma hines shut down in the right order, so as to
avoid wasting time bringing ma hines ba k up so that other ma hines an be shut
down leanly.
The boot sequen e is also riti al to the omprehensive system testing that is
performed at the end of the maintenan e window, as we shall see later.

9.2 Console servi e

All of the equipment in the omputer room that is apable of supporting a serial
onsole should have its serial onsole onne ted to some kind of onsole on en-
trator su h as a networked terminal server. The onsole servers should be running
onsole server software3 whi h se urely allows authenti ated a ess to ma hine
onsoles from authorized hosts a ross the network. This enables people to work
from their own desks, rather than having to try to oordinate a ess for many
people to the very limited number of monitors in the omputer room, or having to
waste omputer room spa e, power and ooling with more monitors. If possible,
the onsole software should log all output to disk for future referen e.
3 http://www. onserver. om
 It is also more onvenient for the individual SAs to work in their own workspa e
with their preparatory notes and referen e materials around them.

9.3 Radios
Sin e the maintenan e window is tightly s heduled, there are many dependen ies,
and system administration work an be unpredi table at times, everyone has to
he k with the ight dire tor to let him or her know when they are nished with
a task, and before they start a new task, to make sure that the prerequisite tasks
have all been ompleted.
We re ommend using hand-held radios to ommuni ate within the group.
Rather than seeking out the ight dire tor, an SA an just all her over the radio
to he k in. Likewise, the ight dire tor an onta t the SAs to nd out status,
and team members and team leaders an nd ea h other, and oordinate, over
the radio. If someone needs extra help, they an also ask for it over the radio.
There are multiple radio hannels, and long onversations an move to another
hannel to keep the primary one free.
The radios are also essential for system-wide testing at the end of the main-
tenan e window, whi h will be des ribed later, in Se tion 11.
If radios won't work, or work badly, in your data enter due to RF shielding
put an internal phone extension with a long ord at the end of every row. That
way SAs in the data enter an still ommuni ate with other SAs while working
in the data enter. At worst, they an go outside the data enter, raise someone
on the radio and arrange to talk to her on a spe i telephone inside the data
enter.

10 Deadlines for hange ompletion

A riti al role of the ight dire tor is tra king how the various tasks are progress-
ing, and de iding when a parti ular hange should be aborted and the ba k-out
plan for that hange exe uted. For a general task that had no other dependen ies,
and where those involved had no other remaining tasks, that time would be 11pm
on Saturday evening, minus the amount of time that it would take to implement
the ba k-out plan, in the ase of a weekend maintenan e window. Though some-
times the sanity and ee tiveness of the people working on the task needs to be
taken into a ount. If they are exhausted and frustrated, the ight dire tor may
de ide to tell them to take a break or to start the ba k-out pro ess early if they
won't be able to implement it as eÆ iently as they would when they were fresh.
If there are other tasks that depend on that system or servi e being opera-
tional, it is parti ularly riti al to pre-dene a ut-o point for task ompletion.
For example, if a onsole server upgrade was going badly, there might not be time
to start the large data moves on Friday night, whi h would prevent them from
happening at all, whi h ould ause a as ade of atastrophi ee ts for people
within the ompany until the next downtime.

11 Comprehensive system testing

A maintenan e window is like taking a very ompli ated pie e of ma hinery om-
pletely to pie es, and then putting it ba k together again, under a time onstraint.
Sometimes there are bits left over when you think you should be nished.
The nal stage of a maintenan e window must involve omprehensive system
testing. Depending on the length of the window, you may only be able to test the
few omponents that you worked on. However, if you have a weekend maintenan e
window, and have a lot of people performing work on may dierent omponents,
you should plan on spending all day Sunday doing system testing.
Sunday system testing starts with a omplete shutdown of all ma hines in the
data enter, and then stepping through the boot sequen e in order. There is an
individual assigned to ea h ma hine on the reboot list. The ight dire tor alls
out the stages during the shutdown sequen e over the radio, and ea h individual
alls in ea h ma hine that they are responsible for when it has ompletely shut
down. When all the ma hines at the urrent stage have shut down, the ight
dire tor alls the next stage. When everything is down, the ight dire tor alls
out the stages on the boot list, and tra ks the progress. If there are problems
with a ma hine at any stage, those must be debugged and xed before moving
on to the next stage. The person assigned to ea h ma hine is responsible for
ensuring that all servi es started orre tly before they all it in as being up.
Finally, when all the ma hines in the data enter have been su essfully boot-
ed in the orre t order, the SA team is split into groups. Ea h group has a team
leader, and is assigned an area in one of the ampus buildings. The teams are
given instru tions on what to do and tests to perform. The instru tions always
in lude rebooting every ma hine to make sure it omes up leanly. They ould
also in lude logging in, he king for a parti ular servi e, or trying to run a par-
ti ular appli ation, for example. Ea h person in the group has a sta k of olored
sti ky tabs (e.g. green) used for marking oÆ es and ubi les that have been om-
pleted, and veried working. They also have a sta k of sti ky tabs of a dierent
olor (e.g. red) that they use to mark ubi les that have a problem. When SAs
run a ross a problem, they spend a short amount of time trying to x it, before
alling it in to the entral ore of people who are assigned to stay in the main
building to help debug problems. As a team nishes its area, it is assigned to a
new area, or to help another team to omplete an area, until the whole ampus
has been overed.
The entral ommand enter, with the ight dire tor and the senior SA
trouble-shooters, keeps tra k of problems on a white-board. The SAs and ight
dire tor de ide who should ta kle ea h problem, based on the likely ause and
who is available. By the end of testing, all oÆ es and ubi les should have tags,
preferably all indi ating su ess. If any oÆ es or ubi les still have tags indi-
ating a problem, there should be a note left for that ustomer explaining the
problem, and someone assigned to meet with that person to try to resolve it rst
thing in the morning.
This method has aught many potentially serious problems before people ame
into work the following day. For example, dis onne ted, or in orre tly onne ted
network segments after wiring loset lean up, a failed push to a software depot
server, inoperable NIS slaves, AppleTalk problems, and li ense server problems.
Be warned, however, that there may be ma hines that weren't working in the
rst pla e. The reboot teams should always make sure to note when a ma hine
did not look operational before they rebooted it. They an still take time to try
to x it, but it is lower on the priority list, and does not have to happen before
the end of the maintenan e window.
Ideally, the system testing, and site-wide rebooting should be ompleted some-
time on Sunday afternoon, giving the SA team time to get some rest after a
stressful weekend, before oming into work the next day.

12 Post-maintenan e ommuni ation

On e the maintenan e work and system testing is ompleted, the ight dire -
tor sends out a message to the ompany informing everyone that servi e should
now be fully restored. The message brie y outlines the main su esses of the
maintenan e window. It also brie y lists any servi es that are known not to be
fun tioning, and when they will be xed. Hopefully there won't be any nonfun -
tional servi es, of ourse.
This message should be in a xed format, and largely written in advan e,
sin e the ight dire tor will be too tired to be very oherent or upbeat if she
writes the message at the end of a long weekend. There is also little han e that
anyone who proofreads the message at that point is going to be able to help,
either.

13 Re-enable remote a ess

The nal a t before leaving the building should be to re-enable remote a ess,
and restore the voi email on the helpdesk phone to normal. Make sure that this
appears on the master plan, and the individual plans of those responsible. It
an be very easily forgotten after an exhausting weekend, but it is a very visible,
in onvenient and embarrassing thing to forget. Espe ially sin e it an't be xed
remotely if all remote a ess was turned o su essfully.
14 Visible presen e the morning after
It is very important for the entire SA group to be in early, and to be visible
to the ompany the morning after a maintenan e window, no matter how hard
they have worked during the outage. Have the people who look after parti ular
departments roam the orridors of the departments they look after, keeping an
eye and an ear open for problems.
Have the ight dire tor and some of the senior SAs (from the entral ore-
servi es group, if there is one) sit in the helpdesk area to listen to what alls are
oming, and listen for things that may be related to the maintenan e window.
They should be able to dete t and x them sooner than the regular helpdesk
sta, who won't have su h a good overview of what has happened.

15 High availability sites

By the very nature of their business, high availability sites annot aord to have
large planned outages.4 This also means that they annot aord to not make
the large investment ne essary to provide high availability. Sites that have high
availability requirements need to have lots of hot redundant systems that ontinue
providing servi e when any one omponent fails. The higher the availability
requirement, the more redundant systems are required to a hieve it.5
These sites still need to perform maintenan e on the systems in servi e. While
the availability guarantees that these sites make to their ustomers typi ally
ex lude maintenan e windows, they will lose ustomers if they have large planned
outages.

15.1 The similarities

Most of the prin iples des ribed above for maintenan e windows at a orporate
site apply at high availability sites.
 They need to s hedule the maintenan e window so that it has the least
impa t on their ustomers. For example, Internet servi e providers often
hoose 2am (lo al time) mid-week; e- ommer e sites need to hoose a time
when they do the least business. These windows will typi ally be quite
frequent, su h as on e a week, and shorter, perhaps 4-6 hours in duration.
4 High availability is anything above 99.9% availability. Typi ally sites will be aiming for
three nines (99.9%) (9 hours down-time per year), four nines (99.99%) (1 hour per year) or ve
nines (99.999%) (5 minutes per year). Six nines (99.9999%) (less than 1 minute a year) is more
expensive than most sites an aord.
5 The term n + 1 redundan y is used for servi es where any one omponent an fail without
bringing the servi e down, n + 2 means any two omponents an fail, and so on.
  They need to let their ustomers know when maintenan e windows are
s heduled. For Internet servi e providers, this means sending an email to
the ustomer onta ts. For an e- ommer e site, this means having a banner
on the site. In both ases it should only be sent to those ustomers who may
be ae ted and should ontain a warning that small outages or degraded
servi e may o ur during the maintenan e window, and give the times of
that window. There should only be a single message about the window.
 Planning and doing as mu h as possible beforehand is riti al be ause the
maintenan e windows need to be as short as possible.
 There still needs to be a ight dire tor who oordinates the s heduling, and
tra ks the progress of the tasks. If the windows are weekly, this may be a
quarter-time or half-time job.
 Ea h item needs to have hange proposal, as des ribed in Se tion 6. The
hange proposal needs to list the redundant systems and in lude a test
to verify that the redundant systems have ki ked in and servi e is still
available.
 They need to tightly plan the maintenan e window. Maintenan e windows
are typi ally smaller in s ope and shorter in time. Items s heduled by
dierent people for a given window should not have any dependen ies on
ea h other. There still needs to be a small master plan that shows who has
what tasks and their ompletion times.
 The ight dire tor must be very stri t about the deadlines for hange om-
pletion.
 Everything must be fully tested before it is de lared omplete.
 Console servers benet all sites.
 The SAs need to have a strong presen e when the site approa hes and enters
its busy time. They need to be prepared to qui kly deal with any problems
that may arise due to the maintenan e.

15.2 The dieren es

There are also a number of dieren es in maintenan e windows for high avail-
ability sites.
 There is a prerequisite that there is redundan y at the site if it is to have
high availability.
 It is not ne essary to disable a ess. Servi es should remain available.
  It is not ne essary to have a full shutdown/boot list, sin e a full shutdown
and reboot does not happen. However, there should be a dependen y list
if there are any dependen ies between ma hines.6
 Sin e the ustomers of Internet servi e providers and e- ommer e sites are
not on-site, being physi ally visible the morning after is irrelevant. However
being available and responsive is still important. Find ways to in rease your
visibility and assure ex ellent responsiveness. Advertise what the hange
was, how to report problems, and so on.
 A post-maintenan e ommuni ation is usually not required, unless there
are remaining problems that the ustomers need to know about. Customers
don't want to be bombarded with email from their servi e providers.
 The most important dieren e is that the redundant ar hite ture of the site
needs to be taken into a ount during the maintenan e window planning.
The ight dire tor needs to make sure that none of the s heduled work an
take the servi e down. The SAs need to make sure that they know how long
fail-over takes to happen.7 If redundan y is implemented within a single
ma hine, the SA needs to know how to work on one part of the ma hine
while keeping the system operating normally.
 Availability of the servi e as a whole needs to be losely monitored during
the maintenan e window. There needs to be a plan for how to deal with
any failure that auses an outage due to temporary la k of redundan y.

6 Usually high availability sites avoid dependen ies between ma hines as mu h as possible.
7 For instan e, how long does the routing system take to rea h onvergen e when one of the
routers goes down or omes ba k up?
Referen es
[All99℄ Je R. Allen. Driving by the rear-view mirror: Managing a network
with ri ket. In First Conferen e on Network Administration (NETA
'99), pages 1{10, Santa Clara, California, April 7-10 1999. USENIX.

[LH01℄ Thomas A. Limon elli and Christine Hogan. The Pra ti e of System
and Network Administration. Addison-Wesley, August 2001, ISBN:

0201702711.
[LRNL97℄ Thomas A. Limon elli, Tom Reingold, Ravi Narayan, and Ralph
Loura. Creating a Network for Lu ent Bell Labs Resear h south. In
Eleventh Systems Administration Conferen e (LISA '97), page 123,

San Diego, California, O tober 26-31 1997. USENIX.

[Oet98℄ Tobias Oetiker. MRTG - the multi router traÆ grapher. In Twelfth
Systems Administration Conferen e (LISA '98) , page 141, Boston,
Massa husetts, De ember 6-11 1998. USENIX.