CIS 188 CCNP TSHOOT (Troubleshooting) Chapter 1 Planning Maintenance for Complex Networks

Rick Graziani Cabrillo College graziani@cabrillo.edu Fall 2011

Chapter Roadmap
Applying Maintenance Maintenance Processes and Methodologies Procedures Maintenance Models Network Task Identification and Methodologies Network Maintenance Determining Planning Procedures and Scheduling maintenance Tools for Formalizing ChangeMaintenance Models Control Procedures Establishing Network Documentation Procedures Establishing Effective Communication Defining Standardization Planning for Disaster Recovery Network Monitoring and Performance Measurement Network Maintenance Tools, Applications and Resources Fundamental Tools, Applications and Resources CLI, debug, GUI, Backup Servers, Log Servers, Time Servers Configuration and Documentation Tools Logging Services Network Monitoring and Performance Measurement Tools Implementing Backup and Restore 2 Disaster Recovery Tools

Without these processes and tools

Network engineers can get into trouble.

Radia Perlman

Troubleshooting analogy (life lessons) Before solving the problem, know the problem you are trying to solve.
4

Planning Maintenance for Complex Networks

Configuration and implementation is sexy Planning and documenting, not as much.

5

Applying Maintenance Methodologies

Maintenance Models and Methodologies Determining Procedures and Tools for Maintenance Models

Applying Maintenance Methodologies

Support and maintenance Objective: Keep the network available with minimum service disruption and at acceptable performance levels. Solution: Structured network maintenance
7

Maintenance Models and Methodologies

Network engineers job description these tasks: Device installation and maintenance Failure response Network performance Business procedures Security

Interrupt Driven

In many smaller networks, the process is largely interrupt driven. User has a problem Application performance problems Security incidents are reported Disadvantage Other tasks beneficial to the long-term health of the network may be delayed or not done. Tasks get done in the order requested instead of priority or urgency. May result in more network downtime in the long run. Cant avoid interrupt driven work entirely because of failures. Can reduce them by proactively monitoring and managing the network. 9

Structured Network Maintenance

Alternative to interrupt-driven is structured network maintenance. Advantages Reduced Network Downtime: Discover problems before they happen More cost effectiveness: Adequate budget for networking needs and choosing proper equipment. Better alignment with business objectives: Upgrades and major maintenance jobs are not scheduled during critical business hours. Higher network security: Observe network vulnerabilities and needs and plan for strengthening network security.

10

Structured Network Methodologies

Several well-known methodologies: IT Infrastructure Library (ITIL) FCAPS - ISO (Fault management, Configuration management, Accounting management, Performance Management, and Security Management) Telecommunications Management Network (TMN) - ITU-T Cisco Lifecycle Services

11

Chapter Roadmap
Applying Maintenance Maintenance Processes and Methodologies Procedures Maintenance Models Network Task Identification and Methodologies Network Maintenance Determining Planning Procedures and Scheduling maintenance Tools for Formalizing ChangeMaintenance Models Control Procedures Establishing Network Documentation Procedures Establishing Effective Communication Defining Standardization Planning for Disaster Recovery Network Monitoring and Performance Measurement Network Maintenance Tools, Applications and Resources Fundamental Tools, Applications and Resources CLI, debug, GUI, Backup Servers, Log Servers, Time Servers Configuration and Documentation Tools Logging Services Network Monitoring and Performance Measurement Tools Implementing Backup and Restore 12 Disaster Recovery Tools

Maintenance Processes and Procedures

Backup and Recovery

Network engineers need to do the following: Identify essential network maintenance tasks Recognize the advantages of scheduled maintenance Evaluate the key decision factors that affect change control procedures Describe the essential elements of network documentation Plan for disaster recovery Describe the importance of network monitoring and performance measurement

13

Network Maintenance Task Identification

Network maintenance plans need to include procedures for: Accommodating Adds, Moves, and Changes Installation and configuration of new devices Replacement of failed devices Backup of device configurations and software Troubleshooting link and device failures Software upgrading or patching Network monitoring Performance measurement and capacity planning Writing and updating documentation

14

Network Maintenance Planning

Scheduling Maintenance Interrupt driven or part of maintenance cycle Schedule guarantees that these tasks will be done regularly and will not get lost in the busy day-to-day work schedule. Change requests need not be handled immediately, but during the next scheduled timeframe. Allows you to properly prioritize tasks Scheduled maintenance, tasks that are disruptive to the network are scheduled during off-hours.

15

Formalizing Change-Control Procedures

Any change you make has an associated risk due to possible mistakes, conflicts, or bugs. Determine the impact of the change on the network Balance this against the urgency of the change High-impact changes are usually made during maintenance windows that are specifically scheduled for this purpose

16

Formalizing Change-Control Procedures

Which types of change require authorization? Which changes have to be done during a maintenance window and which changes can be done immediately? What kind of preparation needs to be done before executing a change? What kind of verification needs to be done to confirm that the change was effective? What documentation or communications need to be taken after a successful change? What actions should be taken when a change has unexpected results or causes problems?

17

Establishing Network Documentation Procedures

Network drawings: Physical and logical structure Connection documentation: Patches, connections to providers, and power circuits Equipment lists: All devices, part numbers, serial numbers, installed software versions, software licenses, warranty/service information IP address administration: IP subnets scheme and all IP addresses Configurations: All current device configurations and archives Design documentation: The why did we do that?

18

19

Establishing Effective Communications

Network maintenance is typically a job that is performed by a team Communications is key Who is making changes and when? How does the change affect others? What are the results of tests that were done and what conclusions can be drawn? One team member can be disruptive to the process handled by another team member

20

Standardization

Important that the tasks are performed consistently: Are logging and debug timestamps set to local time or Coordinated Universal Time (UTC)? Should access lists end with explicit "deny all"? In an IP subnet, is the first or the last valid IP address allocated to the local gateway? Different methods can lead to confusion, especially during troubleshooting.

21

Planning for Disaster Recovery

Always consider the possibility of device failure Building redundancy into the network at critical points and eliminating single points of failure. Due to budgetary limitations, it is not always possible to make every single link, component, and device redundant Natural disasters: flood or fire in the server room
22

Planning for Disaster Recovery

To replace a failed device: Replacement hardware The current software version for the device The current configuration for the device The tools to transfer the software and configuration to the device Licenses (if applicable) Knowledge of the procedures to install software, configurations and licenses

23

Network Monitoring and Performance Measurement

Helps you transform your network maintenance process to a less interrupt-driven, more methodical approach Important to choose the variables to be monitored and measured. Interface status interface load CPU load memory usage More sophisticated metrics include delay, jitter, and packet loss

24

Chapter Roadmap
Applying Maintenance Maintenance Processes and Methodologies Procedures Maintenance Models Network Task Identification and Methodologies Network Maintenance Determining Planning Procedures and Scheduling maintenance Tools for Formalizing ChangeMaintenance Models Control Procedures Establishing Network Documentation Procedures Establishing Effective Communication Defining Standardization Planning for Disaster Recovery Network Monitoring and Performance Measurement Network Maintenance Tools, Applications and Resources Fundamental Tools, Applications and Resources CLI, debug, GUI, Backup Servers, Log Servers, Time Servers Configuration and Documentation Tools Logging Services Network Monitoring and Performance Measurement Tools Implementing Backup and Restore 25 Disaster Recovery Tools

Network Maintenance Tools, Applications and Resources

Choose the tools, applications and resources for doing network maintenance in an efficient manner. Commands Software Servers Some of these we will take a brief look at in this chapter and more detail later. Some in later chapters.

26

Basic components of a network maintenance toolkit

The basic components of a network maintenance toolkit are: CLI device management show commands debug commands Embedded Device Manager (EEM) IP SLA commands

27

Basic components of a network maintenance toolkit

The basic components of a network maintenance toolkit are: Graphical User Interface (GUI) based device management (free) Cisco Configuration Professional (CCP) Secure Device Manager (SDM) Cisco Configuration Assistant (CCA) Cisco Network Assistant CiscoWorks (not so free)

28

Basic components of a network maintenance toolkit

The basic components of a network maintenance toolkit are: Backup server TFTP FTP HTTP Secure Copy Protocol (SCP)

29

Basic components of a network maintenance toolkit

The basic components of a network maintenance toolkit are: Log server Syslog

30

Basic components of a network maintenance toolkit

The basic components of a network maintenance toolkit are: Time server Network Time Protocol (NTP)

31

Brief look at NTP

Vital that the clocks of the network devices are properly set and synchronized. Ensures correct timestamps on logging and debug output. Stratum 1 server: a server that is directly connected to an authoritative time source such as a radio or atomic clock (www.time.gov and www.worldtimeserver.com/atomic-clock ). Statum 2 server: A server that synchronizes its clock to a stratum 1 server. Etc.

32

Brief Look at NTP

Configuring NTP section of the Cisco IOS network management configuration guide: http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/gui de/nm_basic_sys_manage_ps6350_TSD_Products_Configurati on_Guide_Chapter.html#wp1001170

Uses the ntp server command Clock is synchronized to a single timeserver with IP address 10.1.220.3. Time zone is Pacific Standard Time (PST), -8 hour offset to Universal Time Coordinated (UTC). Change to daylight savings time on the 2nd Sunday in March at 2:00 am Change back to standard time on the first Sunday in November at 2:00 am. System logging is configured to use the local date and time in the time stamps and to include the time zone in the time stamp. For log entries generated by debugs, the settings are similar, but milliseconds are included in the timestamps for greater accuracy.

33

Configuration and Documentation Tools

Many web-based (online) maintenance tools and resources that can be helpful during the planning and implementation of network maintenance procedures. Configuration and Documentation Tools Dynamic Configuration Tool Cisco Feature Navigator SNMP Object Navigator Cisco Power Calculator

34

Configuration and Documentation Tools

Dynamic Configuration Tool: This tool aids you in creating hardware configurations. It verifies compatibility of the hardware and software you select, and it gives you a complete Bill of Materials (BoM) that lists all the necessary part numbers. https://apps.cisco.com/qtc/config/html/configureHomeGuest.html

35

Configuration and Documentation Tools

Cisco Feature Navigator: This tool allows you to quickly find the right Cisco IOS software release for the features you want to run on your network. http://tools.cisco.com/ITDIT/CFN/

36

Configuration and Documentation Tools

SNMP Object Navigator: The Simple Network Management Protocol (SNMP) Navigator translates SNMP Object Identifiers (OID) into object names. This tool also allows you to download SNMP Management Information Base (MIB) files and to verify the supported MIBs in a particular Cisco IOS Software version. http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en
37

Configuration and Documentation Tools

Cisco Power Calculator: This tool calculates the power supply requirements for a particular Power over Ethernet (PoE) hardware configuration. Needs Cisco CCO account login

38

Configuration and Documentation Tools

Documentation tools: Wiki: A wiki combines easy web-based access with intuitive editing capabilities.

39

Documentation

Issue tracking system: A.k.a. trouble ticket, support ticket, or incident ticket system. Allows issues to be: Logged, tracked, and documented. Better communications and escalation of incidents Allows a team of people to work on the same incidents in an efficient manner. Can build a historical database of problems, their treatments, and the resolutions.

40

Logging Services

Events on networking devices can be logged. Various events Various levels of severity Events are logged to: Console (default) Console display Buffer Server Examples Interfaces up or down Configuration changes Routing protocol adjacencies

41

Logging Services

Logging severity levels on Cisco Systems devices are as follows: (0) Emergencies (1) Alerts (2) Critical (3) Errors (4) Warnings (5) Notifications (6) Informational (7) Debugging By default, all messages from level 0 to 7 are logged to the console

42

Logging Services

Console

You can also adjust the logging severity level of the console. By default, all messages from level 0 to 7 are logged to the console; You can configure the severity level as an optional parameter: logging console level Limits the logging of messages displayed on the console terminal to the specified level and (numerically) lower levels. 43 You can enter the level number or level name.

Logging Services

Buffer logging buffered [buffer-size|level] May or may not be the default By default, messages of all severity levels are logged to buffer. show logging Displays the content of the buffer The buffer is circular, meaning that when the buffer has reached its maximum capacity, the oldest messages will be discarded to allow the logging of new messages.
44

Logging Services

Server logging ip-address command Some IOS version it is logging host By default, only messages of severity level 6 or lower will be logged to the syslog server. This can be changed by entering the logging trap level command.

45

Network Monitoring and Performance Measurement Tools

GUI- and CLI-based device management tools are used to examine individual devices after the problem is noticed. Network monitoring system continuously checks your network devices availability and status: Detect possible problems as soon as they occur Sometimes before they even become apparent to end users Uses protocols such as SNMP and ICMP Cisco IOS Netflow technology can be leveraged to monitor devices and traffic Gray area between network monitoring and performance measurement

46

Motivations for measuring network performance

The three main motivations for measuring network performance are as follows: Capacity planning: Create a baseline network traffic Recognize trends in traffic growth Predict when you need to upgrade links before congestion and performance problems. Diagnosing performance problems: difficult to troubleshoot because hard to quantify and often intermittent in nature. Application X is really slow lately. What is causing the problem? Where is it occurring? SLA compliance: Guaranteeing a level of service to others through an SLA or 47 certain level of service by a provider, need to have a method to measure.

Measuring network performance

Typical statistics gathered include: Packet and byte counters on interfaces Device CPU and memory utilization Round Trip Time (RTT) Jitter Packet loss Analyzed or graphed using products such as: Cisco Internetwork Performance Monitor (IPM) - part of CiscoWorks LAN Management Solution Multi Router Traffic Grapher (MRTG)

48

Implementing Backup and Restore Services

Essential element of any network maintenance toolkit Simplest and most commonly implemented service is TFTP No configuration on network devices More secure protocols such as FTP, SCP, and HTTP or HTTPS For all of these protocols, the credentials can be specified as part of the Uniform Resource Locator (URL) that is used with the copy command.
49

Copy command
R1# copy startup-config ftp://backup:san-fran@10.1.152.1/R1-test.cfg Address or name of remote host [10.1.152.1]? Destination filename [R1-test.cfg]? Writing R1-test.cfg ! 2323 bytes copied in 0.268 secs (8668 bytes/sec)

copy [/erase] source-url destination-url Copy the startup-configuration to a FTP server at 10.1.152.1 Create a file named RO1-test.cfg username backup and password san-fran The username and password are specified by placing the username and password as username:password@ before the server name or IP address in the URL. For SCP, HTTP and HTTPS you would use a similar syntax, replacing the URL prefix ftp:// with scp://, http:// or https://
50

Copy command
R1(config)# ip ftp username backup R1(config)# ip ftp password san-fran R1(config)# exit R1# copy startup-config ftp://10.1.152.1/R1-test.cfg Address or name of remote host [10.1.152.1]? Destination filename [R1-test.cfg]? Writing R1-test.cfg ! 2323 bytes copied in 0.304 secs (7641 bytes/sec)

Specifying the username and password on the command line is somewhat cumbersome and suffers from the fact that the password is displayed in clear text on the screen. username and password can be specified in the configuration Note: Type of encryption to use on the password. A value of 0 disables encryption. A value of 7 indicates proprietary encryption.
51

Archive command
R1(config)# archive R1(config-archive)# path flash:/config-archive/$h-config R1(config-archive)# write-memory R1(config-archive)# time-period 10080

Configuration Replace and Configuration Rollback A feature for the creation of configuration archives, introduced in Cisco IOS Software Release 12.3(7)T. The only mandatory parameter is the base file path. Local or network path Optional variables: $h for the devices hostname in the filename $t to include a time and date stamp in the filename write-memory option - Triggers an archive copy of the running configuration to be created any time the running configuration is copied to NVRAM. time-period minutes option - Each time the time period elapses, a copy of the running configuration will be archived. 52

Archive command

Router# archive config

archive config Manually saves a copy of the current running configuration to the Cisco IOS configuration archive

53

Archive command
R1(config)# archive R1(config-archive)# path flash:/config-archive/$h-config R1(config-archive)# write-memory R1(config-archive)# time-period 10080

R1# show archive There are currently 3 archive configurations saved. The next archive file will be named flash:/config-archive/R1-config-4 Archive # 0 1 2 5 flash:/config-archive/R1-config-1 flash:/config-archive/R1-config-2 flash:/config-archive/R1-config-3 <- Most Recent Name

54

Configure Replace

R1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. R1(config)# hostname TEST TEST(config)# ^Z TEST# configure replace flash:config-archive/R1-config-3 list This will apply all necessary additions and deletions to replace the current running configuration with the contents of the specified configuration file, which is assumed to be a complete configuration, not a partial configuration. Enter Y if you are sure you want to proceed. ? [no]: yes !Pass 1 !List of Commands: no hostname TEST hostname RO1 end Total number of passes: 1 Rollback Done

configure replace target-url [list] [force] [time seconds] [nolock]

configure replace - Allows you to replace the currently running configuration on the router with a saved configuration. Compares the running configuration with the configuration file Creates a list of differences Set of Cisco IOS configuration commands are generated that changes the existing running configuration to the replacement configuration.

55

Disaster Recovery Tools

Successful disaster recovery is dependent on existence of the following: Up to date configuration backups Up to date software backups Up to date hardware inventories Configuration and software provisioning tools As parts of the fundamental network maintenance toolkit, TFTP, FTP, SCP, HTTP and HTTPS server are useful for creating backups of the configuration and operating system of a router or switch.
56

CIS 188 CCNP TSHOOT (Troubleshooting) Chapter 1 Planning Maintenance for Complex Networks
Rick Graziani Cabrillo College graziani@cabrillo.edu