Zabbix Network Monitoring - Second Edition

Table of Contents

Zabbix Network Monitoring Second Edition

Credits

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Getting Started with Zabbix

The first steps in monitoring

Zabbix features and architecture

Installation

Choosing the version and repository

Hardware requirements

Installing from the packages

RHEL/CentOS

EPEL

The Zabbix repository

OpenSUSE

Installing from source

The server and agent

Software requirements

Downloading the source

Compilation

Dash or underscore?

Initial configuration

Creating and populating the database

Starting up

Using systemd

Verifying the service's state

The web frontend

Prerequisites and setting up the environment

Using the web frontend configuration wizard

Step 1 – welcome

Step 2 – PHP prerequisites

Step 3 – database access

Step 4 – Zabbix server details

Step 5 – summary

Step 6 – writing the configuration file

Step 7 – finishing the wizard

Step 8 – logging in

Summary

2. Getting Your First Notification

Exploring the frontend

The user profile

Monitoring quickstart

Creating a host

Creating an item

Introducing simple graphs

Creating triggers

Configuring e-mail parameters

Creating an action

Information flow in Zabbix

Let's create some load

Basic item configuration

Monitoring categories

Availability

Performance

Security

Management

Efficiency

Item types

How items can be monitored

Using global search

Summary

3. Monitoring with Zabbix Agents and Basic Protocols

Using the Zabbix agent

Passive items

Cloning items

Manually querying items

Active items

An active agent with multiple servers

Supported items

Choosing between active and passive items

Item scheduling

Simple checks

Setting up ICMP checks

Tying it all together

Key parameter quoting

Positional parameters for item names

Using mass update

Value mapping

Units

Custom intervals

Flexible intervals

Custom scheduling

Copying items

Summary

4. Monitoring SNMP Devices

Using Net-SNMP

Using SNMPv3 with Net-SNMP

The engine ID

Authentication, encryption, and context

Adding new MIBs

Polling SNMP items in Zabbix

Translating SNMP OIDs

Dynamic indexes

SNMP bulk requests

Receiving SNMP traps

Using embedded Perl code

Filtering values by received data

Filtering values by originating host

Debugging

Handling the temporary file

SNMP Trap Translator

Using a custom script

Filtering the traps

Custom mapping

Database lookups

Summary

5. Managing Hosts, Users, and Permissions

Hosts and host groups

Host inventory

Editing inventory data manually

Populating inventory data automatically

Host maintenance

Creating maintenance periods

Collecting data during maintenance

Not collecting data during maintenance

Maintenance period options

One-time only maintenance

Daily maintenance

Weekly maintenance

Monthly maintenance

Ad-hoc maintenance

Users, user groups, and permissions

Authentication methods

Creating a user

Creating user groups

Permissions and maintenance

Summary

6. Detecting Problems with Triggers

Triggers

The trigger-and-item relationship

Trigger dependencies

Constructing trigger expressions

Preventing trigger flapping

Checking for missing data

Triggers that time out

Triggers with adaptable thresholds

Triggers with a limited period

Relative thresholds or time shift

Verifying system time

Human-readable constants

Customizing trigger display

Trigger severities

Trigger display options

Event details

Event generation and hysteresis

Summary

7. Acting upon Monitored Conditions

Actions

Limiting conditions when alerts are sent

Additional action conditions

Complex conditions

Dependencies and actions

Media limits for users

Sending out notifications

Using macros

Sending recovery messages

Escalating things

Runner analogy

Using scripts as media

Integration with issue management systems

Bugzilla

Computer Associates Unicenter Service Desk Manager

Atlassian JIRA

Remote commands

Global scripts

Configuring global scripts

Reusing global scripts in actions

Summary

8. Simplifying Complex Configurations with Templates

Identifying template candidates

Creating a template

Linking templates to hosts

Handling default templates

Changing the configuration in a template

Macro usage

User macros

Using multiple templates

Unlinking templates from hosts

Using mass update

Nested templates

Summary

9. Visualizing Data with Graphs and Maps

Visualize what?

Individual elements

Graphs

Simple graphs

Ad hoc graphs

Custom graphs

Working time and trigger line

Graph item function

Two y axes

Item sort order

Gradient line and other draw styles

Custom y axis scale

Percentile line

Stacked graphs

Pie graphs

Maps

Creating a map

Linking map elements

Selecting links

Routed and invisible links

Further map customization

Macros in labels

Link labels

Reflecting problems on map elements

Available map elements

Map filtering

Custom icons and background images

Icon mapping

Other global map options

Displaying host group elements

Numbers as icons

Sharing the maps

Summary

10. Visualizing Data with Screens and Slideshows

Screens

Dynamic screens

Additional screen elements

Templated screens

Slide shows

Showing data on a big display

Challenges

Non-interactive display

Information overload

Displaying a specific section automatically

Summary

11. Advanced Item Monitoring

Log file monitoring

Monitoring a single file

Filtering for specific strings

Monitoring rotated files

Alerting on log data

Extracting part of the line

Parsing timestamps

Viewing log data

Reusing data on the server

Calculated items

Quoting in calculated items

Referencing items from multiple hosts

Aggregate items

Aggregating across multiple groups

User parameters

Just getting it to work

Querying data that the Zabbix agent does not support

Flexible user parameters

Level of the details monitored

Environment trap

Things to remember about user parameters

Wrapper scripts

When not to use user parameters

External checks

Finding a certificate expiry time

Determining certificate validity

Sending in the data

Using an agent daemon configuration file

Sending values from a file

Sending timestamped values

SSH and Telnet items

SSH items

Telnet items

Custom modules

Summary

12. Automating Configuration

Low-level discovery

Network interface discovery

Automatically creating calculated items

Automatically creating triggers

Automatically creating graphs

Filtering discovery results

Filesystem discovery

Introducing the LLD JSON format

Including discovered graphs in screens

Custom thresholds with user macro context

CPU discovery

SNMP discovery

Creating custom LLD rules

Re-implementing CPU discovery

Discovering MySQL databases

Global regular expressions

Testing global regexps

Usage in the default templates

Network discovery

Configuring a discovery rule

Viewing the results

Reacting to the discovery results

Uniqueness criteria

Active agent autoregistration

Auto-registration metadata

Summary

13. Monitoring Web Pages

Monitoring a simple web page

Creating a web-monitoring scenario

Other scenarios and step properties

Alerting on web scenarios

Logging in to the Zabbix interface

Step 1: check the first page

Step 2: log in

Step 3: check login

Step 4: log out

Step 5: check logout

Authentication options

Using agent items

Getting the page

Checking page performance

Extracting content from web pages

Summary

14. Monitoring Windows

Installing the Zabbix agent for Windows

Querying performance counters

Using numeric references to performance counters

Using aliases for performance counters

Averaging performance counters over time

Querying WMI

Monitoring Windows services

Checking automatic services

Service discovery

Windows event log monitoring

Summary

15. High-Level Business Service Monitoring

Deciding on the service tree

Setting up IT services

Creating test items and triggers

Configuring IT services

Sending in the data

Viewing reports

Specifying uptime and downtime

Summary

16. Monitoring IPMI Devices

Getting an IPMI device

Preparing for IPMI monitoring

Setting up IPMI items

Creating an IPMI item

Monitoring discrete sensors

Using the bitwise trigger function

Summary

17. Monitoring Java Applications

Setting up the Zabbix Java gateway

Monitoring JMX items

Querying JMX items manually

What to monitor?

Summary

18. Monitoring VMware

Preparing for VMware monitoring

Automatic discovery

Available metrics

The underlying operation

VMware LLD configuration

Host prototypes

Summarizing default template interaction

Server operation and configuration details

Summary

19. Using Proxies to Monitor Remote Locations

Active proxy, passive proxy

Setting up an active proxy

Monitoring a host through a proxy

Proxy benefits

Proxy limitations

Proxy operation

Proxies and availability monitoring

Method 1 – Last access item

Method 2 – Internal proxy buffer item

Method 3 – Custom proxy buffer item

Setting up a passive proxy

Tweaking the proxy configuration

Summary

20. Encrypting Zabbix Traffic

Overview

Backend libraries

Pre-shared key encryption

Certificate-based encryption

Being our own authority

Setting up Zabbix with certificates

Concerns and further reading

Summary

21. Working Closely with Data

Getting raw data

Extracting from the frontend

Querying the database

Using data in a remote site

Diving further into the database

Managing users

Changing existing data

Finding out when

The when in computer language

Finding out what

Performing the change

Using XML import/export for configuration

Exporting the initial configuration

Modifying the configuration

The XML export format

Scripting around the export

Importing modified configuration

Generating hosts

Importing images

Starting with the Zabbix API

Simple operations

Obtaining the API version

Logging in

Enabling and disabling hosts

Creating a host

Deleting a host

Creating a value map

Obtaining history and trends

Issues with the Zabbix API

Using API libraries

Further reading

Summary

22. Zabbix Maintenance

Internal monitoring

New values per second

Zabbix server uptime

Cache usage

Internal process busy rate

Unsupported items and more problems

Counting unsupported items

Reviewing unsupported items

Internal events and unknown triggers

Backing things up

Backing up the database

Restoring from a backup

Separating configuration and data backups

Upgrading Zabbix

General version policy

Long-term support and short-term support

The upgrade process

Minor version upgrade

Upgrading binaries

Upgrading the frontend

Major-level upgrades

Database versioning

Gathering data during the upgrade

The frontend configuration file

Compatibility

Performance considerations

Who did that?

Exploring configuration file parameters

Zabbix agent daemon and common parameters

Zabbix server daemon parameters

Summary

A. Troubleshooting

Chapter introduction

Common issues

Installation

Compilation

Frontend

Backend

Locked out of the frontend

Monitoring

General monitoring

Monitoring with the Zabbix agent

User parameters

SNMP devices

IPMI monitoring

ICMP checks

Problems with simple checks

Problems with zabbix_sender and trapper items

General issues

Triggers

Actions

Discoveries and autoregistration

Troubleshooting Zabbix

The Zabbix log file format

Reloading the configuration cache

Controlling running daemons

Runtime process status

Further debugging

B. Being Part of the Community

Community and support

Chatting on IRC

Using the Zabbix wiki

Using the Zabbix forum

Filing issues on the tracker

Meeting in person

The Zabbix conference

Local communities

Following the development

Getting the source

Daily snapshots

Accessing the version control system

Looking at the changesets

Translating Zabbix

Commercial support options

Index

Zabbix Network Monitoring Second Edition

Zabbix Network Monitoring Second Edition

Copyright © 2016 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: March 2010

Second edition: August 2016

Production reference: 1030816

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78216-128-8

www.packtpub.com

Credits

Author

Rihards Olups

Reviewers

Werner Dijkerman

Volker Fröhlich

Adail Horst

Raymond Kuiper

Julien Recurt

Acquisition Editor

Sonali Vernekar

Content Development Editor

Samantha Gonsalves

Technical Editor

VIshal K. Mewada

Copy Editor

Madhusudan Uchil

Project Coordinator

Devanshi Doshi

Proofreader

Safis Editing

Indexer

Mariammal Chettiyar

Monica Ajmera Mehta

Graphics

Disha Haria

Production Coordinator

Nilesh Mohite

Cover Work

Nilesh Mohite

About the Author

Rihards Olups has over 15 years of experience in information technology, most of it with open source solutions. His foray into Zabbix, one of the leading open source enterprise-class monitoring solutions, was with the first public release back in 2001, which has allowed him to gain considerable knowledge on the subject. Previously employed by a government agency, Rihards was mostly involved in open source software deployment, ranging from server to desktop-grade software, with a big emphasis on Zabbix. Later, he joined Zabbix SIA, the company behind the software that this book is about, which allowed him to gain even more experience with the subject.

While at Zabbix, he helped Zabbix users and customers get the most value out of the monitoring tool and was responsible for delivering Zabbix training sessions that have been described by some participants as extremely practical and quite challenging.

He started working on the very first book on Zabbix, Zabbix 1.8 Network Monitoring, before joining Zabbix, and he finalized that book with even more in-depth details while helping advance Zabbix.

Rihards departed from Zabbix SIA and ended up seeing more of the user side again, including deployments of Zabbix in real-world environments.

Acknowledgments

I would like to thank my family and friends. Not spotlighting here—but if you think whether this includes you, yes, it does.

I would also like to thank all the people I have worked with. It was a great experience at the State Social Insurance Agency, and it was helped by being surrounded by nice people.

A huge thank you goes to everybody at Zabbix. Besides being experts in their own and many other fields, my Zabbix team members were extremely friendly, helpful, and welcoming. And it is true for everybody in the team. You are all great and will achieve a lot.

The Zabbix community deserves an extra mention. I have met a lot of wonderful people through the Zabbix community, and it is such a pleasure to meet them again every year in Riga, at the Zabbix conference. It is the community that takes Zabbix from the pure product level to something more.

And all you great people at Packt Publishing—thank you for dealing with the delays in Zabbix releases, my own schedule delays, and all the other small things. If you are reading this before the book is out, you know you have helped a lot.

Thank you all.

About the Reviewers

Werner Dijkerman is a systems engineer from the Netherlands. He has more than 10 years of experience in IT operations departments of different organizations. He started working with a leading online retailer in the Netherlands and continued in one of the leading software companies for general practitioners. He now works for iWelcome, the only established IDaaS provider in Europe.

With experience as a Windows and Linux admin, he also know his way around Java applications servers such as Tomcat and Jboss, (No)SQL databases, and monitoring systems. He started with Nagios in the early days and has even tried other tools such as Cacti, but Zabbix has been his main monitoring tool for several years now, with him finding ways and writing (Python) scripts to monitor everything it can.

He is also the author of the most popular Puppet module for Zabbix, wdijkerman-zabbix. It can be found at https://forge.puppetlabs.com/wdijkerman/zabbix on Puppet Forge. Besides this Puppet module, he has also created several Ansible roles under his alias dj-wasabi for four Zabbix components, which can be found on https://galaxy.ansible.com/detail#/role/7626.

Adail Horst is a free software evangelist, Zabbix translator, Zabbix documentation translator, and a lecturer on monitoring in various free software events (such as FISL, Latinoware, and Zabbix Conference). He has authored a book on the tool (De a Zabbix, in Portuguese), created Zabbix-Extras, and has experience in large IT environments monitored by Zabbix.

He is also one of the authors of the book A to Zabbix.

Raymond Kuiper is an IT infrastructure specialist with a strong focus on monitoring, networks, and operations. Raymond started using Zabbix 1.1.3 in 2006, when the company he was working for needed a flexible, integrated solution to monitor infrastructure availability and performance. Over the years, Raymond has become a keen supporter of Zabbix and has implemented monitoring with the tool for various organizations. He also likes to hang out on #zabbix on the Freenode IRC network to help out other Zabbix users.

Julien Recurt is an engineer who has multiples roles depending on the occasion. In the beginning, he worked for SUPINFO to enhance a complex and multisite infrastructure and reduce global costs. His big break was working with Ceph at Cloud-Solution, a French startup, to provide low-cost, scalable storage. Now, he works at Waycom, an Internet and web services provider, and manages about 1200 servers and virtual machines using Ansible.

I would like to thank everybody that contributes to open source software and also my coworkers who supported me in this job.

www.PacktPub.com

eBooks, discount offers, and more

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Preface

Imagine you're celebrating the start of the weekend with Friday-night drinks with a few friends. And then suddenly, your phone rings—one of the servers you administer has gone down, and it needs to be back up before tomorrow morning. So you drag yourself back to the office, only to discover that some logfiles have been growing more than usual over the past few weeks and have filled up the hard drive. While this scenario is very simplistic, something similar has happened to most IT workers at one or another point in their careers. To avoid such situations, this book will teach you to monitor your network's hardware, servers, and web performance using Zabbix, an open source system monitoring and reporting solution.

This book will guide you through setting up almost every feature in Zabbix. With step-by-step instructions, anybody should be able to configure monitoring. You could even say that this is an idiot's guide to Zabbix—but only because we all have had moments of not understanding something. In order to not leave you confused, throughout this book, I will try to hold your hand, push you forward, and explain everything in so much detail that you won't get frustrated at all.

What this book covers

Chapter 1, Getting Started with Zabbix, covers Zabbix installation from scratch, including the initial database, server and agent daemons, and web frontend, all running on the same machine. We will also configure the Zabbix web frontend, using PHP to access the database.

Chapter 2, Getting Your First Notification, covers configuring Zabbix using the frontend to set up data gathering, triggering upon specified conditions, and informing us by sending an e-mail for a single data source.

Chapter 3, Monitoring with Zabbix Agents and Basic Protocols, takes you through setting up the most widely used and basic data-gathering methods—Zabbix agents and simple checks, such as ICMP ping and direct TCP service checking.

Chapter 4, Monitoring SNMP Devices, teaches you how to set up the industry-standard monitoring protocol, SNMP. We will explore both polling by Zabbix and receiving SNMP traps, which will allow us to monitor a large variety of devices, including printers, switches, UPSes, and routers.

Chapter 5, Managing Hosts, Users, and Permissions, covers hosts, users, and permissions, including host and user group functionality and their impact on permissions. We will also explore related topics, including host and host group maintenance.

Chapter 6, Detecting Problems with Triggers, looks at ways to define which conditions are noteworthy by crafting expressions to detect values outside of the normal range, and other cases.

Chapter 7, Acting upon Monitored Conditions, helps you figure out how to react to triggers firing by sending an e-mail, launching an external script, opening a report in a separate bug tracker, or even restarting a faulty service. We will also learn to configure escalations in Zabbix and figure out how hysteresis works.

Chapter 8, Simplifying Complex Configuration with Templates, tells us how we did it all wrong before and teaches us to improve our configuration using templates that allow us to apply uniform configuration to a bunch of hosts while customizing thresholds per host with user macros. We'll also explore template nesting, which allows creating very flexible configuration in a large and mixed environment.

Chapter 9, Visualizing the Data with Graphs and Maps, explains how to create visual elements to display gathered data, including several types of graphs and interactive network maps.

Chapter 10, Visualizing the Data with Screens and Slideshows, helps you configure screens that collect various types of elements to display and slideshows that allow cycling through several screens in an automated fashion.

Chapter 11, Advanced Item Monitoring, explores more advanced ways to gather information using external, calculated, aggregate, and custom item types to retrieve any information.

Chapter 12, Automating Configuration, teaches us how to discover filesystems, network interfaces, and other entities on hosts with low-level discovery. On a higher level, we will explore the ability to automatically manage hosts via network discovery and active agent autoregistration.

Chapter 13, Monitoring Web Pages, takes us through monitoring the accessibility, performance, and availability of web pages using the built-in web scenario feature of Zabbix.

Chapter 14, Monitoring Windows, covers Windows monitoring by installing Zabbix agents, using performance counters, and retrieving data over WMI.

Chapter 15, High Level Business Service Monitoring, demonstrates the IT services feature, which allows aligning monitored services in a tree and calculating SLA conformity. We will fake some data to see how this feature actually looks.

Chapter 16, Monitoring IPMI Devices, teaches us how to set up yet another industry-standard protocol, IPMI. Monitoring both analog and discrete IPMI sensors will be covered.

Chapter 17, Monitoring Java Applications, a separate daemon called Zabbix Java gateway is set up and used to retrieve basic information over JMX.

Chapter 18, Monitoring VMware, covers the built-in VMware monitoring support. We will explore the ability to automatically discover and monitor hypervisors and virtual machines.

Chapter 19, Using Proxies to Monitor Remote Locations, explores the usage of passive and active Zabbix proxies that collect the data on behalf of the Zabbix server and then transmit it back to the server, which helps with remote locations that can't be accessed directly because of firewall concerns and also reduces the load on the Zabbix server.

Chapter 20, Encrypting Zabbix Traffic, looks into one of the major new features in Zabbix 3.0: the ability to encrypt traffic between the Zabbix server, proxies, agents, and the get and sender utilities.

Chapter 21, Working Closely with Data, helps us figure out some details about how data is stored in the Zabbix database and how we can interact with it directly as well as using Zabbix's native XML import and export functionality to more easily create large amounts of configuration. We will also get started with the Zabbix API for more advanced automation needs.

Chapter 22, Zabbix Maintenance, teaches us about the Zabbix upgrade procedure, how different components of various versions can interact, and what database patching between versions involves. We will also explore the internal monitoring options available, discuss backup strategies, and talk in detail about all the configuration parameters Zabbix daemons have.

Appendix A, Troubleshooting, covers common pitfalls with installation, connectivity, configuration, and other areas.

Appendix B, Being Part of the Community, tells us how we are not alone and there's a community around the Zabbix monitoring solution, which we can reach via IRC, forums, and the community wiki.

What you need for this book

You will need at least one Linux system, which could be a virtual machine as well. Depending on specific features discussed, you might also benefit from the following:

Access to an SMTP (e-mail) server

More Linux systems

A device with SNMP support