Escolar Documentos
Profissional Documentos
Cultura Documentos
example)
Introduction
SystemTap is a powerful Linux tool that allows collection of data from both the Linux kernel and user-space applications. SystemTap
includes an extensive library of predefined probes and functions for the kernel (tapsets) and a convenient scripting language to do on-
the-fly data reduction. SystemTap's probing capabilities can be extended to user-space applications.
Debuginfo-based instrumentation
There are two basic ways for user-space probing. One can rely on the basic symbolic probing
style, in which one specifies source code level "co-ordinates", and accesses variables available in context. This style does not require
any changes to the target application binaries, merely preserving their debugging data. (In the usual systemtap way, one can package a
collection of salient probe points into a tapset script that gives abstract names for given functions/statements.)
Compiled-in instrumentation
Another way is to instrument the application itself with embedded markers, which expose selected names and values only to systemtap
scripts. These permit abstraction (easier usage, by exposing only a small number of salient probe points) and sometimes assist reliable
provision of local variable values. This makes it unnecessary to maintain a tapset script, and instead involves adding calls to macros
from <sys/sdt.h> into your program. These calls are source compatible with dtrace on other platforms, so the same source code
compiled on a system that doesn't have systemtap available, but does have a dtrace implementation will be able to discover the same
probe locations. The GNU Debugger (GDB) has also been extended to allow users to set breakpoints on such probes if available.
There are already some applications in Fedora 12 such as java-1.6.0-openjdk and postgresql that support probing by SystemTap. This
simple example uses a heapsort program to show how SystemTap support can added to nearly any user-space application and include
that support in the RPM.
The user-space probes allow you investigate the operation of the program without the need to recompile the program or restart the
program. It also make it very easy to create simple scripts to look at interesting characteristics of program behavior, for example how
long did it take to do the average postgresql query, when did any of the Java virtual machines start doing garbage collection, and how
long did java garbage collection take.
To implement the userspace application probes you will need to have a kernel that supports utrace (Fedora 10 and later kernels) and the
following SystemTap rpms installed on the computer:
systemtap
systemtap-runtime
systemtap-sdt-devel
The process of adding and using the user-space application probes can be broken down into the following steps:
An extremely simple heapsort sort written in C++ is used as a starting point for this example. It reads in an arbitrary number of
integers from stdin terminated by a ctrl-d, sorts the integers using a heapsort algorithm, and then outputs the sorted integers. This
README describes the changes made to add the SystemTap probes to the code. All of this is packaged in heapsort-0.5-1.src.rpm.
#include <sys/sdt.h>
DTRACE_PROBE(provider, name)
provider is an arbitrary symbol identifying your application or subsystem, and name is an arbitrary symbol identifying your probe
point. Markers can include a fixed number of arguments that are either integer or pointer values. Below is an example of a marker with
four arguments:
Systemtap can attach to these markers using this syntax. You may use wildcards or fully spell out the provider and marker names in the
DTRACE_PROBE. Within the probe handlers, arguments may be accessed with $arg1 for values, user_string($arg2) for dereferencing
pointers, or pretty-printed with $$parms. The values $$name and $$provider are also available to match up the current probe pointer.
Process names may also be abbreviated.
Configuration
The first step is to add tests to the configure.ac to enable and disable the SystemTap support. The modification to the application code
should not prevent the code from compiling in environments that do not have the SystemTap user-space support. The following lines in
the configure.ac file control whether SystemTap support is enabled:
AC_CHECK_PROGS(DTRACE, dtrace)
if test -z "$DTRACE"; then
AC_MSG_ERROR([dtrace not found])
fi
AC_CHECK_HEADER([sys/sdt.h], [SDT_H_FOUND='yes'],
[SDT_H_FOUND='no';
AC_MSG_ERROR([systemtap support needs sys/sdt.h header])])
If those dtrace script and sys/sdt.h header are found, then HAVE_SYSTEMTAP is de fined in the config.h with:
SystemTap has library files call tapsets. The configuration needs to determine where to install those with the following in the
configure.ac:
AC_ARG_WITH([tapset-install-dir],
[AS_HELP_STRING([--with-tapset-install-dir],
[The absolute path where the tapset dir will be installed])],
[if test "x${withval}" = x; then
ABS_TAPSET_DIR="\$(datadir)/systemtap/tapset"
else
ABS_TAPSET_DIR="${withval}"
fi], [ABS_TAPSET_DIR="\$(datadir)/systemtap/tapset"])
AC_SUBST(ABS_TAPSET_DIR)
Tapset Skeleton
The tapset/heapsort.stp is a file that will make it easier for people to use the SystemTap probe points in the code. It hides some of the
details about the probe point from the user. The code for the tapset will be placed in tapset/heapsort.stp. Initially, tapset/heapsort.stp
can be empty. There is also a very simple Makefile.am in the tapset directory to indicate how to install and remove the tapset file. The
Makefile.am in the top level directory will need to indicate that there is a subdirectory with the following lines:
SUBDIRS = tapset
DIST_SUBDIRS = $(SUBDIRS)
Probe Point Declaration
The next step is to declare the probes points in the probes.d file and the arguments that they take. The probes.d contents listed below
will be processed to generate the needed include file (probes.h) and stub object file (probes.o):
provider heapsort {
probe input_start();
probe input_done(int); /* (int number of items) */
probe buffer_resize_start();
probe buffer_resize_done();
probe output_start(int); /* (int number of items) */
probe output_done();
probe heap_place(int, int); /* (int position, int value) */
probe heap_build_start();
probe heap_build_done();
};
Some minor changes are needed in the Makefile.am. First, need to add probes.d and a very simple wrapper trace.h to SOURCES list
and indicate the probes.h is a generated file:
Also need some rules to generate the probes.h and probes.o as needed in the Makefile.am:
if ENABLE_SYSTEMTAP
probes.h: probes.d
$(DTRACE) -C -h -s $< -o $@
probes.o: probes.d
$(DTRACE) -C -G -s $< -o $@
heapsort_LDADD += probes.o
endif
What is in STAP_PROBE2() is not important. The important thing is macros are now available to instrument the application code.
#include "probes.h"
The is implemented in the trace.h, a very short include file that conditionally includes the probes.h and has a TRACE macro to
conditionally use the tracepoints:
#include "config.h"
#ifdef HAVE_SYSTEMTAP
// include the generated probes header and put markers in code
#include "probes.h"
#define TRACE(probe) probe
#define TRACE_ENABLED(probe) probe ## _ENABLED()
#else
// Wrap the probe to allow it to be removed when no systemtap available
#define TRACE(probe)
#define TRACE_ENABLED(probe) (0)
#endif
The macros can be placed in any place that normally executable code is placed. They will be inactive until they are used by
SystemTap. The arguments can be used to relay useful state information to SystemTap. For the HEAPSORT_HEAP_PLACE the
location in the heap and the value being inserted into the heap are available to SystemTap.
process("heapsort").mark("heap_place")
The raw probes are not particularly user-friendly. The following section describes how to abstract the interface and hide those details
with a tapset.
Adding a Tapset
Tapsets provide an ABI that hides the details of the probe from the user. The tapsets are typically placed in
/usr/share/systemtap/tapsets. The tapsets consists of aliases and local variables for the probes. The tapset can also definite SystemTap
functions and local variable that make it easier to use the probes.
The following is an example probe alias for the HEAPSORT_HEAP_PLACE() probe used in the source code:
SystemTap Flag
Build dependencies
Configure -Files to install
SystemTap Flag
As with the original source code, it should be possible to build the RPMs with or without the SystemTap support enabled. The variable
sdt in heapsort.spec controls whether SystemTap support is enabled. The following line indicates that the SystemTap support is
enabled by default:
%{!?sdt:%define sdt 1}
The SystemTap support can be turned off with the --define on the following rpmbuild line:
The variable sdt will be used in the rest of the spec file to control whether the package is built with SystemTap support.
Build Dependencies
When the package is built with SystemTap support an additional BuildRequires is needed to supply the tools to generate the probes.h
header and probes.o stub files:
%if %sdt
BuildRequires: systemtap-sdt-devel
%endif
Configure
In the %build section of the heapsort.spec file the configure is extended to:
%configure \
%if %sdt
--enable-systemtap \
--with-tapset-install-dir=%tapsetdir \
%endif
make %{?_smp_mflags}
Files to Install
To make life easier for users the tapset file should be installed. The following is an addition to the %file section of the spec file:
%if %sdt
%{tapsetdir}/*.stp
%endif
probe heapsort* {
printf("%s\n", probestr);
}
This could be run with a specific instance of heapsort and generate something like the following output:
The heap_time_phases.stp tracks statistics about the amount of time spent in the input, heap_build, and output phases for all runs of
/usr/bin/heapsort.
Conclusion
This is an simple example of how incorporate SystemTap probes into an application. For more information check out the SystemTap
webpage:
http://sourceware.org/systemtap/
Also feel free to send email to the mailing list or join the IRC channel to discuss issues with systemtap: