Performance Evaluation of XML-based Network Management

Mi-Jung Choi1, So-Jung Lee1, Dong-Hyun Kim1, James W. Hong1 and Raouf Boutaba2
1

Dept. of Computer Science and Engineering, POSTECH, Korea

2

School of Computer Science, University of Waterloo, Canada

{mjchoi, annie, dhkim03, jwkhong}@postech.ac.kr, rboutaba@waterloo.ca

Abstract
Recently, researchers and developers have been investigating the application of XML technologies to
network and systems management. Although the performance of XML-based network management
systems (XNMS) has been identified as an important issue, it has been difficult to obtain such information
due to a lack of implementations. In this paper, we provide a performance evaluation of XNMS. Our
performance evaluation is based on an XML-based manager managing network devices equipped with
SNMP agents through one or more XML/SNMP gateways. We show the performance evaluation of XNMS
from the perspective of network traffic, response time, and computing resources, namely CPU and memory
usage.

Keywords: Performance Evaluation, XML-based Network Management, XML, SNMP, XML/SNMP

Gateway

1.

Introduction
To benefit from the advantages of XML [1], research on the use of XML in the network management area is

actively in progress [2, 3, 15, 16, 17, 18]. The use of XML in network management offers many advantages. XML
provides powerful modeling features for hierarchical management information in network management, and
XML-based network management (XNM) allows for the reliable transfer of a large amount of data over HTTP.
Also, XNM can be easily implemented using standard APIs and freely available software. This advantageous
applicability of XML technologies to network management alleviates many of the known problems of traditional
SNMP-based management [3].
During the past few years, some research and development have been done in the area of XML-based network
management. Much work has been done in the area of accommodating widely deployed SNMP devices,
specifically in the specification and interaction translation for the gateways between XML-based managers and
SNMP-based devices [4, 5, 6, 7]. Some work focused on extending Web-based management to exploit the
advantages of XML technologies [8, 9, 10, 11] while others focused specifically on the configuration management
of network devices and systems [12, 13, 14, 17, 18]. Also, some work has been done to provide an architectural

framework for XML-based network management [15, 16].

In our previous work [15], we proposed an architectural framework for the development of XML-based
network management systems (XNMS). The framework provides a detailed architecture for an XML-based
manager, agent and XML/SNMP gateway. It also provides methods for interacting between a manager and an agent.
We have validated this architecture by implementing it and using it for managing network devices and systems [6, 7,
15, 16, 17, 18]. We evaluated computing resources such as the CPU and memory usage of an XML-based agent.
Also, we evaluated the performance such as network traffic volume and response time of an XML-based agent. We
compared the performance between the XML-based agent and SNMP agent. Then we showed that our XML-based
agent is as efficient as an SNMP agent in terms of computing resources and processing time [15].
Typically, a large amount of network bandwidth is required to transfer XML data due to its text-based nature
and the use of the HTTP/TCP protocols. An important issue is the ability of the XML-based manager and
XML/SNMP gateway to process information received from multiple managed devices. The development
experience of XNMS is not sufficient and also deficient for wide application to real network management.
Therefore, performance evaluation results for XNM have not been provided until now.
Most IP network devices currently deployed are basically equipped with SNMP agents, and the XML/SNMP
gateway approach is suitable to this situation. The XML-based manager manages the current SNMP-enabled
network devices through this XML/SNMP gateway. The XML/SNMP gateway translates and relays the messages
between an XML-based manager and an SNMP agent. We have applied our XML-based manager and XML/SNMP
gateway to the POSTECH campus network and evaluated the computing resources of the manager and gateway,
and network performance such as traffic volume and response time. We increased the number of SNMP agents from
1 to 100. Also, we increased the number of the XML/SNMP gateways from 1 to 3 and then measured the above
performance parameters. From these performance evaluation results, we show that XNMS does not create much
performance overhead from the perspectives of network traffic, processing time and computing resources. The goal
of our work is to validate the efficiency and scalability of XNMS.
The organization of this paper is as follows. Section 2 describes the testing environment of performance
evaluation in the POSTECH campus and the measurement methods. In Section 3, we present the performance
evaluation results of the XML-based manager and XML/SNMP gateway. In Section 4, we analyze and summarize
the performance test results. Finally, we conclude this paper and discuss some directions for future research in
Section 5.

2.

Measurement Environment and Methods

As depicted in Figure 1, we applied our XML-based manager and XML/SNMP gateway to manage various

network devices deployed in the POSTECH campus network. The XML-based manager and XML/SNMP gateway
are connected by a 100Mbps LAN. However, XML/SNMP gateways and SNMP agents are connected by not only

100Mbps LAN but also a gigabit Ethernet backbone network. Both XML-based manager and the XML/SNMP
gateway run on Linux servers with a Pentium-III 800 MHz CPU and 256M RAM. As a part of our tests to evaluate
the impact of the change in computing resources on the performance of a gateway server, we upgraded our server
running the XML/SNMP gateway to a Pentium-IV 2.8GHz CPU and 512M RAM.
POSTECH Gigabit Ethernet Backbone Network

100Mbps

100Mbps

XML/SNMP Gateway
1Gbps

XML/SNMP Gateway
100Mbps
XML-based
Manager
100Mbps
Linux sever with
Pentium-III 800 MHz CPU
XML/SNMP Gateway
256 MB RAM
Linux severs with
Pentium-III 800 MHz CPU
256 MB RAM
or Pentium-IV 2.8 GHz CPU
512 MB RAM

1Gbps

Switching
hubs

Internet
routers

100Mbps

100Mbps

Backbone
switches 1Gbps

1Gbps

SNMP Agents

Various Network Devices

in POSTECH campus

Figure 1. Performance Testbed

SNMP agents are running in various network devices deployed in the POSTECH campus network. POSTECH
has a gigabit Ethernet backbone network, which is composed of eleven gigabit backbone switches (Catalyst
6500/5500 series) and hundreds of 100 Mbps client switching hubs that are deployed inside the buildings, and two
Cisco 7513 and 7401 Internet routers are connected to the backbone network. We selected one hundred devices
among these Cisco backbone routers and switches that are equipped with SNMP agents and then we carried out
XNMS performance tests.
First, we measured the performance of XNMS using one XML/SNMP gateway. The solid arrow in Figure 1
illustrates this test scenario. To evaluate the XNMS scalability, we increased the number of SNMP agents from 1 to
100. We measured network traffic between the XML-based manager and SNMP agents through the XML/SNMP
gateway in relation to the number of SNMP agents. We separately measured the network traffic volume between the
XML-based manager and XML/SNMP gateway and the network traffic volume between the XML/SNMP gateway
and SNMP agents. The XML-based manager sends an HTTP request message to the gateway to retrieve
management information from multiple SNMP agents. The XML/SNMP gateway processes the HTTP request
message and sends multiple SNMP request messages to each SNMP agent. Since most of the currently deployed
network devices in the real world only support SNMPv1 agents, the gateway simultaneously sends multiple
GetNext requests to the agent for retrieving the group data using a thread mechanism.
Next, we measured the response times as a function of the number of SNMP agents. Also, we separately
measured the latency between the XML-based manager and XML/SNMP gateway, and the latency between the

XML/SNMP gateway and SNMP agent. We divide the latency between the gateway and SNMP agent into two
parts: 1) processing time of translating the message in the gateway, and 2) communication time between the
gateway and agent involving all processing required by the SNMP protocol stack in the gateway and the agent. The
purpose of this division is to determine whether the processing time or the communication time is taking longer in
the gateway. Also, we measured the response time and network traffic volume in relation to the XPath expression.
Various XPath expressions can be applied in the gateway. The complexity of XPath expressions impacts the
network traffic volume and processing time.
Finally, we measured the processing overhead of the XML/SNMP gateway in terms of CPU load and run-time
memory usage. To compare the processing time of the gateway based on the change in the computing resources, we
upgraded the CPU and memory of the gateway. Then, we performed the same performance tests and compared the
response time and processing overhead between the lower-performance gateway (Pentium-III 800MHz, 256MB)
and the higher-performance gateway (Pentium-IV 2.8GHz, 512MB).
In a second test scenario, we increased the number of XML/SNMP gateway from 1 to 3. We measured the
performance of XNMS in relation to the number of XML/SNMP gateways. The dotted arrows in Figure 1 illustrate
this test scenario. The performance evaluation metrics are the same as those used in the single XML/SNMP
gateway scenario. First, we measured network traffic between the XML-based manager and SNMP agents through
multiple XML/SNMP gateways in relation to the number of SNMP agents. We separately measured the network
traffic volume between the XML-based manager and multiple XML/SNMP gateways and the network traffic
volume between multiple XML/SNMP gateways and SNMP agents. The traffic volume between multiple
XML/SNMP gateways and SNMP agents represents the sum of the network traffic that each gateway
communicates with SNMP agents.
Next, we measured the response time in relation to the number of SNMP agents. Also, we separately measured
the latency between the XML-based manager and multiple XML/SNMP gateways and the latency between multiple
XML/SNMP gateways and SNMP agents. The latency between multiple XML/SNMP gateways and SNMP agents
represents the entire time taken for all XML/SNMP gateways to finish communicating with SNMP agents. We
divided the latency calculation into two parts in the same way as done in the previous scenario. Finally, we
measured the processing overhead of XML/SNMP gateways, such as CPU load and run-time memory usage. We
measured the values for each XML/SNMP gateway and then averaged them. In this test scenario with multiple
XML/SNMP gateways, the later two gateways handle same load supported by one single gateway in the first test
scenario (i.e., 100 agents). The load is evenly distributed among the gateways. Assume that we are testing the
performance of XNMS using two XML/SNMP gateways. If one gateway receives information from 100 SNMP
agents, we divide the number of SNMP agents by two. Therefore, each gateway interacts with 50 SNMP agents.
Finally, we compare the performance evaluation results in relation to the number of XML/SNMP gateways. We
identify whether the performance is improved or not. If it has improved, we also identify how much it has improved

by increasing the number of XML/SNMP gateways.

The traffic pattern of POSTECH campus network is constant; there is always a large volume of traffic except
for the period from 4 a.m. to 8 a.m. Since we were not sure of the effect of network traffic volume to the
measurement of latency, we measured the latency at different times and calculated an average from measured
values. Also, for statistical significance, we conducted 10 measurements for each test and averaged them.

3.

Performance Evaluation
First, we evaluate the network traffic volume at the IP layer. Table 1 shows the network traffic between the

XML-based manager and XML/SNMP gateway and between the XML/SNMP gateway and SNMP agents to
retrieve eight objects in the system group of MIB-II (system description, system object ID, system up-time, etc.). If
the manager retrieves the interface group information from the SNMP agent, the management information volume
varies because of the difference in the number of network interfaces of each network devices. However, the system
group information is almost the same for all network devices. Therefore, we chose the system group information for
our XNMS performance tests.
The manager sends an HTTP request message then the gateway processes the request message and sends
multiple SNMP request messages to each SNMP agent [15]. The network traffic between the manager and gateway
is the HTTP message including TCP control information and connection setup. The HTTP request message consists
of gateways information including SNMP information (i.e., gateways IP, SNMP community and version) and
network devices information (i.e., devices IP, XPath expression) [15]. In the case of retrieving 1 object or 8
objects from one device, the HTTP messages are almost the same. Merely, XPath expressions indicating retrieved
data are different. In the case of only 1 object in the system group, the XPath expression is //sysDescr or
//sysObjectID, etc. In the case of 8 objects, the XPath is //system. So, the network traffic generated between the
manager and gateway while retrieving 1 object is slightly greater than that while retrieving 8 objects. Both
XML/HTTP traffic and SNMP traffic are divided into the request and response messages for the Get operation. The
sum of the XML/HTTP traffic and SNMP traffic is equal to the total network traffic volume for the XML-based
manager to retrieve management information from the SNMP agents through the gateway.

# of G atew ays
# of Agents
(# of M IB O bjects)

M anager
G atew ay

2
G atew ay
Agent

Req.

R esp.

1080

691

R eq.

R esp.

R eq.

1 (8)

1078

1106

656

744

10 (80)

2022

6688

6560

7658

2264

6854

6560

7658

20 (160)

2965

13670

13120

15305

3208

13986

13120

30 (240)

3895

20356

19680

22615

4144

20605

19680

40 (320)

4817

27148

26240

30936

5073

27546

50 (400)

5798

33457

32800

38636

6051

60 (480)

6754

39973

39360

45951

70 (560)

7795

45786

45200

80 (640)

8802

52354

90 (720)

9895
10936

M anager
G atew ay

Resp.

R eq.

G atew ay
Agent

Resp.

Req.

Resp.

2502

8687

6560

7658

15305

3462

14364

13120

15305

22615

4401

21018

19680

22615

26240

30936

5315

27996

26240

30936

33998

32800

38636

6298

34482

32800

38636

7012

40575

39360

45951

7268

41094

39360

45951

54214

8066

56345

45200

54214

8325

56598

45200

54214

52480

62305

9068

53012

52480

62305

9316

53624

52480

62305

58248

59040

70117

9150

58938

59040

70117

9435

59624

59040

70117

64857

65600

78148

11196

65553

65600

78148

11455

66268

65600

78148

82

R esp.

G atew ay
Agent

1 (1)

100 (800)

Req.

M anager
G atew ay

95

Unit: bytes

Table 1. Network Traffic of Get Operation for the MIB-II System Group
Manager Gateway (Req.) : 1 Gateway

90000

Manager Gateway (Resp.) : 1 Gateway

80000

Gateway Agent (Req.) : 1, 2 or 3 Gateways

Gateway Agent (Resp.) : 1, 2 or 3 Gateways

Network Traffic (Bytes)

70000
60000
50000
40000
30000
20000
10000
0
0

20

40

60

80

100

Number of SNMP Agents

Figure 2. Network Traffic of Get Operation for System Group

Figure 2 is the graph showing the network traffic in Table 1. The curves show the data using one gateway. The
dotted line with squares shows the request network traffic from the XML-based manager to one gateway. The solid
line with squares is the response traffic to the corresponding request. The network traffic of the request and
response message from the manager to gateway slightly increases as one more gateway is added. The dotted line
with triangles represents the request network traffic from the gateway to the SNMP agents for three different cases,
when the gateway is one, two, or three. As illustrated in Figure 2, the network traffic volume between the

XML-based manager and gateway increases linearly in relation to the number of SNMP agents, which is the
number of MIB objects. The network traffic volume between the manager and gateway increases to almost 500
bytes as the number of gateways increases shown in Table 1. However, the total traffic volume between the
gateways and SNMP agents remains constant no matter how many gateways are added. The sum of request and
response network traffic volume between the XML-based manager and gateway is only 50~60% of that between
the gateway and SNMP agents.
Table 2 shows the response time of the Get operation for the system group in relation to the number of SNMP
agents. We divide the latency between the gateway and SNMP agents into two parts: processing time of XML
message and translation to SNMP message in the gateway and the communication time between SNMP stacks in
the gateway and agents. In the case of retrieving only one SNMP agent, the processing time of the manager is
almost same as that of the gateway. However, as the number of SNMP agents increases, most of the processing
overhead increases in the gateway. The time taken by the gateway to process interaction translation and to
communicate with SNMP agents takes about 80~90% of the total time for the XML-based manager to retrieve
system group information from the SNMP agents through one XML/SNMP gateway. However, the processing time
of the XML-based manager does not increase as much in proportion of the number of SNMP agents. In the gateway,
the processing of XML message and interaction translation occupies more processing time than that of the SNMP
communication does.
1

# of Gateways
# of Agents
(# of MIB
Objects)

1 (1)

Manager
Gateway
512

Gateway
Agent
Proc.
404

Comm.

Manager
Gateway

Gateway
Agent

Gateway
Agent

Proc.

Comm.

Manager
Gateway

Proc.

Comm.

1 (8)

524

832

38

10 (80)

558

1058

236

1296

957

175

1405

912

153

20 (160)

623

1278

361

1364

1103

256

1468

1084

189

30 (240)

659

1463

597

1432

1253

329

1532

1153

262

40 (320)

688

1615

698

1512

1296

384

1625

1204

304

50 (400)

715

1820

814

1527

1386

518

1649

1263

351

60 (480)

744

1996

1285

1604

1492

621

1726

1305

394

70 (560)

986

2230

2942

2095

1591

674

2203

1363

469

80 (640)

1084

2586

3296

2287

1673

712

2398

1435

546

90 (720)

1113

5826

6850

2364

1765

759

2485

1502

625

100 (800)

1148

9053

10154

2438

1876

826

2542

1564

702

Unit: ms

Table 2. Response Time of Get Operation for System Group

Response Time (ms)

22000

Manager Agent (1 Gateway)

20000

Gateway Agent (1 Gateway)

18000

Gateway Agent (2 Gateways)

16000

Manager Agent (3 Gateways)

Manager Agent (2 Gateways)

Gateway Agent (3 Gateways)

14000
12000
10000
8000
6000
4000
2000
0
0

20

40

60

80

100

Number of SNMP Agents

Figure 3. Response Time of Get Operation for System Group

Figure 3 shows the response times in Table 2. The dotted lines illustrate the total response times between the
XML-based manager and SNMP agents through gateways, and the solid lines are the response times between the
gateway and SNMP agents. We summarized the processing time and communication time between the gateway and
agents and drew them as the solid lines. More specifically, the dotted line with triangles represents the total
response time two gateways take to retrieve the system group information from the SNMP agents. Similarly, the
dotted line with circles represents the total response time three gateways take to obtain the system group
information from the SNMP agents. The response time between the XML-based manager and gateway is omitted in
Figure 3 because it is insignificant and constant as the number of gateway increases. However, the response time
between the gateway and SNMP agents decreases as the number of gateway increases. The solid line with triangles
shows the response time between the gateway and SNMP agent when using two gateways and is almost half of the
response time between the gateway and SNMP agents when using one gateway. Similarly, the solid line with circles
shows the response time between the gateway and SNMP agent using three gateways. It is one-third of the response
time between the gateway and SNMP agents using one gateway. Because of the decrease in the response time
between the gateway and SNMP agents, the total response time between the manager and SNMP agents also
decreases as the number of gateway increases.
As shown in Figure 3, in the case of one gateway, the total response time increases linearly until the
XML-based manager retrieves information from 80 SNMP agents; that is, 640 MIB objects. However, if the
number of SNMP agents increases beyond 80, the total response time starts to increase rapidly. On the other hand,
in the case when we apply two or three gateways, the total response time continues to increase linearly until the

number of SNMP agents reaches 100. Therefore, we conclude that 80 SNMP agents are suitable for our XNMS
using a single gateway (Pentium-III 800MHz CPU and 256MB RAM).
We have also evaluated the CPU usage and memory usage of the XML-based manager and the XML/SNMP
gateway in relation to the number of SNMP agents. As shown in Table 3 and Table 4, memory usage increases
slowly in relation to the number of SNMP agents, whereas CPU usage increases almost linearly. Figure 4 and
Figure 5 show the graph of the CPU usage and memory usage in Table 3 and Table 4, respectively. To operate the
gateway, CPU is more utilized than memory. From the perspective of the gateway scalability, as shown in Figure 4,
the memory usage of the XML-based manager increases slowly in relation to the number of gateways. Also, the
CPU usage of the XML-based manager increases as the number of gateways increases because more processing is
required to distribute the requests to the XML/SNMP gateways and to merge the retrieved data from these
gateways.
# of
Gateways
1
2
3

# of SNMP Agents
(# of MIB Objects)
CPU Usage
Memory Usage
CPU Usage
Memory Usage
CPU Usage
Memory Usage

1
(8)

10
(80)

20
(160)

30
(240)

40
(320)

50
(400)

60
(480)

70
(560)

80
(640)

90
(720)

100
(800)

5.1

5.9

6.9

8.7

11.8

16.7

21.7

23.8

25.6

27.4

30.4

10.2

12.5

14.2

14.6

14.7

15.1

15.4

16.0

16.3

17.1

17.7

5.1

7.1

8.5

10.8

14.0

19.0

24.4

26.4

28.0

30.0

32.5

10.2

13.4

15.1

15.5

15.8

16.1

16.6

17.0

17.2

18.2

18.9

5.1

8.1

10.2

12.6

16.0

21.3

27.0

29.0

30.3

33.3

35.7

10.2

14.2

16.2

16.3

16.7

17.3

17.6

18.2

18.4

20.1

20.4

Unit: %

Table 3. Resource Usage of XML-based Manager

CPU Usage (1 Gateway)
Memory Usage (1 Gateway)
CPU Usage (2 Gateways)
Memory Usage (2 Gateways)

100
90

Resource Utility (%)

80

CPU Usage (3 Gateways)

Memory Usage (3 Gateways)

70
60
50
40
30
20
10
0
0

20

40

60

80

100

Number of SNMP Agents

Figure 4. Resource Usage of XML-based Manager

However, as shown in Figure 5, the memory usage of the gateway decreases slowly in relation to the number of

gateways. Also, the CPU usage of the gateway decreases as the number of gateways increases. Obviously, this is
due to the fact that each gateway interacts with a smaller number of SNMP agents as the number of gateways
increases. As illustrated in Figure 5, the dotted line with squares shows that the CPU usage of one gateway
increases linearly until the gateway handles 80 SNMP agents (640 MIB objects). However, when the number of
SNMP agents is above 80, the CPU usage of the gateway increases rapidly. This explains why the response time
between the gateway and SNMP agents increases when the gateway handles more than 80 SNMP agents.
# of
Gateways

# of SNMP Agents
(# of MIB Objects)

1
(8)

CPU Usage

Memory Usage
CPU Usage

Memory Usage
CPU Usage

Memory Usage

10
(80)

20
(160)

30
(240)

40
(320)

50
(400)

60
(480)

70
(560)

80
(640)

90
(720)

100
(800)
79.5

3.1

8.2

10.3

14.6

18

23.1

28.5

33.6

39.5

60.3

12.1

12.2

12.3

12.4

14.4

14.8

15.4

16.8

17.3

18.8

19.7

3.1

5.6

8.4

9.6

10.3

12.1

14.2

16.6

19

22.4

26.5

12.1

12.1

12.3

12.3

12.4

12.4

12.4

13.1

14.1

14.7

15.1

3.1

5.9

6.4

7.2

7.5

9.7

10.3

13.2

14.7

17.5

12.1

12.1

12.2

12.3

12.3

12.3

12.3

12.3

12.4

12.4

12.4

Unit: %

Table 4. Resource Usage of XML/SNMP Gateway

100

CPU Usage (1 Gateway)

Memory Usage (1 Gateway)

90

CPU Usage (2 Gateways)

Resource Utility (%)

80

Memory Usage (2 Gateways)

70

CPU Usage (3 Gateways)

60

Memory Usage (3 Gateways)

50
40
30
20
10
0
0

20

40

60

80

100

Number of SNMP Agents

Figure 5. Resource Usage of XML/SNMP Gateway

As shown in Figure 3, the response time of one gateway sharply increases when handling more than 80 SNMP
agents (640 MIB objects). Also, as shown in Figure 5, the CPU usage of one gateway rapidly increases when
handling more than 80 SNMP agents. Hence, we conclude that the CPU has a significant impact on the processing
overhead in the gateway and we perform more evaluations after upgrading the computing resources from
Pentium-III 800MHz CPU and 256 MB RAM to Pentium-IV 2.8GHZ CPU and 512MB RAM. Table 5 shows the

10

comparison results of the response time on the two different computing environments.
C om puting
R esources of
G atew ays
# of A gen ts
(# of M IB O bjects)

800M H z, 256M B
M anager
G atew ay

2.8G H z, 512M B

G atew ay A gen t
Proc.

C om m .

M anager
G atew ay

G atew ay A gen t
Proc.

C om m .

1 (1)

1012

404

1006

203

1 (8)

1034

832

38

1032

342

22

10 (80)

1188

1058

236

1176

471

129

20 (160)

1245

1278

361

1247

546

184

30 (240)

1318

1463

597

1314

581

277

40 (320)

1398

1615

698

1391

646

331

50 (400)

1408

1820

814

1410

762

402

60 (480)

1484

1996

1285

1483

932

525

70 (560)

1986

2230

2942

1984

1167

965

80 (640)

2165

2586

3296

2161

1420

1329

90 (720)

2226

5826

6850

2227

1757

1694

100 (800)

2295

9053

10154

2295

1998

1821

U nit: m sec

Table 5. Response Time According to Computing Resources

12000

Gateway Agent (Proc.) : 800MHz, 256MB

Gateway Agent (Comm.) : 800MHz, 256MB

Response Time (ms)

10000

Gateway Agent (Proc.) : 2.8GHz, 512MB

Gateway Agent (Comm.) : 2.8GHz, 512MB

8000
6000
4000
2000
0
0

20

40

60

80

100

Number of SNMP Agents

Figure 6. Response Time between the Gateway and Agents according to Computing Resources
Figure 6 shows the graphical representation of Table 5. As shown in Table 5, the processing time between the
manager and the gateway with the lower performance hardware (Pentium-III 800MHz, 256MB) is almost the same
as that between the manager and the gateway with higher performance hardware (Pentium-IV 2.8GHz, 512MB).
Therefore, we do not present the response time between the manager and gateway in Figure 6. After the computing
resources have been upgraded in our case, the message processing time and translation time decreases by half and
the communication time between SNMP stacks in the gateway and agents also decreases by half. Moreover, the
total processing time using the higher-performance gateway increases slightly when the number of SNMP agents

11

exceeds 80 (i.e., more than 640 MIB objects).

Finally, we evaluated the message size and response time in relation to the different XPath expressions using the
800MHz, 256MB gateway. We used four types of XPath expressions as shown in Table 6. In the case of //ifTable
expression, the processing time of XPath and XML message is smaller than the communication time. The
communication time increases because the size of the transferred data is large. In the case of ifInOctets |
ifOutOctets, the processing time of XPath and XML message is almost the same as that of ifTable expression.
However, the transferred message size is a small portion of that of ifTable expression and the communication time
of the second expression is much shorter than that of the first expression. The third and the fourth are the same
XPath expressions. So the transferred message size and communication time between the SNMP stacks in the
gateway and agents are almost the same as one another. However, the processing time of the fourth XPath
expression is much higher than that of the third expression. Therefore, using the appropriate XPath expression is
essential to retrieve the necessary information from SNMP agents considering the processing time and message
size.
As shown in Table 6, the network traffic volumes and MIB objects in these tests are much bigger than those of
the system group tests in Table 1 and Table 2. However, the number of SNMP agents and the processing time in the
gateway are much smaller than in the previous system group tests because the gateway requires more time to
process longer XML messages of the system group tests.
X P ath

//ifT ab le

//ifInO ctets |
//ifO u tO ctets

//ifO u tO ctets

//ifT yp e/
follow ingsib ling ::
ifO u tO ctets

# of SN M P A gents
(# of M IB
O bjects)
P roc. (m s)
G atew ay
A gents
C om m . (m s)
R eq.(bytes)
R es. (bytes)
(# of M IB
O bjects)
P roc. (m s)
G atew ay
A gents
C om m . (m s)
R eq.(bytes)
R es. (bytes)
(# of M IB
O bjects)
P roc. (m s)
G atew ay
A gents
C om m . (m s)
R eq.(bytes)
R es. (bytes)
(# of M IB
O bjects)
P roc. (m s)
G atew ay
A gents
C om m . (m s)
R eq.(bytes)
R es. (bytes)

1810

3634

4444

6268

2553

2076

3486

3825

8078
7235

4230

7770

11253

14866

18605

148420

297988

364408

513976

662396

155764

304802

376410

530902

684210

164

330

494

660

824

1987

1986

2203

2232

2396

393

786

1080

1291

1863

13448

27060

40508

54120

67568

13776

27720

41496

55440

69216

82

165

247

330

412

1946

1985

2165

2206

2344

196

379

526

777

898

6724

13530

20254

27060

33784

6888

13860

20784

27720

34608

82

165

247

330

412

2863

3355

3965

4710

5271

192

376

589

739

880

6724

13530

20254

27060

33784

6888

13860

20784

27720

34608

Table 6. Response Time according to XPath Expression

12

4.

Discussion of Results
In summary, we evaluated the scalability of the XML-based manager and the XML/SNMP gateway to manage

existing SNMP agents. First, we measured the network traffic volume for XML/HTTP communication and SNMP
communication. The network traffic overhead in SNMP communication is twice as much as

XML/HTTP

communication. Because the SNMP agents supported by current network devices in the POSTECH campus
network implement the SNMPv1 protocol, the XML/SNMP gateway retrieves information from SNMP agents by
using GetNext operations. Consequently, the network traffic volume generated by SNMP communications is high.
The network traffic volume between the XML-based manger and gateway is becoming much smaller than that
between the gateway and SNMP agents as the number of SNMP agents increases. Even if we add more
XML/SNMP gateways, the total traffic volume between the gateways and SNMP agents remains the same. On the
other hand, the traffic volume between the XML-based manager and gateway increases by about 500~1000 bytes
for each added gateway.
Next, we measured the XML/HTTP and SNMP communications time. When we retrieve information from only
one SNMP agent, the response time between the XML-based manager and XML/SNMP gateway is almost the same
as the response time between the XML/SNMP gateway and SNMP agent. The latency between the XML/SNMP
gateway and SNMP agents increases rapidly as the number of SNMP agents increases, whereas the latency between
the XML-based manager and XML/SNMP gateway increases more slowly. The response time between the gateway
and SNMP agents is about 90% of the total response time between the XML-based manager and SNMP agents if
the number of SNMP agents is more than 80. From the perspective of the gateway scalability, if we use one
gateway, the total response time increases in scale by about 1000ms every time we add ten SNMP agents until the
number of SNMP agents is about 70~80. However, if the number of SNMP agents exceeds 80, the total response
time rapidly increases by about 15000ms. In contrast, when two or three gateways are used, the total response time
increases almost linearly until the number of SNMP agents reaches 100. The processing time between the gateway
and SNMP agents decreases considerably as the number of gateways increases. Therefore, the total processing time
between the XML-based manager and SNMP agents decreases. The total response time when using two gateways is
almost half the response time when using a single gateway. When three gateways are used, the total response time
is almost one-third of that when a single gateway is used. 70~80 SNMP agents are suitable for one gateway
(Pentium-II 800MHz and 256MB) to manage.
Also, we measured the computing resources of the XML-based manager and XML/SNMP gateway. When the
network devices are managed through one gateway, the resource utilization of the XML-based manager increases as
the number of SNMP agents increases. The CPU usage increases by 1~5% every time ten SNMP agents are added,
whereas memory usage increases by less than 2%. The increase rate of memory usage is not large compared to that
of CPU usage, because the management information loaded to the DOM tree is almost regular although the number
of SNMP agents increases. In terms of the gateway scalability, CPU usage of the XML-based manager increases by

13

2~3% and memory usage of the XML-based manager increases by 1~2% every time we add one more gateway.
The resource utilization of the XML/SNMP gateway also increases as the number of SNMP agents increases.
When the network devices are managed through a single gateway, the CPU usage of this gateway increases almost
linearly by 3~6% until the number of SNMP agents reaches 80. However, it rapidly increases to more than 20%
when the number of SNMP agents reaches 80. This is why the response time between the gateway and SNMP
agents increases considerably after the number of SNMP agents is above 80. The gateway simultaneously makes
requests for multiple MIB objects to multiple SNMP agents using multi-threading mechanisms. When the number
of SNMP MIB objects exceeds 640, the CPU cannot support more threads. Therefore, we also need to upgrade the
CPU of the gateway or add another gateway as the number of the SNMP agents grows. The memory usage of the
gateway increases by 1~2%, however, the change is negligible.
From the perspective of the gateway scalability, the amount of computing resources assigned to each gateway
decreases as the number of gateways increases. The CPU usage of the gateway decreases to a half when two
gateways are used in the management system. Also, the CPU usage of the gateway decreases to one-third when
three gateways are used. The memory usage of the gateway also decreases by about 1~4% each time a new gateway
is added.
We performed more tests after upgrading the computing environments from 800MHz CPU and 256MB RAM to
2.8GHz CPU and 512MB RAM. The processing time of the gateway decreased by a half after upgrading the
computing resources. Also, the processing time of the gateway with 2.8GHz and 512MB did not lead to a
significant increase when the number of SNMP agents is above 80. The upgrade of computing resources allows to
support more threads and hence to retrieve thousands of MIB objects simultaneously.
Finally, we evaluated the processing time of different XPath expressions. Based on different XPath expressions,
the transferred message size and processing time vary significantly. Though they are the same XPath expressions,
the difference in the complexity of their XPath expressions leads to different processing times. Therefore, one must
carefully select an appropriate XPath expression for exactly specifying the target objects to retrieve and avoid
delivering unnecessary data to the manager. The XML/SNMP gateway must provide acceptable performance when
concurrent calls from managers are made. Therefore, it is more desirable to delegate these overheads of XPath
parsing to the managers, provided that most manager systems have sufficient computing resources. Also, it can be
considered that the XML/SNMP gateway supports limited XPath expressions, and time-consuming XPath parsing
is performed in the manager for reducing the overhead of the gateway.
Even though the network traffic volumes and the number of MIB objects are small, the processing time of the
gateway is much higher and larger when there are more SNMP agents due to the processing overhead of the XML
messages parsing in the gateway. The gateway calls the snmpwalk function to retrieve multiple MIB objects from
the SNMP agent and the snmpwalk function internally uses

a series of GetNext operations. Calling GetNext

operations when there are a number of SNMP agents (i.e., 10 operations for each of 10 agents) takes more time than

14

when there are smaller number of SNMP agents (i.e., 100 operations for 1 agent), assuming the total number of
GetNext requests (i.e., 100 times) is the same in both situations.

5.

Conclusion and Future Work

In this paper, we have presented a performance evaluation of XNMS and validated its scalability. From the

performance evaluation results, we showed that the performance overheads of XNMS is not severe. We have
deployed XNMS for managing various network devices in the POSTECH gigabit campus network. We monitored
100 Cisco routers and switches among the POSTECH network devices. Because these network devices are
equipped with SNMPv1 agents, we used an XML/SNMP gateway approach to manage them. As the number of the
gateway increases, the total processing time is reduced considerably. More the gateways are used, shorter the
response time is. Also, the selection of an appropriate XPath expression can reduce the transferred message size and
processing time of the gateway. The performance evaluation results show that the volume of network traffic
between the XML/SNMP gateway and SNMP agents is larger. Also, the latency between the XML/SNMP gateway
and SNMP agents is longer. Therefore, SNMP processing overhead between the gateway and agents is much higher
than XML/HTTP processing overhead between the manager and gateway.
To reduce the network traffic overhead and response time between the gateway and SNMP agents, one needs to
upgrade the SNMP agents to support SNMPv2 using GetBulk operations. However, most of the widely-deployed
network devices are equipped with SNMPv1 agents. So, it is difficult to employ SNMPv2 agents and obtain
performance gain of the network traffic in the current situation. An interesting future work is to implement a tuning
process to optimize the CPU of the XML-based manager and the gateway. Also, as a future work, we would like to
devise a method for replacing the DOM parser with a SAX parser in order to reduce memory usage.

Acknowledgements
The authors would like to thank the NMRG members at the NOMS 2004 NMRG meeting who gave us
valuable comments on our work for improving the quality of our paper.

References
[1] T. Bray, J. Paoli and C. M. Sperberg-McQueen, Extensible Markup Language (XML) 1.0, W3
Recommendation REC-xml-19980210, Feb. 1998.
[2] F. Strauss, and T. Klie, Towards XML Oriented Internet Management, Proc. IFIP/IEEE International
Symposium on Integrated Network Management (IM 2003), Colorado Springs, USA, Mar. 2003, pp.505~518.
[3] J. Schonwalder, A. Pras, J.P. Martin-Flatin, On the Future of Internet Management Technologies, IEEE
Communications Magazine, October 2003, pp.90~97.
[4] Frank Strauss, et al, A Library to Access SMI MIB Information, http://www.ibr.cs.tu-bs.de/projects/libsmi/.
[5] Avaya Labs., XML based Management Interface for SNMP Enabled Devices, http://www.research.
avayalabs.com/user/mazum/Projects/XML/.

15

[6] J. H. Yoon, H. T. Ju and J. W. Hong, Development of SNMP-XML Translator and Gateway for XML-based
Integrated Network Management, International Journal of Network Management (IJNM), Vol. 13, No. 4,
July-August 2003, pp. 259-276.
[7] Y. J. Oh, H. T. Ju, M. J. Choi, J. W. Hong, Interaction Translation Methods for XML/SNMP Gateway,
Proc. of DSOM 2002, Montreal Canada, Oct. 2002, pp. 54~65.
[8] WBEM, WBEM Initiative, http://www.dmtf.org/wbem/.
[9] J.P. Martin-Flatin. Web-Based Management of IP Networks and Systems, Ph.D. Thesis, Swiss Federal
Institute of Technology, Lausanne (EPFL), Oct. 2000.
[10] H. T. Ju, M. J. Choi, S. H. Han, Y. J. Oh, J. H. Yoon, H. J. Lee, and J. W. Hong, An Embedded Web Server
Architecture for XML-based Network Management, Proc. of IEEE/IFIP Network Operations and
Management Symposium (NOMS 2002), Florence, Italy, Apr. 2002, pp.1~14.
[11] A. John, K. Vanderveen and B. Sugla, XNAMI-An extensible XML-based paradigm for network and
application management instrumentation, Proc. of IEEE International Conference on Network, 1999. pp.
115124.
[12] P. Shafer and R. Enns, JUNOScript: An XML-based Network Management API,
http://www.ietf.org/internet-drafts/draft-shafer-js-xml-api-00.txt, Aug. 27, 2002.
[13] Cisco Systems, Cisco Configuration Registrar,
http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/ie2100/cnfg_reg/index.htm.
[14] IETF, Network Configuration (Netconf), http://www.ietf.org/html.charters/Netconf-charter.html.
[15] M. J. Choi, J W. Hong and H. T. Ju, XML-based Network Management for IP Networks, ETRI Journal,
Vol. 25, No. 6, Dec. 2003, pp. 445~463.
[16] M. J. Choi, J. M. Oh, and J. W. Hong, Design and Implementation of XML-Based Management Agent,
Proc. Of IEEE/IFIP Asia-Pacific Network Operations and Management Symposium (APNOMS2003),
Fukuoka Japan, Oct. 2003, pp 331-342.
[17] H. M. Choi, M. J. Choi, and J. W. Hong, Design and Impelmentation of XML-based Configuration
Management System for Distributed Systems, Proc. of IEEE/IFIP Network Operations and Management
Symposium (NOMS 2004), Seoul, Korea, Apr. 2004, pp. 831-844.
[18] M. J. Choi, H. M. Choi, H. T. Ju and J. W. Hong, XML-based Configuration Management for IP Network
Devices, Accepted to Appear in a Special Issue on XML-based Management of Networks and Services in
IEEE Communications Magazine, July 1, 2004.

16