COMPETITIVE EDGE

COMPETITIVE EDGE: The Performance Claims of the SG135 Tested

Competitive Edge: The Performance

Claims of the SG135 Tested
When comparing the performance of different solutions on the market, it is easy just to read Datasheet Claims
off the datasheet. However, should we believe the claims on a product datasheet? Fortinet Sophos
extensively certifies and rigorously tests its products, but if we compare those results against SG135
the datasheet values of a competitive solution, is that a fair and even comparison? Usually
datasheet numbers are incomplete, or the results stem from a test where the conditions
are unknown. If we use datasheet claims to compare to our own solutions, this is not a fair
comparison for Fortinet or the customer.
nn6000 Mbps firewall throughput
In this paper we will run the Sophos SG135 through a gauntlet of tests that we typically run
nn750 Mbps IPS throughput
our own solutions through. Even with Fortinet solutions, performance can vary based on the
nn1000 Mbps IPSec VPN throughput
features activated. But how does that affect competitors’ solutions? How accurate are their
datasheet claims? nn350 Mbps AV-proxy throughput

The Product Tested – Sophos SG135

The SG135 is one of Sophos’ SMB appliances and it is something we see come up in SMB
opportunities. The SG135 is billed as a Unified Threat Management appliance (UTM) and Next
Generation Firewall (NGFW), and has been on the market for almost a year. For this test we
used the latest OS from Sophos available for the SG135, version 9.310-11. From Sophos’
website, we can find their performance claims for the SG135 (see Details sidebar). For the
SG135, two different performance numbers are listed in two different sections. Specifically,
for IPS throughput there is a number 750 Mbps, and also 1.5 Gbps. It’s unclear which is the
‘right’ one, but it’s a good bet it’s neither.

While it is not entirely clear under what conditions these values were attained, these will be the
metrics that we will use to compare with actual results. Fortinet thoroughly and fairly tests its
own devices, and we will apply the same tests fairly to the SG135 to get the most apples-to-
apples, honest comparison possible. The tests are exactly the same as we use for customer-
facing FortiGate POCs – we are not trying to do anything special here to tilt the results in our
favor.

What’s in the Box?

The SG135 comes with a 320 GB laptop hard drive, 64 MB flash, 6 GB memory, and an
Intel Atom C2558 4-core CPU. This CPU is what performs all datapath and management
functions. There are eight 1 GB copper ports, as well as a console port, two USB ports, and
a VGA port.

The underlying OS itself is running a Linux 3.12 kernel. There is a kernel module that does all
the packet filtering (stateful inspection) functions of the firewall, while a plethora of user space
programs (both open source and proprietary) performs all the deeper packet inspection.

© 2015 Fortinet. All Rights Reserved – For Internal/Partner Use Only 1

COMPETITIVE EDGE: The Performance Claims of the SG135 Tested
The Tests and Methodology
A standard set of performance tests was performed on the SG135
– the same tests Fortinet runs on all of its FortiGate appliances. The
tests were run on the device twice, once in Firewall-only mode and
again with the device in ‘Firewall + IPS’ mode. Certainly customers
deploy firewalls in their networks in both modes, so we tested
the platform under both scenarios. Traffic for all the tests was
generated on an Ixia BreakingPoint device.
Specifically, the (8) tests performed include:
– (1) Stateful/HTTP traffic throughput (32K file size)
– (4) Non-stateful/UDP traffic throughput (4 varying packet
sizes)
– (1) Maximum Connections Per Second
– (1) Maximum Concurrent Connections
– (1) Security Effectiveness
It should be noted that the SG135 cannot granularly apply
different UTM via policy. While you can try to configure overrides
or exclusions via IP to the different modules (Intrusion Prevention,
Web Filtering, etc.), that just places configuration options for snort
and other user-level programs that are doing these functions. The
packets are still delivered to these different user space processes
and cause a large performance hit. The only way we could really
isolate the different UTM functions was by enabling or disabling
those functions with their large on/off buttons for that entire feature.
This resulted in slightly different configs for each test scenario.
Throughput Tests – Raw throughput alone may not always be the
truest indicator or test of real-world performance and performance
limits, but it’s clearly a popular and recognizable test. We use an
HTTP-32K session for stateful traffic, and a number of UDP tests at
different packet sizes for non-stateful traffic. The four UDP packet
size scenarios tested are 1518 bytes, 512 bytes, 64 bytes, and a
mixed packet size scenario (IMIX). The HTTP-32K test is the test
that most represents real traffic in a customer environment. The
small packet UDP test might be very relevant for Financial Services
and other high-transaction environments.
Connections per Second (CPS) Test – We measured maximum
CPS by transferring a 1 byte file over HTTP repeatedly as fast as
possible. This exercises the CPU and is usually the best indicator
of overall performance, even when there is hardware acceleration.
We performed a 10-packet connection (3-way handshake,
get+ack, 200ok+ack, 4-way close = 10 packets). Many vendors
try to improve their CPS number by defining a connection to be
comprised of fewer packets, but the overall number of packets
able to be processed remains the same.
© 2015 Fortinet. All Rights Reserved – For Internal/Partner Use Only 2
Concurrent Sessions Test – Measures how many connections
can be open simultaneously. The results of this test are mostly a
function of memory. To find the maximum concurrent sessions, we
just have to be able to open connections, leave them open, transfer
data, and then close the connections. All parts need to occur and
be successful. If any of them fails, it means we were unable to
open or maintain open some of the connections.
Security Effectiveness Test – Measures how many attacks can
be recognized and blocked by a device. The test we use is the
BreakingPoint Strike Level 3 test. It has 468 different attacks that it
tries to perform. In order to make sure that traffic is inspected fully,
we run these attacks bidirectionally, giving us a total of 938 attacks.
Determining Pass vs. Fail
To determine whether the device passed (or failed) each test, we
looked for consistent and stable performance results. The device
was stressed until results began to destabilize. When a device
becomes overloaded, the metrics appear unstable in performance
charts. Even if certain performance metrics had potential/theoretical
“headroom” to go higher, the user experience would be suffering
due to the instability in another aspect of performance that had hit
its limit.
For example, in our stateful throughput test there are several
criteria that we must inspect beyond pure throughput itself.
Unstable connections per second and/or concurrent connections
performance are both often leading indicators of a device starting
to struggle at a given throughput level. For UDP, an increase in
latency can be that indicator.
Normally when we find something is failing a test at a certain value,
we reduce the value and test again, repeating this process until a
stable test is seen. Sometimes, a device can exhibit strange or bad
behavior even when reducing the value. In those cases we might
talk about a higher value that the device achieved, but those higher
values would be in a failure state. Reporting the higher value is
really just for informational purposes, but are not considered valid
results.
Of the eight tests that the SG135 was subjected to, almost all
important tests failed to live up to the SG135 datasheet claims. We
will focus the remainder of this Competitive Edge on some of these
critical tests where the product failed to deliver versus the claimed
metrics.
COMPETITIVE EDGE: The Performance Claims of the SG135 Tested

Firewall-Only Tests
For the firewall-only tests, we set up three port pairs of the 1 G
ports. Since the datasheet claims 6 G, three pairs should give us
that. At first, using link aggregation on the three pairs of links was
tried (one link agg per side). Unfortunately, that never would give
us above 4 Gbps of throughput. So instead we just used individual
port pairs with separate subnets.

SUMMARY OF RESULTS
Test FW Mode FW+IPS Comment
Mode
Stateful Throughput FAIL FAIL Constant application
HTTP-32K *(2.7 Gbps) *(141 Mbps) failure
Non-stateful Throughput FAIL FAIL Constant traffic failure
UDP-1518 Byte *(5.1 Gbps) *(1.8 Gbps)
Non-stateful Throughput 1.7 Gbps 420 Mbps No claim on datasheet
UDP-512 Byte
Figure 1
Non-stateful Throughput 300 Mbps 60 Mbps No claim on datasheet
UDP-64 Byte
Even reducing the throughput to 800 Mbps, we still saw strange
Non-stateful Throughput FAIL FAIL No claim on datasheet
drops of traffic during the test runs.
UDP - IMIX *(1.4 Gbps) *(420 Mbps)
Connections Per 7130 1840 No claim on datasheet Other packet sizes did not have the same kind of failures as the
Second UDP-1518B test. This leads us to believe it’s something about
Concurrent Connections 2M 1.1M No claim on datasheet handling large packets that gives the SG135 problems. While there
Security Effectiveness 2.3% 5.7% No claim on datasheet were no errors, the throughput results for the smaller packet sizes
were also correspondingly small.
We uncovered problems with the SG135 living up to its datasheet
For stateful throughput, the SG135 achieved zero, a failed result.
specs when we subjected it to just about any test that was on it.
No matter how little traffic was sent, there were always 10-40%
There are lots of tests and metrics which are not even listed on their
application failures that were occurring. Other devices did not
public datasheet. There seemed to be some fundamental problems
exhibit this behavior, but a different model Sophos device did, again
with the device’s ability to sustain any kind of traffic. During almost
leading us to believe it was a problem with the OS (see Figure 2).
every throughput-based test, there were intervals where the device
would stop passing traffic on one or more of its interface pairs.

Even when dramatically lowering the traffic volume, this still

occurred. This is why for the summary these tests are listed as
FAIL. To make sure it was not a bad hardware device or bad
configuration, the same tests were conducted with different
devices. Using a larger Sophos model (SG310) got the same types
of results. Using a FortiGate, and a second competitor’s device,
the tests ran fine. This leads us to believe it’s a problem with the
underlying OS.

For the UDP-1518B throughput test, if we would try to push 6 Gbps,

it would never handle that much traffic (even ignoring the massive
failures). Bringing the test to 5.1 Gbps showed that the device could
at least (for a short time) pass 5.1 Gbps (see Figure 1).

Figure 2

© 2015 Fortinet. All Rights Reserved – For Internal/Partner Use Only 3

COMPETITIVE EDGE: The Performance Claims of the SG135 Tested

If we were to ignore the application failures and try to see what

the device could do, 2.7 Gbps at 7958 connections per second
seemed sort of stable (minus the application failures that were
being dropped) (see Figure 3).

Figure 4: HTTP-CPS FW for the Sophos SG135

The stateful CPS tests were able to be performed without any

major problems. 7130 CPS was the result, with only 31% CPU
(see Figure 4). The device itself was capable of more (almost 21K),
Figure 3
but there were unacceptable session failures. ~21K CPS is not a
bad number for an SMB-sized box. But the fact that the CPS tests
The CPU utilization was 92% on average, some cores higher and
did not have the same level of application failures as the stateful
some lower. Not all cores were used evenly, especially because
throughput tests was very interesting.
there were a number of other user space processes competing for
CPU time. After investigating, it seemed like the 32K file size for our stateful
throughput test was what the SG135 could not handle. Our CPS
Because this test was measuring firewall-only performance of the
test uses a 1 byte file, and did not have the failures. After some
SG135, the throughput is a function of the Intel Atom processor
trial and error, we determined that a 5500 byte file was the largest
inside. For non-stateful traffic, the maximum packet-per-second
we could transfer without having the application failures. So if a
rate that the CPU could do was around 400K to 440K, no
customer has files that will go through the firewall that are larger
matter what packet size. This means very low performance for
than 5500 bytes, then the Sophos SG135 is definitely not for them.
smaller packet sizes. This is probably why Sophos only lists
the performance of stateless UDP-1518 byte packets as their The Concurrent Sessions test is the last test we performed. This
throughput number. It’s intentionally misleading, as no serious big one is pretty cut and dried. The test shows the SG135 is able to
player uses UDP-1518B as their main performance benchmark. reach 2M concurrent sessions. This meets the datasheet value.
Also, because this is purely CPU based, the latency (~150 us) is
much higher than on a device with hardware-based acceleration.

© 2015 Fortinet. All Rights Reserved – For Internal/Partner Use Only 4

COMPETITIVE EDGE: The Performance Claims of the SG135 Tested

Firewall + IPS Tests

For these tests, we enabled the Intrusion Prevention and Advanced
Threat Protection big on/off buttons. We could observe processes
relating to these functions consuming CPU when this was enabled.

The results are summarized in the second column of our summary

table, which lists the overall performance for our tests when the
SG135 is in firewall and has Intrusion Prevention and Advanced
Threat Protection enabled. There was no significant difference if
Advanced Threat Protection was disabled.

SUMMARY OF RESULTS
Test FW Mode FW+IPS Comment
Mode
Stateful Throughput FAIL FAIL Constant application
HTTP-32K *(2.7 Gbps) *(141 Mbps) failure
Non-stateful Throughput FAIL FAIL Constant traffic failure
UDP-1518 Byte *(5.1 Gbps) *(1.8 Gbps) Figure 5

Non-stateful Throughput 1.7 Gbps 420 Mbps No claim on datasheet

UDP-64B is even lower at only 60 Mbps (see Figure 6).
UDP-512 Byte
Non-stateful Throughput 300 Mbps 60 Mbps No claim on datasheet
UDP-64 Byte
Non-stateful Throughput FAIL FAIL No claim on datasheet
UDP - IMIX *(1.4 Gbps) *(420 Mbps)
Connections Per 7130 1840 No claim on datasheet
Second
Concurrent Connections 2M 1.1M No claim on datasheet
Security Effectiveness 2.3% 5.7% No claim on datasheet

In the non-stateful UDP throughput tests with IPS enabled, we can

see that while the SG135 is inspecting UDP traffic for IPS, it takes
a huge performance hit when it is enabled. With UDP-1518B we
are only seeing 780 Mbps (see Figure 5). This is about half of their
claimed datasheet number. And remember, their datasheet claims
are only using the easiest test, non-stateful UDP-1518 byte.

Figure 6

© 2015 Fortinet. All Rights Reserved – For Internal/Partner Use Only 5

COMPETITIVE EDGE: The Performance Claims of the SG135 Tested
Moving on to TCP throughput with IPS, the SG135 achieved only
141 Mbps, and that was with 15% of the connections failing. Even
at these levels the test result was not stable. The CPU usage was
only 67% average, some cores were higher, some were lower. Not
all cores were used evenly (see Figure 7).
Figure 7: IPS - HTTP-32K Test Result
These results are wholly unacceptable. While 141 Mbps is still
below their claimed value of 750 Mbps (or even 1500 Mbps),
having these kinds of application failures represent traffic in a
customer’s environment that would be dropped!
For the Firewall + IPS connection rate test, the maximum CPS that
was achieved with stability was 1840 connections per second.
Past that, we were getting connection failures. This test seemed to
have variable results when various reporting or logging processes
would kick in (see Figure 8).
Figure 8: IPS - HTTP-CPS Test
© 2015 Fortinet. All Rights Reserved – For Internal/Partner Use Only 6
And finally, the final performance test, Firewall + IPS Concurrent
Sessions, showed results less than the Firewall-only test – 1.1M
connections. The interesting part here is that the memory usage
of other processes was a definite factor in this one. Because
these additional processes are taking up large chunks of memory,
it leaves less available for the system to use. There were a few
instances where the Web UI locked up entirely and had to be
rebooted.
Security Effectiveness
Performance aside, another important aspect of any security device
is security effectiveness. How much protection will this device give
you? In order to look at this, we run a standard set of security
attacks through the device to see what kind of results we get.
The test we use is the Breakingpoint Strike Level 3 test. It has 468
different attacks that it tries to perform. In order to make sure that
traffic is inspected fully, we run these attacks bidirectionally, giving
us a total of 938 attacks.
As a base line, we run the security attacks with firewall only first, as
there are some attacks that are blocked by the nature of a stateful
inspection firewall, then we run the test again with different security
features enabled.
When running the base line test, we see that the Sophos SG135
blocked 22 attacks out of 938 (2.3%). Not great, but most of these
attacks occur at higher application levels than a stateful inspection
firewall is going to see (see Figure 9).
Figure 9
COMPETITIVE EDGE: The Performance Claims of the SG135 Tested
Next, Intrusion Prevention was turned on and the test was run
again. The number went up from 22 attacks blocked to 53 out of
938 (5.7%). That is not a big improvement. Looking at the logs, a
small number of the attacks that were blocked were logged with
what looked like something correct, but for the rest there was no
log entry. Still, 5.7% is not a very good blocking rate (see Figure 10).
According to the SG135, not all IPS rules are enabled. From the
Web UI, all signatures were enabled with no age restriction, but that
does not enable all rules.
Figure 10
As part of further testing, it was also decided to try turning on
Web Filter. This turns on the proxy function of the device, and
leads to even more degraded performance. But when the security
effectiveness test was run again, the blocking rate improved,
going from 53 attacks blocked to 395 out of 938 attacks blocked
(42.1%). This is a large improvement from before (though it’s still
a VERY low block rate). It seems like turning on WF helped the
SG135 block more attacks, but looking at the logs showed no
information about these attacks (see Figure 11).
Figure 11
© 2015 Fortinet. All Rights Reserved – For Internal/Partner Use Only 7
What the logs did show (sometimes) was that the connection was
blocked due to the URL being too long. This turns out to be a case
of the proxy blocking long URLs. Many attacks can be blocked
in this way because there’s a buffer overflow attempt. However,
these are NOT blocked because the SG135 detected this specific
attack. There’s no log message about the type of attack or any
useful forensic information. There are also potentially false positives
generated because something does have a long URL.
Conclusion
When evaluating security devices in the market, sometimes the
only information available is on a datasheet. Knowing that different
vendors test their devices in different ways makes it very difficult to
accurately draw comparisons to the datasheet or even between
multiple vendors. Being able to have the same tests run on
separate devices gives us a basis for comparison.
The Sophos SG135 claims to be a 6 Gbps Firewall. It’s not clear if
they are also claiming it’s a 6 Gbps NGFW. Even trying to handle
5.1 Gbps of the simplest UDP large packet firewall-only traffic is
difficult for this device. Even at numbers as low as a few hundred
Mbps, the SG135 drops traffic.
When even turning on just one of the functions of NGFW (IPS), the
performance drops dramatically. And while a performance drop
is expected even in the Sophos datasheet (from 6 Gbps down
to 750 Mbps), the real value of 141 Mbps (with all its associated
transaction failures) is far below that.
When looking at actual testing of stateful TCP throughput, there
are even worse kinds of failures. Trying to transfer files that are
over 5500 bytes in length causes massive amounts of transaction
failures. It’s not clear why this occurs, but this is something that
should never be placed into a real-world environment.
Perhaps the only good thing that can be said about the Sophos
SG135 is that during testing the IPv4 and IPv6 test results were
mostly similar. This is expected with a device that does everything
in CPU. There is an expectation that some performance may drop
slightly, which we see, though all the same problems that plague
IPv4 on this device are the same for IPv6.
Lastly, the poor security effectiveness of this device cannot be
stressed enough. The ultra-low block rates (less than 6% of
attacks) of the IPS + APT combination make it seem like these
functions are doing nothing but slowing your throughput down with
no appreciable gain. Turning on Web Filtering as a way to boost
this percentage is questionable at best, and third-party security
effectiveness tests like NSS CAWS also reflect this same pattern.