IP Infusion White Paper

ZebOS Internet Route Server Solution

IP Infusion White Paper

ZebOS Internet Route Server Solution

Introduction To support heavy traffic on the Internet resulting from increased data being sent from mobile phones and web-enabled devices to telecommunications data centers, Internet Service Providers (ISPs) have to peer with each other, which has led to the emergence of Internet Exchange (IX) junctions or Internet Exchange points (IXPs). These exchange points are used to hand off traffic between service provider domains. As Internet traffic moves across different autonomous systems, it is important to ensure that routes are properly propagated through provider peers while tracking and monitoring routes to ensure that bogus and malicious information is not propagated. The ZebOS Internet Route Server is designed to address the need for a scalable, secure and low-cost server which can be used for route-viewing and monitoring, as well as for policy-controlled route propagation in the control plane. The ZebOS Internet Route Server features a Linux-based platform optimized for handling a large number of peers and routes. The platform can be easily used for writing custom applications for monitoring and viewing routes. In addition to this, the ZebOS Internet Route Server platform can be used for virtualization and BGP Route Reflections. The ZebOS Internet Route Server has been developed to simplify communications routing protocols between routers, while at the same time mitigating any system-wide outages, also known as black holing, that might be caused by a malicious attack or operator error. Border Gateway Protocol (BGP) is one of most commonly used protocols on the Internet. It depends heavily on the exchange of information between routers. BGP maintains a table of IP networks or prefixes to facilitate network routing between Autonomous Systems (AS). BGP is utilized to transfer routes across the autonomous systems Route Reflector in the Internet. With the growth of the Internet driven by the proliferation of small web-enabled devices, BGP routers are required to handle an increasing number of routes. ISPs peer with each other through other service providers to offer many different services to their customers. BGP maintains a table of IP networks or prefixes to facilitate network routing among multiple ASs. The IXPs, where Internet providers exchange their Internet traffic, are a critical point in the network. As the number of routes and IP addresses increase, maintaining IP network stability, security and scalability at the same time is a tough challenge. Although BGP route servers operate very efficiently, they are vulnerable to attacks by malicious hackers or operator errors in the routing prefixes. This can result in black holing, where the traffic is directed towards the offending peer router away from the intended endpoint. The ZebOS Internet Route Server solution paired with support for anti-BGP hijacking provides a filtering mechanism that is a robust solution to mitigate the creation of these black holes. One of the advantages with this route server versus other solutions is that it attaches to the network non-intrusively without causing any disruptions during installation or during an alarm event. Operator Requirements to Administer BGP Router Operators usually want to get necessary information from the routing tables regarding an invalid route whenever invalid routes are flagged, as outlined below: a) b) c) When did the route become invalid? Why is it an invalid route? Who announced it as invalid?

This enables administrators to fix any problems. BGP requires a lot of information to be exchanged periodically, causing a significant amount of overhead. Usually, this is handled within an AS utilizing route reflectors, as shown in Figure 1.

BGP session

BGP session

BGP session

BGP session BGP session

Figure 1: Topologies of BGP Routers and Servers

October 2009

IP Infusion White Paper

Topologies of BGP Routers and Servers A service provider may peer with many other providers BGP routers to share the load of the large number of routes. Sometimes this means multiple O(n x n) connections among BGP routers at the edge of Internet junctions, since BGP edge routers belonging to different ASs may want to exchange each others routes. The problem is similar inside a single provider network, where internal BGP (iBGP) full mesh connections between routers exist. Route reflectors are used to reflect routes among the clients. In this case, each client manages one connection instead of a number of full-mesh connections. Route servers follow a similar principle, but are used with external BGP (eBGP) connections at the junctions of different autonomous systems.

While creating TCP/IP connections is easy, maintaining so many TCP/IP connections can overload system resources and network bandwidth. Route servers (as well as route reflectors) reduce the number of TCP/IP connections by centralizing the route service. Each node is connected to the route server through one eBGP TCP/IP connection; these nodes are called clients of the route server. The route server receives the routes from all of its clients. Depending on the route server configuration, it may be used only for monitoring and route-viewing purposes, or it can be configured through Access Control Lists (ACLs) and route maps to apply policies on behalf of each client and forward the routes to other clients accordingly. Note that the route servers are typically used to forward the control plane traffic. It is assumed that the actual data traffic flows directly between the routers for which the route server is distributing the routes. Thus typically, the route servers can be located multi-hops away from the clients. Multi-hop eBGP connections should be used between the client and its route server. The route server also preserves the next hop, MED and ASPATH attributes of BGP updates across clients, since it does not take part in the data forwarding path. The clients are either a single BGP peer or a peer group. ZebOS Internet Route Server Features BGP Route servers are still being standardized at IETF. The ZebOS implementation is described in RFC 1863, which defines a BGP/IRDP route server alternative to mesh routing. RFC 1863 was replaced by RFC 4223, which has been used as the guideline for the ZebOS Internet Route Server. Historically, IP Infusions ZebOS Advanced Routing protocol suite has offered BGP route server features, and further improvements are planned in the future.

Figure 2: A Full Mesh of BGP Routers BGP protocol is specified in RFC 4271. A standard BGP requires a TCP connection to all the peers to and from which a BGP router wants to send and receive routes. Thus, if there are 100 eBGP peers participating in peering with each other, each BGP speaker has to maintain 99 TCP/IP connections with its peers.

AS1 BGP Route Server with policy filters

Route-Server Clients

AS2 AS3

AS4

Figure 3: Example of a BGP Route Server

ZebOS Internet Route Server Solution

IP Infusion White Paper

RR/RS

BGP session Operators can see all routing information which can be handled by native BGP Operators can check the validation of all routing entries by using anti-hijacking module The amount of routing information which operators can see is greater than if they only use SNMP

Figure 4 The ZebOS implementation supports instantiation of a BGP view for multiple clients. By doing so, network operators can easily group their BGP peers based on their actual requirements and assign unified policies for inbound and outbound traffic, as shown in Figure 4 above, which illustrates the use of BGP route servers. The ZebOS Internet Route Server is a Linux-based system route server optimized for serving large numbers of clients. Optionally, it can also be used as a route reflector (RFC 4566). However, it is recommended that the BGP be configured either as a route server or as a route reflector. This document concentrates on route server services. The key features of the ZebOS Internet Route Server are support for routes of IPv6, IPv4 and VPN address families, High Availability, security and scalability features. The route server is useful for BGP-MPLS-VPN as well as regular BGP network environments. Route Server as a Monitor or Viewer The ZebOS Internet Route Server can be configured such that it does not forward any routes, but updates the local BGP table with all the routes information. It attaches to the network nonintrusively, without causing any disruptions during installation or during an alarm event. The ZebOS Internet Route Server can be used to view IPv4, IPv6 and BGP-MPLS-VPN routes. BGP views configurations can be used to group a set of clients in a particular view and watch their route characteristics, origin and path information, etc. It is also possible to view a single peer as part of multiple views. With ZebOS, each view maintains its own Routing Information Base (RIB). If desired, inbound and outbound routing policies may be applied using the route maps. Route Server as a Proxy for Distributing Routes In this mode, the route server receives routes from the clients, applies inbound and outbound policies based on their configurations and then distributes the best routes to the peers. In this process it preserves the next hop, ASPATH and MED BGP attributes of the clients. Thus, the route server acts as a proxy for distributing the BGP routes between two of its client routers. This mode of operation is useful for separating the control path from data paths at the Internet junctions for business reasons and load-sharing. Applying Policies at the Route Server The ZebOS Internet Route Server is designed to apply inbound and outbound policies based on the clients policies or service level agreements. The clients may be a single BGP peer or a peer-group. ZebOS uses per-policy BGP RIBs and instances as opposed to using per-peer BGP RIBs. This design provides a scalable architecture which can save computation and space and create policies or views as the operator needs. The following diagram explains how different policies may be configured for route services. We apply Policy 1 to include R1, R2 and peergroup PR, and Policy 2 to R2, R1 and R4, R5. Policy 1: Apply weight 500 to routes when peer is matched with R1; no routes are accepted from PR but it is only used for sending routes. No routes are sent to R1, R2 from route server. Apply weight 500 to the routes received from R2. Only accept routes from R1 and R2 and send routes to R4 and R5.

Policy 2:

Figure 5 describes the policy applications and policy members clearly. Both routers R1 and R2 belong to Policy1 and Policy 2. If both R1 and R2 advertise the same prefix, P1 and Policy 1 are applied. Route P1 (red arrow) from R1 is advertised to peer-group. But when Policy 2 is applied to the other members

October 2009

IP Infusion White Paper

AS1 BGP Route Server with policy filters

R4 R1 R2

Route-Server Clients

P1 from R1 AS2 Policy 1 AS3 AS4 Policy 2 R5 P1 from R2

Peer-Group AS5 AS6

Figure 5: ZebOS Internet Route Server Policy Filtering and Views of the route server, R4, R5 receive Prefix P1 (blue arrow) from router R2. Policies are placed using BGP ACL and route-map configurations. The following example shows that we are setting different preferences (weight) for routes lan02 and lan03 coming from the peer r2_NH. access-list lan02 permit 60.1.1.0/24 access-list lan03 permit 70.1.1.0/24 access-list r2_NH permit 60.60.60.60/32 access-list r3_NH permit 90.90.90.90/32 ! ip prefix-list lan02 seq 5 permit 60.1.1.0/24 ip prefix-list lan03 seq 5 permit 70.1.1.0/24 ! route-map to_R1_P1 permit 1 match ip peer r2_NH match ip address prefix-list lan02 set weight 100 ! route-map to_R1_P1 permit 3 match ip peer r2_NH match ip address prefix-list lan03 set weight 50 ! Management Interfaces BGP command CLI, route map and ACL are the interfaces through which the route server configuration is managed. BGP MIB information for SNMP interfacing is also available through the regular BGP implementation. The relevant ZebOS Internet Route Server statistics are available through the BGP views. The ZebOS Internet Route Server does not install BGP routes in the kernel, as it does not perform data forwarding. Scalability and Performance The ZebOS Internet Route Server inherits ZebOS BGP scalability and performance. When the ZebOS Internet Route Server is configured, the routes are stored in BGP RIBs only; no corresponding routes are installed in the kernel. Since route server configuration is done under BGP views and no data forwarding takes place, we expect that the route server will be able to accommodate processing/updating millions of routes since it does not spend cycles to co-ordinate with lower layer modules or FIB for route installation. The ZebOS Internet Route Server design offers a scalable solution for adding clients in policy-based views. Memories are allocated dynamically when a BGP view is added to the configuration. Thus, it maintains one BGP RIB per view as opposed to spending memory and cycles in maintaining per-peer RIB entries. ZebOS BGP also incorporates co-operative style scheduling for long operations of route processing. We have taken some measurements on ZebOS Internet Route Server performance on ZebOS 7.7 using simulation tools to generate a large number of routes. The laboratory tests were performed using the system configuration as shown in Table 1. It was observed that the route server system was able to handle: 100 BGP views configurations with 100 clients in each view

ZebOS Internet Route Server Solution

IP Infusion White Paper

Processor and processor speed Number of processors Memory size Swap space NIC card type, model and driver information (for each one separately) Table 1 25 clients each advertising with 100,000 routes with 2 views or policy configurations

Intel Pentium 4 CPU 3.06GHz 1 1024MB 2048MB Intel EtherExpressTM 100 NICs

It was also observed that BGP memory usage and route installation time increase with the addition of large numbers of routes. The ZebOS Internet Route Server should support 1000 peerclient eBGP connections with 100 peers sending 100,000 IPv4 or IPv6 or VPNv4 routes each at a time.

configure with two route servers and process the data at least twice. This method also uses more network traffic, although it offers safety in terms of a single point of failure. Another alternative is to use the ZebOS BGP GR mechanism. If the route
ZebOS Advanced Routing Suite

BGP Route Server Policy1 Policy2 _____ Policy n BGP Route Reflector BGP View 1 BGP View 2 BGP View n

Convergence By default, ZebOS BGP maintains a copy of outgoing advertisement information (adjacency RIB). Although it occupies some amount of run-time memory, the changes in the updates are generated quickly in order to minimize the convergence time among the peers. The ZebOS Internet Route Server can be also configured in such a way that it always receives the same set of routes from two different peers, but chooses the routes from one set. If the active set of routes becomes unavailable, the ZebOS Internet Route Server uses the BGPs built-in faulttolerance mechanism to choose the alternate set of routes and announces them to its clients. Moreover, the ZebOS BGP offers configurable knobs for keep-alive timer, hold-timer, minimum route advertisement interval (MRAI) and Graceful Restart (GR) timer, etc., for fine-tuning the BGP convergence according to the deployment network requirements. Virtualization ZebOS provides a virtual routing capability, so the route server system is capable of virtualization as ZebOS. The ZebOS Internet Route Server also uses BGP multi-instances to provide BGP views, which are also virtualized containers to hold each BGP instance and routing table, etc. ZebOS virtual routing domains can be used to maximize the system capacity. For example, one domain may be used for a route server operation, and the second domain can be used as a route reflector. BGP views are possible in each domain to accommodate BGP multiple virtual instances. High Availability BGP protocol itself offers fault tolerance characteristics using best path algorithms. Thus, multiple route servers can be configured for redundancy. However, this requires each client to

Virtual Router A

Virtual Router B

server is configured to do the GR capability exchange with its peers, then its peers will preserve the routes for the specified restart time by when the Route-Server BGP may be restarted with the saved configuration. It re-populates the BGP routing tables from the client updates. However, this offers only partial High Availability. In the near future, a transparent route-server High Availability feature is planned, where one route server will be the primary device and a second will be used as a backup. The backup server can take over the operation of the primary route server transparently should the primary system or BGP process go down. Security ZebOS BGP offers TCP-MD5 (RFC 2385) name-password style security for securing BGP sessions. Thus, the route server or route reflector clients should be able to create BGP sessions with TCP-MD5 security. TCP-MD5 protection ensures authenticity of BGP information between a client and the route server. It also protects the session from TCP segment spoofing such as TCP reset and TCP session hijacking. However, this does not protect a BGP speaker from receiving a black hole

October 2009

IP Infusion White Paper

route or a malicious route from its peer AS, which may have already received a bad prefix. Therefore, more checks in BGP are required. BGP Hijacking The BGP protocol inherently has a few security weaknesses. It does not protect the integrity, freshness or origin authentication of the messages, nor does it check for the ASs authority to announce a route. Further, there is no way of ensuring validity of ASPATH attributes during the course of its travel from the origin to the receiver. The consequences of BGP attacks are diverse and may result in delay or unreachability of the network or of a destination. They may also bring down a section of the Internet, contaminating large sections of routes in different ISPs, thereby creating black holing. There are multiple solutions to address BGP security problems today, for example, TCP-MD5, IPSecenabled BGP sessions or BGP sessions over IPSec VPNs, generalized TTL check in BGP, defensive routing policies, etc. The Public Key Infrastructure for authentication of BGP routes has been discussed in the Internet Engineering Task Force (IETF) Secured Inter Domain Routing (SIDR) Working Group. In addition to TCP-MD5 and the ability to apply defensive routing policies, ZebOS BGP offers a unique immediate solution for prefix AS origin validation through the Internet Routing Registry (IRR) service as the route server receives the update. If the route is valid, then the route server follows a set of pre-determined rules to install the routes in the local RIB and forward to the clients. If the route is not found through the IRR service database or is invalid, the route server logs those routes, and alerts may be generated. This prefix origin validation check will address most of todays BGP prefix hijacking and network outage issues due to mis-configuration and human error. Here are some examples of BGP prefix security issues: A BGP peer announces a prefix that it is not supposed to originate A BGP peer announces a more specific prefix than the true originator A BGP peer announces that it can route traffic for the hijacked prefix (black hole routing or hijacked route path)

With the secure ZebOS Internet Route Server, it will be easy to identify or filter the malicious prefixes and the attacker origin AS. Moreover, only the route server needs an upgrade with the security software rather than all the client routers, which still get the benefit of BGP security. Multi Protocol Support The route server fully supports IPv4, IPv6 and VPNv4 unicast address families. Additional address families as shown in Table 2 are partially supported at the route servers. Writing Applications The route server platform software is based on a Linux operating system enriched with socket API, different user interfaces and libraries. Thus, an operator can write or port open networking software to monitor/view routes on the same server real-time. Any other applications can be written with portability to ensure operators custom environments. Product Availability The ZebOS Internet Route Server will be available in two product categories. Immediately Out-of-the-box server solution: A fully configured out-of-the-box server will be delivered with route server application loaded and ready to run. Future Binary: The software code for running the route server will be delivered in binary format customized to the customers requirements. Conclusion IP Infusions ZebOS Internet Route Server is the industrys leading low-cost, scalable, multi-function platform which offers innovative control plane routing services for BGP routing packet monitoring, without any disruptions to the network, and is a scalable alternative to route exchanges at large Internet junctions. We believe its ability for virtualization and secure communication will provide a competitive edge to ISPs in deploying routing services at IXPs and the borders of their autonomous systems.

AFI 1 : IPv4 2 : IPv6 1 : IPv4 2 : IPv6 2 : IPv6 Table 2

SAFI 1 : NLRI for Unicast forwarding (regular IPv4 routes) 1 : NLRI for Unicast forwarding (regular IPv6 routes) 128 : NLRI for MPLS labeled VPN (used in BGP-MPLS-VPN) 128 : NLRI for MPLS labeled VPN (used in 6VPE) 4 : NLRI with MPLS labels (used in 6PE)

ZebOS Internet Route Server Solution

About IP Infusion IP Infusion Inc. delivers advanced software solutions that power communications equipment for packet-based Next Generation Networks (NGNs). With a unique modular architecture and the industrys broadest suite of communication protocols, IP Infusion enhances product differentiation and market agility for many of the worlds leading network equipment vendors. Incorporated in Delaware in October 1999, IP Infusion is headquartered in Sunnyvale, California, and is a wholly owned subsidiary of ACCESS CO., LTD., of Tokyo, Japan. For more information about IP Infusion, please visit www.ipinfusion.com. 2009 ACCESS CO., LTD. All rights reserved. ACCESS is a registered trademark or trademark of ACCESS CO., LTD. in the United States, Japan and/or other countries. IP Infusion, the IP Infusion logo and ZebOS are either registered trademarks or trademarks of IP Infusion Inc. in the United States and/or other countries. The registered trademark LINUX is used pursuant to a sublicense from Linux Mark Institute, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other trademarks, logos and trade names mentioned in the document are the property of their respective owners. Specifications are subject to change without prior notice.