Linux Networking Internals

The Linux Network Subsystem
Unable to handle kernel paging request at virtual address 4d1b65e8
Covers Linux version 2.6.25
Unable to handle kernel paging request at virtual address 4d1b65e8
pgd = c0280000
pgd = c0280000
Version 1.1
<1>[4d1b65e8] *pgd=00000000[4d1b65e8] *pgd=00000000
Internal error: Oops: f5 [#1]
Internal error: Oops: f5 [#1]
Modules linked in:Modules linked in: hx4700_udc hx4700_udc asic3_base asic3_base
CPU: 0
CPU: 0
PC is at set_pxa_fb_info+0x2c/0x44
PC is at set_pxa_fb_info+0x2c/0x44
LR is at hx4700_udc_init+0x1c/0x38 [hx4700_udc]
LR is at hx4700_udc_init+0x1c/0x38 [hx4700_udc]
pc : [<c00116c8>] lr : [<bf00901c>] Not tainted
sp : c076df78 ip : 60000093 fp : c076df84
pc : [<c00116c8>] lr : [<bf00901c>] Not tainted
© Copyright 20062004, Michael Opdenacker For full copyright information see last page.
© Copyright 20032006, Oron Peled Creative Commons AttributionShareAlike 2.0 license
© Copyright 20042006 Codefidence Ltd. 1
Rights to copy
This kit contains work by the
following authors:
Attribution – ShareAlike 2.0
You are free © Copyright 20042006
to copy, distribute, display, and perform the work Michael Opdenacker
to make derivative works
michael@freeelectrons.com
to make commercial use of the work
Under the following conditions
http://www.freeelectrons.com
Attribution. You must give the original author credit.
© Copyright 20032006
Share Alike. If you alter, transform, or build upon this work,
Oron Peled
you may distribute the resulting work only under a license
identical to this one. oron@actcom.co.il
For any reuse or distribution, you must make clear to others the http://www.actcom.co.il/~oron
license terms of this work.
Any of these conditions can be waived if you get permission from © Copyright 2004 – 2008
the copyright holder.
Codefidence ltd.
Your fair use and other rights are in no way affected by the above.
info@codefidence.com
License text: http://creativecommons.org/licenses/bysa/2.0/legalcode
http://www.codefidence.com
What is Linux?
Linux is a kernel that implements the POSIX and Single Unix
Specification standards which is developed as an Open Source project.
When one talks of “installing Linux”, one is referring to a Linux
Distribution: a combination of Linux and other programs and library that
form an operating system.
Linux runs on 24 main platforms and supports applications
ranging from ccNUMA super clusters to cellular phones and
micro controllers.
Linux is 15 years old, but is based on the 40 years old Unix design
philosophy
Layers in a Linux system
●
Kernel
User programs
●
Kernel Modules
●
C library
●
System libraries Kernel
●
Application libraries
C library
●
User programs
Kernel architecture
App1 App2 ...
User
space
C library
System call interface
Process Memory Filesystem Device

management Networking
management support control Kernel
space
Filesystem
types
CPU support CPU / MMU Storage Character Network

code support code drivers device drivers device drivers
Hardware
CPU RAM Storage

Kernel Mode vs. User Mode
All modern CPUs support a dual mode of operation:
User mode, for regular tasks.
Supervisor (or privileged) mode, for the kernel.
The mode the CPU is in determines which instructions the CPU is
willing to execute:
“Sensitive” instructions will not be executed when the CPU is in
user mode.
The CPU mode is determined by one of the CPU registers, which stores
the current “Ring Level”
0 for supervisor mode, 3 for user mode, 12 unused by Linux.
The System Call Interface
When a user space tasks needs to use a kernel service, it will make a
“System Call”.
The C library places parameters and number of system call in registers
and then issues a special trap instruction.
The trap atomically changes the ring level to supervisor mode and the
sets the instruction pointer to the kernel.
The kernel will find the required system called via the system call table
and execute it.
Returning from the system call does not require a special instruction,
since in supervisor mode the ring level can be changed directly.
Linux System Call Path
do_name() Function call

Kernel
Trap
sys_name()
entry.S
Glibc
Task
Task
Linux networking Subsystem Overview
Stack <> App App 1 App2 App3
Socket Layer
UDP TCP ICMP
Networking Stack IP
Bridge
Stack Driver Interface
Driver <> Stack
Driver
Driver <> Hardware
Network Device Driver Hardware Interface
packet packet packet packet packet
Memory Driver
Access
Tx
Send Send Send SentOK SendErr Free Memory
mapped
registers
access
Rx
Free Free RcvOk RcvErr RecvCRC RcvOK
Interrupts
packet packet packet packet
●Driver allocates Ring Buffers. DMA

●Driver resets descriptors to initial state.
●Driver puts packet to be sent in Tx buffers.
●Device puts received packet in Rx buffers.
●Driver/Device update descriptors to indicate state.
●Device indicates Rx and end of Tx with interrupt,
unless interrupt mitigation techniques are applied.
Network Device Registration
Each network device is represented by a struct net_device
These are allocated using:
struct net_device *alloc_netdev(size, mask,
setup_func);
size – size of our priv data part
mask – a naming pattern (e.g. “eth%d”)
setup_func – A functionthat set ups the rest of net_device.
And is registered via a call to:
int register_netdev(struct net_device *dev);
Network Device Initialization
The net_device structure is initalized with numerous methods
and flags by the setup function:
open – request resources, register interrupts, start queues.
stop – deallocates resources, unregister irq, stop queue.
get_stats – report statistics
set_multicast_list – configure device for multicast
hard_start_xmit – called by the stack to initiate Tx.
IFF_MULTICAST – Device support multicast
IFF_NOARP – Device does not support ARP protocol
Packet Representation
We need to manipulate packets through the stack
This manipulation involves efficiently:
Adding protocol headers/trailers down the stack.
Removing protocol headers/trailers up the stack.
Packets can be chained together.
Each protocol should have convenient access to header
fields.
To do all this the kernel uses the sk_buff structure.
Socket Buffers
The sk_buff structure represents a single packet.
This structure is passed through the protocol stack.
It holds pointers to a buffers with the packet data.
It holds many type of other information:
Data size.
Incoming device.
Priority.
Security ...
struct sk_buff
next: Next buffer in list ip_summed: Driver fed us an IP checksum
prev: Previous buffer in list priority: Packet queuing priority
sk: Socket we are owned by users: User count see {datagram,tcp}.c
tstamp: Time we arrived protocol: Packet protocol from driver
dev: Device we arrived on/are leaving by truesize: Buffer size
input_dev: Device we arrived on head: Head of buffer
h: Transport layer header data: Data head pointer
nh: Network layer header tail: Tail pointer
mac: Link layer header end: End pointer
dst: Destination route cache entry destructor: Destruct function
sp: Security path, used for xfrm nfmark: Netfilter hooks private data
cb: Control buffer. Private data. nfct: Associated connection, if any
len: Length of actual data ipvs_property: skbuff is owned by ipvs
data_len: Data length nfctinfo: Connection tracking info.
mac_len: Length of link layer header nfct_reasm: Netfilter conntrack reassembly pointer
csum: Checksum nf_bridge: Saved data about a bridged frame
local_df: Allow local fragmentation flag tc_index: Traffic control index
cloned: Head may be cloned (see refcnt) tc_verd: Traffic control verdict
nohdr: Payload reference only flag dma_cookie: DMA operation cookie
pkt_type: Packet class secmark: Security marking for LSM
fclone: Clone status
Socket Buffer Diagram
headroom frag1
Note
Ethernet Network chip

IP must support
TCP frag2 Scatter/Gather
len to use of frags.
Payload
...
Otherwise
head kernel must
data Padding copy buffers
tail before send!
end frag3
struct
... sk_shared_info
dev
struct sk_buff
Socket Buffer Operations
skb_put: add data to a buffer.
skb_push: add data to the start of a buffer.
skb_pull: remove data from the start of a buffer.
skb_headroom: returns free bytes at buffer head.
skb_tailroom: returns free bytes at buffer end.
skb_reserve: adjust headroom.
skb_trim: remove end from a buffer.
Operation Example: skb_put
unsigned char *skb_put
(struct sk_buff * skb, unsigned int len)
Adds data to a buffer:
skb: buffer to use
len: amount of data to add
This function extends the used data area of the buffer.
If this would exceed the total buffer size the kernel will
panic.
A pointer to the first byte of the extra data is returned.
Socket Buffer Alignment
CPUs often take a performance hit when accessing unaligned
memory locations.
Since an Ethernet header is 14 bytes network drivers often
end up with the IP header at an unaligned offset.
The IP header can be aligned by shifting the start of the
packet by 2 bytes. Drivers should do this with:
skb_reserve(NET_IP_ALIGN);
The downside is that the DMA is now unaligned. On some
architectures the cost of an unaligned DMA outweighs the
gains so NET_IP_ALIGN is set on a per arch basis.
Socket Buffer Padding
The networking layer reserves some headroom in skb data.
This is used to avoid having to reallocate skb data when the
header has to grow.
In the default case, if the header has to grow 16 bytes or less
we avoid the reallocation.
Unfortunately, this headroom changes the DMA alignment of
the resulting network packet. As for NET_IP_ALIGN, this
unaligned DMA is expensive on some architectures.
Therefore architecture can override this value, as long as at
least 16 bytes of free headroom are there.
Socket Buffer Allocations
dev_alloc_skb: allocate an skbuff for Rx
netdev_alloc_skb: allocate an skbuff for Rx, on a
specific device.
Allocate a new sk_buff and assign it a usage count of one.
The buffer has unspecified headroom built in. Users should allocate
the headroom they think they need without accounting for the built
in space. The built in space is used for optimizations
NULL is returned if there is no free memory.
Although these functions allocates memory it can be called from an
interrupt.
sk_buff Allocation Example
Immediately after allocation, we should reserve the needed
headroom:
struct sk_buff*skb;
skb = dev_alloc_skb(1500);
if(unlikely(!skb))
break;
/* Mark as being used by this device */
skb>dev = dev;
/* Align IP on 16 byte boundaries */
skb_reserve(skb, NET_IP_ALIGN);
Softnet
Network stack is implemented as a pair of softirqs for
parallelize packet handling on SMP machines:
NET_TX_SOFTIRQ Feeds packets from network stack to
driver.
NET_RX_SOFTIRQ Feeds packets from driver to network
stack.
Like any other softirq, these are called on return from
interrupt or via the low priority ksoftirqd kernel thread.
Transmit/receive queues are stored in percpu softnet_data.
Linux Contexts
Interrupt Handlers
Interrupt SoftIRQs
Context
Kernel
Regular
tasklets
tasklets
Hi prio
Timers
Net
... Space
Stack
Network Process
Interface User Context Thread
Device
Driver Kernel
User Space Thread
Packet Reception
The driver allocates an skb and sets up a descriptor in the
ring buffers for the hardware.
The driver Rx interrupt handler calls netif_rx(skb).
netif_rx deposits the sk_buff in the percpu input queue. and
marks the NET_RX_SOFTIRQ to run.
At SoftIRQ processing time, net_rx_action() is called by
NET_RX_SOFTIRQ, which calls the driver poll() method to
feed the packet up.
Normally poll() is set to proccess_backlog() by net_dev_init().
Packet Rx Overview
Packet Transmission
Each network device defines a method:
int (*hard_start_xmit) (struct sk_buff *skb, struct
net_device *dev);
This function is indirectly called from the NET_TX_SOFTIRQ
Call are serialized via the lock dev>xmit_lock_owner
The driver manages the transmit queue during interface up
and downs or to signal back pressure using the following
functions:
void netif_start_queue(struct net_device *net);
void netif_stop_queue(struct net_device *net);
void netif_wake_queue(struct net_device *net);
Packet Tx Overview
NAPI
Network “New API”
Provides interrupt mitigation
Requirements:
A DMA ring buffer.
Ability to turn off receive interrupts or events.
It is used by defining a new method:
int (*poll) (struct net_device *dev, int * budget);
which is called by the network stack periodically when
signaled by the driver to do so.
NAPI (cont.)
When a receive interrupt occurs, driver:
Turns off receive interrupts.
Calls netif_rx_schedule(dev) to get stack to start
calling it's poll method.
The Poll method
Scans receive ring buffers, feeding packets to the stack via:
netif_receive_skb(skb).
If work finished within budget parameter, reenables interrupts
and calls netif_rx_complete(dev)
Else, stack will call poll method again.
Routing
After the socket buffer is delivered to a protocol handler the
handler may decide to route the packet.
The default routing uses the normal destination based routing
with single table and a FIB destination cache.
For each packet the routintg destination is looked up in the
FIB cache.
If found, the packet is sent to that interface driver.
Otherwise a more costly routing decision based on rules occurs
and the result is stored in the FIB.
What is Netfilter?
Netfilter is a framework for packet mangling
Each protocol defines "hooks" (IPv4 defines 5) which are
welldefined points in a packet's traversal of that protocol
stack.
At each of these points, the protocol will call the netfilter
framework with the packet and the hook number.
Parts of the kernel can register to listen to the different hooks
for each protocol.
When a packet is passed to the netfilter framework, it will
call all registered callbacks for that hook and protocol.
Netfilter Architecture
Ingres Egres
Pre Post
Route Forward
Routing Routing
Route
Local Local
In Out
Local Sockets
Netfilter Hook
Kernel code can register a call back function to be called
when a packet arrives at each hook. and are free to
manipulate the packet.
The callback can then tell netfilter to do one of five things:
NF_ACCEPT: continue traversal as normal.
NF_DROP: drop the packet; don't continue traversal.
NF_STOLEN: I've taken over the packet; stop traversal.
NF_QUEUE: queue the packet (usually for userspace handling).
NF_REPEAT: call this hook again.
IP Tables
A packet selection system called IP Tables has been built
over the netfilter framework.
It is a direct descendant of ipchains (that came from
ipfwadm, that came from BSD's ipfw IIRC), with
extensibility.
Kernel modules can register a new table, and ask for a packet
to traverse a given table.
This packet selection method is used for packet filtering (the
`filter' table), Network Address Translation (the `nat' table)
and general preroute packet mangling (the `mangle' table).
IP Tables and Netfilter Hooks
Egres
Ingres
Pre Post
Route Forward
Routing Routing
Conntrack Mangle Conntrack

Mangle Filter Mangle
Destination NAT Source NAT
Route
Filter Conntrack
Conntrack Local Local
Mangle
Mangle In Out Destination NAT
Filter
Local Sockets
BSD Sockets Interface
User space network interface:
socket() / bind() / accept() / listen()
Initalization, addressing and hand shaking
select() / poll() / epoll()
Waiting for events
send() / recv()
Stream oriented (e.g. TCP) Rx / Tx
sendto() / recvfrom()
Datagram oriented (e.g. UDP) Rx / TX
Simple Client/Server
Clients Server
socket s1, s2 ... sn;
char buf[256];
socket s; s =socket()
char buf[256]; bind(s1, IP:port)
listen(s1)
s =socket()
while {
connect(s, IP:port)
select(s1,s2 ... sn)
while(ret !=0)
ret = recv(s, buf) if(s1)
sn = accept(s1)
else
while(ret !=0)
ret = send(sn, buf)
}
Simple Client/Server Copies
Client Server
Rx Tx Kernel
Kernel
Copy Copy from

to user user
... ...
ret = recv(s, buf) ret = send(s, buf)
... ...
User space User space

Application Application
BSD Sockets Interface Properties
Originally developed by UC Context switch for every
Berkeley research at the Rx/Tx
dawn of time
Buffer copied from/to user
Used by 90% of network space to/from kernel
oriented programs
Standart interface across
operating systems
Simple, well understood by
programmers
Zero Copy
Inkernel buffer that the user has control over.
The buffer is implemented as a set of referencecounted pointers which
the kernel copies around without actually copying the data.
splice() moves data to/from the buffer from/to an arbitrary file descriptor
tee() Moves data to/from one buffer to another
vmsplice() does the same than splice(), but instead of splicing from fd to
fd as splice() does, it splices from a user address range into a file.
Can be used anywhere where a process needs to send something from
one end to another, but it doesn't need to touch or even look at the data,
just forward it.
Zero Copy of Example 1
Splice() * User space
Only pointer is copied
File Socket Buf Kernel

Pointer to Pointer to page Memory
page as part of frag list
cache
page Data
Copy (using DMA)

Hardware
HD Controller Network Chip
* In relaity you have to do two splice calls: one from the file to an intermediate pipe and one
from the pipe to the socket buffers.
Zero Copy of Example 2
Mem write VMSplice() * User space
Only pointer is copied

Proccess
page tables skb Kernel
Pointer to page Memory
as part of frag list
Data
Copy (using DMA)

Hardware
Network Chip
* In relaity you have to do two vmsplice to an intermediate pipe and one splice from the pipe
to the socket buffers.
Hardware Offloading
Large receive offload supported (in software)
TCP / Large Segment Offload supported (e.g. e1000 driver)
No TCP Offload Engine support
Security updates
Pointintime solution
Different network behavior
Hardwarespecific limits and resourcebased denialofservice
attacks
http://www.linuxfoundation.org/en/Net:TOE
More Information
Linux Foundation Net:Kernel Flow
http://www.linuxfoundation.org/en/Net:Kernel_Flow
Zero Copy I: UserMode Perspective
http://www.linuxjournal.com/article/6345
“Understanding Linux Network Internals”, O'Reilly Media
Use the Source, Luke!
Many resources and tricks on the Internet find you will, but
solutions to all technical issues only in the Source lie.
Thanks to LucasArts
Copyrights and Trademarks
© Copyright 20062004, Michael Opdenacker
© Copyright 20032006, Oron Peled
© Copyright 20042008 Codefidence Ltd.
Tux Image Copyright: © 1996 Larry Ewing
Linux is a registered trademark of Linus Torvalds.
All other trademarks are property of their respective owners.
Used and distributed under a Creative Commons AttributionShareAlike 2.0 license

Linux Networking Internals

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Linux Networking Internals

Enviado por

Direitos autorais:

Formatos disponíveis

The Linux Network Subsystem

Process Memory Filesystem Device

CPU support CPU / MMU Storage Character Network

CPU RAM Storage

do_name() Function call

Stack <> App App 1 App2 App3

UDP TCP ICMP

Driver <> Hardware

packet packet packet packet

●Driver allocates Ring Buffers. DMA

●Driver puts packet to be sent in Tx buffers.

●Device puts received packet in Rx buffers.

●Driver/Device update descriptors to indicate state.

●Device indicates Rx and end of Tx with interrupt,

unless interrupt mitigation techniques are applied.

Ethernet Network chip

Conntrack Mangle Conntrack

Copy Copy from

User space User space

Splice() * User space

Only pointer is copied

File Socket Buf Kernel

Copy (using DMA)

Mem write VMSplice() * User space

Only pointer is copied

Copy (using DMA)

Você também pode gostar