CASE STUDY IN RT-LINUX

Ajith Kumar P. Shetty,

School of Electronics Engineering, VIT University, Vellore, Tamilnadu 63201, INDIA
{ajithkumarpshetty.2013, }@vit.ac.in

Abstract Embedded application is a hotspot at present and Linux

gradually becomes the most important operating system for embedded
applications. Aiming at the real-time problems of Linux and from five
performance parameters of real time operating system, this paper
analyzes and concludes that scheduling latency and interrupt
latency are the fundamental constraints for improving real-time
performance of Linux 2.6 kernel, then designs and implements a
new task model and new interrupt operations to solve the above
problem. Hard real-time task scheduling algorithm which is named as
Priority Bitmap Algorithm, new interrupt response and new interrupt
operations are emphasized and main codes are given out. Through
testing, response time of real-time task is indicated to be shortened
largely and meets the initial expectation.
Keywords: Real-time, Interrupt Latency, Scheduling Strategy

I. INTRODUCTION.

RT Linux is a hard real time RTOS microkernel that runs the

entire Linux operating system as fully preemptive process. It is one
of the hard real-time variants of Linux, among several, that makes
it possible to control robots, data acquisition systems,
manufacturing plants, and other time-sensitive instruments and
machines.
It was developed by Victor Yodaiken, Michael Barabanov, Cort
Dougan and others at the New Mexico Institute of Mining and
Technology and then as a commercial product at FSMLabs. Wind
River Systems acquired FSMLabs embedded technology in
February 2007 and made a version available as Wind River RealTime Core for Wind River Linux. As of August 2011, Wind River
has discontinued the Wind River Real-Time Core product line,
effectively ending commercial support for the RTLinux product.
The key RTLinux design objective was to add hard real-time
capabilities to a commodity operating system to facilitate the
development of complex control programs with both
capabilities. For example, one might want to develop a real-time
motor controller that used a commodity database and exported a
web operator interface. Instead of attempting to build a single
operating system that could support real-time and non-real-time
capabilities, RTLinux was designed to share a computing device
between a real-time and non-real-time operating system so that (1)
the real-time operating system could never be blocked from

execution by the non-real-time operating system and (2)

components running in the two different environments could easily
share data. As the name implies RTLinux was the first computer
designed to use Linux as the non-real-time system but it eventually
evolved so that the RTCore real-time kernel could run with either
Linux or BSD UNIX.
Multi-Environment Real-Time (MERT) was the first example
of a real-time operating system coexisting with a UNIX system.
MERT relied on traditional virtualization techniques: the real-time
kernel was the host operating system (or hypervisor) and Bell
Systems UNIX was the guest. RTLinux was an attempt to update
the MERT concept to the PC era and commodity hardware. It was
also an attempt to also overcome the performance limits of MERT,
particularly the overhead introduced by virtualization.
The technique used was to only virtualize the guest interrupt
control. This method allowed the real-time kernel to convert the
guest operating system into a system that was completely
preemptible but that could still directly control, for example,
storage devices. In particular, standard drivers for the guest worked
without source modification although they needed to be recompiled
to use the virtualization "hooks". See also paravirtualization. The
UNIX "pipe" was adapted to permit real-time and non-real-time
programs to communicate although other methods such as shared
memory were also added.
From the programmer's point of view, RTLinux originally
looked like a small threaded environment for real-time tasks plus
the standard Linux environment for everything else. The real-time
operating system was implemented as a loadable kernel
module which began by virtualizing guest interrupt control and
then started a real-time scheduler. Tasks were assigned static
priorities and scheduling was originally purely priority driven. The
guest operating system was incorporated as the lowest priority task
and essentially acted as the idle task for the real-time system. Realtime tasks ran in kernel mode. Later development of RTLinux
adopted the POSIX threads application programming interface
(API) and then permitted creation of threads in user mode with
real-time threads running inside guest processes. In multiprocessor
environments threads were locked to processor cores and it was

possible to prevent the guest thread from running on designated

core (effectively reserving cores for only real-time processing).
RTLinux provides the capability of running special real-time
tasks and interrupt handlers on the same machine as standard
Linux. These tasks and handlers execute when they need to execute
no matter what Linux is doing. The worst case time between the
moment a hardware interrupt is detected by the processor and the
moment an interrupt handler starts to execute is under 15
microseconds on RTLinux running on a generic x86 (circa 2000). A
RTLinux periodic task runs within 25 microseconds of its
scheduled time on the same hardware. These times are hardware
limited, and as hardware improves RTLinux will also improve.
Standard Linux has excellent average performance and can even
provide millisecond level scheduling precision for tasks using the
2.
POSIX soft real-time capabilities. Standard Linux is not, however,
designed to provide sub-millisecond precision and reliable timing
guarantees. RTLinux was based on a lightweight virtual machine
where the Linux "guest" was given a virtualized interrupt controller
and timer, and all other hardware access was direct. From the point
of view of the real-time "host", the Linux kernel is a thread.
Interrupts needed for deterministic processing are processed by the
real-time core, while other interrupts are forwarded to Linux, which
runs at a lower priority than real-time threads. Linux drivers handle
almost all I/O. First-In-First-Out pipes (FIFOs) or shared memory
can be used to share data between the operating system and
RTLinux.

4) rtl fifo connects RT tasks and interrupt handlers to Linux

processes through a device layer so that Linux processes
can read/write to RT components.
5) semaphore is a contributed package by Jerry Epplin which
gives RT tasks blocking semaphores.
6) POSIX mutex support is planned to be available in the
next minor version update of RTLinux.
7) mbuff is a contributed package written by Tomasz
Motylewski for providing shared memory between RT
components and Linux processes.

Kernel module:
RTLinux is in fact a Linux kernel module. It is the same type of
module that Linux uses for drivers, file systems, and so on. The
main difference between RTLinux module and an ordinary Linux
module is that RTLinux module calls functions, which are offered
by RTLinux kernel whereas ordinary module uses only Linux
kernel functions. The source code of a simple Linux module is
given below. It contains two functions: init_module, which is called
when
the
module
is
inserted
to
the
kernel
by insmod or modprobe command, and cleanup module, which is
called before module is removed by rmmod.
3. Architecture:

RTLinux is structured as a small core component and a set of

optional components. The core component permits installation of
very low latency interrupt handlers that cannot be delayed or
preempted by Linux itself and some low level synchronization and
interrupt control routines. This core component has been extended
to support SMP and at the same time it has been simplified by
removing some functionality that can be provided outside the core.
Functionality:
The majority of RTLinux functionality is in a collection of
loadable kernel modules that provide optional services and levels
of abstraction. These modules include:
1) rtl sched a priority scheduler that supports both a "lite
POSIX" interface described below and the original V1
RTLinux API.
2) rtl time which controls the processor clocks and exports an
abstract interface for connecting handlers to clocks.
3) rtl posixio supports POSIX style read/write/open interface
to device drivers.

Fig 1: Architecture of RTLinux.

Standard time sharing OS and hard real time executive running
on same machine Kernel provided with the emulation of Interrupt
control H/W. RT tasks run at kernel privilege level to provide direct
H/W access. RTLinux runs Linux kernel for booting, device
drivers, networking, FS, process control and loadable kernel
modules. Linux kernel runs as a low priority task on the RT kernel
hence making sure it cannot preempt any RT task.
RT task allocated fixed memory for data and code.RT tasks
cannot use Linux system calls , directly call routines or access
ordinary data structures in Linux Kernel. RTProcesses without
memory protection features (RTLinux Pro has some PSDD now).
The RT kernel is actually patched over the Linux kernel and then

recompiled to work as a RTLinux system. Completely configurable

RTLinux kernel.
VM layer only emulates the Interrupt Control. The 0 level OS
does not provide any basic services that can be provided by Linux Only RT services - No primitives for process creation, switching or
MM. Uses software stack for switching and not the expensive H/W
switching. RTKernel is not preemptable.

include time-sharing scheduling, virtual memory system timing

unpredictability, and lack of high-granularity timers. It turns out
that using software interrupts, together with several other
techniques, it is nevertheless possible to modify Linux so as to
overcome these problems. The idea to use software interrupts so
that a general-purpose operating system could coexist with a hard
real-time system is due to Victor Yodaiken (personal
communications). This thesis details how this idea and several
others were applied to build a hard real-time version of Linux.
IV. SOLUTION METHODOLOGY.

II. OBJECTIVES
The key RTLinux design objective is that the system should be
transparent, modular, and extensible. Transparency means that
there are no unopenable black boxes and the cost of any operation
should be determinable. Modularity means that it is possible to
omit functionality and the expense of that functionality if it is not
needed. The base RTLinux system supports high speed interrupt
handling and no more. And extensibility means that programmers
should be able to add modules and tailor the system to their
requirements. It has simple priority scheduler that can be easily
replaced by schedulers more suited to the needs of some specific
application. When developing RTLinux, it was designed to
maximize the advantage we get from having Linux and its
powerful capabilities available.

RTLinux is very much module oriented. To use RTLinux, you

load modules that implement whatever RT capabilities you need.
Two of the core modules are the scheduler and the module that
implements RT-fos. If the services provided by these modules
dont meet the requirements of the application, they can be
replaced by other modules. For example, there are two alternative
scheduling modules a earliest deadline rst scheduler
implemented by Ismael Rippol and a rate-monotonic scheduler.

III. DETAILED PROBLEM DEFINITION

There are numerous real-time operating systems available
today. What was missing, however, is an open, standard, supported,
efficient, and inexpensive multitasking system with hard real-time
capabilities. Many UNIX systems meet the first three requirements.
Linux 1, a relatively recent free UNIX-like OS, originated by Linus
Torvalds, features excellent stability, efficiency, source code
availability, not restrictive license, and substantial user base.
Source code availability is essential for verification of the system
correctness, adaptation to specific problems, and mere bug fixing.
Linux can run on the most widely available computers: IBM PCs
and compatible ma-chines (most PC hardware is supported). Since
many computer and engineering labs have computers of this type, it
is possible for them to use Linux without any further investment.
Linux has all features of a modern UNIX system: several X
Window System implementations, graphical user interface toolkits,
networking, databases, programming languages, debuggers, and a
variety of applications. It is embeddable and is able to perform
efficiently using relatively small amounts of RAM and other
computer resources. In short, Linux has the potential to make an
excellent development platform for a wide variety of applications,
including those involving real-time processing. However, Linux
has several problems preventing it from being used as a hard realtime OS, most notably the fact that interrupts are often disabled
during the course of execution of the kernel. Other problems

Fig.2: Flow of data and control

The basic scheduler simply runs the highest priority ready RT
task until that task suspends itself or until a higher priority task
becomes ready.
The original RTLinux scheduler used the timer in one shot
mode so that it could easily handle a collection of tasks with
periods that had a small common divisor. For example, if one task
must run every331time units and the other runs every1027time
units, there is no good choice for a timer period. In one-shot mode,
the clock would be set rst to generate an interrupt after 331 time
units and then reprogrammed after the interrupt to generate a
second interrupt in another691time units (minus the time needed to
reprogram the clock). The price we pay is that we reprogram the
clock on every interrupt. For x86 generic motherboards,
reprogramming the clock is relatively slow. It turns out, how- ever,
that many applications dont need the generality of a one-shot timer
and can avoid the expense of reprogramming. Professor Paolo
Mantegazza of the Aerospace Engineering Department in
Politecnico di Milano wrote a scheduler that demonstrated the
utility of periodic mode and encouraged us to put it into the
standard scheduler6. The current RTLinux scheduler offers both

periodic and one shot modes. On SMP systems the problem gets
simpler because there is an on-processor high frequency timer chip
that is available to the RTL system.

V. DETAILS OF EXPERIMENTATION, ANALYSIS, MODELLING.

In order to measure the performance of Real-Time Linux, I
have conducted several experiments. The experiments were
performed on two IBM PC compatible computers running Linux
version 2.0.29 and Real-Time Linux version 0.5a. The results are
summarized in Table. To measure the maximum interrupt latency,
an additional machine running Real-Time Linux (Machine 1) was
used to send interrupt requests to the machine being tested
(Machine 2) and to measure the response time of the latter (Figure
1.2).
The D0 output of the parallel port of Machine 1 is connected to
the ACK parallel port input of Machine 2. When the ACK input
goes from a logic one to logic zero, an interrupt request is sent to
the processor. To provide feedback, the D0 output of Machine 2 is
connected to a parallel port input (PE) on Machine 1.

exits the busy loop, obtains the current time, and computes how
long it took Machine 2 to respond. This sequence is performed
periodically over a substantial interval of time. The maximum
response time encountered is taken to be an estimate of the worstcase interrupt latency.
As seen from Table, the interrupt handling latency in plain
Linux is substantially higher than in Real-Time Linux. Moreover, it
is quite possible that some device drivers that I have not used
disable interrupts for longer periods, further increasing Linux
interrupt processing latency.
To measure scheduling precision a periodic real-time task was
run. On each wake-up the current time was obtained and compared
to the estimate. Maximum deviations were recorded.
I found it impossible to reliably run periodic tasks as standard
Linux processes, partly because Linux does not provide periodic
timers facility, and also because of other problems .
During all tests the system was heavily loaded with disk and
network I/O operations. Although device drivers in Real-Time
Linux do not disable hardware interrupts, heavy I/O does increase
interrupt latency. This fact can be attributed to the DMA cycle
stealing.
Several stress tests have also been performed. In one of them
the system has successfully performed a backup of a local system
over the local network, scheduling two periodic real-time tasks
each with 1 millisecond period at the same time. This experiment
demonstrates that the presence of a real-time system has no
adverse effect on the functioning of the Linux kernel.
Overall, the results show that Real-Time Linux is a viable
platform for hard real-time processing.
VI. APPLICATIONS

Fig.2: Measuring interrupt latency

Table 1: interrupt latency and scheduling precision for different

machines.
Machine A: Intel 80486 33 MHz, ISA bus, 16 MB of RAM,
Western Digital Caviar 2340 IDE hard drive, 3C509 Ethernet card.
Machine B: Intel Pentium 120 MHz, PCI bus, 32 MB of RAM,
EATA-DMA DPT SCSI adapter, Conner CP30200 SCSI hard drive,
NE2000 Ethernet card (in an ISA slot).
The measurement is performed as follows. A real-time process
on Machine 1 records the current time, sends a pulse to the D0
output, and enters a busy loop waiting for the PE input to change.
An interrupt request is sent to the CPU of Machine 2. From an
interrupt handler (both real-time and ordinary Linux handlers were
used) the output D0 is toggled. The real-time process on Machine 1

The first one is that of Harald Stauss 2 from the department of

physiology at the Hum-boldt University in Berlin, Germany. He
has implemented a system for the recording and display of
hemodynamic measurements in rats. It uses an analog-to-digital
converter card to acquire the signals from sensors. The card based
on the MAX192BCPP chip has eight 12-bit channels that are
multiplexed to one serial line connected to the serial port of the
computer. The system runs under Real-Time Linux and consists of
a real-time task and a user process. The real-time task polls the
card and passes the received data through an RT-FIFO to the user
process. The process records the data to a _le and at the same time
displays it graphically in a Motif drawing widget. Mr. Stauss
reports that the system is able to reliably acquire data with the total
sampling rate of 3000 Hz on a i486/33MHz-based machine. On the
same computer, Labtech Notebook for DOS, a commercial data
acquisition program, could only provide sampling rates of less than
400 Hz.
Bill Crum at New Mexico Tech has developed an embedded
control and monitoring software for the Tech Sunrayce car [3]. The
system specification required data collection from 70 sensors with
total sampling rate of 80 Hz. The needed response times ranged
from 200 to 300 milliseconds. These requirements were satisfied
by using one real-time task executing with 0.0125 seconds period
on a 20 MHz i386{based computer running Real-Time Linux. To

facilitate the debugging of the system, Bill Crum has written a set
of programs that he describes as an \engineering workbench". The
programs simulate tools commonly found in electrical engineering
laboratories: an oscilloscope, a logic analyzer, and a signal
generator. The complex uses a data acquisition card that provides
analog and digital I/O. Real-time tasks are used to communicate
with the card, and Linux processes implement graphical interfaces
using the Qt widget set.
There are several other applications. A task running under RealTime Linux is used for real-time communication with the Phantom,
a force feedback device. This is a part of a system that creates
virtual worlds. The system allows users to navigate through these
worlds, to feel and manipulate objects. Dan Samber 4 from Mount
Sinai Medical School in New York City uses Real-Time Linux to
reliably communicate with a patient monitor over a serial line for
recording and display of physiological parameters.
VII. CONCLUSIONS
By observing various features of the RTLinux and outcomes of the
various analysis and experiments yields a conclusion that the RealTime Linux is a viable platform for hard real-time processing
which involves low interrupt latency and requires
more
scheduling precision.