Debugging tools for embedded systems Investing in the right development tools up front results in faster development and

debug cycles and a shorter time to market BY Microchip Chandler, http://www.microchip.com JOHN DAY Technology AZ

Tools for debugging embedded systems range from software monitors and simulators, which are independent of the hardware under development, to custom silicon for in-circuit emulation, which replaces the target processor for maximum hardware control. Several tools combine these approaches and vary greatly in cost and performance. In general, the higher the level of integration with the end-product hardware, the greater the benefit of a tool in shortening product development time, but the greater the cost as well. Simulators Software instruction simulators provide simulated program execution with read and write access to the internal processor registers. Beyond that, the tools vary in capability. Some are limited to simulation of instruction execution only. Most offer breakpoints, which allow fast execution until a specified instruction is executed. Many also offer trace capability, which shows instruction execution history. While instruction simulation is useful for algorithm development, embedded systems by their very nature require access to peripherals such as I/O ports, timers, A/Ds, and PWMs. Advanced simulators help verify timing and basic peripheral operation, including I/O pins, interrupts, and status and control registers. These tools provide various stimulus inputs ranging from pushbuttons connected to I/O pin inputs, logic vector I/O input stimulus files, regular clock inputs, and internal register value injection for simulating A/D conversion data or serial communication input. Many embedded systems can effectively be debugged using proper peripheral stimulus. Simulators offer the lowest-cost development environment. However, many real-time systems are difficult to debug with simulation only. Simulators also typically run at speeds 100 to 1,000 times slower than the actual embedded processor, so long timeout delays must be eliminated when simulating. In-circuit emulators In-circuit emulators are plugged into a system in place of the embedded processor. They offer real-time code execution, full peripheral implementation, and breakpoint capability. High-end emulators also offer real-time trace buffers, and some will time-stamp instruction execution for code profiling (see figure).

An in-circuit emulator is plugged into a target system in place of the embedded processor. Some in-circuit emulators have special ASICs or FPGAs that imitate core processor code execution and peripherals, but there may be behavioral differences between the actual device and the emulator. The problem is sidestepped with "bond-out" emulation devices that provide direct I/O and peripheral access using the same circuit technology as the processor and that provide access to the internal data registers, program memory, and peripherals. This is accomplished by using emulation RAM instead of the processor's internal program memory. The microcontroller firmware is downloaded into the emulation RAM and the bond-out processor executes instructions using the same data registers and peripherals as the target processor. The I/Os of the bond-out silicon are made available on a socket that is plugged into the system under development instead of the target processor being emulated. State-of-the-art emulators provide multilevel conditional breakpoints and instruction trace, including code coverage and timestamping. Some even allow code tracing without halting the processor. There is a large gap in performance and price between software simulators and hardware emulators. Between the two extremes, there is a lot of room for intermediate solutions. Burn-and-learn method Simulation is most often used with the burn-and-learn method of run-time firmware development. A chip is burned with a device programmer; and after plugging it into the hardware, the system crashes. At this point, an attempt is made to figure out what went wrong; the source code is changed, the executable is rebuilt, and another chip is burned. This cycle is repeated until the chip works properly. Routines can be added to dump vital debugging information to a serial port for display on a terminal. I/O pins can be toggled to indicate program flow.

If more symptoms are provided by the system as it runs, then more logical changes to the source code can be made. Overall, this method of debugging is inefficient, slow, and tedious. In-circuit simulators When simulating code where branches are conditional on the state of an input pin or other hardware condition, a simulator can get information on that state from the hardware using simulator stimulus. This is the first step away from software-only simulation, allowing a certain amount of hardware debug. This approach is taken by integrating the capability of a simulator with a communication module that acts as the target processor. Stimulus is provided to the simulation directly from the processor's digital input pins, which allows the simulator to set binary values on the output pins. However, such tools run at the speed of a simulator, so they can't set and clear output pins fast enough to implement timing-critical features like a software UART. They often don't support complex peripheral features such as A/Ds and PWMs. Run-time monitors In-circuit simulators provide a communication channel between the host development system and the hardware, with something like a UART in the hardware to provide the communication. The host can then issue commands to the processor to get it to perform debug functions such as setting or reading memory contents. Some code execution control is also possible. This multiplexed execution of code under development and communication of debug information with a cross-development host is usually referred to as a run-time monitor. Software breakpoints can be built in at compile time. Other features, such as hooking into a periodic timer interrupt to copy data of interest from the target to the host, can also be included when building the executable. Since these techniques don't require writing to program memory at run time, they can be used with burn-and-learn debug using one-time-programmable devices. The routines required to transfer debug data to the host or to download a new executable--the monitor code itself--occupy a certain amount of program memory on the target device. They also use data memory and bandwidth in addition to the UART or other communication device. However, most of this overhead occurs when code execution is not in progress. Use of a software run-time monitor is a great step forward from simulation in isolation from the target hardware. Adding just a few simple support features in the silicon of the target processor can turn the monitor into a system that provides all the basic features of an emulator. In-circuit debuggers

The system becomes more than a software monitor when custom silicon features are added to the target processor. Development tools with special silicon features to support code debug and having serial communication between host and target are typically referred to as "debuggers." In the past, some systems have used external RAM program memory to support this feature. Now reprogrammable flash memory has made in-circuit debuggers (ICDs) practical for singlechip embedded microcontrollers. ICDs allow the embedded processor to "self-emulate." A good example of an in-circuit debugger is the MPLAB-ICD, a powerful, low-cost, run-time development tool. MPLAB-ICD uses the in-circuit debugging capability built into the PIC16F87X or PIC18Fxxx microcontrollers. This feature, along with the In-Circuit Serial Programming (ICSP) protocol, offers cost-effective in-circuit flash program load and debugging from the graphical user interface of the MPLAB Integrated Development Environment. The designer can develop and debug source code by watching variables, single-stepping, and setting breakpoints. Running the device at full speed enables testing hardware in real time. The in-circuit debugger consists of three basic components: the ICD module, ICD header, and ICD demo board. The MPLAB software environment is connected to the ICD module via a serial or full-speed USB port. When instructed by MPLAB, the ICD module programs and issues debug commands to the target microcontroller using the ICSP protocol, which is communicated via a five-conductor cable using a modular RJ-12 plug and jack. A modular jack can be designed into a target circuit board to support direct connection to the ICD module, or the ICD header can be used to plug into a DIP socket. The ICD header contains a target microcontroller and a modular jack to connect to the ICD module. Male 40- and 28-pin DIP headers are provided to plug into a target circuit board. The ICD header may be plugged into either the included ICD demo board or the user's custom hardware. The demo board provides 40- and 28-pin DIP sockets that will accept either a microcontroller device or the ICD header. It also offers LEDs, DIP switches, an analog potentiometer, and a prototyping area. Immediate prototype development and evaluation are feasible even if hardware isn't yet available. The complete hardware development system, along with the host software, provides a powerful run-time development tool at a very reasonable price. Run-time operation The debug kernel is downloaded along with the target firmware via the ICSP interface. A nonmaskable interrupt vectors execution to the kernel under three conditions: when the program

counter equals a preselected hardware breakpoint address, after a single step, or when a halt command is received from MPLAB. As with all interrupts, this pushes the return address onto the stack. On reset, the breakpoint register is set equal to the reset vector, so the kernel is entered immediately when the device comes out of any reset. The ICD module issues a reset to the target microcontroller immediately after a download. Thus, after a download the kernel is entered, and control is passed to the host. The user can now issue commands to the target processor and modify or interrogate all RAM registers, including the PC and other special function registers. The user can single-step, set a breakpoint, animate, or start full-speed execution. Once started, a halt of program execution causes the PC address prior to kernel entry to be stored, which allows the software to display where execution halted in the source code. When the host is commanded to run again, the kernel code executes a return from interrupt instruction, and execution continues at the address that pops off the hardware stack. Silicon support requirements The breakpoint address register and comparator make up most of what is needed in silicon, along with some logic to single-step and recognize asynchronous commands from the host. Since the ICSP programming interface is already in place to support programming of the device, this doesn't constitute an additional silicon requirement. When this channel is used for in-circuit debug, these two I/O pins may not be used for other run-time functions. Embedded system development tools Emulators, simulators, and in-circuit debuggers have features useful to the design engineer (see table). Features of emulators, simulators, and in-circuit debuggers Parameters Real-time execution Low cost Full device peripheral implementation Automatic Burn and learn Yes Yes/Free Yes Software simulation No Yes/Low/Free No/Limited In-circuit simulation No Yes/Low Yes/Limited In-circuit emulator Yes No/High Yes In-circuit debugger Yes Yes/Low Yes

No

Yes

Yes

Yes

Yes

download of new program No loss of target device I/O pins when debugging Target system cabling View and modify RAM and peripherals I/O resources needed to support debugging No loss of program memory or data RAM Simple connectivity of debugger to target Real-time trace buffer Single stepping Hardware breakpoints Yes Yes/Sometimes Yes/Sometimes Yes No

None No

None Yes

Complex Yes/Limited

Complex Yes

Simple Yes

None

None

Serial port, some I/O pins No

None

Two or fewer I/O No

No

Yes

Yes

None

None

No/Complex

No/Complex

Yes

None None None

No/Limited Yes Yes/Unlimited

No/Limited Yes One/Unlimited

Yes Yes One/Unlimited

No Yes Yes/One

With the basic run-time features of single-step, full-speed execution and watching variables, an in-circuit debugger can help in a lot of designs where only an emulator would have done before. Investing in the right development tools up front will pay back handsomely in faster development and debug cycles and a shorter time to market. EMBEDDED SYSTEM DEBUGGING

Debugging tools Application Debugging: Simulators and emulators are two powerful debugging tools which allow developers to debug (and verify) their application code. These tools enable programmer to

perform the functional tests and performance tests on the application code. Simulator is a software which tries to imitate a given processor or hardware. Simulator is based on the mathematical model of the processor. Generally all the functional errors in an application can be detected by running it on the simulator. Since simulator is not actual device itself, it may not be an exact replica of the target hardware. Hence, some errors can pass undetected through the simulator. Also, the performance of an application can not be accurately measured using Simulator (it only provides a rough estimate). Generally most development tools come under an integrated environment, where Editor, Compiler, Archiver, Linker and Simulator are integrated together. Emulator (or Hardware Emulator) provides a way to run the application on actual target (but under the control of a emulation software) hardware. Results are more accurate with emulation, as the application is actually running on the real hardware target. Hardware Debugging: Developer of an Embedded System often encounters problems which are related to the Hardware. Hence it is desirable to gain familiarity with some Hardware Debugging (probing tools). DVM, Oscilloscope (DSO or CRO) and Logical Analyzer (LA) are some of the common debugging tools, which are used in day to day debugging process. Memory Testing Tools There are a number of commercially available tools which help programmers to test the memory related problems in their code. Apart from Memory leaks, these tools can catch other memory related errors - e.g. freeing a previously allocated memory more than once, writing to uninitialized memory etc. Here is a list of some freely (no cost) available Memory Testing tools:
y y y y y

dmalloc DUMA valgrind memwatch memCheckDeluxe

Debugging an Embedded System (a) Memory Faults One of the major issue in embedded systems could be memory faults. Following types of Memory Fault are possible in a system (i) Memory Device Failure: Some times the memory device may get damaged (some common causes are current transients and static discharge). If damaged, the memory device needs replacement. Such errors can occur in run time. However such failures are very rare. (ii) Address Line Failure: Improper functioning of address lines can lead to memory faults. This could happen if one or more address lines are shorted (either with ground or with each other or with some other signal on the circuit board). Generaly these error occur during the production of circuit board, and post-production testing can catch such errors. Some times the address line

drivers might get damaged during run time (again due to current transients or static discharge). This can lead to address line faults during run time. (iii) Data Line Failure Can occur if the data lines (one or more) are shorted (to ground or with each other or with some other signal). Such errors can be detected and rectified during postproduction testing. Again, the electric discharge and current transients can damage can damage the data line drivers, which might cause to memory failures during run time. (iv) Corruption of few memory blocks : Some time a few address locations in the memory can be permanently damaged (either stuck to Low or stuck to High). Such errors are more common with Hard-disks (less common with RAMs). The test software (power on self test) can detect these errors, and avoid using these memory sectors (rather than replacing the whole memory). (v) Other Faults : Some times the memory device may be loosely inserted (or may be completely missing) in to the memory slot. Also there is a possibility of Fault in Control Signals (similar to Address and Data Lines).

There are two types of sections in System Memory - Program (or code) sections, and Data sections. Faults in program sections are more critical because even the corruption of one single location can cause the program to crash. corruption of data memory also could lead to program crashes, but mostly it only cause erratic system behavior (from which the application could gracefully recover - provided that software design takes care of error handling).

Memory Tests Following simple tests can detect memory faults: (a) Write a known patter "0xAAAA" (All odd data bits being "1" and even bits being "0") in to the memory (across all address ranges) and read it back. Verify that the same value (0xAAAA) is read back. If any Odd Data line is shorted (with even data line or with Ground), this test will detect it. Now repeat the same test with data pattern "0x5555". This test will detect any shorting of the even Data line (short with ground or with odd data line). Also, these two test in combination can detect any bad memory sectors. (b) Write a unique value in to each memory word (across entire memory range). Easiest way to choose this unique value is to use the address of given word as the value. Now read back these values and verify them. If the verification of read back values fails (whereas the test-a passes), then there could be a fault in address lines.

The tests "a" and "b" can be easily performed as part of power on checks on the system. However it will be tricky to perform these tests during run time, because performing these test will mean loosing the existing contents in the memory. However certain systems run such memory tests during run time (once in every few days). In such scenarios, the tests should be performed on smaller memory sections at a time. Data of these memory sections can be backed up before performing the test, and this data can be restored after test completion. Tests can be run one by one on each section (rather than running the test on entire memory at a time).

(b) Hardware Vs Software Faults In Embedded System, Software is closely knit with the Hardware. Hence, the line dividing the Hardware and Software issues is very thin. At times, you may keep debugging you software, whereas the fault may lie somewhere in the Hardware (and vice versa). The problem becomes more challenging when such faults occur at random and can not be reproduced consistently. In order to disect and debug such tricky issues, a step-wise approach needs to be followed.

* Prepare a test case: When you observing frequent application crash because of some unknown issues, you should plan to arrive at a simpler test case (rather than carrying on debugging with entire application). There are two benefits of this approach: A simpler application will take less time to reproduce the error and hence total debugging time is fairly reduced. Secondly, there are less unknown parameters (which might be causing the error) in a smaller application, and hence less debugging effort is needed. You can gradually strip down the application (such that the error is still reproducible) to get a stripped down version of the application which can act as a preliminary version of the test case. * Does the error change with change in operating condition: Does the error change (frequency of error or type of error) with change in the operating conditions (e.g. board temperature, processor speed etc)? If so, then your errors might be hardware related (critical timings or glitches on signals). * Is the error reproducible: Arriving at a test case which can reliably reproduce the error greatly helps the debug process. A random error (not reproducible consistently) is hard to trace down, because you can never be sure as to what is causing the system failure. * Keep your Eyes Open: Always think lateral. Do not be inclined towards one possibility of error. The error could be in the application software, in the driver software, in the processor hardware on in one of the interface. Some times there could be multiple errors to make your life miserable - such errors can be caught only with stripped down test cases (each test case will catch a different error). * Mantain Error Logs: While writing the application code, you should add the provision for

system log in Debug version (Log can be disabled for the release version). Log of the events, just prior to system crash can tell you a great deal about the possible causes of failure.

(c) POST Generally most systems perform a POST (Power On Self Test) on start up. POST may be during the pre-boot phase or during the boot phase. POST generally includes tests on memory and peripherals of a system.