Você está na página 1de 83

he Sun SPARC Enterprise T5120/T5220 Servers are the first of Suns servers to

implement the Sun UltraSPARC T2 processor. This processor uses four, six, or eight
64-bit SPARC cores, each of which supports eight threads running concurrently. The
servers support one processor, and therefore support a maximum of 64 threads.
The Sun SPARC Enterprise T5120/T5220 Servers use the on-chip multithreaded
processor to minimize the CPU idle time, thus ensuring that multiple threads have
access to the processor and do not have to wait for the processor to be completely
freed to gain CPU access time.
The processor is designed for highly threaded transactional processing. This means
that the time usually spent waiting for memory access has been minimized, which
maximizes the core utilization processes.
The Sun SPARC Enterprise T5120/T5220 Servers implement the PCI-Express, or PCIe,
as the primary I/O bus on the server.
PCIe offers an advantage of speed and data transmission rate; it operates at 2.5
GHz and transmits data at approximately 250 Mbytes/ sec in each direction. PCIe
provides a true bidirectional link, so with one link available, the server offers a data
transmission rate of approximately 500 Mbytes/sec.
The Sun SPARC Enterprise T5120 Server has two PCIe x4 slots and one PCIe x8 slot
while the Sun SPARC Enterprise T5220 Server has four PCIe x4 and two PCIe x8
slots.
Note: The x4 and x8 nomenclature refers to the number of lanes. Each lanes
supports 250 Mbytes/second

he Sun SPARC Enterprise T5120/T5220 Servers utilize the UltraSPARC sun4v


architecture, a hyper-privileged architecture.
Enhancements in this line include:
Use of the Sun UltraSPARC T2 processor which supports multiple threads on multiple
cores and its conformance to the new architecture by segregating hardware-specific
drivers from the OS.
Implementation of virtualization, which introduces a layer between the operating
system and the platform that removes the need for the operating system to have
direct register access to the processor, memory, and critical I/O devices. This
benefits customers because hardware can be upgraded without changing the
software infrastructure.
The Sun SPARC Enterprise T5120/T5220 Servers provide support for a new Ethernet
driver, referred to as e1000g. This new driver provides the following features:

Support for data link provider interface (DLPI) version 2, which enables a data link
service user to access and use a variety of conforming data link service providers
without special knowledge of the provider's protocol;
Physical layer static configuration using FORTH code, or FCode properties;
Portability to Solaris X86 and SPARC platforms through device driver interface, or
DDI frameworks;
Fault management infrastructure support, which provides error handling and
management capabilities;
Message signaled interrupts, or MSI support, which allows the devices to
communicate in a peer-to-peer manner without the involvement of the host CPU.
These implementations bring the gigabit Ethernet interface in line with Suns other
network interface drivers.

The Sun SPARC Enterprise T5120/T5220 Servers implement RAS features, which
enable the system to maintain a higher uptime rate. RAS features are incorporated
across both hardware and software components, including:
System busses
PCI bridges
Processors
Memory management
System power
And system cooling
The inclusion of RAS features in these entry-level servers continues Suns
commitment to providing enterprise-level services to lower-cost solutions. The RAS
features included in the Sun SPARC Enterprise T5120/T5220 Servers include:
N+1 redundant hot-pluggable power supplies and hot-swappable fan modules
Extended ECC protection on L2 cache data path and memory interface
DRAM extended ECC, which allows the detection of up to 4-bits in error, as long as
they are on the same DRAM
Standardized error message generation
Environmental monitoring with the inter- integrated circuit (I2C) control serial bus

Remote access using ILOM-based hardware and software components


Remote monitoring using Sun Net Connect

The Sun SPARC Enterprise T5120/T5220 Servers implement several key


technologies that give it a great advantage in the entry-level server market for
companies that require database servers, web servers, and application servers.
In this module, we have discussed:
An overview of the Sun SPARC Enterprise T5120/T5220 Servers, including their
features, target markets, and target applications
The key technologies driving the Sun SPARC Enterprise T5120/T5220 Servers,
including the new processor, architecture, and use of a faster PCI-based bus.
The Sun SPARC Enterprise T5120 server is a 1U rack mountable server.
The Sun SPARC Enterprise T5220 server is a 2U rack mountable server.
Both servers implement the newly available chip multithreading (CMT), UltraSPARC
T2 processor. These entry-level servers are enterprise servers, designed to meet the
needs of Suns customers who use Java servers and databases, such as Oracle, on
the back end.
These Sun servers implement the sun4v architecture and on-chip multithreading
technology that allows them to process more information per processor, in highly
threaded workloads, than competitors. The Sun SPARC Enterprise T5120/T5220
Servers implement the PCI-Express (PCIe) I/O bus, which provides a greater data
transfer rate than previous PCI standards.
The Sun SPARC Enterprise T5120/T5220 Servers implement several new features
and enhancements. Overall, the servers feature improvements in processor
throughput and IO technology.
The next few slides describe the system specifications for the Sun SPARC Enterprise
T5120/T5220 Servers, starting with the processor.
The Sun SPARC Enterprise T5120/T5220 Servers are equipped with a single
UltraSPARC T2 processor on-board that operates at 1.2 and 1.4 gigahertz per second
(GHz/sec). The processor contains a 4 megabyte Level 2, or L2, cache, and four, six,
or eight cores, each of which has one floating point unit, or FPU, and supports eight
threads. Each core has a 16 kilobyte instruction cache and an 8 kilobyte data cache.
The CPU also supports up to 16 FBDIMM slots, each of which supports 1, 2, and 4
gigabyte DIMMs with a maximum of 64 gigabytes supported in the Sun SPARC

Enterprise T5120/T5220 Servers. The DIMMs are controlled by four memory


channels, each of which controls four DIMM slots.
The system I/O is built on the PCIe bus, which provides support for:
Three standard half length/half height PCIe expansion slots on three riser boards in
the Sun SPARC Enterprise T5120 server or
Six standard half length/half height PCIe expansion slots on three riser boards in the
Sun SPARC Enterprise T5220 server
In addition to PCI Express expansion cards, two of the expansion slots can
alternately accept Sun proprietary XAUI-based cards to support 10Gbps Ethernet
networking.
Note: XAUI cards are only supported in slots 0 and 1.
Internal storage in the Sun SPARC Enterprise T5120 server is handled by four small
form factor (SFF), 2.5, internal SAS hard disk drives. Both the Sun SPARC Enterprise
T5120 and T5220 Servers currently support 73 or 146 gigabyte disks at 10,000
revolutions per minute, or RPM. The disks are attached to a SAS and SATA disk
backplane and managed through an LSI 1068E SAS/SATA disk controller.
The Sun SPARC Enterprise T5220 server supports eight small form factor (SFF) SAS
disk drives.
System components that affect the Sun SPARC Enterprise T5120 servers
environment include two 650 watt, hot-swappable power supply units (PSUs) and
four hot-swappable system fan trays, 2 fans/ tray, providing N+1 redundancy.
System components that affect the Sun SPARC Enterprise T5220 servers
environment include two 750 watt, hot-swappable power supply units (PSUs) and
three hot-swappable system fan trays, 2 fans/tray, providing N+1 redundancy.
For monitoring environmental conditions, both servers have an Integrated Lights
Out Management, or ILOM, controller with an RJ-45 serial management port and a
10/ 100BaseT Ethernet Management port
System status LEDs provide system and component status information to
administrators.
The Sun SPARC Enterprise T5120/T5220 Servers incorporate several reliability,
availability, and serviceability, or RAS features, which have become standard in
Suns offering. These features include extended error correction code, or ECC, and
CRC checking on busses, cache, memory channels, and address and data paths,
power supply and fan module redundancy, the inclusion of a remote access tool,
environmental monitoring solutions, and remote monitoring solutions.

The Sun SPARC Enterprise T5120/T5220 Servers are designed to fulfill the needs of
customers who require real-world application performance while optimizing the
computing power that can be achieved within the expensive and shrinking data
center space. As servers become increasingly more power hungry, maintaining an
environment to balance power input and thermal output for each rack becomes
increasingly important.
The Sun SPARC Enterprise T5120/T5220 Servers utilize Cool Computing to meet the
needs of these customers. With the Sun SPARC Enterprise T5120/T5220 Servers,
changes in the architecture have improved the power input to thermal output ratio.
CMT- based processors are designed for the thread-rich network computing
environment. These processors maximize application throughput by processing
multiple threads simultaneously on a single chip.
Essentially, any application that is vertically and/or horizontally scalable today can
benefit from CMT.
These include real-world applications such as:
Databases
Applications servers
Transaction processing
And web-based services
Primary market segments that require these services include:
Financial services
Telecommunications
Government
Retail organizations
And professional services

With the introduction of the new processor and architecture, the Sun SPARC
Enterprise T5120/T5220 Servers are the right servers for customers whose
applications are highly threaded, using parallel threads, or require large instruction
or data working sets. The CMT chips can handle certain types of software and tasks
better than other high-end 64-bit chips. Any application that creates network traffic,
such as Web services software, Java, Web servers, and application servers, can
benefit from spreading lots of software threads across low-powered processor cores.

Several servers compete in the same market as the Sun SPARC Enterprise
T5120/T5220 Servers. Competition arises primarily from IBM, HP, Fujitsu, and Dell.
The trend toward multi-core processors is nothing new: IBM currently ships servers
with the dual-core Power4+ and Power5 chips; HP uses the dual-core with PA-RISC
processor; Fujitsu implements the SPARC64-V chip; Dell ships servers with the Intel
Xeon MP and EM64T processors; while Advanced Micro Devices is supplying an
Opteron processor to the industry.
Sun is well positioned to handle the boost in performance with the Solaris
Operating System. Unlike most desktop operating systems, the Solaris OS can
leverage CMT to run 64 simultaneous threads. For threaded applications, future
UltraSPARC processors will deliver up to 50 times the performance of today's
fastest UltraSPARC processor, without a significantly higher cost-per-chip.
And just as important as speed, CMT enables administrators to consolidate the
infrastructure of dozens of servers onto a single server. Imagine the savings in
administration, maintenance, power, cooling, and floor space. The management and
maintenance savings are as phenomenal as the technology itself.
Several key technologies are being used in the Sun SPARC Enterprise T5120/T5220
Servers. These technologies help to increase bandwidth, connectivity, system
throughput, and computing throughput through enhancements to the architecture,
processors, and system busses.
The Sun SPARC Enterprise T5120/T5220 Servers offer the following major features:
Dense packaging in a reduced footprint
Implementation of a four, six, or eight core Sun UltraSPARC T2 processor
On-chip multithreading technology to maximize the use of the system processor
High performance busses and memory interconnect systems
Improved input and output (I/O) throughput through the incorporation of PCIe
Architectural enhancements to system platforms to improve system upgrades,
memory management, and the I/O infrastructure
Increased on-board network connectivity

========================
This module, Sun SPARC Enterprise T5120/ T5220 Servers Architectural Overview,
presents information on the hardware architecture for the Sun SPARC Enterprise
T5120/T5220 servers.

The references shown here provide additional information about the Sun SPARC
Enterprise T5120/T5220 Servers.
Most of the documentation shown is available on the www.sun.com/ documentation
web site. A link is provided on your screen.
Well pause for just a bit so that you can review the information on this slide.
he Sun SPARC Enterprise T5120/T5220 servers have the following circuit boards
installed in the chassis:
Motherboard
Power distribution board (2 in the T5220)
Paddle board
USB board
Disk backplane
Fan boards (2)
PCIe riser cards (3)

The motherboard is actually an assembly made up of the motherboard itself and a


tray or carrier. The motherboard assembly comes in several different versions, with
the only difference being processor speed and the number of cores.
The motherboard includes a direct-attach CPU module, slots for 16 DIMMs, memory
control subsystems, and all system controller (ILOM) logic.
In addition, a removable NVRAM contains all Mac addresses, host ID, and OpenBoot
PROM configuration data. When replacing the motherboard, the NVRAM can be
transferred to a new board to retain system configuration data.
The service processor (ILOM) subsystem contains a PowerPC Extended Core, and a
communications processor that controls the host power and monitors host system
events (power and environmental). The ILOM controller draws power from the hosts
3.3V standby supply rail, which is available whenever the system is receiving AC
input power, even when the system is turned off.

The power distribution board distributes main 12v power from the power supplies to
the rest of the system. It is directly connected to the paddle card, and to the
motherboard via a bus bar and ribbon cable.

The paddle board is an assembly made up of the board, a metal mounting bracket,
and a top cover interlock or kill switch.
The paddle board serves as the interconnect between the fan connector boards, and
SAS backplane.
The paddle board is an assembly made up of the board, a metal mounting bracket,
and a top cover interlock or kill switch.
The paddle board serves as the interconnect between the fan connector boards, and
SAS backplane.
The USB board connects directly to the SAS backplane. It is packaged with the DVD
drive as a single customer-replaceable unit (CRU).
The fan boards carry power to the system fan modules.
In addition, they contain fan module status LEDs, and transfer I2C data for the fan
modules.
Note: The fan boards in the Sun SPARC Enterprise T5120 and T5220 are different
boards, but function the same.
There are three PCI riser cards per system, each attached in slots to the rear of the
motherboard.
In 1U systems, each riser card supports one card.
In 2U systems, each riser supports two cards.
PCI riser cards come in 2 different versions.
One that can support both the x8 PCIe card or an XAUI card, and
One that supports x16 PCIe.
Slots on the motherboard are keyed so that you can only plug in the correct type of
riser card into the motherboard.
Note: The slots that you see on the motherboard are not industry standard PCIe
slots. They are Sun proprietary slots that only accommodate the Sun riser cards.
Most of the electrical connectivity in the Sun SPARC Enterprise T5120/T5220 servers
is accomplished through connectors on the systems infrastructure boards.
The only system cables in the chassis are:
PDB to MB ribbon cable
Horizontal to vertical PDB ribbon cable (2U only)

Disk backplane to MB cable (1 in the 1U, 2 in the 2U)


Top cover, interlock switch cable
The diagram shown illustrates the overall system architecture for the Sun SPARC
Enterprise T5120/T5220 servers.
The top, center portion of the diagram depicts the UltraSPARC T2 CPU and the
memory architecture. The processor uses four memory channels, each of which
manages one of four banks, each with four FBDIMM slots.
There are three DC-DC converters required to deliver power to the processor and
FBDIMM connectors.
To the right of the UltraSPARC T2 processor, you will find the service processor
architecture. The service processor connects to the CPU through the serial system
interface (SSI) communications bus.
To the left of the UltraSPARC T2 processor, is the LSI 1068E SAS/SATA disk controller.
Finally, the lower half of the architectural diagram shows the I/O architecture. The
UltraSPARC T2 CPU interfaces with the I/O through three PCIe switches.
We will provide an in depth explanation of all of these sections.

The illustration depicts all of the on-die components and data paths among the
different components for the UltraSPARC T2 processor.
Here, you have eight cores communicating through the Cache Crossbar (CCX) to the
4 MB total of 16-way associative L2 cache and then through the MCU controller
channel, labeled MCU 0 through 3.
Youll notice in the diagram that each CPU core has its own floating point unit (FPU).
Each core also has its own crypto unit.
Also there are two 10GB (XAUI) interfaces and two PCI-Express interfaces are
integrated onto the chip.
The UltraSPARC T2 processor is designed to operate with the first generation
industry standard Fully Buffered Dual In-line Memory Modules (FBDIMMs).
This feature dictates the memory architecture of the Sun SPARC Enterprise
T5120/T5220 servers.
The UltraSPARC T2 processor has four memory controllers, referred to as memory
branches. Each branch has two channels. Each channel supports 10 Southbound

(from MCU to memory) and 14 Northbound (from memory to MCU) high-speed serial
lanes utilizing differential pair signaling.
The UltraSPARC T2 processor allows up to a maximum of 32 FBDIMMs to be
accessed, but is limited to 16 FBDIMMs in the Sun SPARC Enterprise T5120/T5220
servers.
An FBDIMM module is comprised of standard DDR2 memory chips with an additional
high speed serial link device, called an Advanced Memory Buffer (AMB) chip, which
buffers all memory chip address, data and control signals from the outside world.
The AMB serializes this information into high-speed, differentially driven data links
that form point to point connections from the processor to the first FBDIMM, or from
one FBDIMM to another FBDIMM in a daisy-chain fashion. Since the Sun SPARC
Enterprise T5120/ T5220 Servers utilize FBDIMM memory modules that contain the
AMB chip within the module, only the processor-to-DIMM and DIMM-to-DIMM link
signals must be routed on the motherboard.
The Sun SPARC Enterprise T5120/T5220 servers support three DIMM sizes at
revenue release, including:
1 GB, 2 GB and 4 GB.
Because all four branches need to be populated, the following three memory
configurations are supported:
4 FBDIMMs, 8 FBDIMMs and 16 FBDIMMs.
In total, the system supports a maximum total system memory of 128GB. The
DIMMs supported are FBDIMMs which must all be the same density within a memory
branch..
Individual, faulty FBDIMMs can be replaced, meaning FBDIMMS do not have to be
replaced in pairs.
Well start our discussion of the I/O architecture with the two XAUI ports on the
UltraSPARC T2 processor. These control the two XAUI slots on the motherboard.
To the left of the XAUI ports is a PCIe port on the UltraSPARC T2 processor.
The first component on the PCIe port is the ST Probe. This is a Soft Touch probe
which is used for diagnostics.
Below the ST Probe is the first PCIe switch.
Coming off the bottom of the first switch is a PCIe x8 slot. Topologically, this is the
closest to the processor. It will have the least latency. All the other slots, which
includes two x4 slots in the Sun SPARC Enterprise T5120 and an additional x8 slot

and two additional x4 slots in the case of the Sun SPARC Enterprise T5220, are
connected to another PCIe switch, which is cascaded off the first PCIe switch.
The third PCIe switch, shown on the left side of the diagram, controls the remaining
on-board I/O which includes:
Four Gigabit Ethernet interfaces
Four USB ports
Two in the front
Two in the rear
DVD-ROM drive

The SAS/SATA controller provides support for embedded mirroring for the internal
disks of the Sun SPARC Enterprise T5120/T5220 servers. It supports RAID levels 0
and 1, striping, and mirroring, respectively. This allows you to mirror the boot drive.
Support for mirroring would not require software intervention, but instead relies on
the controller.
The SAS/SATA controller also provides external 32-bit support for Flash ROM and
non-volatile static random access memory (NVSRAM).
The controller implements the Fusion-MPT, or message passing technology
architecture, which features a performance-based message passing protocol. By
managing all I/O and coalescing interrupts to minimize system bus overhead, the
controller requires small device drivers that are independent of the I/O bus. This
represents a savings as a single device driver is used for SAS/SATA, SCSI, and fiber
channel (FC).
The service processor is the hardware portion of the lights-out-management system
implemented on the Sun SPARC Enterprise T5120/T5220 servers. Unlike previous
versions of Sun service processors, the service processor hardware is not on a
separate card, but is integrated on the motherboard.
The hardware components of the service processor include:
A field-programmable gate array (FPGA) device that controls aspects of system
power and acts as the primary ILOM to host server communications gateway
I2C devices responsible for monitoring the servers environment and FRUID data
A Motorola MPC885 microprocessor, which contains its own instruction and data
caches, and a built-in memory controller

Management ports
Click the links provided to view additional information on the topics presented.
The I2C bus is used on the service processor for the TOD functionality, several
SEEPROMs, and to monitor the host servers environmental monitoring devices.

The OSP card supports several devices monitored on the I2C bus for itself, including:

IMAX
ILOM
NVRAM
FRU PROM
TOD clock
Socketed system configuration PROM containing host identification and MAC
address
The I2C bus is also responsible for monitoring the host servers monitoring and
control devices, which includes temperature sensors, power supply and fan status,
and LEDs.
Motorola MPC885 Microprocessor

The Motorola MPC885 microprocessor offers a highly integrated, high computing


power, lower power dissipation device that acts as the mini-computer for the
service processor. The microprocessor contains an embedded PowerPC core that
includes its own cache, system interface unit, and an interface to a communications
processor. The microprocessor includes the following features:

Support for 66 MHz, 80 MHz, and 133 MHz core frequency


2 KB of data cache
1 KB of instruction cache
Support for an 8-bit, 16-bit, and 32-bit external bus

Built-in memory controller


Two serial communications controllers, one of which provides an external RS-232
UART channel
Two serial management channels used to communicate with the host server
Two 10/100BaseT Ethernet controllers, where one provides an external network
management interface and the second one communicates with the host server
An I2C port for providing information on the configuration and status
Low-power mode
357-pin plastic ball grid array (PBGA), which is a method of packaging high I/O
devices
Management Ports

The service processor supports two management ports: a serial port and a network
management port. Both ports use RJ-45 connectors.

The serial port implements the full complement of RS-232-style modem controls. It
can be connected to a terminal server or to a modem to gain access to the service
processor and therefore console access to the Sun SPARC Enterprise T5120/T5220
servers.
The service processor draws power from the 3.3V standby that is routed through the
motherboard from the PSUs, regardless of whether the server is on.
When power is applied to the system and the 3.3VDC receives power, the service
processor boots.
==============================
As predicted by Gordon Moore in his article in Electronics magazine in 1965, the
number of transistors on a chip has doubled approximately every two years. This
has come to be known as Moore's Law.
In addition, the law shows an equivalent doubling of the clock frequency within the
same time frame. However, the memory speeds have not kept pace. DRAM doubled
in speed every 6 years, leaving a growing speed gap between processors and
memory.

High speed processors can execute instructions at a fast rate. This execution
instruction, called a thread, is made up of compute time and, for memory access
instructions, memory latency time.
The compute time can be fast, but overall processor performance is stalled
whenever a memory access instruction is performed. Due to the speed gap that
exists between processors and memory, the processor can spend up to 75 percent
of its processing cycles waiting on memory.
One way to take advantage of this memory latency is to start another thread that
the processor can execute, while the first thread is stalled.
This concept, known as multithreading, or MT, allows multiple instruction streams to
be executed within the same period of time.
An MT processor has multiple sets of registers and other thread states which allow
threads to execute either simultaneously, if the processor can physically support
this, or to be switched off when one thread is delayed waiting for data or
instructions, such as memory access.
Adding more transistors within the chip to support more threads in parallel,
improves the overall processor performance, and significantly lowers the effect of
the memory latency. With multiple threads, cycles that would otherwise be wasted
are available as compute cycles for another thread.
Another method of improving processor throughput is through chip multiprocessing,
or CMP.
CMP duplicates the processing unit of the processor, where multiple cores are
included on a single chip. Functionally, this allows one thread to be active at a time
on each core. This improves utilization of chip resources.
Because physical resources are now available with a multiple cores, that core
processes its thread independently from the other cores. The number of parallel
threads that can be executed has increased.
Chip multithreading combines the concept of multithreading and chip
multiprocessing. This combination provides an increase in throughput and a gain in
thread-level parallelism, or TLP, so you can now process multiple threads on a core
in a processor with multiple cores. For example, a processor with 4 cores and 4
threads per core can execute 16 parallel threads.
Sun's UltraSPARC T2 processor implements the CMT design. The UltraSPARC T2
processor is available with 4, 6, or 8 cores. Each of these cores supports up to eight
threads. This provides support for up to 64 threads on the server.

Each thread within a core contains a full set of system registers but shares the Level
1 instruction cache, which is 16 Kbytes, and the data cache, which is 8 Kbytes. By
providing each core with its own L1 cache, we can dramatically decrease memory
latency as compared to having L1 cache shared by all cores.
L1 data and instruction cache communicates with the 4 megabyte L2 cache through
a cache crossbar interface that enables L2 cache sharing among the available
cores.
To realize the large opportunity for throughput that the processor provides, the
operating system must be able to effectively manage the individual cores and
threads of the processor.
Applications and the operating system should be able to fully utilize all cores so that
the CPU utilization rate is high. Threads are switched in and out quickly, depending
on their need. If a thread requires memory access as a result of a memory cache
miss, another thread can continue to execute its instructions until it has been
moved off to make room for another thread. Keep in mind that each thread has its
own set of registers, so its state is maintained.
The operating system, specifically the scheduler, is vital in scheduling threads for
the CMT environment. While for all appearances, the operating system appears to
treat these cores as individual processors, support is added to various OS
subsystems to ensure that they are CMT-aware.
The Solaris OS treats these logical processors like traditional SMP processors or
symmetrical multiprocessing processors, scheduling runnable threads across them.
However, there is an awareness of grouping, in which threads are organized by the
core they are associated with. This helps the kernel with scheduling, as it alerts the
system to which threads are sharing resources so that it can better manage
scheduling policies to improve performance, primarily through cache utilization and
maximizing aggregate data path bandwidth.
Unless bound to a processor by the user, the OS scheduler must decide which
threads should be run on the same processor or on separate processors.
The scheduler uses several methods to balance loads and improve performance,
including:
Load balancing in which the scheduler uses a scheduling policy to distribute the
workload across the logical processors to help maximize per-chip resource
availability. It looks at the current load on a core to determine the best fit for the
thread requiring execution time. If the core is underloaded, it can handle another
thread without straining access to resources.
Thread-to-cache affinity looks at a threads timestamp to see the last time it ran on
its prior CPU. If it is less than a predetermined time, the thread is reassigned to the

CPU. This method also takes into account the size of the cache to ensure the thread
has the amount of resources required to complete its tasks efficiently.
Shared run queues provide an advantage with shared cache. Once a thread runs on
a strand within a specific core, there may be no disadvantage to it running on any
other strands in the same core. Hence, the logical processors on a core share a
dispatch queue.
Updating CPU performance counters, or CPCs, to make them more flexible and
available to the kernel, as well as to help make better scheduling decisions based
on workload characterizations
Field Programmable Gate Array (FPGA) Device

The FPGA device controls system power and acts as the primary gateway for ILOMto-host server communications. It provides the interface to the UltraSPARC T2 CPU
so that it can gain access to the boot Flash ROM, SRAM, which acts as a mailbox and
POST scratch pad. The FPGA also provides the following functions for the server:

Reset control
Clock control
Power control
Interrupts for the ILOM and UltraSPARC T2 processor
A mailbox communications system is used by both the UltraSPARC T2 processor and
the ILOM to gain indirect access to these functions. The mailbox pointers are held in
the FPGA registers while data is maintained in the SRAM. Both the processor and the
ILOM use the mailbox, and therefore the SRAM, to pass data between each other.

The Virtual Blade System Controller (vBSC) controls most FPGA functionality and
provides interfaces for ILOM to call these functions when required, such as booting
the system and configuring specific options.
==================================================
==
he following Sun SPARC Enterprise T5120/ T5220 components can be serviced by
the customer:
Fan modules

Power supply units of which there are 2 units accessible from the rear of the server.
PCI cards and riser cards
DDR2 FBDIMMs represents the memory available in 1G, 2GB and 4G FBDIMMs.
System battery
SAS disk drives
Rail mount kit which is used to rack mount the Sun SPARC Enterprise T5120/T5220
server
Cable management arm which is used to streamline the cables in the rear of the
server.
The following Sun SPARC Enterprise T5120/ T5220 server components should be
serviced by an authorized, trained engineer:
DVD ROM which is replaceable from the front of the unit.
Disk backplane board
Fan power board which distributes power to the system fans.
Paddle board
Power distribution board which takes the power from the power supplies and
distributes it to the rest of the boards in the server
Bus bars
System configuration PROM
Cable Kit which contains two SAS cables
Motherboard (MB) assembly
It should be noted that in the event of a CPU failure, the entire motherboard must
be replaced.
When servicing the internal components to the Sun SPARC Enterprise T5120/T5220
server, be sure to:
Turn off all peripheral devices connected to the server.
Turn off the server itself, except in the case of hot-swap components.
Label and disconnect all of the cables coming into the server.
And finally, ensure that you follow ESD precautions.

Before working on the Sun SPARC Enterprise T5120/T5220 server be sure to have:
A Flat blade No.1 screwdriver and a No. 2 Phillips head screwdriver
An Electrostatic discharge (ESD) mat
A grounding wrist or foot strap
Hot-swappable components are those that you can install or remove while the
system is running, without affecting the systems performance. However, you might
have to prepare the operating system before the hot-swap operation is performed.
The following components are hot-swappable in a Sun SPARC Enterprise
T5120/T5220 server:
The two power supply units
The hard disk drives
Note: The system fans are hot-pluggable. No preparation of the operating system is
required before removing and replacing system fans.
he top cover for both the Sun SPARC Enterprise T5120 and T5220 server includes an
integrated, latched door for access to the hot plug fans.
Depending on the component that you are servicing, you might need to remove the
top cover.
Click the top link on your screen for a printable text based procedure on the removal
and replacement of the top cover.
Click the bottom link on your screen for an animated demonstration of the removal
of the top cover.
For each of the CRUs listed, click the component name for a printable text based
procedure on the removal and replacement of that component.
Click the link provided to view an animated demonstration on servicing CRUs.
Procedure for Removing and Installing a Fan Module

Fans are replaced in pairs (2 fans per module).

To remove a fan module:

Slide the system out of the rack.

Push the 2 fan door latches toward the rear of the server and open the fan door.

The faulty fan module is identified with a corresponding Service Required LED.

On the Sun SPARC Enterprise T5120, the Fan Fault indicators are located on the fan
board.

On the Sun SPARC Enterprise T5220, the Fan Fault indicators are located on the fan
modules.

Pull up on the fan module handle until the fan module is removed from the chassis.

Note: The removal procedure for the Sun SPARC Enterprise T5120 and T5220 fan modules is
the same. The only difference is that the Sun SPARC Enterprise T5120 server has 4 fan
modules (8 fans) and the Sun SPARC Enterprise T5220 server has 3 fan modules (6 fans).

To install a fan module:


1. With the top cover door open, install the replacement fan module into the server. The fan modules
are keyed to ensure they are installed in the correct orientation.
2. Apply firm pressure to fully seat the fan module.
3. Verify that the Fan Fault indicator on the replaced fan module is not lit.
4. Close the fan door.
5. Verify that the Top Fan indicator, Service Required indicators, and the Locator indicator/Locator
button are not lit.

6. Procedure for Removing and Installing a FBDIMM


7.
8.

Complete the following steps to remove an FBDIMM:


1. Power off the system and slide it out of the rack.

Note: FBDIMMs should only be removed with the power cord disconnected from the chassis.

9. 2. Remove the server top cover.


10. 3. Lift the hinged air baffle.
11. 4. Locate the memory module socket in which you will remove an FBDIMM. Press the
FBDIMM fault button.
12. The FBDIMM fault button is located on the motherboard near the FBDIMMs.

13. Faulty FBDIMMs are identified with a corresponding amber LED on the motherboard.
14. 5. Push down on the ejector tabs on each side of the FBDIMM until the FBDIMM is
released

15. 6. Carefully lift the FBDIMM straight up to remove it from the socket.
16. Complete the following steps to install an FBDIMM:
17. 1. Power off the system and slide it out of the rack.
18. 2. Remove the top cover.
19. 3. Lift the hinged air baffle.
20. 4. Locate the memory module socket in which you will install an FBDIMM.
21. 5. Ensure that the ejectors, at each end of the memory socket, are fully open (rotated
downward) to accept the new FBDIMM.

22. 6. Align the FBDIMM with the key in the socket.


23. 7. Press the FBDIMM straight down until it snaps into place and the ejectors engage the
cutouts in the FBDIMM's left and right edges.

Procedure for Removing and Installing a Power Supply Unit


Complete the following steps to remove a power supply:
1. If the server is in a rack with a cable management arm attached, swivel open the cable management
arm to view the power supplies.
2. Identify which power supply you will replace. Each power supply has an amber LED that you can view
from the rear of the server. If the amber LED is on, the power supply is faulty and should be replaced.
3. Disconnect the AC power cord from the power supply that you are replacing. The power supplies are
hot-swappable, so you do not have to shut down the server or disconnect the other power supply.

Note: The Service Action Required LEDs on the front panel and back panel blink when a
power supply is unplugged.

4. Remove the power supply:


a. Grasp the power supply handle and push the thumb latch toward the center of the power supply.
b. While continuing to push on the latch, use the handle to pull the power supply from the chassis.
Remove the power supply:
1. Align the power supply with the empty bay in the chassis.
2. Press the power supply into the bay until it firmly engages the connector on the power distribution
board. It is fully seated when the thumb-latch clicks into place.
3. Connect the AC power cord to the new power supply.
4. Swivel any cable management arm back into the closed position.

Procedure for Removing and Installing the


System Battery
Complete the following steps to remove the system battery:
1. Power off the system and slide the system out of the rack.
2. Remove the top cover.
3. Remove PCIe/XAUI riser 0.
4. Using a small (No. 1 flat-blade) screwdriver, press the latch and remove the
battery from the motherboard.
To install the battery, reverse this procedure.

Note: Install the new battery with the plus sign (+) facing up.

Procedure for Removing and Installing a PCI Card and


Riser Card
Procedure for Installing and Removing a PCI Card and Riser Card PCIe/XAUI cards are installed on
vertical risers. You must remove the relevant riser to access a PCIe/XAUI card.
Complete the following steps to install a PCI card:
1. Unpackage the replacement PCIe or XAUI card and place it on an antistatic mat.
2. Locate the proper PCIe/XAUI slot for the card you are replacing or
3. If necessary, review the PCIe and XAUI Card Guidelines to plan your installation.
4. Disconnect any data cables connected to the cards on the PCIe/XAUI riser being removed. Label
the cables to ensure proper connection later.
5. Remove the riser board.
a. Remove the #2 Phillips screw securing the riser to the motherboard.
b. Slide the riser forward and out of the system.
6. Insert the PCIe/XAUI card into the correct slot on the riser board.
7. Replace the riser board.
a. Slide the riser back until it seats in its slot in the back panel.
b. Replace the #2 Phillips screw securing the riser to the motherboard.
8. Re-install any data cables connected to the cards on the PCIe/XAUI riser being installed.
9. Install the top cover.
To remove a PCI card, reverse this procedure.

Note: The procedures are the same for both the Sun SPARC Enterprise T5120 and the Sun
SPARC Enterprise T5220.

Procedure for Removing and Installing a Hard Disk Drive


Complete the following steps to remove and replace a hard disk drive:
1. On the drive you plan to remove, push the hard drive release button.
The latch opens.

Caution: The latch is not an ejector. Do not bend it too far to the left. Doing so can damage
the latch.

2. Grasp the latch and pull the drive out of the drive slot.
To install a hard disk drive:
1. If necessary, remove the blank panel from the chassis. Press the FBDIMM's fault button.

Note: Sun SPARC Enterprise T5120 servers might have three blank panels covering
unoccupied drive slots. Sun SPARC Enterprise T5220 servers might have as many as seven
blank panels covering unoccupied hard drive slots.

2. Align the replacement drive to the drive slot.


Hard drives are physically addressed according to the slot in which they are installed. If you removed an
existing hard drive from a slot in the server, you must install the replacement drive in the same slot as the
drive that was removed.
3. Slide the drive into the drive slot until it is fully seated.
4. Close the latch to lock the drive in place.

Procedure for Removing and Installing a DVD


Assembly
To remove a DVD assembly:
1. Remove the following hard drive:
o

Sun SPARC Enterprise T5120: HDD3

Sun SPARC Enterprise T5220: HDD7

2. Release the DVD/USB module from the disk drive backplane.


Use the finger detent in the disk drive bay below the DVD/USB module to detach the module from
the backplane.

3. Slide the DVD/USB module out of the disk drive cage.


To install the DVD assembly:
1. Slide the DVD/USB module into the front of the chassis until it seats.
2. Install the hard drive you removed during the DVD/USB module removal procedure.

Procedure for Removing and Installing the Power Distribution Board to Motherboard Ribbon
Cable
To remove the PDB to MB ribbon cable:
1. Power off the system and slide the system out of the rack.
2. Remove the server top cover.
3. Lift the hinged air baffle.
4. Disconnect one end of the ribbon cable from the power distribution board and the other end from
the motherboard.
To install the PDB to MB ribbon cable, reverse this procedure.

Procedure for Removing and Installing a Fan Board


To remove a fan board:
1. Power off the system and slide the system out of the rack.
2. Remove the top cover.
3. Remove the fan modules.

Note: If you are replacing a defective fan module connector board, remove only the fan
modules that are necessary to remove the defective fan module connector board.

4. Remove the Phillips screw that secures the fan module connector board to the chassis.
5. Slide the fan board toward the left side of the chassis approximately 0.5 inch (12 mm) to disengage the
fan board from the paddle board and the bottom of the chassis.
6. Lift the fan board up and out of the chassis.

To install a fan board, reverse this procedure.

Procedure for Removing and Installing the Paddle Board


To remove the paddle board:
1. Power off the system and slide the system out of the rack.
2. Remove the top cover.
3. Remove the motherboard assembly.
4. Remove both power supplies from the chassis.
5. Remove all the disk drives from the server.
6. Remove the DVD, DVD carrier and USB board from the server.
7. Remove the 4 screws securing the disk cage assembly to the chassis. There are 2 screws on the side
of the chassis near the right front and 2 screws on the side of the chassis near the left front.
8. Disconnect the disk cable from the motherboard, so that it does not obstruct access to the power
distribution board.
9. Remove the PDB to MB ribbon cable.
10. Disconnect the top cover intrusion switch cable connector from the power distribution board.
11. Remove the 4 screws connecting the power distribution board to the bus bar.
12. Remove the single screw securing the power distribution board to the bottom of the chassis.
13. Slide the power distribution board toward the left approximately 0.5 inch (12 mm) to release the power
distribution board from the paddle board and the captive standoffs on the bottom of the chassis.
14. Lift the power distribution board out of the chassis.
15. Remove the 2 screws that secure the paddle board bracket to the chassis.

Note: Do not remove the 2 screws that secure the paddle board to the paddle board bracket.

16. Slide the paddle board bracket toward the rear of the chassis approximately 0.5 inch (12 mm) to
release the paddle board bracket from the chassis.
17. Lift the paddle board and bracket out of the chassis.
To install the paddle board, reverse this procedure.

Procedure for Removing and Installing a Disk Backplane


To remove a disk backplane:
1. Power off the system and slide the system out of the rack.
2. Remove the top cover.
3. Remove all the disk drives from the server.
4. Remove the DVD, DVD carrier and USB board from the server.
5. Remove the 4 screws securing the disk cage assembly to the chassis. There are 2 screws on the
side of the chassis near the right front and 2 screws on the side of the chassis near the left front.
6. Slide the disk cage assembly toward the front of the chassis approximately 0.5 inch (12 mm).
This releases the disk cage assembly from the chassis bottom.
7. Disconnect the disk cable from the disk backplane. The Sun SPARC Enterprise T5120 server has
1 disk cable. The Sun SPARC Enterprise T5220 server has 2 disk cables.
8. Remove the 2 screws that secure the disk backplane to the disk cage assembly. The Sun SPARC
Enterprise T5220 server has 4 screws securing the disk backplane to the disk cage assembly.
9. Slide the disk backplane down approximately 0.25 inch (6 mm) to release the disk backplane
from the "fingers" on the disk cage assembly that protrude through keyhole slots in the disk
backplane.
10. Pull the disk backplane away from the disk cage assembly.
To install a disk backplane, reverse this procedure.

Procedure for Removing and Installing the System


Configuration PROM
To remove the system configuration PROM:
1. Power off the system and slide the system out of the rack.
2. Remove the top cover.

3. Remove the right most PCI and riser card.


4. The system configuration PROM is the chip located at j7901 on the motherboard. Lift it straight up
off the motherboard.
To install the system configuration PROM, reverse the procedure.

Note: The PROM is keyed, so it can only be installed one way.

Procedure for Removing and Installing the


Motherboard Assembly
The Sun SPARC Enterprise T5120 and Sun SPARC Enterprise T5220 use the same motherboard.
The motherboard assembly consists of the motherboard and the tray that the motherboard sits in. They
should be removed and installed as a single unit.
To remove the motherboard:
1. Power off the system and slide the system out of the rack.
2. Remove the top cover.
3. Remove the air baffle.
a. Open the air baffle.
b. Disengage the rear of the air baffle from the motherboard and rotate the air baffle
forward.
c.

Press in the edges of the air baffle to disengage its pins from the chassis.

4. Disconnect the motherboard to power distribution board ribbon cable.


5. Disconnect the disk cable from the motherboard. The Sun SPARC Enterprise T5220 server has 2
disk cables.
6. Remove all PCI boards and riser cards.
7. Remove all FBDIMMs.
8. Remove the 4 screws connecting the motherboard to the bus bar.
9. Loosen the captive screw securing the motherboard to the chassis.
The captive screw is colored green, and is located to the left of the bus bar screws.

10. Using the green handles, slide the motherboard back and tilt the motherboard assembly to lift it
out of the chassis.
Grab the handles and move the motherboard toward the back of the system and lift it out of the
chassis.
To install the motherboard assembly, reverse this procedure.

Procedure for Removing and Installing the Front Panel Light Pipe Assembly
To remove the front panel light pipe assembly:
1. Remove the hard drive cage.
2. Remove the hard drive backplane.
3. Remove the #2 Phillips screw securing the front control panel light pipe assembly to the hard
drive cage.
4. Slide the light pipe assembly out of the hard drive cage.
To install the front panel light pipe assembly:
1. Align the light pipe assembly with the mounting holes on the hard drive cage.
2. Secure the light pipe assembly with a #2 Phillips screw.
3. Install the hard drive backplane.
4. Install the hard drive cage.

Procedure for Removing and Installing a Power Distribution Board

Note: It is easier to service the power distribution board (PDB) with the bus bar assembly
attached.

To remove a power distribution board:


1. Power off the system and slide the system out of the rack.
2. Remove the top cover.

3. Remove the motherboard assembly.


4. Remove both power supplies from the chassis.
5. Remove all the disk drives from the server.
6. Remove the DVD, DVD carrier and USB board from the server.
7. Remove the 4 screws securing the disk cage assembly to the chassis. There are 2 screws on the side
of the chassis near the right front and 2 screws on the side of the chassis near the left front.
8. Disconnect the disk cable from the motherboard, so that it does not obstruct access to the power
distribution board. Note: There are 2 disk cables connected to the motherboard in the Sun SPARC
Enterprise T5220 server.
9. Remove the PDB to MB ribbon cable.
10. Disconnect the top cover intrusion switch cable connector from the power distribution board.
11. Remove the 4 screws securing the power distribution board to the bus bar.

Note: The Sun SPARC Enterprise T5220 has 4 additional screws to remove. They are
connected to 2 additional bus bars that connect to a vertical power distribution board.

12. The Sun SPARC Enterprise T5220 server also has a ribbon cable that connects the horizontal and
vertical power distribution boards. This must be disconnected also.
13. Remove the single screw securing the power distribution board to the bottom of the chassis.
14. Slide the power distribution board toward the left approximately 0.5 inch (12 mm) to release the power
distribution board from the paddle board and the captive standoffs on the bottom of the chassis.
15. Lift the power distribution board out of the chassis.
To install a power distribution board, reverse this procedure.

To set up the service processor with initial network configuration information, you
must establish a connection through ILOM to the service processor. Until the service
processor has an IP address assigned to it, you must use a serial connection to
communicate with the service processor. After establishing a serial connection to the

service processor, you can choose to configure the service processor with a static or
DHCP IP address.
The default action for the service processor is to try and use DHCP for its network
configuration information. When you apply power to the system for the first time,
ILOM broadcasts a DHCPDISCOVER packet. If you have an established DHCP
server on the network, the DHCP server returns a DHCPOFFER packet containing an
IP address and other network configuration information to the service processor.
If you prefer to configure the service processor with a static IP address, and you have
a DHCP server established on your network, you can configure the static IP address
prior to attaching a LAN cable to the NET MGT port of the server.
Note: Sun recommends a static IP address for the service processor.
Whether static or DHCP IP addresses are assigned, you must initially establish a serial
console connection to communicate with ILOM.
You can access the ILOM CLI at any time by connecting a terminal or a PC running
terminal emulation software to the serial management port on the chassis.
To connect to the ILOM using a serial connection:
Step 1. Verify that your terminal, laptop, or terminal server is operational.
You need a display device to interface to the serial port. This can be one of the
following devices:
Laptop or desktop computer
Terminal device
Personal Digital Assistant

Step 2. Before trying to connect to the port, make sure that your display device is
properly configured. In the case of the laptop, desktop computer, and PDA, make sure
that you have a terminal simulation program loaded and started.

In the case of all the devices, make sure that they are configured with the following
default parameters:
8N1: eight data bits, no parity, one stop bit
9600 baud
Disable hardware flow control

Step 3. Connect a serial cable from the serial management port on the rear of the
chassis to a terminal device.
The serial management port is the left-most RJ-45 port, as viewed from the rear of the
chassis.
The pinout requirements for the serial cable connected to the serial port are displayed
in the table on your screen.

Step 4. Press Enter on the terminal device.


This establishes the connection between the terminal device and the ILOM.
Note: If you connect a terminal or emulator to the serial port before it has been
powered up or during its power-up sequence, you will see bootup messages.
When the system has booted, the ILOM displays its login prompt:
SUNSP 00:12:2F:4A:7A:3B login:
The first string in the prompt is the default host name. It consists of the prefix SUNSP
and the ILOMs MAC address. The MAC address for each ILOM is unique.

Step 5. Log in to the CLI:

Type the default user name, root.


Type the default password, changeme.
Once you have successfully logged in, the ILOM displays the ILOM default
command prompt:
->
You can now run CLI commands.
Note: CLI is the only available user interface on the serial port.

The CLI architecture is based on a hierarchical namespace, which is a predefined tree


that contains every managed object in the system. This namespace defines the targets
for each command verb.
The ILOM includes three namespaces:
The /SP namespace, the /SYS namespace, and the /HOST namespace.
The /SP namespace manages the ILOM. For example, you use this space to manage
users, clock settings, and other ILOM issues.
The /SYS namespace manages the host system. For example, you can change the host
state, read sensor information, and access other information for managed system
hardware.
The /HOST namespace - Used for monitoring and managing the host operating
system.
A syntax diagram for the ILOM CLI is shown on your screen.
An ILOM CLI command is made up of the following components:
verb - The CLI supports a pre-defined list of verbs.

options - All options are not supported for all commands. See a specific command
section for the options that are valid with that command.
target - Every object in your namespace is a target. All targets are not supported for all
commands.
properties - Properties are the configurable attributes specific to each object. An object
can have one or more properties.
Note: For information on using the CLI, see the Integrated Lights Out Manager
(ILOM) Administration Guide.
Use the set command to change properties and values for network settings.
Network settings have two sets of properties:
pending - which are the updated settings, not currently in use
active - read-only settings, currently in use by the ILOM
To change settings:
First, enter the updated settings as the pending settings.
Then, set the commitpending property to true.
To display network settings, type the command:
show /SP/network
If you are already in the /SP/network directory, type:
show

Follow these steps to assign a static IP address to the network management port:
Step 1. At the ILOM prompt, type the following command to set the working
directory.

-> cd /SP/network
Step 2. Type the following commands to specify a static Ethernet configuration.
-> set pendingipaddress=129.144.82.26
-> set pendingipnetmask=255.255.255.0
-> set pendingipgateway=129.144.82.254
-> set pendingipdiscovery=static
-> set commitpending=true
Note: The network values shown are samples only. You must specify the IP address,
netmask, and gateway appropriate for your network configuration.

Ensure that the same IP address is always assigned to an ILOM by either assigning a
static IP address to your ILOM after initial setup, or configuring your DHCP server to
always assign the same IP address to an ILOM.
This enables the ILOM to be easily located on the network.
When using the network management port, the wire speed is set to 10/100 megabit,
full duplex. Connection through this Ethernet port is allowed only after you have
configured the service processor using the serial port to have a valid IP address on
your network.
The service processor accepts ssh only through its Ethernet port.
The following interfaces can be used as ILOM management interfaces:
CLI using ssh which meets the DMTF SMASH industry standard
SNMP v1, v2c, v3
Web browser

IPMI 2.0
In general, feature compatibility exists between all user interfaces.
In addition, the CLI is backward compatible to ALOM which is the Advanced Lights
Out Management firmware application used on previous versions of Suns service
processors.
The first time the service processor is initialized, the following default conditions
exist:
The network is enabled
DHCP is enabled
ssh service is enabled
The shell for the root user is the DMTF CLI

As previously mentioned, there is an ALOM backward compatibility shell to


accommodate users that are more comfortable with an ALOM interface.
Some differences in the ALOM compatibility shell in Sun SPARC Enterprise
T5120/T5220 service processor include:
ssh access only - no telnet
tftp only - no ftp
A tftp server must be configured to support commands like:
- flashupdate
- FlashupdateFPGA
- frucapture As these commands use tftp as an underlying protocol
User names and passwords

Passwords have to be 8 to 16 characters (versus 8 characters)


Maximum of 10 users allowed (compared to 16)
Two user roles (compared to 4 in ALOM). They are:
CUAR (Administrator)
No bits set (Operator)
Two CLI modes:
default (ILOM), and
alom
A new parameter - netsc_commit
Using this, network settings take immediate effect, and there is
No need to reboot the SP And, finally,
setdefaults -a The -a stands for all. This resets all configuration parameters to
defaults, including all usernames and passwords, which means that the only user that
is left is the root user.
To create a new user, with the ALOM shell as their interface, follow the procedure on
your screen. This procedure creates the user admin to emulate the ALOM
administrator.
Step 1. Log in as the root user.
Step 2. Create the admin user.
-> create /SP/users/admin
Step 3. Set the role and cli mode.
-> set /SP/users/admin role=Administrator

-> set /SP/users/admin cli_mode=alom


Step 4. Log out and log back in as the admin user. In subsequent logins as admin,
youll get the ALOM shell.
Over the course of a products life cycle, new versions of firmware are released. To
verify which version of firmware your system is running, issue the version command.
In the command output, you are looking for the ILOM version, the SC firmware
version, and the OBP version.
If you are considering updating the ILOM firmware, be aware of the following:
It is likely that the firmware images available to download from the SunSolve
database are more current than the image installed on your service processor at the
factory.
The BIOS and the SP firmware are simultaneously updated. A single firmware image
contains both the BIOS and the SP firmware.
A firmware upgrade will cause the server and ILOM to be reset. It is recommended
that you perform a clean shutdown of the server prior to the upgrade procedure.
An upgrade takes about five minutes to complete.
ILOM will enter a special mode to load new firmware. No other tasks can be
performed in ILOM until the firmware upgrade is complete and ILOM is reset.
Note: Ensure that you have reliable power before upgrading your firmware. If power
to the system fails (for example, if the wall socket power fails or the system is
unplugged) during the firmware update procedure, the ILOM could be left in an
unbootable state.
To update the ILOM firmware using the CLI:
Step 1. Log in to the service processor as root.
Step 2. Update the ILOM firmware (and consequently the BIOS) as follows:
-> load -source tftp:// servername:port/path/to/image

Note This functionality also exists in the ILOM Web GUI.


From the service processor, you can power on the server by typing:
start /SYS
You can use the stop command to perform an orderly shutdown of the server followed
by a power off of the server, as shown on your screen.
You can also skip the orderly shutdown and force an immediate power off with the
-force option, as shown on your screen.
Output to POST, OBP, and the Solaris OS are displayed to the system console, which
is accessible through the service processor.
To initiate a connection to the server console, execute the start /SP/console command
from the service processor prompt.
To terminate a connection to the server console, execute the stop /SP/console
command from the service processor prompt.

Working in a data center with thousands of servers in racks can sometimes pose a
problem when you are trying to locate one.
The locate LED is a white LED that you can light to help you find your server in a
crowded equipment room. The Locate LED has two states, fast blink and Off.
To turn on the locate LED, type:
-> /SYS/LOCATE value=Fast_Blink
To turn off the locate LED, type:
-> /SYS/LOCATE value=Off

The system event log accumulates various events, including administration changes to
the ILOM, software events, warnings, alerts, and events from the IPMI log.
You should note that the ILOM tags all events or actions with LocalTime=GMT (or
UDT). Browser clients show these events in LocalTime. This can cause apparent
discrepancies in the event log. When an event occurs on the ILOM, the event log
shows it in UDT, but a client shows it in local time.

To view and clear the system event logs, perform the following steps:
Step 1. Navigate to /SP/logs/event.
-> cd /SP/logs/event
Step 2. From the CLI, enter the show list command: -> show list
The event log scrolls onto your screen.
Step 3. To scroll down, press any key except q.
Step 4. To stop displaying the log, press q.
Step 5. To clear the system event log, use the command:
set clear=true
Step 6. The CLI asks you to confirm.
Type y.
The CLI clears the system event log.
Note: The system event log accumulates many types of events, including copies of
entries that IPMI posts to the IPMI log. Clearing the system event log clears all
entries, including the copies of the IPMI log entries. However, clearing the system
event log does NOT clear the actual IPMI log.

You must use IPMI commands to view and clear the IPMI log.
The service processor also allows you to create additional users. The tasks that can be
performed by a user are determined by the privileges that you assign to that users
account. You can have up to a maximum of ten user accounts, including root.
Each user account consists of a user name, a password, and a role.
As the root user, you can add, delete, and list users on the service processor.
To add a user, execute the create command, providing the following information:
username
password, and
role - either administrator or operator
You can remove users from the service processor using the delete command, as shown
on your screen.
To display information about all local user accounts, type show /SP/users.

As root user, you can use the set command to change passwords and roles for existing
user accounts.
For example, when changing the role for user1 from Administrator to Operator, type:
-> set /SP/users/user1 role=operator
To change user1's password, type:
-> set /SP/users/user1 new_password
To log into the Sun ILOM Web GUI, follow these procedures:
Step 1. Using secure http, type the IP address of the ILOM service processor into your
web browser, as the example on your screen shows.

The JavaTM Web Console login screen is displayed.


Step 2. Type your user name and password.
Step 3. Click OK.
The ILOM Web GUI login page is displayed.
Click the link provided on your screen to view a demonstration using the Web GUI.

The Sun SPARC Enterprise T5120/T5220 server has a virtual keyswitch with four
different modes.
The first mode is normal in which the service processor uses the diagnostic settings
that you specified with the set command to determine how POST is executed.
The second mode is diag in which the vBSC sets the diagnostic level to the
interactive menus.
The third mode is stby. In this mode, the service processor prevents you from
powering up the server.
And in the fourth mode, locked, the service processor does not allow you to send a
reset to the server.
To set the virtual keyswitch, execute the set command with /SYS/keyswitch_state as
the target and provide a mode as a value for the target.
To view the current keyswitch setting, execute the show /SYS/ keyswitch_state
command.

The Sun SPARC Enterprise T5120/T5220 server has a virtual keyswitch with four
different modes.

The first mode is normal in which the service processor uses the diagnostic settings
that you specified with the set command to determine how POST is executed.
The second mode is diag in which the vBSC sets the diagnostic level to the
interactive menus.
The third mode is stby. In this mode, the service processor prevents you from
powering up the server.
And in the fourth mode, locked, the service processor does not allow you to send a
reset to the server.
To set the virtual keyswitch, execute the set command with /SYS/keyswitch_state as
the target and provide a mode as a value for the target.
To view the current keyswitch setting, execute the show /SYS/ keyswitch_state
command.
You can power on the system by executing the start /SYS command at the service
processor prompt, or by pressing the power button located in the bottom left corner on
the front of the machine.

Output from POST, OBP, and the Solaris OS are displayed to the system console,
which is accessible through the service processor. To acquire the console, execute the
start /SP/ console command from the service processor.
When you issue the command the system will prompt you Are you sure you want to
start /SP/console (y/n)?. You can suppress this prompt by using the option -script.
Although multiple users can connect to the system console from ILOM, only one user
at a time has write access to the console. This is referred to as a write-locked session.
Any characters that other users type are ignored. These are referred to as read-only
mode sessions, where users can only view the console.
If no other users have access to the system console, then the user entering the console
session first obtains the write lock automatically.

To see if there are other users connected into the service processor and to see whether
they are connected to the console, execute the show /SP/sessions command.
Note: Terminate a console session by typing #. (pound, period).

If the Solaris OS is running, you can use the stop /SYS command from the ILOM
shell to issue a graceful shutdown to Solaris. It is similar to one of the Solaris OS
commands, such as shutdown, init, or uadmin.
It can take up to 65 seconds for the poweroff command to completely shut down the
system. This is because ILOM attempts to wait for a graceful shutdown to complete
before the server is powered off.
You can also force the server to power down by executing the -> stop -force /SYS
command or by pressing and holding the power button on the front of the server. This
does an immediate shutdown regardless of the state of the host.
From the service processor console, you can issue the command: -> set
/HOST/send_break_action true to bring down the server to the Open Boot PROM
prompt, or ok prompt. It is the equivalent of executing an L1-A or Stop-A on a system
with a keyboard attached.
For the system to accept a break, the virtual keyswitch must not be in the locked
position. If it is in the locked position, ILOM returns an error message.

To reset the service processor or the server, execute the reset command from the
service processor.
To reset the service processor, type reset /SP
If you reset /SYS, the server reboots using the Open Boot PROM settings that you
have configured.The reset command does not perform a graceful shutdown of the
operating system.
You will be prompted to confirm a reset operation.

The -script option instructs ILOM to proceed without prompting the confirmation
question.

Whenever a device is physically added or removed from a Sun SPARC Enterprise


T5120/T5220 server, structures that describe this device to the hardware and the
operating system must be created or removed. The process of creating and removing
these structures is the foundation of device configuration.
In the Sun SPARC Enterprise T5120/T5220 servers, device configuration is initiated
through the Hypervisor layer. The Hyrpervisor then passes this structure to the Open
Boot PROM and then ultimately up to the Solaris OS. The Solaris OS maintains its
knowledge of available devices using a set of hierarchically organized device files.
These files are located in the /devices directory of the root file system. In addition, the
Solaris OS uses the path_to_inst file in the /etc directory to manage device instance
names. It uses links in the /dev directory to enable logical device addressing.

The Sun SPARC Enterprise T5120/T5220 servers run OBP 4.x. The capabilities of the
Open Boot PROM in these servers have decreased as a lot of its functionality has been
moved to the Hypervisor layer. Its key functions are to allow you to boot the operating
system, modify system startup parameters, load and execute programs, and get help in
troubleshooting.
The Solaris OS comes pre-installed on the Sun SPARC Enterprise T5120/T5220
servers on the disk in slot 0. The operating system is not configured, that is, the sysunconfig command was run after the OS was installed. When you boot the system for
the first time from the disk, you are prompted to configure it for your environment. At
the ok prompt, boot from the disk that contains the Solaris operating system. You
might want to configure an alias for this as your boot disk.

There are some instances when you might want to reinstall or upgrade the Solaris OS
in your server. The Sun SPARC Enterprise T5120/T5220 servers support a minimum
OS release of the Solaris 10 Update 4 OS. It has an architecture type of sun4v, to

coincide with the ILOM firmware running on the service processor. The v represents
the virtualization of the hardware to the Open Boot PROM and the Solaris OS, which
is performed by the Hypervisor layer. You can install or upgrade the operating system
through traditional jumpstart procedures or using the DVD-ROM.
Next, were going to take a look at some standard Solaris OS commands to see what
is reflected differently in their output.
The first is the date command. The system TOD (time of day) is managed by the
service processor and is provided to the Solaris OS by the Hypervisor layer. You must
set the date on the service processor using the set /SP/clock/datetime command. If you
set the date using the date command within the Solaris OS, it holds until your next rePOST, when the service processor will then pass its date over to the Solaris OS once
again.
The second command is prtdiag. The prtdiag command displays system configuration
and diagnostic information. Youll notice in the prtdiag output that the Solaris OS sees
the UltraSPARC T2 processor as 64 CPUs. It is an 8-core processor with 8 threads
running per core. Youll also see in the prtdiag output that you can obtain memory
specifics from the service processor. You only see the total memory size in the prtdiag
output.
The next command is psrinfo. The psrinfo command reflects the same changes from a
CPU perspective as the prtdiag command. You will see that the Solaris OS is
presented as 64 processors.
And finally, well take a look at the output to ifconfig -a. This output represents the
four gigabit Ethernet interfaces as ipge interfaces, ipge0 through ipge3.
Multipathing software lets you define and control redundant physical paths to I/O
devices, such as storage arrays and network interfaces. If the active path to a device
becomes unavailable, the software can automatically switch to an alternative path and
maintain its availability. This is known as automatic failover. To take advantage of
multipathing capabilities, you must first configure the server with redundant hardware
for example, multiple network interfaces going to the same subnet or two controllers
attached to the same storage array and then configure the software to make use of it.

For the Sun SPARC Enterprise T5120/T5220 servers, there are three different types of
multipathing software that are available.
The first is the Solaris OS IPMP (IP Multipathing) which provides multipathing and
load-balancing capabilities for IP network interfaces.
Next, we have Sun StorEdge Traffic Manager Software (STMS), which enables I/O
devices to be accessed through multiple host controller interfaces from a single
instance of the I/O device. This software is fully integrated into the Solaris operating
system.
And finally, we have Veritas Volume Manager (VxVM), which includes Dynamic
Multipathing (DMP). This software provides disk multipathing as well as disk load
balancing to optimize I/O throughput.

Within the Sun SPARC Enterprise T5120/ T5220 servers, the SAS controller supports
hardware mirroring and striping using the Solaris OS raidctl utility.
A hardware RAID volume created using the raidctl utility behaves differently than one
created using software RAID. When volumes are created using hardware RAID, only
one device appears in the device tree. Member disk devices are invisible to the
operating system and are accessed only by the SAS controller.
Executing the raidctl command tells you whether there are any RAID volumes found.
To create a RAID volume, execute the raidctl -c primary_drive secondary_drive
command. The secondary drive will disappear from the device tree.
To set up a striped volume, execute the raidctl -c -r 0 primary_drive secondary_drive
tertiary_drive, and so on command. In this case, all but the primary drive disappears
from the Solaris OS device tree.
To delete the hardware RAID volume, execute the raidctl -d mirrored_volume
command.

To perform a disk replacement operation through dynamic reconfiguration, verify


which hard drive corresponds to the physical device you want to remove.
Ensure that no applications or processes are accessing the hard drive. To view the state
of the SCSI devices, execute the cfgadm -al command. To remove a hard drive from
the device tree, execute the cfgadm -c unconfigure Ap_Id command, where the Ap_Id
is the attachment point identifier represented in the left-most column of the cfgadm -al
output. When this operation completes, the blue OK-to_remove LED lights on the
drive. It is now safe to remove and replace the disk.
After installing the new drive, execute the cfgadm -c configure Ap_Id command to
configure the drive back into the Solaris OS.The green activity LED flashes as the
new disk is added to the device tree.
LEDs are placed throughout the Sun SPARC Enterprise T5120/T5220 servers to
help pinpoint problem components in the server, as well as to give a visual
indicator of the overall server status.
LEDs are found on the following chassis locations and server components:
The front and rear panels
The system fan modules
The power supplies
The disk drives
The DVD drive
Service indicators are categorized as either system status LEDs or component
status LEDs.
On the Sun SPARC Enterprise T5120/T5220 servers front panel, system status
LEDs are on the left, and component status LEDs are on the right.
Looking at the system status LEDs on the front panel of the server, starting on the top,
and moving down, the system status LEDs are the:

White-colored Locator indicator,


The Amber-colored Service Required indicator, and the
Green-colored running or Power OK Indicator
Click the indicator name on your screen for additional information about the function
of that indicator.
The system status LEDs on the front panel of the server are replicated on the rear of
the chassis, as shown on your screen. Starting on the left, and moving to the right,
they are:
a White-colored Locator indicator
Amber-colored Service Required indicator, and
Green-colored running or Power OK Indicator
Each power supply has three LEDs. From top to bottom, they are:
a Green-colored PSU OK indicator,
Amber-colored PSU fault indicator, and
Green-colored A/C PSU power indicator

The gigabit Ethernet ports each have two LEDs to show their current status. They are
the:
Green-colored Link/activity indicator, and the
Amber-colored Speed indicator
Click the indicator name on your screen for additional information about the function
of that indicator.

The Sun SPARC Enterprise T5120/T5220 status indicators conform with the
American National Standards Institute (ANSI) Status Indicator Standard (SIS).
The table on your screen describes the SIS standard LED behaviors and their
meanings.
You can also obtain the LED status information through the service processor.
To view LED status information in the ALOM CLI, use the showenvironment
command. In the ILOM cli, use the show /SYS/<component>/<property> command.
Click as indicated on your screen to view a demonstration of these commands.
Each DIMM slot on the motherboard has an associated fault LED that identifies a
faulty FBDIMM diagnosed by POST or FMA.
When power is removed, the FBDIMM fault LEDs are lit by pressing a fault reminder
button located on the motherboard.
To identify faulty FBDIMMs, follow this procedure:
Step 1. Unplug all power cords.
Step 2. Press the FBDIMM fault button.
The FBDIMM fault button is located on the motherboard near the FBDIMMs as
shown.
Step 3. Note the location of faulty FBDIMMs.
Faulty FBDIMMs are identified with a corresponding amber LED on the
motherboard.
Step 4. Ensure that all FBDIMMs are seated correctly in their slots. If re-seating the
FBDIMM does not fix the problem, remove and replace the faulty FBDIMM.
Note: The DIMM fault LEDs can be lit for only a minute or so with the fault reminder
button.

Firmware diagnostic tests are executed on both the Sun SPARC Enterprise T5120/
T5220 server host and on its service processor. The purpose of these tests is to verify
the core functionality of the service processor and the host.
The output displayed during the resetting of the service processor is an excellent
source of diagnostic information.
Click as indicated on your screen to view a demonstration of the power reset sequence
of the service processor.

When the service processor has finished booting and the ILOM firmware has loaded,
you can log in to the service processor and power on the host server. You can do this
using the poweron command in the ALOM cli, executing start /SYS in the ILOM cli,
or by pressing the power button on the front panel of the chassis.
When power is applied to the host server, the vBSC, which is responsible for calling
POST and collecting POST status on completion, is initialized.
Hosted on the service processor, vBSC can be thought of as an extension of
Hypervisor. The functionality that OBP once provided has been moved to the service
processor. This eliminates the need to tie OBP-specific settings to the entire server.
When the server powers on, the following actions take place:
vBSC is initialized from the service processor.
POST is called from vBSC to perform a sanity check of the server as well as testing of
the components based on the settings passed to the host server by vBSC.
The ASR database is updated based on diagnostic tests performed.
And Hypervisor is started for the host server.

While POST is called automatically, the output is not displayed by default. Likewise,
higher levels of POST testing are not called by default. The server must be configured
for these actions to happen.
The ILOM variables that affect POST are diag_trigger, diag_verbosity, diag_level and
diag_mode.
You can modify them with the set command in the ILOM or the setsc command in
ALOM and view them with their current settings by executing the show commnad in
ILOM and showsc command in ALOM.
The service processors setkeyswitch command also affects the behavior of POST
execution. If the keyswitch is set to the DIAG position, then vBSC sets the diagnostic
levels to the interactive menus. If the keyswitch is set to the NORMAL position, then
the service processor variables are used to determine if and how POST is executed.
Click as indicated on your screen to view a table of these ILOM variables and their
possible settings.
OBP tests are executed once POST has completed and the Hypervisor has been
loaded. OBP tests are affected by the diag-switch? OBP variable.
The value of diag-switch? affects the verbosity of OBP. If diag_switch? is set to true,
the output from the OBP initialization and probing is sent to the console. If diagswitch? is set to false, no initialization or probing output is displayed.
In addition to normal POST and OBP output, there are several commands and
interfaces that you can use to analyze and test the system hardware to assist in
troubleshooting.
In the remainder of this module, we will look at analyzing POST output, running
POST menus, analyzing OBP diagnostic output, analyzing POST error messages,
managing the ASR database, and viewing logs, console messages, and faults.
There are two levels of POST tests that are executed on the host server. These are the
integrity POST and, if configured, EPOST.

Integrity POST executes the following tests on the server each time the system is
powered on:
Register tests on window registers and the Niagara CPU scratchpad
Floating point unit access to check the path from all threads
L2 access to check the path from all threads
Quick FPGA check by the master thread, the first thread to jump to POST, to check
the integrity of the SRAM
All threads returned from testing
EPOST executes tests for the remainder of the server based on the diag_level setting.
When executed with a diag_level set to min, EPOST:
Initializes registers and global variables.
Runs the basic memory tests in which all memory cells are touched with unique
patterns and checked with hardware ECC.
Tests I2C operation and clock frequency for the master thread only.
Performs a basic test on the JBUS-to-PCI-E bridge for the master thread only.
When executed with a diag_level set to max, EPOST:
Performs all the steps of the minimum diagnostic mode.
Runs memory tests on any arrays not covered by BIST, such as L1 tags and the
internal register arrays.
Tests the functional operation of the L1 and L2 caches and instruction and data
memory management unit and interrupts for all of the hardware strands.
Performs an extended memory test in addition to hardware ECC checks.

Tests the functionality of the JBUS-to-PCI-E bridge for the master hardware thread,
specifically direct memory access and interrupt testing.
The POST interactive menu mode is called when two conditions are met. The first is
that the systems virtual keyswitch is set to DIAG mode and the second is that the
service processor diag_mode variable is set to menu. You can verify that these
conditions are met with the output to the showkeyswitch and showsc commands.
The POST menu mode provides access to all tests available within POST, including:
built in self test
Tests performed when diag_level is set to min
Tests performed when diag_level is set to max

When POST has completed, OBP initializes and works with Hypervisor to build the
device tree structure. Even though the Open Boot PROM does not have as much
functionality as it previously did, you can still gather some information from it to help
you troubleshoot.
From the output to the banner command, you can see how much memory has passed
POST.
From the output to show-devs, show-disks and show-nets, you can see the device tree
that has been built, along with the disks and network interfaces that the Open Boot
PROM sees.
POST displays a great deal of information on errors and warnings to assist you in
troubleshooting the cause of a problem. The level of information given is affected by
the value of the service processor diag_verbosity variable. Errors follow the standard
format that identifies:
The test that was executing.
The hardware that was being tested at the time of the failure

The suggested repair instructions for which component to replace


And the error message generated by the fault.
Automatic System Recovery, or ASR, lets you manually manage blacklisted items
from the service processor as well as lets the service processor manage the blacklisted
items through the vBSC.
An ASR database is maintained with a list of any blacklisted items. When POST
completes, it returns a status of the components tested to the vBSC. If any of the
components are reported back as failed, the service processor attempts to unconfigure
them and map them out of the server. An event is then logged to the service processor
and to the console regarding this issue.
ILOM provides commands to manage the ASR database. The command that you use
is cli dependent.
In the ILOM cli, you can use the show /SYS command to query the component_state
of a specific component.You use the set command to disable and enable a specific
component.
You also use the set /SYS command to clear the ASR database for a specific
component.

Both the ILOM and ALOM cli provide commands for viewing console messages,
system messages, and system faults. The types of messages include errors, faults,
notices and general system information.
In the ILOM cli, execute the show /SP/ logs/event/list command to display messages,
notices, and events that have been sent to the service processors event log.
The equivalent ALOM CLI command is the showlogs command.
The ALOM showfaults command displays current valid system faults.
The ALOM cli also provides a command called consolehistory that shows the contents
of the boot log and the run log. The boot log contains the messages from POST, OBP,

and the booting of Solaris. The run log contains everything in the boot log plus the
Solaris runtime messages.
The ILOM cli does not provide the console history functionality.
Click the links on your screen to view a demonstration of these commands.
Several tools and features are available on the Sun SPARC Enterprise T5120/T5220
servers to help administrators monitor, troubleshoot, and diagnose issues. In this
section we will discuss the following tools:
Software-based commands
Applications
Log files
You can monitor and analyze the server status using a combination of these tools.
Click as indicated on your screen to view a block diagram showing the diagnostic
components on both the host server and the service processor.
The Sun SPARC Enterprise T5120/T5220 servers implement the fault management
architecture introduced in Solaris 10 OS. Incorporated into both the hardware and
software of the Sun SPARC Enterprise T5120/T5220 servers, the FMA helps the
server maintain a greater uptime rate by:
Automatically and silently diagnosing underlying problems
Using predictive self-healing
Disabling faulty components if necessary and possible
Issuing alerts on problems and logging events
Providing data to higher-level management services and, in the future, remote services
Be sure to have your server fully patched for the latest FMA agents.

The fault management architecture incorporates the following components to help


achieve its goals:
Error report creation, where the Hypervisor, vBSC, and hardened device drivers each
create reports that in the end are handed off to the Solaris OS fault management
daemon.
The Solaris OS fault management daemon, fmd, is responsible for forwarding reports
generated by the vBSC from the hypervisor and hardened system device drivers to a
diagnosis engine.
The diagnosis engine contains profiles with fault trees and rules for devices on the
server. These rules determine how a fault on a device should be handled and who
should be handling it.
Fault response agents help the server and system administrator manage the hardware
or software fault.
Click as indicated on your screen to view the fault management architecture block
diagram for the Sun SPARC Enterprise T5120/T5220 servers.

There are several Solaris OS commands associated with FMA. These include:
The fmadm command, which lets you view, load, and unload modules. It also lets you
view and update the resource cache, which is a list of faulty resources as seen by the
fault management daemon, fmd.
The fmdump command, which enables system administrators to view any log files
associated with fmd and retrieve specific details of any diagnosis issued. By default,
the fmdump command lists the fault log, displaying the time, the ID associated with
that fault, and the message ID, SUNW-MSG-ID, that can be viewed on Suns message
lookup website.
The fmstat command, which reports the statistics of the fault management system. By
default, the fmstat command lists the active modules and the statistics associated with
the modules.

Intermittent problems can often times be difficult to diagnose. Diagnostic tools


exercise the components to the point where they display an emerging failing
condition. These tools are designed to stress the components to the point of failure. On
the Sun SPARC Enterprise T5120/T5220 servers, the Sun Validation Test Suite
(SunVTS) is used to exercise the server, as well as for hardware validation and repair
verification.
The minimum version of SunVTS that supports the Sun SPARC Enterprise T5120/
T5220 servers is SunVTS 6.4 and it ships with the Solaris 10 Update 4 OS.
The following Sun SPARC Enterprise T5120/ T5220 components can be diagnosed
through SunVTS:
CPU
FBDIMMs
I/O
Gigabit Network Ports
SAS Disks, Controller and Cables
DVD Device
Host-to-Service Processor Interface
SunVTS was modified for the Sun SPARC Enterprise T5120/T5220 as follows:
CPU/memory tests were updated to test the UltraSPARC T2 specific features.
cryptotest was enhanced to test the cryptographic unit on the UltraSPARC T2.
A new SunVTS test developed for the UltraSPARC T2 is Xnetlbtest, which provides
testing coverage for the two 10 Gigabit ports on the network interface unit of the
UltraSPARC T2 processor.

Additional complex testing of nettest and netlbtest include the following features:
Spawns off continuous Tx/Rx asynchronously to force the driver to exercise different
DMA channels
Provides classification, IP fragmentation, and variable length packets to cover jumbo
frame test
Supports back to back (port to port) loopback tests
Transmit rate can be varied by using delay between sends
Soft error threshold allows limit of packet drop in pass/fail criteria
Provides options of different payload data patterns

You can collect Sun SPARC Enterprise T5120/T5220 servers status and configuration
information from several sources within the Solaris 10 OS. You can find system
status, such as error and information messages, in the log files. You can obtain system
status by executing specific Solaris OS commands.
The following section describe the utilities that you can use to collect status and
configuration information on the Sun SPARC Enterprise T5120/T5220 servers.
The main system log for the Solaris 10 OS is the messages file located in the /var/adm
directory. Here, you can locate system status and error and informational messages by
filtering this file. This file can grow to be large, so it is important to select key values
to filter on, for example, cpu, mem, error, and so on.
You can also obtain system status and configuration data through the use of the Solaris
OS utilities, such as:
prtdiag, which lists the available CPUs, the I/O configuration, and the PROM and
ASIC versions
iostat, which displays information on each I/O device, including I/O errors

prtconf, which displays the system device drivers


prtpicl, which displays platform-specific information stored in the platform
information and control library (PICL)
psrinfo, which displays which CPUs are available and their status
raidctl, which tells us if there are any RAID sets configured and, if so, what their
members are
and the Sun Explorer Data Collector

The Sun Explorer Data Collector is a utility, made up of shell scripts and some
binaries, that automates the collection of system configuration data from Sun Solaris
servers. It collects a summary of installed software, firmware, and storage subsystem
components and saves it in a compressed tar format. This tool is accessible from
SunSolve, located at
http://sunsolve.sun.com.
This web site provides more information on the utility, any patches that are needed for
the Sun SPARC Enterprise T5120/T5220 servers, and the link to the software
download website.

Another source of system status and configuration information is a system core dump.
The Solaris 10 server is enabled by default to save a core dump when it occurs.
To verify that core dumps are enabled, run the dumpadm command. If core dumps are
enabled, you also need to verify that you have enough swap space and file system
space for the core dump to be stored. You can do this with the swap -l and df -k
commands, respectively.
To test the save dump utility, perform a graceful shutdown of the Solaris OS. From the
OBP prompt, perform a sync followed by a reset-all. Watch for the savecore messages
during boot and then verify that the savecore files are in the savecore directory.

(http://oukc.oracle.com/sun_content/secure_content/wzt-6094/courses/wzt6094/techtalks/mod_08/demos/m8_s04a_popup.html)

The Sun SPARC Enterprise T5140, code name Maramb1U, server is a 1U rack
mountable server.
The Sun SPARC Enterprise T5240, code name Maramba2U, server is a 2U rack
mountable server.
It should be noted that both servers share the same motherboard.
The Sun SPARC Enterprise T5140/T5240 servers are the first servers to expand chip
multithreading (CMT) to include multi-processor systems.
They are also the first servers to implement the UltraSPARC T2 Plus processor.

Each CPU has 4, 6, or 8 cores, for up to 128 threads.


These servers have twice the threads of the Sun SPARC Enterprise T5120/T5220
servers in the same footprint.

The Sun SPARC Enterprise T5140 server implements the 1U, Jake chassis used by
other Sun 1U servers.
The chassis measures 1.75 inches high by 18 inches wide by 24 inches deep, and
weighs approximately 20.5 pounds or 9.3 kilograms without optional equipment.
The Sun SPARC Enterprise T5240 server implements the 2U, Elwood chassis used
by other Sun 1U servers.
The chassis measures 3.5 inches high by 18 inches wide by 24 inches deep and weighs
approximately 40 pounds or 18 kilograms without optional equipment.
Both the Sun SPARC Enterprise T5140 and T5240 servers can be fitted into a 19-inch,
4-post rack and are shipped with the rackmount kit for that type of rack.
The Sun SPARC Enterprise T5140/T5240 offers the following features:
Two UltraSPARC T2 Plus CPUs.
Fully buffered dual inline memory module (FBDIMM) slots:
There are 16 in the Sun SPARC Enterprise T5140 server.
And 32 in the Sun SPARC Enterprise T5240 server, when equipped with a mezzanine
assembly.
Support for 1, 2, and 4 gigabytes (Gbytes) FBDIMMs.
Hot-plug, 2.5-inch Serial Attached SCSI, (SAS) Hard Disk Drives (HDDs) or 2.5-inch
Serial ATA (SATA) HDDs.

Up to 8 in the Sun SPARC Enterprise T5140 server, however at revenue release,


T5140 only offers 4 disk configurations.
Up to 16 in the Sun SPARC Enterprise T5240 server.
Note: SATA disk drives will not be supported at revenue release (RR).
Peripheral Component Interconnect - Express (PCIe) slots:
With three in the Sun SPARC Enterprise T5140 server.
And six in the Sun SPARC Enterprise T5240 server. Note that all slots accept x8 low
profile PCIe cards, but two slots also accept Sun XAUI-based 10 gigabit Ethernet
cards.
Two hot-swap power supplies.
Hot-plug fan modules, with
Seven in the Sun SPARC Enterprise T5140 server,
And six in the Sun SPARC Enterprise T5240 server.
Four gigabit Ethernet ports.
Four Universal Serial Bus (USB) 2.0 ports
A Motorola MPC885 remote service processor.
Remote lights out management support,
And finally, extensive reliability, availability, and serviceability (RAS) functionality.

The next few slides describe the system specifications for the Sun SPARC Enterprise
T5140/T5240 servers, starting with the processor.

The Sun SPARC Enterprise T5140/T5240 servers are equipped with two UltraSPARC
T2 Plus processors on-board. These processors operate at 1.2 and 1.4 gigahertz. The
1.4GHz will be post RR.
The processor has four, six, or eight cores.
Note: The UltraSPARC T2 Plus processor uses the same SPARC core as the
UltraSPARC T2 (code name: Niagara2) processor. Each core has a separate floating
point unit. The processor contains a 4 megabyte Level 2, or L2, cache, and supports
eight threads per core.
Each CPU also supports up to 16 FBDIMM slots, each of which supports 1, 2, 4, or 8
gigabyte DIMMs, although 8 GB FBDIMMs will not be supported at RR. The
FBDIMMs are controlled by two memory control units per processor, compared to
four memory control units per processor in the UltraSPARC T2 processor.
As mentioned earlier, the Sun SPARC Enterprise T5140 and T5240 servers share the
same motherboard. There are sixteen FBDIMM slots on the motherboard.
If a Sun SPARC Enterprise T5240 server customer chooses 32 memory slots, a
memory mezzanine is required.
The Sun SPARC Enterprise T5240 has an additional 16 memory slots on the
mezzanine board.

The mezzanine board assembly is a customer removable assembly that consists of:
A removable mezzanine chassis, and
A mezzanine board containing 16 memory slots

Note that not every Sun SPARC Enterprise T5240 server will have a mezzanine
assembly. It will only be shipped with the server when the additional memory is
ordered by the customer.

The Sun SPARC Enterprise T5140/T5240 servers support DIMM sizes of 1Gbyte, 2
Gbytes, or 4 Gbytes.
A maximum of 128 Gbytes of main memory is supported at revenue release.
The availability of 8 Gbyte DIMMs will be determined after Revenue Release.
Supported memory configurations in the Sun SPARC Enterprise T5140 server
include:
8 FBDIMMs
12 FBDIMMs, and
16 FBDIMMs.
Supported memory configurations in the Sun SPARC Enterprise T5240 server
include:
8 FBDIMMs
12 FBDIMMs
16 FBDIMMs
24 FBDIMMs, and
32 FBDIMMs.
Configuration rules for memory include:
The minimum number of FBDIMMS is eight.
Memory is added in sets containing two FBDIMMs.
Individual FBDIMMs can be replaced.
All FBDIMMs must be the same size.
Mixing of FBDIMMs from different vendors is allowed.

Each unpopulated FBDIMM slot in the Sun SPARC Enterprise T5140/T5240 servers
must contain a DIMM filler.
The use of DIMM fillers ensures uniform airflow throughout the server. They also
reduce air flow bypass, and consequently lower fan speeds, which in turn improves
noise emissions and increases the fan modules mean time between failure.
System I/O is built on the PCIe bus.
In the Sun SPARC Enterprise T5140 server, the PCIe bus supports 3 standard half
length/ half height PCIe expansion slots on 3 riser boards.
In the Sun SPARC Enterprise T5240 server, six standard half length/half height PCIe
expansion slots on three riser boards are supported on the PCIe bus.
In addition to PCI Express expansion cards, two of the expansion slots can alternately
accept Sun proprietary XAUI-based cards to support 10Gbps Ethernet networking.
Note that XAUI cards are only supported in slots 0 and 1.
Internal storage in the Sun SPARC Enterprise T5140 server is handled by up to eight
small form factor (SFF), 2.5-inch, internal SAS hard disk drives. Both the Sun SPARC
Enterprise T5140 and T5240 Servers currently support 73 or 146 gigabyte disks at
10,000 revolutions per minute, or RPM. The disks are attached to a SAS and SATA
disk backplane and managed through an LSI 1068 SAS/SATA disk controller.
The Sun SPARC Enterprise T5240 server supports up to sixteen small-form-factor
SAS disk drives.
The system interfaces offered by the Sun SPARC Enterprise T5140 and T5240 servers
are as follows:
Four 10/100/1000Base-T Ethernet ports.
One RS-232 serial port.
Four external Universal Serial Bus 2.0 ports. Two located in the front and two located
in the rear.

One DB-9 serial port.


And three internal, low profile, PCI-Express slots.
Note that the 3 PCIe slots on the motherboard support 3 PCIe riser boards. Each PCIe
riser board provides 2 slots for PCIe cards, for a total of 6 PCIe cards.
The Sun SPARC Enterprise T5140 riser boards support 3 half length, half height PCIe
cards.
The Sun SPARC Enterprise T5240 riser boards support 6 half length, half height PCIe
cards.
System components that affect the Sun SPARC Enterprise T5140 servers
environment include two 720 watt, hotswappable power supply units (PSUs) and
seven hot-swappable system fan trays, 2 fans per tray, providing N+1 redundancy.
System components that affect the Sun SPARC Enterprise T5240 servers
environment include two 1100 watt, hotswappable power supply units (PSUs) and six
hot-swappable system fan trays, 2 fans per tray, providing N+1 redundancy.
Note: The Sun SPARC Enterprise T5240 server configuration with 16 disk drives and
1.4 GHz CPUs require 1200W/208-240VAC power supplies.
For monitoring environmental conditions, both servers have an Integrated Lights Out
Management, or ILOM, controller with an RJ- 45 serial management port and a
10/100BaseT Ethernet management port. System status LEDs provide system and
component status information to
administrators.
The Sun SPARC Enterprise T5140/T5240 servers offer a number of reliability,
availability, and serviceability, or RAS, features, including:
Predictive failure analysis by monitoring fan speeds, processor errors, DIMM errors,
and processor temperatures.
N+1 redundant fans.
N+1 redundant power supplies.

Independent host operation - This means that service processor malfunctions will not
cause the server to stop functioning once the server is powered on.
Indicator Light Emitting Diodes, or LEDs, on the front and back of the chassis, which
allow problems to be detected and easily isolated.
And available software-based RAID levels 0, 1, 1E, and 10, which can enhance disk
availability and performance.
The Sun SPARC Enterprise T5140/T5240 servers have the following circuit boards
installed in the chassis:
A motherboard.
One power distribution board.
One power supply backplane.
One paddle board.
One disk backplane.
One USB board.
Two fan boards.
Three PCIe riser cards.
And a memory board assembly.

The motherboard is actually an assembly made up of the motherboard itself and a tray
or carrier. The motherboard assembly comes in several different versions, with the
only difference being processor speed and the number of cores.
The motherboard includes two UltraSPARC T2 Plus processors, slots for 16 DIMMs,
memory control subsystems, and all system controller, or ILOM logic.

In addition, a removable NVRAM contains all Mac addresses, host ID, and OpenBoot
PROM configuration data. When replacing the motherboard, the NVRAM can be
transferred to a new board to retain system configuration data.
The motherboard has various port connectors for USB, serial, and network.
The service processor (ILOM) subsystem contains a PowerPC Extended Core, and a
communications processor that controls the host power and monitors host system
events (power and environmental). The ILOM controller draws power from the hosts
3.3V standby supply rail, which is available whenever the system is receiving AC
input power, even when the system is turned off.

The power distribution board distributes main 12v power from the power supply
backplane to the rest of the system.
It is directly connected to the paddle board and the power supply backplane through
bus bars, and to the motherboard through a ribbon cable.
The power supply backplane distributes main 12v power from the power supplies to
the power distribution board.
The power supplies plug into connectors on the power supply backplane.
The paddle board is an assembly made up of the board, a metal mounting bracket, and
a top cover interlock, or kill switch.
The paddle board serves as the interconnect between the fan connector boards and the
disk drive backplane.
The disk backplane includes the connectors for the SATA or SAS drives, as well as the
interconnect for the USB board, Power and Locator buttons, and system component
status LEDs.
The USB board connects directly to the disk backplane. It is packaged with the DVD
drive as a single customer-replaceable unit (CRU).
The fan boards carry power to the system fan modules.

In addition, they contain fan module status LEDs, and transfer I2C data for the fan
modules.
Both the T5140 and T5240 systems have three PCIe riser cards which are each
inserted in a slot to the rear of the motherboard.
Note: that the slots you see on the motherboard are not industry standard PCIe slots.
They are Sun proprietary slots that only accommodate the Sun riser cards.
The Sun SPARC Enterprise T5240 server has a mezzanine board assembly which has
sixteen additional FBDIMM slots.
Most of the electrical connectivity in the Sun SPARC Enterprise T5140/T5240 servers
is accomplished through connectors on the systems infrastructure boards.
The only system cables in the chassis are:
the power distribution board to motherboard ribbon cable.
the power supply backplane to power distribution board ribbon cable.
the hard drive data cables.
and the top cover, interlock switch cable.
The diagram shown illustrates the overall system architecture for the Sun SPARC
Enterprise T5140/T5240 server.
As previously mentioned, The Sun SPARC Enterprise T5140 server and Sun SPARC
Enterprise T5240 server use the same motherboard. The motherboard has two
UltraSPARC T2 Plus CPUs.
CPU0 connects to PCIe switch 0, PLX 8548.
CPU1 connects to PCIe switch 1, PLX 8548.
The Sun UltraSPARC T2 Plus processors feature integrated, on-die memory controller
units, optimizing memory performance and bandwidth per CPU.

Each processor supports up to:


8 DIMM slots in the Sun SPARC Enterprise T5140
16 DIMM slots in the Sun SPARC Enterprise T5140
Below the CPUs, are the two PCIe switches, PCIe switch 0 and PCIe switch 1.
In terms of PCIe slots, PCIe switch 0 supports:
PCIe slot 3, which is x8 electrically, T5240 only
PCIe slot 1, which is x8 electrically
and PCIe slot 2, which is x16 physically, x8 electrically
PCIe switch 2 supports:
PCIe slot 0, which is x8 electrically
PCIe slot 4, which is x8 electrically, T5240 only
and PCIe slot 5, which is x8 electrically, T5240 only
Note: The Sun SPARC Enterprise T5140 server has PCIe slots 0, 1 and 2. The Sun
SPARC Enterprise T5140 server has PCIe slots 0, 1, 2, 3, 4, and 5.
Continuing with the connections to the PCIe switches...
PCIe switch 0 has a connection for:
The LSI 1068E SA/SATA disk controller.
A PCIe/PCI bridge, that in turn connects to a PCI/USB bridge, supporting the USB
ports on the rear of the chassis, the USB ports on the front of the server, and the DVD
drive.
Another port off PCIe switch 1 connects to the Neptune Ethernet controller, which in
turn supports:

Four gigabit Ethernet ports on the rear of the server, as well as XAUI slot 0 and slot 1.
The service processor connects to the CPUs through the serial system interface (SSI)
communications buses.
The service processor uses the Motorola MPC885 PowerPC chip.

The service processor is the hardware portion of the lights-out-management system


implemented on the Sun SPARC Enterprise T5140/T5240 servers. Unlike previous
versions of Sun service processors, the service processor hardware is not on a separate
card, but is integrated on the motherboard.
The hardware components of the service processor include:
A field-programmable gate array (FPGA) device that controls aspects of system
power and acts as the primary ILOM to host server communications gateway
I2C devices responsible for monitoring the servers environment and FRUID data
A Motorola MPC885 microprocessor, which contains its own instruction and data
caches, and a built-in memory controller
Management ports
Click the links provided to view additional information on the topics presented.
The illustration on your screen depicts on-die components and data paths among the
different components for the UltraSPARC T2 Plus processor.
Within each CPU, you have eight cores communicating through the Cache Crossbar to
the 4 MB total of 16-way associative L2 cache and then through the MCU, or memory
controller unit, channels.
Youll notice in the diagram that each CPU core has its own floating point unit (FPU).
Each core also has its own crypto unit.

Youll also notice that each CPU has two MCU, compared to four in the Sun
UltraSPARC T2 processors. Each MCU has two Coherency and Ordering Units,
labeled CU on the diagram. The CUs are the blocks that get all the memory and IO
requests, serializes them for global visibility and routes them to the right node's MCU
based on interleave factor.
Connecting the two CPUs are four coherence planes, labeled Coherence Plane 0
through Coherence Plane 3. These are the only communication paths between the
CPUs. All the memory cache and I/O traffic flowing between the two CPUs uses a
coherence plane.
Click as indicated on your screen to view a summary of the features and differences
between the Sun SPARC Enterprise T5140 and T5240 servers.
The Sun SPARC Enterprise T5140/T5240 is available in five configurations. Each
configuration is differentiated by:
CPU,
Memory,
And hard disk drive availability.
The following base components are common to all available configurations:
A chassis that is
A 1-rack unit in the T5140,
And a 2-rack unit in the T5240.
Two sockets for Ultra SPARC T2 Plus processors.
Sixteen memory slots in the T5140,
And 32 memory slots for the T5240.
Two 146 Gbyte disk drives.

One DVD drive assembly.


or PCI-Express (PCIe) slots,
There are three available in the T5140,
And six available in the T5240 server.
Four gigabit Ethernet ports.
Four USB 2.0 ports
An MPC885 integrated service processor.
Two power supplies.
And a standard accessory kit with documentation.
Click as indicated on your screen to view the details of each available configuration.
The Sun SPARC Enterprise T5140/T5240 servers are the latest offering in Suns
family of CoolThreads servers. CoolThreads servers are the fastest and most space
and energy efficient systems that Sun offers.
Among the CoolThreads servers are:
The Sun SPARC Enterprise T1000 (code name: Erie) and T2000 (code name: Ontario)
servers. These servers were the first to introduce CoolThreads technology. They have
a single UltraSPARC T1 (code name: Niagara) processor, can process up to 32 threads
simultaneously, and support up to 64 Gbytes of memory.
The second generation of CoolThreads servers is represented by the Sun SPARC
Enterprise T5120/T5220 (code name: Huron) servers. These servers have a single
UltraSPARC T2 (code name: Niagara II) processor. The processor has integrated onchip cryptographic acceleration and 10 Gigabit Ethernet, enabling secure computing
at wire speed. It can process up to 64 threads simultaneously and supports up to 64
Gbytes of memory.

We now have the next generation of CoolThreads servers: the Sun SPARC Enterprise
T5140/T5240 servers, which as previously stated, have two UltraSPARC T2 Plus
processors, can process up to 128 threads simultaneously, and support up to 128
Gbytes of memory
The Sun SPARC Enterprise T5140/T5240 servers are targeted at IT managers looking
for more cost-effective computing solutions to meet their organizations growing
computing needs.
The Sun SPARC Enterprise T5140/T5240 servers also meet the needs of customers
requiring a basic but high-performance computing platform in a small footprint that
provides investment protection with its market-leading price and performance.
The T5140/T5240 servers will also appeal to system administrators because of their
ease of system growth and manageability, and their support of multiple operating
systems.

The Sun SPARC Enterprise T5140/T5240 servers are well suited for virtualization and
consolidation, database, and business processing applications in fields including
manufacturing, financial services, service providers, video rendering, government,
and education.

LDoms are currently supported on the following UltraSPARC T1-based systems: the
Sun Fire T1000, the Sun Fire T2000, the Netra T2000, the Netra CP3060 Blade, and
the Sun Blade T6300 Server Module.
There are many methods for virtualizing or partitioning a system into multiple discrete
operating environments. Each method has different underlying functionality
and therefore different applications. Three methods of virtualization offered by Sun
are Solaris Containers, Sun Fire Dynamic System Domains, and Logical Domains.

Solaris Containers can create multiple virtualized environments within one


Solaris kernel structure. This keeps the memory footprint low and provides
flexibility in fine-grained resource controls, which are good for consolidating large
numbers of dynamically resourced environments within a single kernel or version of
Solaris.
Solaris Containers can create multiple virtualized environments within one
Solaris kernel structure. This keeps the memory footprint low and provides
flexibility in fine-grained resource controls, which are good for consolidating large
numbers of dynamically resourced environments within a single kernel or version of
Solaris.
Sun Fire Dynamic System Domains allow you to create electrically isolated domains
on midrange and High-End Sun Fire systems. These domains offer the maximum
security isolation and availability in a single chassis, and combine many redundant
hardware features for high availability. They are great for consolidating a small
number of mission-critical services with security and availability.
The features of Logical Domains sit somewhere in the middle of Solaris Containers
and Sun Fire Dynamic System Domains offering isolation between the various
domains, but achieved via a firmware layer. They lower the
hardware infrastructure requirements drastically. They also are great for cost effective
security and consolidation, with sun4v support for multiple operating environments.
A Logical Domain or LDom is a full virtual machine, with a set of resources such as
a boot environment, CPU, memory and I/O devices and ultimately its own operating
environment. It is isolated by virtue of the Hypervisor's capability to be an
intermediate step between the
operating environment and the hardware to virtualize.

First released on the Sun Fire T1000 and T2000 systems, the Hypervisor, a firmware
layer on the flash PROM of the motherboard, is a thin software layer with a stable
interface sun4v - between the operating system and the hardware. The Hypervisor
provides a set of support functions to the operating system, so that the OS does
not need to know intricate details of how to perform functions with the hardware.
This allows the operating system to simply call the Hypervisor with calls to the sun4v
platform. Because of this stable interface which does not change, the operating
environment does not require updating even if a new generation of machines with, for
example, faster CPUs is introduced therefore
creating a consistent programming model .
The Hypervisor layer is very thin and exists only to support the operating environment
for hardware-specific details. More importantly, as the Hypervisor is the engine that
abstracts the hardware, it can choose to expose or hide various aspects of the hardware
to the operating environment. For example, it can expose some CPUs but not others,
and some amount of memory but not all to a specific operating environment. The
Hypervisor can then create a so-called virtual machine, which also provides the
OpenBoot stack.

The slide shows the relationship between the hardware layer and the created LDoms,
shown here as a Virtual machine.
The Hypervisor sits between the hardware and the virtual machines.

There are several different types of LDoms whose names derive from their usage.
A Control Domain is an LDom that creates and manages other LDoms.

An LDom that serves up devices, virtual network switches, and SANs to other LDoms
is called a Service Domain.
And if you simply want to run an operating environment like Solaris - without
sharing devices or controlling domains, you create a Guest Domain.
You can also create combinations of domain types for instance, combining the
functions of a Control Domain and Service Domain.
The Control Domain forms the basis for communications among the Hypervisor, the
sun4v platform, and the other domains, allowing for the creation and control of
LDoms.

The Control Domain contains the SUNWldm package, which consists of the LDoms
Manager utility and the associated daemon processes required for LDoms. It is the
first domain created during the LDoms installation procedure.

Interface to the Hypervisor is made via the command line with the LDoms Manager.
This application understands the mapping between the physical and virtual devices,
and interacts with the various components to sequence changes - such as the addition
or removal of resources, and even creation of an LDom. The LDomS Manager
communicates these changes to proxy agents located in the supported operating
environments of the Guest Domains that are undergoing changes.
Because the Control Domain can interact with and control other domains even
remove them entirely - this domain is viewed from a security perspective similarly to
the system controller, which should be hardened and secured. One method for doing
so is to apply the Solaris Security Tool kit.

The role of the Proxy Agent in the operating environment is to allow the
communication of events from the Hypervisor, notifying the operating environment of
actions such as the addition and removal of devices.
An OS that supports such features can then signal back to the Hypervisor that it is
ready for the action to occur.
A Service Domain is one that shares out or virtualizes physical devices to other
logical domains on the system. The Service Domain takes ownership of a component,
for example, a PCI controller - and then shares one of the interface cards found under
that controller to another domain via a Logical Domain Channel. You will learn more
about these devices further in this module.
Since the Service Domain must support specific LDoms functions such as
virtualization services, it must run the same version of the Solaris operating system as
the Control Domain. Unlike the Control Domain, where you can only have one
running at any one time, you may have multiple Service Domains currently up to a
maximum of two.
A Guest Domain is a domain that does not run the control structures for LDoms, such
as daemons and processors that interact with the Hypervisor. A guest domain also does
not share out devices to other domains. It is simply a consumer of the resources
allocated to it. It must run an operating system that recognizes both the sun4v
platform and the virtual devices presented by the Hypervisor. The Minimum
supported operating system is Solaris 10 Update 3 (11/06).

Logical Domains communicate via Logical Domain Channels or LDCs. These are
channels of communication by which data can be moved from one domain to another.
LDCs are the mechanism by which virtual networks can be established between
LDoms, and they are the conduit for services such as I/O provided to a Guest
domain.
LDCs are explicitly created. They are defined by the Control Domain and bound to
the designated LDoms with specific services at each end of the channel. It is a strict
point-to-point link rather than the traditional networking paradigm of a port opening

upon request. This helps to make LDCs more secure, and as they are created logically
within the Hypervisor, they are flexible and fast to set up.
The goal of the Service Processor or the Advanced Lights Out Manager, known as
ALOM, is to manage the hardware so it does not know about LDoms.
However there are some small interactions between the Hypervisor and the service
processor for example, storing the LDoms configuration so that they are persistent.
The Service Processor requires firmware 6.4.X.
The UltraSPARC T1-based systems support three different Logical Domains types
and a total of 32 LDoms. The LDom types are the Control Domain, the Service
Domain, and the Guest domain.

The virtualized devices are any hardware resources on the system that are abstracted
by the Hypervisor and presented to the LDoms on the system. They can take the form
of directly virtualized devices such as CPU and memory, and those devices like I/O
and Network devices that are proxied from a Service Domain for use by other
domains. These are physical devices that are translated to virtual devices by the
Hypervisor and presented by the Service Domain to other domains. They also include
console and cryptographic devices.
The OpenBoot environment forms the basis for initial program loading and execution,
typically an operating environment. It also provides other features such as diagnostics
and boot time parameters to control operation. In Logical Domains, the OpenBoot
environment is virtualized and made available to multiple partitions as discrete boot
environments. OpenBoot is the basis for running the operating environment in an
LDom.
The OpenBoot ok prompt is the first thing you will see when connecting to the
console of a newly created LDom, and a familiar sight for those familiar with Sun's

SPARC hardware. All logical domains in a system will have the same version of
OpenBoot.
All CPUs presented by the Hypervisor are referred to as Virtual CPUs or VCPUs.
On a Sun Fire T1000 or T2000 system, each of the cores of the system has 4
executing threads, represented as Virtual CPUs by the Hypervisor. Thus an 8 core Sun
Fire T2000 would have 32 Virtual CPUs able to be partitioned among the various
LDoms on the system.
Similar to CPUs, the memory contained in the Niagra-based servers is virtualized to
be presented in various amounts to guest LDoms by the Hypervisor. The memory can
be allocated in increments as small as 8k chunks. Most importantly, memory is
represented to the Guest Domains as starting from the same offset.
The process of translating memory from the platform to domains is referred to as
mapping. This happens in most operating environments. In Solaris, applications
already see memory that is remapped by the kernel from a real address to a virtual
one. The Hypervisor, working with the memory management units in the hardware,
simply takes an additional step of mapping from the hardware, or physical
environment, to what is presented to the operating environment.
The I/O devices on a sun4v system, such as internal disks, PCI controllers, and
attached adapters and devices, can be presented to the various LDoms in several ways
through the Hypervisor, depending upon the application requirements and the
administrative model that is needed.
The virtual disk server is a way to present an LDom with a device from which to boot
an
operating system. The image can take many forms, including:
An entire physical disk this could also be a logical storage unit from a SAN device,
which is sometimes referred to as a LUN,
A Single Slice of a disk,
A File via the loop-back file system called lofi, or

A ZFS (Zettabyte File System) volume.


The traditional model of direct device control by an operating environment is
maintained by the LDoms model. It uses a mode in which the Hypervisor creates a
mapping from the device to a virtual interface. It then allows the logical domain to
maintain ownership of the device.
The maximum number of logical domains having Direct I/O devices is 2.
This is based upon the systems having 2 PCI-e controllers, PCI-e A and PCI-e B, each
able to be owned independently.
So in the case of 2 Service Domain deployments, each could own a PCI root and the
devices in the tree. Because this limitation can be restricting, it is one of the reasons
for having a virtualized approach to I/O providing the flexibility for more logical
domains to have access to devices such as PCI cards and their devices by sharing
them without direct ownership.
In the direct I/O method, two domains could each control or own a single PCI-e
controller. The diagram shows a Logical Domain owning a PCI-E controller.
In contrast to the Direct Devices model, the Virtual Devices model provides the
capability for devices to be shared out to multiple domains.
This allows for the creation of virtual SANs thereby providing additional
consolidation benefits in the form of rationalization of storage and interfaces, and the
reduction in administrative burden.
The concept of Virtual Devices is based upon at least one domain having ownership of
a device through the Direct Devices model.
This domain is designated as a Service Domain, and it establishes a path to the other
domains via a Logical Domain Channel.
The operating system in a Guest Domain sees a virtual device driver as if it were a
local device.

In this diagram, the Service Domain owns a PCI-e controller and allows Logical
Domain A to access a virtual form of I/O via the Logical Domain Channel (LDC).
With LDoms, the network is virtualized along with everything else for the virtual
machine. This means that network adapters are virtualized --they are called vnet -and a service is created to share them from the Control domain to the Guest Domains.
Additionally it is possible to create a virtual switch, called a vsw, which is able to
provide a channel of communication that multiple domains can use in this case via a
networking protocol such as TCP/IP.
Networking devices other than those on-board the system such as PCI-e networking
cards can be owned directly by a domain or virtualized as a service. The on-board
network ports are shared as a service by default.
The console has traditionally been the conduit for accessing system-level messages
for administrative purposes. These include reviewing boot-up messages during an
intervention when other methods cannot be used such as when networking services
are down.
The console device as a connection to the OpenBoot environment is also virtualized
via the Hypervisor. Connection is made through a network service in the Control
Domain at a specific port.
For example, connecting to the local host within the Control Domain on port
5000 will by default connect to the first logical domain other than the Control
Domain. It is also possible to specify a virtual console concentrator, or vcc, to group
virtual consoles to assist in administration.
The cryptographic devices on the Sun Fire T1000 and T2000, referred to as Modular
Arithmetic Units (MAUs), provide high performance, dedicated cryptographic
engines to perform tasks such as encrypting and decrypting network traffic that could
occur between a secure sockets layer, or SSL, web server and an application server.

In LDoms, the MAUs are virtualized, so they are referred to as virtual MAU, r
VMAU.

There are 8 VMAU units on the Sun Fire T1000 and T2000, 8 core model servers 1
per core of 4 VCPUs. As they are part of a core, they may only be bound to a domain
that contains at least one strand from the parent core.