The MX000 servers:

Are a family of symmetric multiprocessing or SMP systems

Are Sun SPARC binary-compatible servers to run mission-critical and highperformance computing applications
Incorporate mainframe features, such as dynamic domains, partitions, and robust
hardware redundancies
Offer unparalleled performance, scalability, and reliability, availability, and
serviceability, or RAS

The MX000 server product line consists of five unique models that are categorized
as either mid-range or high-end servers.
The M4000 and M5000 are mid-range servers. These servers are designed as a
follow-on to the Sun Fire V490 and V890 servers. The mid-range server product
family is comprised of the M4000, a 6-Rack Unit, or RU, and the M5000, a 10-RU,
rack-optimized servers with dual- core SPARC64 VI processors.
The M8000, M9000 and M9000+ are high-end servers. These servers are designed
to replace the Sun Fire mid-range and high-end servers (E4900, E6900, E20K &
E25K). They use the dual core (each core with two threads) SPARC64 VI processors.
Each of the MX000 servers merge mainframe technologies for high reliability and
take advantage of the knowledge accumulated over time with high-speed
technologies of super computers and the flexibility of UNIX server development.
Key technologies for this product line include:
Resource flexibility: Each M8000, M9000, and M9000+ server contains SPARC64 VI
microprocessors. They use dual or multi-core SPARC64 VI processors, each core
contains 2 threads. The system uses a partition function that enables the
microprocessor to function as multiple servers to facilitate flexible use of resources,
thereby enabling efficient execution of job operations.
For error isolation, the MX000 servers feature a robust fault isolation mechanism. If
an error does occur, the corresponding error can be corrected or isolated, possibly
without halting the server. This feature minimizes problems in many cases, thereby
improving job continuity.
The MX000 servers are the result of an alliance between Sun Microsystems and
Fujitsu. These servers:
Are Sun and Fujitsus next generation, mid-range and high-end server models

Have been jointly designed by Sun and Fujitsu

Are being jointly manufactured by Sun and Fujitsu
Run the Solaris 10 OS and are SPARC v9-compliant for binary compatibility with Sun
Fire and PRIMEPOWER servers
Are fully compliant with Suns standards for software, storage, and services
This alliance entails shared investment of MX000 servers product development
costs and has allowed Sun to invest more heavily in chip
multi-threading (CMT) technology for future enterprise computing products.
The M4000 and M5000 mid-range servers are designed with the following key
factors in mind:
Price
Performance
Competitive reliability
Reliability, Availability, and Serviceability or RAS
Rackmount installation
The M4000 and M5000 mid-range servers provide the following robust capabilities:
Application acceleration capabilities include:
Dual-core SPARC64 VI microprocessors, the next generation SPARC-based processor.
The CPU is designed to have the highest reliability, with guaranteed data path
integrity, automatic recovery and instruction retry, plus all SRAMs and registers are
protected.
An upgrade option to the SPARC64 VI+ microprocessor with mixed speed support.
A High speed interconnect of 304.2 gb per second
And industry standard Peripheral Component Interconnect PCI Express or PCIe I/O.
These servers use fast Chip Multi-threading or CMT processors with a great deal of
memory and the latest industry-standard I/O.
Improved RAS features include:
Increased uptime. The product line was designed with the highest RAS standards in
mind. The servers are end-to-end ECC protected with massive health checks in
place.

Its modular design capabilities include: 6 RU and 10 RU, rack-optimized enclosures

Support for Dynamic reconfiguration (DR)
And support for two or four bootable domains
Lastly, world class OS support includes:
Solaris 10 OS. These servers will run on Solaris the industry leading UNIX operating
system.

The M4000 and M5000 mid-range servers share the same architecture and use the
same or similar components. There are, however,
distinctive differences in the way the servers are configured and managed. These
differences are described later in this module.
The M4000 and M5000 share the following set of common features. These include:
Rack-mounting capability
Multiple central processing unit or CPU modules
Symmetrical Multi-processing or SMP, which makes multiple CPUs available to
complete individual processes simultaneously
eXtended System Control Facility Unit or XSCFU board
PCIe and PCI-X I/O bus
PCI cassettes
Operator panel
Hard disk drives or HDDs, an optional tape drive unit, and a DVD drive
Redundant power and cooling
Hot field-replaceable unit or FRU removal/replacement capability
And lastly, I/O expansion with the External I/O Expansion Unit
Click on the link provided for details on the M4000 and M5000 server common
feature set.
Next, let's take a high-level look at the high-end servers in this product line. The
M8000, M9000, and M9000+ high-end servers are designed with:
Highest availability

Highest absolute performance

Sophisticated RAS features, and
Highest scalability
Similar to the mid-range products in this product line, the M8000, M9000, and
M9000+ servers provide a set of robust, state of the art capabilities. These
capabilities include:
Application acceleration capabilities such as:
Dual-core SPARC64 VI+ microprocessors
An upgrade option to the SPARC64 VI+ microprocessor with mixed- speed support
A high-speed interconnect at 304.2 gigabits per second, and
An industry-standard PCI and PCIe I/O
Mainframe-class RAS capabilities such as redundancy for:
Power
Cooling, and
Service processors
Unmatched configuration flexibility in the areas of:
Modular building block architecture, which includes CPU/Memory Unit or CMU, I/O
Unit or IOU, expansion box, and eXtended System Controller Facility, or XSCF,
software.
Symmetric multiprocessing, or SMP, architecture with as many as 128 cores, and
As many as 24 bootable domains
Lastly, as described earlier in this module, Sun delivers the most powerful, open
operating systems on the market with the Solaris 10 OS.
The M8000, M9000 and M9000+ servers hardware configuration includes the
following components:
Centerplane
CPU/memory subsystem or CMU
I/O subsystem
PCIe I/O bus

PCI cassettes
System bus
Multiple CPU modules
SMP, which makes multiple CPUs available to complete individual processes
simultaneously
eXtended System Control Facility Unit board or XSCFU_B
Operator panel
HDDs such as an optional tape drive unit and a DVD drive
Redundant features including power and cooling
Hot FRU removal/replacement capability
And, I/O expansion with the External I/O Expansion Unit
Note Hardware for the M8000, M9000, and M9000+ servers is configured
differently from the hardware in the M4000 and M5000 servers, even though all five
servers are in the same server series.
Click on the link provided to learn more about high-end server features.

Before continuing with the next topic, common features and components for the
MX000 product line, let's take a look at some key terms. These terms will help you
better understand the topics presented in the section.
The first term is CPU/memory unit or CMU. The CMU consists of the CPU and
memory.
CPU Core The CPU chip contains multiple processor units. The CPU is considered to
be a virtual processor.
I/O Unit or IOU. The IOU contains PCI card slots and HDDs
The next term is Physical System Board or PSB. The PSB consists of CMU and the
IOU.
Extended System Board or XSB. The XSB consists of either the entire CMU with UniXSB and IOU or a quarter of the CMU and IOU with Quad-XSB, a quad-XSBs. Note
that the uni-XSB is the system board with the undivided hardware resource on a
physical system board. While the quad-XSB is the divided system board configured
with the hardware resource on a physical

system board, which is segmented into four.

The last term in this section is Logical System Board or LSB. The LSB number is
assigned to an XSB to identify it in a domain.
We are pausing the presentation to allow you time to more closely review this
terminology. Click the play button to resume the training.
Caution Although the M4000 server can be configured with four XSBs and the
M5000 server can be configure with eight XSBs, the servers I/O resources only
support two XSBs on a M4000 server and four XSBs on a M5000 server.
Now, let's take a look at the common features and components for this product line
in more detail. At the end of this slide you will be provided with links to additional
information for many of the features discussed.
The common features and components include:
CPU/memory board, typically referred to as CMU consists of the CPUs and memory
DIMMs. In the MX000 servers, the CPU and memory are located on different boards
and in different slots. In the MX000 high-end servers, the CPUs and memory are
located on the same physical board.
Note Mixing of CPUs running at different clock frequencies is supported across the
CMUs within a single system. The latest CPUs can therefore be installed when
available. This allows for further performance scaling capabilities.
The next feature common across the product line is the use of the SPARC64 VI
microprocessor. All CPUs in MX000 servers use the SPARC64 VI high-performance
multiprocessor. The microprocessor implements an instruction retry function
allowing an operation to continue when an error has been detected. The
architecture also implements multithreading technology which is a combination of
chip multithreading or CMT and Vertical Multithreading or VMT technology. The CPUs
have two physical cores and each core has two strands with VMT structures. This
allows four threads to run in parallel. The two strands that belong to the same
physical core share most of the physical resources, while the two physical cores do
not share physical resources except the Level 2 (L2) cache and system interface.
Another feature common across the product line is the implementation of the
memory boards. The memory subsystem controls memory access and cache
memory. The memory subsystem uses double-data-rate (DDR)-II memory and can
implement up to eight-way interleaving, providing higher-speed memory access.
Memory mirror mode is also supported for every pair of memory buses in a CMU.
Additionally,this platform supports chipkill operations which are essentially the
same as RAID for disk subsystems.

In terms of I/O features, the product line provides I/O functionality using a
component called the IOU. Each IOU has one I/O controller, which manages four PCI
buses (also called PCI bridges). The IOU also contains cassettes that support PCI
cards.
The M4000 and M5000 servers four upper slots of the IOU support PCIe adapters
while the lower slots support PCI-X adapters. On the M8000, M9000, and M9000+
servers only PCIe slots are supported.
Lastly, MX000 servers support a mix of PCIe and PCI-X cards. The industry standard
PCI cards used are not hot-pluggable, so the PCI cards are installed into a hotpluggable PCI Cassette unit.
In terms of disk and tape options,
all MX000 servers support the insertion and removal of PCI cards for certain PCIe
and PCI-X hot-plug controllers. PCI cards must be un-configured and disconnected
using the Solaris OS cfgadm command before the cards can be physically removed.
Additionally, the multi-port 80 I/O Box provides a host system with additional slots
for PCI cards. The single I/O boat configuration provides six additional slots and the
optional two I/O boat configuration provides twelve slots.
Next, let's take a look at Extended System Control Facility or XSCF features.
Note For simplicity, the XSCFU is also referred to as the service processor, and is
referred to as such for the remainder of this module.
The MX000 servers all use the Extended System Control Facility to provide system
monitoring and control. The XSCF is actually just one part of the total solution, the
firmware component, there is also a physical component where the XSCF firmware
is installed. This hardware component is called the Extended System Control Facility
unit (XSCFU). A list of service processor functionality is provided in the link on this
slide.
Additionally, two types of interfaces for XSCF firmware are available:
A command-line interface (CLI) called the XSCF shell.
A Browser User Interface (BUI) called the XSCF web.
Click on the link provided to learn more about these interfaces.
The Fault Management Agent or FMA is incorporated by default into the Solaris OS
software that resides in a domain. In the MX000 servers the service processor also
has a communication path with the domain and some fault management
information is forwarded and diagnosed by the service processor.

The service processor has a full-featured, enhanced FMA implementation which

provides fault diagnosis & reconfiguration capabilities.
Dynamic Reconfiguration or DR features. The high-end server models allow hot
insertion and removal of CMUs and IOUs. The M4000 and M5000 servers do not
allow hot insertion and removal. In addition, the system architecture and XSCF
firmware allows a servers hardware resources to be dynamically partitioned into
independent hardware domains, each running a different instance of the Solaris OS.
A domain can be configured and managed through DR while in Uni-XSB or Quad-XSB
mode.
Information on DR Basic functions is available from the link provided.
We'll pause the presentation for a moment. Click on the links provided to learn more
about that topic or subtopic. Click play to continue the presentation.
The MX000 servers provide a number of advanced features that are typically found
on mainframe-class servers. Let's take a closer look at the Reliability, Availability,
and Serviceability or RAS features incorporated into the MX000 server product line.
Reliability features incorporated into the MX000 servers include:
Constant monitoring of the server to ensure that the server is operating normally.
Providing an historical log of all pertinent environmental conditions and error
conditions when encountered.
Dynamically re-allocating CPU resources into an OS using DR without interrupting
the applications that are running. If a process has been bound to a specific CPU,
dynamic CPU de-allocation is not possible.
Providing ECC protection for the address and data paths.
Mitigating failures by providing a rich check/recovery mechanism in the processor.
Cooperating with the XSCF firmware to periodically check if software, including the
domain OS, is running. This is referred to as Alive Check.
Checking the status of each component to detect signs of an imminent fault, such
as system down occurrences which Helps to prevent system failure.
Periodically performing memory patrol to detect memory errors. Memory patrol
prevents faulty areas from being used which helps to prevent system failures.
Availability features incorporated in the MX000 servers include:
Using redundant components as a strategy to improve availability. When one
component fails, the redundant component(s) ensure that services that were

delivered by the failed component continue to be available. Power supplies and fans
are examples of redundant components.
Maintaining a log and threshold of errors and taking proactive recovery actions that
minimize the likelihood of a system outage. This is referred to as predictive failure
analysis.
Providing a limited single hardware point of failure prevents any domain from
rebooting the OS.
Using fault isolation of interconnect segments
Providing chip-kill memory support which allows recovery from memory chip failures
such as Dynamic Random Access Memory or DRAM failures. These types of failures
result in a correctable error.
Providing dual power grid capability. The alternating current or AC power subsystem
is completely duplicated, which allows a redundant connection to a different power
grid. This is not available on the M4000 or M5000 servers, although they do offer
power supply redundancy.
Using an automated email notification system to detect a problem and to send an
event notification through email, the SNMP trap, the BUI interface, or to remote
system monitors.
Supporting an enhanced retry and degradation function for the detected faults.
Shortening the system downtime by using automatic system reboot.
Collecting XSCF fault information about faults, log files, configurations, and
environmental conditions.
Serviceability features incorporated in the MX000 servers include:
Supporting first fault isolation based on in-line error detection with the capability to
detect errors within FRU boundaries.
Logging errors immediately.
Supporting multi-stage alerts and reporting through which errors are logged and
reported to higher software levels. They are made available to system management
and remote monitoring.
Automatically directing the maintenance activities of the service personnel.
Performing maintenance activities while the system is in operation. This is referred
to as limited concurrent maintenance.
Electronically reading FRU-ID information, such as part numbers and serial numbers.

Providing chip or ASIC test as well as processor tests or BIST.

Supporting dynamic system domains, including domain isolation. This strategy
provides multiple domains. Each domain is capable of running different OS
instances and each domain operates independently from other domains.
Using system and FRU-level fault indicators, or light-emitting diodes (LEDs) to
visually identify the system status including FRU operational status and service
required.
And using XSCF firmware or service processor firmware for platform administration,
platform loggingmem, and service related actions.

With its rack optimized enclosure, the M4000 server is a 19 inch, 6 rack unit or RU
system which converts to 10.35 inches or 263 millimeters .It is the smallest server
in this server line. Let's take a look at the components inside an M4000 server.
From the front you can see the operator control panel. Inside the enclosure, the
server contains a single motherboard, 2 CPU boards and 4 memory boards with up
to 8 DIMMs per board. It also provides redundant power and cooling through 2
power supplies, setup in an N+1, configuration and 4 system fans. Internal storage
consists of one DVD-ROM, two Serial Attached SCSI or SAS hard disk drives and one
optional tape drive unit.
From the rear, you can see the IOU which contains PCI card slots and HDDs. There
are 5 PCI cassettes per IOU which consists of four PCIe and one PCI-X. The number
of supported domains is two. This number is limited by the IOUs. The last
component we will locate in the M4000 server is the XSCFU or service processor.
As with the M4000 server, the M5000 server provides a rack optimized enclosure. It
is a 10 RU system which converts to 17.25 inches or 438 millimeters. It is the
equivalent of two M4000 servers and is the largest in the mid-range server product
line.
From the front you can see the operator control panel. Inside the enclosure, the
server contains a single motherboard, 4 CPU boards and 8 memory boards with up
to 8 DIMMs per board. It also provides redundant power and cooling through 4
power supplies, setup in an N+1, configuration and 4 system fans. Internal storage
consists of one DVD-ROM, four Seriel Attached SCSI or SAS hard disk drives and one
optional tape drive unit.
From the rear, there are 2 IOUs each containing 5 PCI cassettes per IOU. Each PCI
cassette consists of four PCIe and one PCI-X. The number of supported domains is
four. This number is limited by the IOUs. The last component we will identify is the
XSCFU or service processor.

Each CPU board found on a mid-range server contains two physical processor chips.
Each processor chip contains two CPU processor cores which are effectively, two
independent CPUs. While each core has two threads or instruction streams.
To the Solaris operating system, a thread is seen and mapped as a processor. A fully
configured Sun SPARC M4000 server shows a total of 16 CPUs installed. A
breakdown of how this is implemented is as follows:
A fully configured MX000 mid-range server has 2 CPU boards, each CPU board
contains 2 CPU chips, each CPU chip contains 2 cores, and each core contains 2
threads or instruction streams for a total of 16 processors mapped to the Solaris
operating system.
Note The CPU board architecture is described later in this course.

As noted earlier in this module, there are three high-end servers in this product
family. In this section, we'll take a closer look at the components found in the highend servers in the product family.
The first is the M8000 server. It is the smallest in the high-end server line. This
server provides SMP/CMT scalability using as many as 32 processor cores and
flexible, dynamic system domain configurations.
The operator panel is on the front cabinet door. The panel contains, from left to
right, the power LED, XSCF LED, fault LED, power button, and keyswitch.
Accessible from the front of the machine are eight power supplies and rack space
for the dual power grid option along the top. Underneath are a DAT drive, a DVD
drive, a connection to the operator panel on the front door, two DCA boards, and
two XSCFU boards. Four CPU memory boards reside to the right along with a ground
connection. Six fan trays reside along the bottom of the machine and three filters
run under the fan trays.
From the rear of the server you have some open rack space along the top. Towards
the middle are six fan trays. Along the bottom are four IO units with three filters
running underneath.
Click the link provided to learn more about the Sun SPARC Enterprise M8000 server
hardware configuration.

The M9000 server is next in the high-end server line. This server provides SMP/CMT
scalability using up to 64 processor cores and flexible dynamic system domain
configurations.

As with the M8000 server, the operator panel resides on the front door and opens in
the same manner. Accessible from the front along the top of the system are 15
power supplies, ground and 4 fan trays as well as AC inlets with corresponding
breakers for grid zero power. In the middle are eight filler panels for the crossboards
used in the M9000+ server. Under the crossbar board locations are two clock boards
on the top and two XSCFU boards below. Underneath there is a second ground
connection. Near the bottom of the server are the optional tape drive, DVD drive AC
connection to the operator panel on the front door and four IO units with three
filters running below.
On the rear of the system are 12 fan trays, two ground connections and locations
for eight CPU memory boards. Additionally there is space for four more IO units as
well as filters along the bottom of the server.
Click the link provided to learn more about the Sun SPARC Enterprise M9000 server
hardware configuration.

The M9000 server comes next and is the largest system in the high-end server line.
This server provides SMP/CMT scalability using up to 128 processor cores and
flexible dynamic system domain configurations.
The M9000+ server consists of two M9000 servers that are bolted together. The
M9000+ server include a total of 64 interconnect cables between the two system
frames. Of the 64 cables, 62 of the cables are data and clock cables, one is an
XSCFU_B board cable, and one is an XSCFU_C board cable. Although the system
cabinets are mostly the same, there are small differences between the two sides.
Therefore, one side is referred to as the main cabinet and the other side is called
the expansion cabinet.
The operator panel on the left chassis door acts on behalf of both servers. Each
cabinet has its own optional power cabinet. On the front of the machine along the
top are 30 power supplies and eight fan trays. There are filler panels for AC inlets
used for single phase power. Below the power supplies are four ground connections.
In the middle are 16 crossbar units with interconnect cables between the two
chassis. Under the crossbar boards are 4 clock boards and 4 XSCFU boards with
interconnect cables between the two chassis. Along the bottom of the server are
four more ground connections, a DAT drive, a DVD drive and a connection to the
operator panel on the front door. To the right are eight IO units with six filters
running underneath.
From the rear, there are twenty-four fan trays along the top. Underneath the fan
trays are four ground connections. In the middle there are slots for sixteen CPU
memory boards and along the bottom are locations for eight more IO units. Six
filters run along the bottom of the server.

Click the link provided to learn more about the Sun SPARC Enterprise M9000+
server hardware configuration.
This module presents the Sun SPARC Enterprise server architecture model, which
includes differences and similarities between the mid-range and high-end server
architecture. It also reviews physical components and presents information that
relates the architecture to how the product family performs.
To better help you learn the content in this module, be sure to keep the following
questions in mind:
How do the MX000 servers differ from other Sun servers?
What components make up the MX000 server models?
What role does each component play in the operation of the system?
What functions do the various ASICs provide?
The MX000 server architecture builds and improves on the success of previous Sun
high-end and mid-range product lines by using a new CPU and bus architecture. This
enhanced architecture includes:
A new system bus architecture based on the Jupiter bus interconnect
An enhanced CPU architecture running at speeds of 2 Ghz, or greater, and
Multi-core, multi-threaded CPUs
The Jupiter interconnect is a high-speed system interconnect between the CPU,
memory, and I/O subsystems in both the mid-range and high-end servers. It
provides superior memory and I/O bandwidth, which helps the SPARC64 VI, or
Olympus-C processor deliver superior performance.
The Jupiter interconnect provides 7.5 times more system bandwidth than the
previous generation of servers. Its maximum theoretical bandwidth is 304.2
Gigabytes per second with a latency of 258 nano second for local transactions and
498 nano second for remote transactions.
The Jupiter interconnect is central to the architecture found in both the mid-range
and high-end servers.
The Jupiter interconnect performs the following functions:
Allows the system to be divided into logically isolated domains
Enables hot-plugging and unplugging of individual boards
Let's look at the Jupiter interconnect in more detail.

The diagram shown provides a graphical representation of the M4000 server

interconnect architecture. The M4000 server is implemented within a single
motherboard. The M4000 features one logical system board with two system
controllers. Both system controllers connect to each other, as well as CPU modules,
memory address controllers, and the IOU.
Next, let's look at the interconnect diagram for the M5000 server. The M5000 server
is implemented within a single motherboard, but features two logical system
boards. Similar to the M4000 server design, each logical system board contains two
system controllers, which connect to each other, as well as CPU modules, memory
access controllers, and an IOU. In addition, each system controller connects to a
corresponding system controller on the other logical system board.

As shown in the graphic, the M8000 and M9000 servers feature multiple system
boards that connect to a common crossbar.
Well pause for a few moments so that you can review the information on this slide.
CPU and memory are organized differently on the mid-range and high-end servers.
In the mid-range server line, CPUs reside on CPU modules, or CPUMs, and memory
resides on memory boards, or MEMBs. There are two distinct boards in the midrange servers.
In the high-end server line, both the CPUs and memory reside on a CPU memory
unit, or CMU.
NOTE: Both the mid-range and high-end servers support the same type and speed
of CPUs and the same type and sizes of DIMMs. They just reside on different boards.

Sun SPARC Enterprise servers utilize the SPARC64 VI processor, which boasts an
innovative design that incorporates the latest in dual-core and multi-threaded
technology, as well as extensive reliability features.
The SPARC64 VI CPU is a high-performance, highly integrated microprocessor
implementing two SPARC V9 cores. These processors implement a combination of
CMP, or chip multiprocessing, and VMT, or vertical multi threading. The CPUs each
have two physical cores and each core has two strands or threads with VMT
structure, meaning that four strands are able to run in parallel. As mentioned
previously, the Solaris OS sees each strand as a complete processor. The two
strands that share the same core share most of the physical resources. However,
the two physical cores do not share the physical resources, except the L2 cache and

the system interface. These CPUs interconnect through the Jupiter bus, which is
scalable to 64 sockets.
The illustration shows the SPARC64 VI processor chip block diagram. C1 and C2
represent core 1 and core 2. Each core has its own floating point unit, or FPU, and
data cache, or D$. They do share the same L2 cache, which can be four or six
MBytes.
The next topic presented in this module is SPARC64 VI microprocessor memory.
The DIMM modules supported in these servers are high-capacity, dual side-mounted
units. They are Double Data Rate II (DDR II) DIMMs. The supported DIMM sizes are:
1-Gbyte DIMMs
2-Gbyte DIMMs
4-Gbyte DIMMs (stacked and non-stacked modules)
8-Gbyte DIMMs currently in progress, but not supported at release
DIMMs are installed in sets of four. You cannot mix stacked DIMMs with non-stacked
DIMMs in the same set. Stacked DIMM modules use two RAM wafers that are
stacked on top of each other. This allows larger modules to be manufactured using
cheaper low-density wafers. Stacked DIMM modules typically draw more power.
Both the mid-range and high-end servers support memory mirroring. Memory
mirroring operates by pairing DIMMs together for both reads and writes. Mirroring
divides the available memory in half and is turned off by default on all systems.
On writes, the data and ECC are written to both halves of the pair. On reads, data is
read from both halves and the ECC is compared. Errors during memory mirroring are
handled as follows:
If both copies of the pair have correctable ECC and have matching data, the data is
returned.
If one copy has uncorrectable ECC and the other has correctable ECC, the data is
returned.
If both copies have uncorrectable ECC errors, an error data packet is returned.
If both copies have correctable ECC, but the data does not match, an error data
packet is returned.
Memory mirroring is supported both in uni- and quad-XSB mode in the mid-range
servers. However, it is only supported in uni-mode on the high-end servers. This is
because in the high-end server models, there is striping between the MACs on a
board, where as in the mid-range server models there is no striping.

In the high-end models configured in quad-XSB mode, you get half of MAC0 and
MAC2 for XSB-0. This does not allow the memory mirror mode because you can only
mirror within a MAC. In the mid-range servers configured in quad-XSB mode, you get
one full MAC per quad-XSB. This makes it possible for mirroring in the mid-range
servers.
We will discuss memory access controllers next and uni- and quad-XSB modes later
in this module.

The MEMBs contain MACs in the mid-range servers and the CMUs contain MACs in
the high-end servers. The MACs provide an interface from memory to the SC ASICs.
For example, in the figure shown, you can see that each MEMB in the M5000 server
has a MAC that serves as the memorys interface to the SCs in the system.

The system controller, not to be confused with the service processor, is an ASIC that
provides an interface between the MACs, the CPUs, and the I/O controllers. The
connection on the high-end servers goes through the crossbar units (XBUs). On the
mid-range servers, the connection goes through the motherboard unit (MBU).
In the graphic provided, the SC and MAC ASICS can be seen on the CMUs found in
the high-end servers.
Let's take a closer look at the I/O boards for the MX000 product line. The I/O boards
in both the mid-range and high-end servers are called I/O units, or IOUs. However,
they are different components and support different numbers and types of cards.

The IOU in the mid-range servers supports both PCIe and PCI-X cards. The figure
shows the layout of the IOUs in an M5000 server, which supports one or two IOUs.
The M4000 server supports only a single IOU with only half of the IOU resources
shown, making this figure applicable across the mid-range server line. Each I/O unit
contains the following:
PCI cards including four short PCIe (8-lane) slots (upper four slots) and one short
PCI-X slot (133 MHz) (lowest slot).
One I/O controller (IOC) chip, which acts as the bridge chip between the system bus
and the I/O bus
And PCIe switches or bridges connected to the slots
Now let's take a look at the high-end server IOU.

The IOU in the high-end servers supports only PCIe cards. In the figure shown, you
can see the layout of the IOUs in a high-end server. All of the high-end servers use
the same board.
Each IOU contains the following:
PCI cards, including eight short PCIe 8-lane slots
Two I/O controller, or IOC, chips, which act as the bridge chip between the system
bus and the I/O bus, and
PCIe switches or bridges connected to the slots
The IOU in the high-end servers supports only PCIe cards. In the figure shown, you
can see the layout of the IOUs in a high-end server. All of the high-end servers use
the same board.
Each IOU contains the following:
PCI cards, including eight short PCIe 8-lane slots
Two I/O controller, or IOC, chips, which act as the bridge chip between the system
bus and the I/O bus, and
PCIe switches or bridges connected to the slots

Before continuing with the discussion of the MX000 server architecture, let's take a
moment to review the new naming conventions used for the boards in this server
line.
The first new board term is the Physical System Board, or PSB, which consists of
CPU, memory, and I/O.
The other new board name change is the eXtended System Board, or XSB, which
allows you to configure your PSB as
a uni-board (00-0) with all of the components assigned to it, or a quad-board (00-0,
00-1, 00-2, 00-3) with the components being divided
Now, let's take a closer look at the board architecture for MX000 mid-range servers.

The M4000 server contains one PSB. The M5000 server contains two PSBs. Each PSB
consists of one IOU with one PCI-X slot, four PCIe slots and built-in I/O consisting of
two disks and two Gigabit Ethernet ports. The PSB also contains two CPUMs with
two CPUs each, as well as four MEMBs with eight DIMMs each.

The diagram provided is a mid-range server quad-XSB block diagram. This diagram
shows two PSBs:PSB 0 and PSB 1. Quad-XSB number 00-0 on PSB 0 and quad-XSB
number 01-0 on PSB 1 both contain one CPU, one MEMB with eight DIMMS, an I/O
channel with one PCI-X slot and two PCIe slots (0 and 1), and built-in I/O consisting
of two disks and two Gigabit-Ethernet ports.
The quad-XSB number 00-1 on PSB 0 and quad-XSB number 01-1 on PSB 1 both
contain one CPU, one MEMB with eight DIMMS, and an I/O channel with one PCI-X
slot and two PCIe slots (0 and 1).
The quad-XSB numbers 00-2 and 00-3 on PSB 0 and quad-XSB numbers 01-2 and
01-3 on PSB 1 both contain one CPU and one MEMB with eight DIMMS.
We'll pause this presentation to allow you time to review the block diagram and
click the link that provides a summary of configuring a PSB as a quad-board on a
mid-range server. Click play to continue the presentation.
In the MX000 high-end server product line the M8000 server contains four PSBs, the
M9000 server contains eight PSBs, and the M9000+ contains 16 PSBs.
Each PSB consists of one IOU with eight PCIe slots and built-in I/O with four disks. It
also contains one CMU with four CPUs and 32 DIMMs.
The table shows the result of configuring a PSB as a quad-board. Notice the pattern
that emerges within the XSBs.
On quad-XSB XX-0, where XX refers to the PSB number, there is: one CPU (CPU
zero); the CPUs associated memory, which consists of eight DIMMs, two PCIe slots,
zero and one; and built-in I/O consisting of two disks, HDD zero and HDD one.
On quad-XSB XX-1, there is: one CPU (CPU one); the CPUs associated memory,
which consists of eight DIMMs, and two PCIe slots, two and three.
On quad-XSB XX-2, there is: one CPU (CPU two); the CPUs associated memory,
which consists of eight DIMMs, two PCIe slots, four and five; and built-in I/O
consisting of two disks, HDD two and HDD three.
Lastly, on quad-XSB XX-3, there is: one CPU (CPU three); the CPUs associated
memory, which consists of eight DIMMs, and two PCIe slots, six and seven.
This same pattern holds true for the remaining PSBs when you quad them.
The diagram shows a combination of uni-XSB and quad-XSBs divided among four
domains in a high-end server.
We'll pause this presentation to allow you time to review the diagram. Click play to
continue the presentation.

The mid-range server architecture is designed to support one integrated service

processor. The high-end server architecture is designed to support two integrated
service processors.
The integrated service processor enables system control through the maintenance
bus for:
I2C master
JTAG controller
Monitor power supplies
Interrupt handling
Reset control
The MX000 servers use a global maintenance bus to monitor the environmental
integrity of the platform. The diagram provides a graphical representation of how
the maintenance bus interfaces within the system. The monitoring structure is
controlled by the service processor and uses the I2C bus architecture. The I2C bus
monitors the server voltages, temperatures, and fan speeds.
We'll pause this presentation to allow you time to review the diagram. Click play to
continue the presentation.

Next, let's take a look at the service processor components and their functions. The
service processor is comprised of a variety of electrical components, and, in effect,
is a small computer unto itself.
The major components that make up a service processor include:
A processor consisting of a freescale MPC8541E PowerQUICC III CPU, which contains
a 64-kilobyte L1 cache, 256-kilobyte L2 cache, a core speed of 533 megahertz that
supports a 32-bit PCI bus, I2C bus, local bus, and SDRAM bus.
Next, the lattice field programmable gate array, or FPGA, which is known as the
MBCF-SCF, or maintenance bus control. The FPGA interfaces to the MPC8541E local
bus, maintenance bus, and TTY bus. It also monitors the PSU and UPS. The FPGA has
an operating frequency of 340 megahertz and an operating voltage of 1.14v to
1.26v.
Service processor main memory uses a Micron MT46V32M16P-6T:F. It consists of
256 megabytes of DDR-SDRAM and supports an eight megabyte by 16 by 4 bank
configuration that provides 64 bits of data and 8 bits of ECC. It has an operating
frequency of 133 megahertz.

Flash memory, or FMEM, is based on a Spansion S29JL064H70TFI000 with 8

megabytes and 64 megabits. It uses two banks of memory, one of which is used for
backup. It has an operating voltage of 3 volts.
The maintenance bus acts as the communication mechanism for various
components. It provides I2C master control, JTAG controller functionality, power
supply monitoring, interrupt handling, and reset control.
The service processor provides ports, such as one USB 1.1 port, two 1/0100Base-T
Ethernet ports, one serial port, two uninterruptable power control units, and one
RCI, which is not used in Sun's configuration of these servers.
The last component is the clock, which is based on Epson Q414574B1000102. It is a
highly accurate, real-time crystal clock with automatic leap year adjustment.
Click the link provided to view a functional block diagram of the service processor
used in the MX000 mid-range servers.
Now, let's take a minute to review the topics that were presented in this module.
First, we compared and contrasted the Jupiter bus Interconnect with bus
interconnects for Sun's predecessor mid-range and high-end servers as well as
documented the architectural similarities and differences across the MX000 server
product lines.
Next, we classified and labeled architectural components across the entire MX000
server product line. We also discussed the differences between the uni-XSBs and
quad-XSBs.
We also discussed key terms and definitions used in the board nomenclature for the
MX000 server line.