INTRODUCTION TO DYNAMIC

RECONFIGURATION AND CAPACITY

ON DEMAND FOR
SUN SPARC ENTERPRISE™
SERVERS
Nick Kloski, Sun Technical Marketing

Sun BluePrints™ On-Line — April 2007

Part No 820-2084-10
Revision 1.0, 4/22/07
Edition: April 2007
 Sun Microsystems, Inc.

Table of Contents

Dynamic Reconfiguration and Capacity on Demand . . . . . . . . . . . . . . . . . . . . . . . . . 1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Enhanced Domain Granularity with the
eXtended System Board (XSB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Enhancements to Dynamic Reconfiguration (DR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Enhancements to Capacity on Demand (COD). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Ordering Sun Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Accessing Sun Documentation Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.

Dynamic Reconfiguration and Capacity on Demand

Dynamic Domains1, Dynamic Reconfiguration (DR), and Capacity on Demand (COD)

provide technology that lets organizations effectively manage server resources in a
highly available and cost-effective fashion. Sun SPARC Enterprise™ servers provide
enhancements over previous-generation implementations of these technologies,
building upon over 10 years of experience with Dynamic Reconfiguration in particular.
These enhancements give organizations more flexibility in the granularity of
components available for Dynamic Reconfiguration, while adding new options to the
process. This article introduces the following topics:
• “Enhanced domain granularity with the eXtended System Board (XSB),” page 2
introduces the eXtended System Board and describes the different ways that it can
be configured.
• “Enhancements to Dynamic Reconfiguration (DR),” page 3 describes specific new
capabilities of DR with respect to Sun SPARC Enterprise servers.
• “Enhancements to Capacity on Demand (COD),” page 5 describes Capacity on
Demand licenses and how they work with Sun SPARC Enterprise servers.

Introduction
The underlying goal of Dynamic Reconfiguration is to allow organizations to gain higher
hardware utilization of their servers while achieving greater resiliency against hardware
failures. The DR architecture allows for components within the server (CPUs, memory,
and I/O resources) to be assigned, reserved, and moved between different Dynamic
Domains (hardware domains). The need to move components can come from business
needs (dynamic provisioning) or from the need to replace faulty or failed hardware.
Dynamic Domains represent logical partitions within the server chassis, allowing for
separation of resources and allowing multiple instances of the Solaris™ Operating
System (Solaris OS) to run concurrently on the system. Each instance of the Solaris OS
running in a Dynamic Domain has access to its own resources, providing complete
isolation of one Solaris instance from any other instance running on the same server.

Dynamic Domain technology represents just one of several categories of server

virtualization that Sun offers in its portfolio. Unlike other virtualization approaches,
Dynamic Domains partition the components of the chassis itself, before the Solaris OS is
loaded. Other virtualization methods (such as Solaris Containers) can be applied and
configured within a running Solaris OS instance in a Dynamic Domain. A discussion of
Solaris Containers is beyond the scope of this document, but more resources can be
found in the references section.

1.Formerly known as Dynamic System Domains


System board taxonomy
A number of acronyms are used to describe
the capability of the system boards on Sun
SPARC Enterprise servers:
• Physical System Board (PSB): The PSB is
the physical board that is inserted into
the system, with sockets for four SPARC®
processors.
• eXtended System Board (XSB): The PSB in
Sun SPARC Enterprise servers can be set
to one of two logical configurations,
comprising either the entire PSB
(Uni-XSB-mode) or a quarter of the PSB
(Quad-XSB mode).
• Logical System Board (LSB): An LSB
represents a number that assigns an XSB
to a Dynamic Domain.
2 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.
Enhanced Domain Granularity with the
eXtended System Board (XSB)
In previous Dynamic Reconfiguration implementations, resources such as system
boards, memory, and I/O could be swapped in and out of running domains only as
complete units. For example, Sun Fire™ E4900-E25K system boards (with up to four
processors and associated memory) were the base unit for DR operations in those
systems. As a result, using DR to remove a specific processor and its associated memory
from a domain required removing all of the processors and memory on the Physical
System Board (PSB).
This approach has advantages for some, since treating an entire system board as the
base unit for DR maximizes hardware isolation. However, this large granularity can
make other situations more complex, such as when applications running on the server
expect, require, or reserve specific amounts of finer-grained hardware resources. For
example, the requirement to remove an entire PSB sometimes complicate DR
operations on systems running databases, where Intimate Shared Memory (ISM) was
configured and expected by the database application.
Sun SPARC Enterprise servers (including Sun SPARC Enterprise M4000, M5000, M8000,
and M9000 servers) mitigate this requirement with the introduction of what Sun calls
the eXtended System Board (XSB) architecture. This architecture facilitates flexible
configuration of domains. By dividing the PSB into smaller discrete components,
Logical System Boards (LSBs) consisting of one or four processors can then be assigned
to individual domains.
Extended System board (XSB) functionality is easily understood when thinking about
the smallest discrete unit that any given DR operation can address. Combining all
processors on one PSB into a single discrete unit is called running that PSB in “Uni-XSB”
mode. In Uni-XSB mode, all of the processors on one PSB are treated as one Logical
System Board (LSB) when that board is configured into a domain. Uni-XSB mode is
identical to how Sun has traditionally implemented resource utilization on earlier
systems, and is ideal for allowing complete isolation of hardware resources since the
domain controls all of the resources of the system board.
For situations where flexibility is required over complete hardware isolation, the XSB
architecture can also be configured in “Quad-XSB” mode. Since each PSB has sockets
for four processors, setting a system board into Quad-XSB mode will allow each
processor (and associated memory and I/O) to become a discrete XSB onto which DR
operations can be performed. System boards in the various Sun SPARC Enterprise
servers present different configuration options depending on their architecture.
Figure 2 illustrates a Sun SPARC Enterprise M4000 server in Quad-XSB configuration.
3 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.

PCI-X Slot

Memory Module Quad-XSB 0 DVD/DAT IO Channel PCIe Slot 0

Disks 0, 1
PCIe Slot 1

CPU Module

IO Channel PCIe Slot 2

Memory Module Quad-XSB 1
PCIe Slot 3

Memory Module Quad-XSB 2

CPU Module
Memory Module Quad-XSB 3

System Board I/O Board

Figure 1. Sun SPARC Enterprise M4000 server in Quad-XSB configuration

Figure 2 illustrates a Sun SPARC Enterprise M8000 or M9000 system board in Quad-XSB
mode.

CPU PCIe
DDR2
Slot
DIMMS Quad-XSB 00-0 FLP
(8) Memory Controller PCIe
PCIe Slot
Bridge
(8)
CPU PCIe
DDR2 Slot
DIMMS Quad-XSB 00-1 FLP
(8) PCIe
Memory Controller
Slot

CPU PCIe
DDR2 Slot
DIMMS Quad-XSB 00-2 FLP
PCIe
(8) Memory Controller
Slot
PCIe
Bridge
(8)
CPU PCIe
DDR2 Slot
DIMMS Quad-XSB 00-3 FLP
PCIe
(8) Memory Controller
Slot

System Board 0 I/O Board

Figure 2. Sun SPARC Enterprise M8000 or M9000 system board in Quad-XSB

configuration

Figure 3 illustrates three configurations of system boards in the Sun SPARC Enterprise
M5000 server, showing full Uni-XSB, full Quad-XSB, and a combination of both.
4 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.

IOU #0

Memory Module Uni-XSB 00-0 DVD/DAT IO Channel PCI-X Slot PCIe Slot 0
CPU Module Disks 0,1
Memory Module PCIe Slot 1
IO Channel

PCIe Slot 2

Memory Module PCIe Slot 3

CPU Module
Memory Module
SB0

SB1
Memory Module Uni-XSB 01-0 Disks 2,3 IO Channel PCI-X Slot PCIe Slot 0
CPU Module
Memory Module PCIe Slot 1
IO Channel

PCIe Slot 2

Memory Module PCIe Slot 3

CPU Module
Memory Module

System Board I/O Board

IOU #0

DVD/DAT IO Channel PCI-X Slot PCIe Slot 0

Memory Module Quad-XSB 00-0 Disks 0,1
PCIe Slot 1
CPU Module
IO Channel PCIe Slot 2
Memory Module Quad-XSB 00-1
PCIe Slot 3

Memory Module Quad-XSB 00-2

CPU Module
Memory Module Quad-XSB 00-3
SB0

SB1
Disks 2,3 IO Channel PCI-X Slot PCIe Slot 0
Memory Module Quad-XSB 01-0
PCIe Slot 1
CPU Module
IO Channel PCIe Slot 2
Memory Module Quad-XSB 01-1
PCIe Slot 3

Memory Module Quad-XSB 01-2

CPU Module
Memory Module Quad-XSB 01-3

System Board I/O Board

IOU #0

DVD/DAT IO Channel PCI-X Slot PCIe Slot 0

Memory Module Quad-XSB 00-0 Disks 0,1
PCIe Slot 1
CPU Module
IO Channel PCIe Slot 2
Memory Module Quad-XSB 00-1
PCIe Slot 3

Memory Module Quad-XSB 00-2

CPU Module
Memory Module Quad-XSB 00-3
SB0

SB1
Uni-XSB 01-0
Memory Module Disks 2,3 IO Channel PCI-X Slot PCIe Slot 0
CPU Module
Memory Module PCIe Slot 1

Memory Module IO Channel PCIe Slot 2

CPU Module
PCIe Slot 3
Memory Module

System Board I/O Board

Figure 3. Sun SPARC Enterprise M5000 server in various Uni-XSB and Quad-XSB
configurations

Hot Plug and Hot Swap:
• Hot plug: Hot plug denotes the capability
to remove a component in an electrically
safe manner. Hot plug components allow
replacement without removing power
from the system.
• Hot swap: Hot swap implies that
necessary drivers and the OS can be
configured to allow removal or
replacement of components within the
domain. Hot swap components allow
replacement components or additional
resources to be added into a running
domain without rebooting.
5 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.
Instead of setting up a one-to-one relationship between PSBs and LSBs (as with Uni-XSB
mode), Quad-XSB mode allows four LSBs to exist on the physical system board. This
configuration allows the addition or removal of individual processors (and their
resources) into any desired domain on the system. In this way, one PSB can be logically
divided into fine-grained resources that can be assigned independently into four
separate domains if desired, or separately into the same domain.
Note – PSBs in Quad-XSB mode cannot interleave memory between processors, even if they
are placed into the same domain. For full memory interleaving in the domain, Uni-XSB mode
must be employed. In Quad-XSB mode, memory interleaving is 2-way. In Uni-XSB mode, inter-
leaving is 8-way (depending on full memory installation).
An increase in granularity for controlling processors also allows for more flexibility in
the assignment of memory and I/O resources. As each Uni- or Quad-XSB is assigned to a
domain, it brings along its own specific memory and I/O resources. The control and
assignment of these resources is thus directly related to the specific XSBs that are
assigned in the domain. As a result, memory and I/O resources cannot be assigned
independently from the XSB, as the relationship of XSB to memory and I/O is fixed.
Enhancements to Dynamic Reconfiguration (DR)
Dynamic Reconfiguration on Sun SPARC Enterprise servers provides an integrated
method for managing DR operations across the server line. DR operations on CPU and
memory resources as well as chassis devices such as the DVD/DAT drives are performed
exclusively from the Service Processor (XSCF). DR operations affecting I/O resources
occur from individual Solaris domains. System Administration commands are also
identical through the entire server family, reducing both complexity and training
requirements.
Additional enhancements to DR are described in the sections that follow:
• I/O resources
As mentioned, I/O slots on the Sun SPARC Enterprise sever line are fixed to specific
XSBs. As XSBs are moved to and from domains, associated
I/O resources are also dynamically configured along with them. The I/O
subsystem on the Sun SPARC Enterprise server family has the capability of
supporting both PCI-X as well as up to eight lane (x8) PCI Express cards1. I/O
resources can be moved from domain to domain, subject to driver awareness of
the DR architecture and redundancy of I/O paths to ensure system availability.
1.PCI Express x2 and x4 cards are also supported.
6 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.

Coupling this capability with the External I/O Expansion Unit allows for greatly
enhanced I/O capacity, flexibility, and redundancy. Each External I/O Expansion
Unit adds up to 12 expansion slots. Both the external and the internal I/O
modules support hot plug of both PCI-X and PCI Express cards, a notable evolution
from earlier server offerings.

Note – While the entire Sun SPARC Enterprise server line supports hot swap and DR of I/O,
only the M8000/M9000 servers support hot swap of CPU and memory.

• DVD/DAT
Throughout the Sun SPARC Enterprise server family, each chassis has the option of
built-in DVD and DAT drives. The assignment of these drives in Sun SPARC
Enterprise M4000 and M5000 servers is directly linked to specific XSB
configurations (namely tied to a specific processor on the PSB). In contrast, Sun
SPARC Enterprise M8000 and M9000 servers can assign the DVD and DAT resources
into any running domains that have access to the internal boot drives.

Note – Access to specific chassis resources such as internal boot disks and DVD/DAT drives is
dependent on several factors. On Sun SPARC Enterprise M4000/M5000 servers, chassis
resources are assigned through domain ownership of specific XSBs.On Sun SPARC Enterprise
M8000/M9000 servers, chassis resources are assigned through placement of Base I/O cards
(IOUA cards) in specific IOU slots. For more information, please see the Sun SPARC Enterprise
DR User’s Guide.

• Kernel cage
A kernel memory board is a system board on which kernel memory is loaded.
Kernel memory consists of memory internally used by the Solaris OS and contains
an OpenBoot PROM program. Kernel memory is allocated on a single system board
as much as possible. If all memory on the system board is allocated to kernel
memory and more kernel memory is required, the memory on an additional
system board is used. The Solaris OS limits the allocation of kernel memory to a
minimum set of system boards, using a function called “kernel cage memory”.

The Solaris OS is configured to hold kernel pages preferentially in LSBs that are
configured in Uni-XSB mode rather than Quad-XSB mode. This preference
facilitates DR operations on additional boards (in either Quad- or Uni-XSB mode)
since the kernel typically doesn’t need to be quiesced in order to move the kernel
memory to another board. If the kernel requires more memory than exists on a
single XSB, then the kernel will span multiple boards. In this scenario, DR of those
system boards is still possible, but will require slightly more time due to the extra
overhead of temporarily paging out kernel memory to other free/available
memory. These operations are also more complex since several more restrictions
or dependencies have to be checked for a copy-rename operation.
7 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.

• Float board
Any XSB can be designated as a “float” board. Boards so designated are
considered as the lowest priority for holding kernel memory, and are thus easier to
DR out of domains. By designating system boards as float boards, resources can be
moved more easily from idle domains to busy domains.

• Reservations
Any LSB can be reserved for use in a specific domain. This capability is helpful if
limitations exist for using live DR to move resources into a running domain (such
as per-CPU application licensing). Boards can that are reserved for the domain are
configured and available for processing upon the next reboot.

• Omit memory and omit I/O

Any LSB can have either “Omit Memory” and/or “Omit I/O” attributes set. Boards
so designated will not add their memory or I/O into the domain to which they
have been configured. This capability simplifies system administration decisions
regarding DR operations since the domain ignores the memory and I/O on the LSB.
As a result, DR of LSBs into and out of domains can be accelerated with these
options.

Enhancements to Capacity on Demand (COD)

The Sun SPARC Enterprise server family also provides enhancements to the Capacity On
Demand (COD) architecture over earlier rack-mount versions of the Sun Fire server
family. COD allows organizations to purchase only the processing and memory they
need, while providing the flexibility to upgrade their systems quickly to adapt to
growing business needs. COD lets organizations grow in a measured fashion, helping
them respond to urgent business needs with on-demand hardware upgrades. A system
using COD is provisioned for the future, letting organizations grow and adapt to data
processing needs over time, while allowing for flexibility in day-to-day system use.

When a server is purchased under a COD agreement, a certain number of processors

are installed and reserved, but not licensed for use. Before licensing, these processors
are present, but not functionally enabled in the system. As processing demands grow,
licenses can be purchased that allow unused processors to come online and be
Dynamically Reconfigured into running domains (or be used to create new domains).
Furthermore, licenses available for the Sun SPARC Enterprise server family only limit the
number of COD processors that can be used at any given time, and are not tied to
specific processors in specific slots. For example, if an organization has a license to use
six COD processors, but has 12 COD processors physically installed in their system, any
six processors could be used at any given time, on any number of Physical System
Boards.
8 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.

Because the system boards have a per-processor granularity of domain usage in Quad-
XSB mode, the COD architecture allows for granular use of licenses as well. Extending
the example above, if a domain utilizes four of the 12 COD licenses installed on the
server, then those four processor licenses would be released back into the COD license
pool when that domain is taken offline. Other active domains could then use any of the
four licensed COD processors that have not yet been reassigned. This equivalency holds
true for DR as well. If a COD processor is taken out of a running domain (or fails due to
hardware issues), one COD license becomes free, and any COD processor in the system
would be available for use in any domain using that released license.

A Headroom feature is also provided with COD on Sun SPARC Enterprise servers
(disabled by default). Enabling the headroom feature will allow for instant use of up to
four previously unused/unassigned COD processors without the installation of an
appropriate license. This feature allows for emergency mitigation of computational
shortfalls, and requires that the customer purchase a Right To Use (RTU) license within
30 days. The headroom feature also allows four processors (one PSB in Uni-XSB mode or
four Quad-XSB mode LSBs on one or more COD-enabled system boards) to be used to
supplement a hardware failure on a PSB in a running domain. In such a scenario, this
flexible capability provides equal computing power using the temporary headroom of
COD processors while a replacement PSB is dispatched to the customer site.

Summary
Sun SPARC Enterprise servers provide a range of enhancements that give considerable
flexibility to configuring resources for Dynamic Domains. With the option to use the
eXtended System Board in either Uni-XSB mode or Quad-XSB mode, organizations have
the opportunity to select for either fault isolation or maximum domain granularity.
Improved granularity of hardware resources (processors, memory, and I/O) along with
new COD per-processor licensing gives administrators the tools they need to respond to
changing conditions in the enterprise without compromising performance and uptime.

About the Author

Nick Kloski has been working for Sun Microsystems since 1998, and has held a variety of
roles including Field Service, Systems Administrator, QA Test Engineer, and his current
role in Technical Marketing as a SPARC technology specialist.
9 Dynamic Reconfiguration and Capacity on Demand Sun Microsystems, Inc.

References
Foxwell, Harry J.; Lageman, Meno; van Hoogeveen, Joost Pronk; Rosenfeld, Isaac; Setty,
Sreekanth; and Victor, Jeff. “The Sun BluePrints Guide to Solaris Containers:
Virtualization in the Solaris Operating System,” Sun BluePrints OnLine, October 2006,
To access this article online, go to http://www.sun.com/blueprints/1006/
820-0001.html

Sun SPARC Enterprise Mx000 Product Notes. Sun Microsystems, 2007.

Sun SPARC Enterprise User Guide. Sun Microsystems, 2007.

Sun SPARC Enterprise Administration Guide. Sun Microsystems, 2007.

Sun SPARC Enterprise XSCF User Guide. Sun Microsystems, 2007.

Sun SPARC Enterprise DR User Guide. Sun Microsystems, 2007.

Sun SPARC Enterprise server Web site:

http://www.sun.com/sparcenterprise

Ordering Sun Documents

The SunDocsSM program provides more than 250 manuals from Sun Microsystems, Inc.
If you live in the United States, Canada, Europe, or Japan, you can purchase
documentation sets or individual manuals through this program.

Accessing Sun Documentation Online

The docs.sun.com web site enables you to access Sun technical documentation
online. You can browse the docs.sun.com archive or search for a specific book title
or subject. The URL is
http://docs.sun.com/

To reference Sun BluePrints OnLine articles, visit the Sun BluePrints OnLine Web site at:
http://www.sun.com/blueprints/online.html
Introduction to Dynamic Reconfiguration and Capacity on Demand for Sun SPARC Enterprise Servers On the Web sun.com/sparcenterprise

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com
© 2007 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, BluePrints, Solaris, Sun Fire, and Sun SPARC Enterprise, are trademarks or registered trademarks of Sun Microsystems, Inc. in
the United States and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the US and other countries. Products bearing SPARC
trademarks are based upon an architecture developed by Sun Microsystems, Inc. Information subject to change without notice. Printed in USA 04/07