HPC COMPUTING, FROM THE INSIDE OUT

An Insiders Perspective
Philip Pokorny, CTO Penguin Computing, Inc.

PENGUIN COMPUTING HOW DID WE GET HERE?

2 | HPC Computing, from the Inside Out

| June 14, 2012

BEGINNING AT THE END

Worlds First APU Cluster Delivered September 2011 Collaboration among AMD, Sandia National Lab, Penguin Computing 104 Altus 2A00 nodes A8-3850 APU, 16GB of memory, SSD scratch drive, Qlogic Infiniband Complete solution with: Management nodes (boot, login, storage gateways) Networking (Ethernet and Infiniband) Six racks Power distribution with per outlet switching Serial console monitoring

3 | HPC Computing, from the Inside Out

| June 14, 2012

HISTORY OF PENGUIN COMPUTING

Founded in 1998 by Sam Ockman Focus on systems running Linux Focus on reliable systems Customized systems First Linux 1U rackmount server Short depth 1U 2U Storage/Virtual Tape server with six IDE drives Customer specified filesystem layout Customer specified RAID configurations

4 | HPC Computing, from the Inside Out

| June 14, 2012

MEMORABLE ADVERTISING CAMPAIGNS

5 | HPC Computing, from the Inside Out

| June 14, 2012

AMD USHERS IN THE 64-BIT PC ERA

AMD Opteron Processor AMD64 architecture 64-bit registers (and more of them) Orthogonal instruction set Integrated memory controller HyperTransport Interconnect Linux 64-bit ready and tested

Altus 1000e Dual socket system Fan control and monitoring Temperature sensor driver (written and contributed to kernel) Penguin Remote Control
6 | HPC Computing, from the Inside Out

| June 14, 2012

FROM SERVERS TO CLUSTERS

Large orders for Altus 1000e Beowulf style clusters

Scyld Software Don Becker Merged with Penguin Computing in 2003 Complete clusters with all the accessories Racks, UPS, keyboard & monitor Networks, cabling, management Factory integrated and tested Software installed and configured Tested as a complete cluster Shipped in a crate Ready to plug into network and power on-site
7 | HPC Computing, from the Inside Out

| June 14, 2012

FROM THE INSIDE OUT

8 | HPC Computing, from the Inside Out

| June 14, 2012

CPU CORES AND MEMORY SIZES OVER TIME

Dual Socket AMD Opteron Processor Memory Configurations

9 | HPC Computing, from the Inside Out

| June 14, 2012

MEMORY PER CORE OVER TIME

Dual Socket AMD Opteron Processor Memory Configurations

10 | HPC Computing, from the Inside Out

| June 14, 2012

WHAT'S THE TREND?

Dual Socket AMD Opteron Processor Memory Configurations

11 | HPC Computing, from the Inside Out

| June 14, 2012

LONG-TERM TRENDS IN CPU AND MEMORY

Dual Socket AMD Opteron Processor Memory Configurations

12 | HPC Computing, from the Inside Out

| June 14, 2012

MEMORY BANDWIDTH OVER TIME

13 | HPC Computing, from the Inside Out

| June 14, 2012

THOUGHTS ON THE FUTURE

14 | HPC Computing, from the Inside Out

| June 14, 2012

STORAGE PERFORMANCE
7200 RPM drives from 2002 to 2012 80 GB to 3000 GB 40 MB/sec to 160 MB/sec Lots more space, not much more performance RAID Less than perfect scaling Reliability level impacts performance Motherboard SATA ports limit number of drives Solid State Drives Limited writes

May need ATA TRIM for better results

15 | HPC Computing, from the Inside Out

| June 14, 2012

STORAGE PERFORMANCE
SATA really? Compatible with 1985 MFM drive controller! Assumes compatibility at register level
Butquirks to every controller

One drive per physical port Hot swap added on Oops, forgot the drive activity signal SAS Higher performance drives (15,000 RPM) Flexible number of drivers per controller Can host SATA drives Multiple ports for redundancy and performance

Ask for it

16 | HPC Computing, from the Inside Out

| June 14, 2012

IS THAT A GPU IN YOUR SOCKET?

Mixed number of CPU/GPU

Socket compatible FPGAs Spansion flash controller and NOR flash DIMMs

Just need the hardware

17 | HPC Computing, from the Inside Out

| June 14, 2012

POWER EFFICIENCY AND IDEAS FOR IMPROVEMENT

CPUs well-covered elsewhere Motherboards More efficient voltage converters (VRM) Better VRM tuning Generate voltages only where needed

Power supply Higher efficiency

Beware of hidden power distribution boards

Single rail (one voltage, not five or more)

Common infrastructure 12.5V distribution Shared fans Match power supplies to server load
18 | HPC Computing, from the Inside Out

| June 14, 2012

WHERE DOES THE POWER GO?

You cant control what you cant measure Need to compare multiple systems Need high-speed capture for some analysis Diagnose problems Spot trends

CPU versus motherboard versus anything else

Monitor each voltage rail individually
CPU, motherboard, fans, hard drives

High speed (1000 samples per second or faster) Correlate samples to code execution

19 | HPC Computing, from the Inside Out

| June 14, 2012

COMING SOON
A glimpse of what we might learn CPU power draw during make j4 Different phases of compilation Clearly different power levels Larger files can load CPU 100% for long periods of time

20 | HPC Computing, from the Inside Out

| June 14, 2012

THANK YOU QUESTIONS?

21 | HPC Computing, from the Inside Out

| June 14, 2012