Blue Gene: Planning and Methods of Disaster Recovery

Abstract

Blue Gene is a Supercomputer that operating speed is far from other computers. Blue stand for the corporate colour of IBM. It performs large scale weather forecasting for global climate change or pattern matching in the biosciences. It is an project by IBM. The purpose of this project is to reach operating speeds in the PFLOPS (Peta 10¹⁵ Floating-point Operations Per Second) range. With low power consumption. The IBM project generated four supercomputers: Blue Gene/L, Blue Gene/C, Blue Gene/P and Blue Gene/Q. In this article, we will mainly cover one of Blue Gene series – Blue Gene/Q.

1. Hardware

1.1 Basic Components

Blue Gene/Q system is made up of multiple components.

Each single Module is using A2 1.6GHZ with 16-Cores processor. The module is inside the computer card that also included 16GB DDR3 SDRAM.

Compute card comprises of one application-specific integrated circuit memory chips.

32 Compute Cards are embedded into one Node board.

Node board involves dc-dc converters provided as well as water cooled for the compute card. Also, there is a 16 GB SDRAM DDR3 1333Mtps per node.

A Rack includes 1 OR 2 midplane and maximum 4 I/O drawers. Each midplane should have 32 compute cards. That's mean there are 32*16*2=1024 chip (16 core) in One rack inside.

I/O drawer involves 8 I/O compute card that have air-cooled heat sink, different cooling system inside.

We can have 256 racks to form a Multi-rack system in blue gene.

1.2Basic Cables

There are 6 types of basic cable in Blue Gene/Q:

-Optical 48-channel (use for extending torus connections between midplanes)

-Electrical coaxial cables (use to distribute differential clock signals between racks)-->

-Electrical Ethernet cables (between control switch to service cards and I/O drawers)

-Active optical cables (PCIe 2.0 adapter cards <----> federated switch)

-10 Gb fiber Ethernet

-Optical, 12-fiber (I/O drawers and node boards connection)

1.3 Local Memory bus

The application-specific integrated circuit (ASIC) of Compute chip includes 2 memory controllers by running parallel mode. And each memory controller involves a 144 bit memory bus interface, 128 bit data bus.

In the writing period of the on-die termination (ODT), data bus is terminated in a DRAM. As a result of the short data net, only the lowest current terminator values are used. The compute card and I/O circuit card terminate the address bus at the end of daisy chain to DRAM destination.

Also, base on the effects of memory corruption, IBM must use the Error-correcting code (ECC) in the memory. There are 72 symbols long for the ECC. The symbols’ arrangement is 64 + (1+7). 64 are the data symbols, 1 is for other purposes, and 7 are using to check error. This ECC can correct even single or double symbol failure and detect any triple-symbols failure.

1.4 Power Supply

According to the Blue Gene/Q data sheet, the typical Blue Gene/Q rack power consumption is about 80kW per rack. Of cause this data should be an estimate. If there are any hardware failure, huge workload application running, etc…. the power consumption must be larger than 80kW.

Also, Blue Gene/Q encloses power cords, circuit breaker and Uninterruptible power supply (UPS). Because of providing backup power and power condition, UPS should be served. In the Blue Gene planning, the system availability is very important. If there is no power suddenly, it may product data lose. Another consideration is the quality of power feeds. In this case, UPS can provide the best quality of power.

1.5 Cool down System

In the Blue Gene/Q, there are 90% using water cooling and 10% using air cooling.

2. Software

2.1 System overviews

Basic Blue gene software can run applications by using large number of across chips. The Blue Gene project solve a lot of fundamental problems such as the logic integrated processor, parallel systems, power management, error recovery, programming models, etc… .

In the Blue Gene system, I/O nodes provide the services to multiple compute nodes. I/O nodes and compute nodes can connect to each other by using torus network. Users use it from a front end node. There are some compilers for producing user app and some exactable jobs. These jobs connect with the Control System in Service node and pass the job to system.

2.2 Compilers in Blue Gene/Q

-IBM XL compilers

These compilers can be used in C, C++ and Fortran applications

-GNU compilers

These compilers include gcc, binutils, glibc, gdb.

Also, Blue Gene system has Mathematical Acceleration Subsystem libraries and Engineering & Scientific Subroutine libraries.

Developer can create two types of libraries themselves: static libraries and dynamically loaded libraries.

2.3 Debuggers

There are few part of code are involved: Compute Node Kernel, tool control daemon, tool running on I/O nodes and tool running front end node. The communication provided by Tool control daemon, the I/O nodes via Code Development and Tools interface (CDTI) debugger running on the compute nodes.

-GNU Project Debugger tool

GDB debugs programs are written in Ada, C, C++, Objective-C, Pascal and many other languages by specifying executing behavior of both the debugging program and other programs. IBM also provides a “gdbserver” for remote-debugging.

-Coreprocessor debugger

Coreprocessor debugger is a reliable debugger as it covers whole Blue Gene/Q system including hardware, kernel, and Application. JTAG interface of the Coreprocessor debugger can read and organize hardware information. Even the operating system is not working, users can still perform debug. It is independent to the application code, so failure node will be isolated by sorting the node based on their stack traceback and kernel status.

-Addr2line utility

The addr2line utility provides debugger information by analyzing the core file which will be generated while program did not run successfully.

It helps to find out problem of the source line by compiling with the address found in the

core file and the -g executable

3. Processes and memory (optional)

3.1 Process creation and count

Once a Job is submitted, CNK will load the application into memory at process initialization time.

And user needs to indicate how many processes should be created at the time of job initialization.

Deciding number of processes to run on each node is critical as it will directly affect the performance and the memory allocation of each process. After the number of process count was indicated, the memory will be allocated evenly for each process. There are types of application, memory requirement and the parallel paradigm, they are critical to determine how many processes per node to use. For single-threaded applications, the memory requirement per process is the main consideration. For more additional processes running in a node, we need to consider the replicated data segment, storage for the main stack, and storage for the heap.

3.2 Memory system

The distributed storage system of BlueGene / Q system can be divided into on-chip cache hierarchy and an off-chip main store. The on-chip cache hierarchy supporting the communication of 17 ASIC processors. Base on multi-machine style, the memory were distributed with no hardware sharing between nodes and each of a Blue Gene/Q node has 16 GB SDRAM.

Q&A

1) Why it call "Blue Gene" ?

A) It is because blue is the corporate colour of IBM.

2) Why we have to invent Blue Gene ?

A) In some scientistic activities, fast computers are required for program and control massively parallel systems.

3) What is included in blue gene compilers for compiling C, C++ and FORTRAN applications?

A) Blue Gene compilers include gcc, binutils, glibc, gdb

4) What is addr2line utility?

A)The addr2line utility provides debugger information by analyzing the core file which will be generated while program did not run successfully.