Abstract
Blue Gene is a Supercomputer that operating speed is far from other computers. Blue stand for the corporate colour of IBM. It performs large scale weather forecasting for global climate change or pattern matching in the biosciences. It is an project by IBM. The purpose of this project is to reach operating speeds in the PFLOPS (Peta 1015 Floating-point Operations Per Second) range. With low power consumption. The IBM project generated four supercomputers: Blue Gene/L, Blue Gene/C, Blue Gene/P and Blue Gene/Q. In this article, we will mainly cover one of Blue Gene series – Blue Gene/Q.
1. Hardware
1.1 Basic
Components
Blue Gene/Q system is made up of multiple components.
Compute
card comprises of one application-specific integrated circuit memory chips.
32 Compute
Cards are embedded into one Node board.
A Rack
includes 1 OR 2 midplane and maximum 4 I/O drawers. Each midplane should have
32 compute cards. That's mean there are 32*16*2=1024 chip (16 core) in One rack
inside.
I/O drawer involves 8 I/O compute card that have air-cooled heat sink, different cooling system inside.
We can
have 256 racks to form a Multi-rack system in blue gene.
There are 6 types of basic cable in Blue Gene/Q:
-Optical 48-channel (use for extending torus connections between midplanes)
-Electrical
coaxial cables (use to distribute differential clock signals between racks)-->
-Electrical
Ethernet cables (between control switch to service cards and I/O drawers)
-Active optical
cables (PCIe 2.0 adapter cards <----> federated switch)
-10 Gb fiber
Ethernet
-Optical, 12-fiber (I/O drawers and node boards connection)
1.3 Local Memory bus
The application-specific
integrated circuit (ASIC) of Compute chip includes 2 memory controllers by
running parallel mode. And each memory controller involves a 144 bit memory bus
interface, 128 bit data bus.
In the
writing period of the on-die termination (ODT), data bus is terminated in a
DRAM. As a result of the short data net, only the lowest current terminator
values are used. The compute card and I/O circuit card terminate the
address bus at the end of daisy chain to DRAM destination.
Also, base on the effects of memory
corruption, IBM must use the Error-correcting code (ECC) in the memory. There
are 72 symbols long for the ECC. The symbols’ arrangement is 64 + (1+7). 64 are
the data symbols, 1 is for other purposes, and 7 are using to check error. This
ECC can correct even single or double symbol failure and detect any
triple-symbols failure.
1.4 Power Supply
According to the Blue Gene/Q data sheet, the typical Blue Gene/Q rack power consumption is about 80kW per rack. Of cause this data should be an estimate. If there are any hardware failure, huge workload application running, etc…. the power consumption must be larger than 80kW.
Also, Blue Gene/Q encloses power cords, circuit breaker and Uninterruptible power supply (UPS). Because of providing backup power and power condition, UPS should be served. In the Blue Gene planning, the system availability is very important. If there is no power suddenly, it may product data lose. Another consideration is the quality of power feeds. In this case, UPS can provide the best quality of power.
1.5 Cool down
System
In the Blue Gene/Q, there are 90% using water cooling and 10% using air cooling.
2. Software
2.1 System overviews
Basic Blue gene software
can run applications by using large number of across chips. The Blue Gene
project solve a lot of fundamental problems such as the logic integrated
processor, parallel systems, power management, error recovery, programming
models, etc… .
In the Blue Gene system, I/O nodes provide the services to multiple compute nodes. I/O nodes and compute nodes can connect to each other by using torus network. Users use it from a front end node. There are some compilers for producing user app and some exactable jobs. These jobs connect with the Control System in Service node and pass the job to system.
2.2 Compilers in Blue Gene/Q
-IBM XL compilers
These compilers can be used in C, C++ and
Fortran applications
-GNU compilers
These compilers include gcc, binutils, glibc, gdb.
Also, Blue Gene system has Mathematical Acceleration Subsystem libraries and Engineering & Scientific Subroutine libraries.
Developer can create two types of libraries themselves: static libraries and dynamically loaded libraries.
2.3 Debuggers
There are few part of code are involved: Compute Node Kernel, tool control daemon, tool running on I/O nodes and tool running front end node. The communication provided by Tool control daemon, the I/O nodes via Code Development and Tools interface (CDTI) debugger running on the compute nodes.
-GNU Project Debugger tool
GDB debugs programs
are written in Ada, C, C++, Objective-C, Pascal and many other languages by specifying
executing behavior of both the debugging program and other
programs. IBM also provides a “gdbserver” for remote-debugging.
-Coreprocessor debugger
Coreprocessor debugger is a reliable debugger as it covers whole Blue Gene/Q system including hardware, kernel, and Application.
JTAG interface of the Coreprocessor debugger can read and organize hardware information. Even the
operating system is not working, users can still perform debug. It is independent
to the application code, so failure node will be isolated by sorting the node based on their stack traceback and kernel status.
-Addr2line utility
The addr2line utility provides debugger information by analyzing the core file which will be generated while program did not run successfully.
It helps to find
out problem of the source line by compiling with the address found in the
core file and the -g executable
3. Processes and memory (optional)
3.1 Process creation and count
Once a Job is submitted, CNK will load the
application into memory at process initialization time.
And user needs to indicate how many processes should be
created at the time of job initialization.
Deciding number of processes to run on each node is critical
as it will directly affect the performance and the memory allocation of each
process. After the
number of process count was indicated, the memory will be allocated evenly for
each process. There are types of application, memory requirement
and the parallel paradigm, they are critical to determine how many processes per node to use. For single-threaded
applications, the memory requirement per process is the main consideration. For
more additional processes running in a node, we need to consider the replicated
data segment, storage for the main stack, and storage for the heap.
3.2 Memory system
The distributed storage system of BlueGene / Q system can be divided
into on-chip cache hierarchy and an off-chip main store. The on-chip cache hierarchy
supporting the communication of 17 ASIC processors. Base on multi-machine style, the memory were distributed
with no hardware sharing between nodes and each of a Blue Gene/Q node has 16 GB
SDRAM.
Q&A
1) Why it call "Blue Gene" ?
A) It is because blue is the corporate colour of IBM.
2) Why we have to invent Blue Gene ?
A) In some scientistic activities, fast computers are required for program and control massively parallel systems.
3) What is included in blue gene compilers for compiling C, C++ and FORTRAN applications?
A) It is because blue is the corporate colour of IBM.
2) Why we have to invent Blue Gene ?
A) In some scientistic activities, fast computers are required for program and control massively parallel systems.
3) What is included in blue gene compilers for compiling C, C++ and FORTRAN applications?
A) Blue Gene compilers include gcc, binutils, glibc, gdb
4) What is addr2line utility?
A)The addr2line utility provides debugger information by analyzing the core file which will be generated while program did not run successfully.
Reference:
http://public.dhe.ibm.com/common/ssi/ecm/en/dcd12345usen/DCD12345USEN.PDF
http://www.redbooks.ibm.com/redpieces/pdfs/sg247869.pdf
