Over the last 18 months, Pan, the New Zealand eScience Infrastructure Auckland cluster has grown from a single iDataplex rack, with 2 support racks to four iDataplex racks housing 1056 Westmere and 3296 Sandybridge i386 cores, with and additional 72 cores in 3 large memory nodes. There are 16,384 Tesla M2090, 26,880 K20X and 240 Intel Phi 5110P cores. Growing pains, and operational experience have shaped the current cluster topology.
The InfiniBand network provides a 40Gb/s communication backbone for the cluster, and has been optimized for low latency MPI. Pan quickly outgrew its single infiniband switch, and now has two large switches, in a two MPI island topology. The MPI islands are bridged to allow the full cluster to be used for extremely large MPI jobs, with a small latency hit for traffic crossing the bridge. We can add one more iDataplex rack before needing to move to a three island topology.
The GPFS file system also evolved, as more nodes were added to the cluster. Initially, both files and file system meta data were served through four GPFS NSD servers, backed by four fibre channel SANS. We have since placed the file system meta data onto SSDs, served through two dedicated GPFS NSD meta data servers and have added two additional backend data SANs. The original four GPFS servers now serve only the data, with a portion of the faster disk now providing a shared scratch space to the cluster nodes.
NeVE, the New Zealand eScience Infrastructure Virtual Environment, provides support VMs, and is backed by its own SAN, used to host the virtual machine images. The GPFS file system access is via direct connections to the fibre channel, bypassing the GPFS NSD servers. Initial attempts to use the infiniband proved unsuccessful, though newer drivers may make this viable again.