Without any doubt, one of if not the most interesting HPC products seen on the show floor is comes from Russia. T-Platforms is a very progressive company that engineered the densest GPGPU cluster we have seen so far.
T-Platforms created a 7U design that fits no less than 16 blade boards with 32 Tesla 2050 boards. In order to reach that level of density, the company took the existing MXM design from the limited series of GeForce GTX 480M boards, modified the part to fit the blade design, increased the amount of memory to 3GB GDDR5 memory and placed a GF100-875-A3 [Tesla] instead of GF100-375-A3 [GeForce] chip.
Ultra-dense motherboard packs two Tesla MXM modules, two Intel processors and custom memory connectors. Infiniband connection is located on the lower center.
Given the extreme reliability demands in a dense system, each GPU has a dedicated Intel 5520 Northbridge and a dedicated single port QDR Infiniband chip, bypassing the CPU as much as possible in order to achieve high bandwidth and reduce the latency for the intra- and inter-node traffic.
This custom approach naturally, has its price. The level of complexity on this multi-layered mainboard is nothing short of amazing, with the board carrying equal amount of layers as the most complex graphics processing units.
GPUs are paired with Intel quad or sexa-core Xeons with 3GB DDR3 per CPU. Single 7U node fits up to 192 CPU cores, 384 threads, 96GB of DDR3 memory in proprietary format and 32 nVidia Tesla MXM2050 or MXM2070 boards yielding 32.96 TFLOPS in Single-Precision, or 16.48 TFLOPS in Double Precision performance per single 7U rack.
16 blades pack two Tesla and two Xeon processors each, for currently the highest density compute node in the world
By using this density, T-Platforms offers 1PFLOPS of performance in ten 42U racks, which we predict could cause a lot of havoc on the HPC market. The company is working about building the clusters in space-challenged datacenters.
The price of the system is very interesting… Alexey Nechuyatov, Director of Product Marketing told us they’re targeting a $300,000 price for the node. All in all, reaching 1PFLOPS computing and the price coming down from hundreds of millions of dollars and putting the price in budgets reachable by science and commercial community.