Supercomputing company Cray has joined the US Department of Energy’s Big Data Center (BDC) initiative at the National Energy Research Scientific Computing Center (NERSC).

Established in August 2017, the BDC is a collaboration between NERSC, Intel, and five Intel Parallel Computing Centers (IPCCs) - the University of California-Berkeley, the University of California-Davis, New York University (NYU), Oxford University and the University of Liverpool. 

Its purpose is to determine whether or not current HPC systems can support a new generation of data-intensive workloads - 100 terabytes plus, using 100,000 CPU cores or more, and to develop new, scalable computing algorithms. 

Developing at scale

To research BDC, NERSC uses its recently-installed Cray XC40 “Cori” supercomputer, which contains 2,388 Intel Xeon ’Haswell’ processor nodes and 9,688 Intel Xeon Phi ’Knight’s Landing’ nodes.

In teaming up with Cray. the public-private partnership plans to focus on three areas:

  • Advancing the state-of-the-art in scalable, deep learning training algorithms,
  • Developing a framework for automated hyper-parameter tuning
  • Exploring the use of deep learning techniques and applications against a diverse set of important scientific use cases, such as genomics and climate change

“We are really excited to have Cray join the Big Data Center,” Prabhat, Director of BDC, and group lead for data and analytics services at NERSC.

“Cray’s deep expertise in systems, software, and scaling is critical in working towards the BDC mission of enabling capability applications for data-intensive science on Cori. Cray and NERSC, working together with Intel and our IPCC academic partners, are well positioned to tackle performance and scaling challenges of deep learning.”

Per Nyberg, Cray’s senior director of AI and Analytics, added: “Deep learning is increasingly dependent on high performance computing, and as the leader in supercomputing, Cray is focused on collaborating with the innovators in AI to address present and future challenges for our customers.

“Joining the Big Data Center at NERSC is an important step forward in fostering the advancement of deep learning for science and enterprise, and is another example of our continued R&D investments in AI.”