Innovating the Next Discontinuity Tight Coupling for AI and HPC

For more than thirty years, the fastest supercomputer in the world doubled in performance every 1.2 years exponentially outpacing Moore’s Law of a performance doubling every two years. By some measures, AI is growing even faster. That performance increase was achieved in part by a series of technical discontinuities, from vector computing to massively scaled computation to GPUs. Each innovation brought a performance level almost unattainable with previous technologies. Using holistic system design and leveraging 3D integration opens the door for the next discontinuity of tight coupling. This paper describes the key challenges facing AI and HPC computing including the memory wall, productively and effectively leveraging accelerators, and the communication wall. It describes an architecture using holistic system design that demonstrates how the tightly coupled technologies enabled via 3D integration helps address those challenges.