High-Performance Communication Strategies in Parallel and Distributed Deep Learning

Recorded talk [best effort]. Speaker: Torsten Hoefler Conference: DFN Webinar Abstract: Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this talk, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. Specifically, we present trends in DNN architectures and the resulting implication