Problems and opportunities in training deep learning software systems
Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, Nachiappan Nagappan
Abstract
Deep learning (DL) training algorithms utilize nondeterminism to improve models' accuracy and training efficiency. Hence, multiple identical training runs (e.g., identical training data, algorithm, and network) produce different models with different accuracies and training times. In addition to these algorithmic factors, DL libraries (e.g., TensorFlow and cuDNN) introduce additional variance (referred to as implementation-level variance) due to parallelism, optimization, and floating-point computation.
Topics & Concepts
Computer scienceTraining (meteorology)Variance (accounting)ComputationParallelism (grammar)Training setArtificial intelligenceMachine learningPoint (geometry)Deep learningSoftwareData parallelismFloating pointComputer engineeringParallel computingAlgorithmProgramming languageMathematicsGeometryMeteorologyPhysicsBusinessAccountingAdversarial Robustness in Machine LearningMachine Learning and Data ClassificationAnomaly Detection Techniques and Applications