Fast Learning of Deep Neural Networks via Singular Value Decomposition
MetadataShow full item record
In this paper, we propose a new fast training methodology for learning of Deep Neural Networks (DNNs) via Singular Value Decomposition (SVD). The fast training methodology uses a supervised pre-adjusting process to adjust roughly parameters of weight matrices of DNNs and change distributions of singular values. SVD is applied to pre-adjusted DNNs, reducing quantities of parameters in DNNs. An unconventional Back Propagation (BP) algorithm is used to train the models restructured by SVD, which has lower time complexity than the conventional BP algorithm. Experimental results indicate that on Large Vocabulary Continuous Speech Recognition (LVCSR) tasks, using the fast training methodology, the unconventional BP algorithm achieves almost 2 times speed-up without any loss of recognition performance and almost 4 times speed-up with only a tiny loss of recognition performance.
13th Pacific Rim International Conference on Artificial Intelligence, Proceedings
Pattern Recognition and Data Mining