Deep Neural Networks with Multistate Activation Functions

Loading...
Thumbnail Image
File version

Version of Record (VoR)

Author(s)
Cai, Chenghao
Xu, Yanyan
Ke, Dengfeng
Su, Kaile
Griffith University Author(s)
Primary Supervisor
Other Supervisors
Editor(s)
Date
2015
Size
File type(s)
Location
Abstract

We propose multistate activation functions (MSAFs) for deep neural networks (DNNs). These MSAFs are new kinds of activation functions which are capable of representing more than two states, including the N-order MSAFs and the symmetrical MSAF. DNNs with these MSAFs can be trained via conventional Stochastic Gradient Descent (SGD) as well as mean-normalised SGD. We also discuss how these MSAFs perform when used to resolve classification problems. Experimental results on the TIMIT corpus reveal that, on speech recognition tasks, DNNs with MSAFs perform better than the conventional DNNs, getting a relative improvement of 5.60% on phoneme error rates. Further experiments also reveal that mean-normalised SGD facilitates the training processes of DNNs with MSAFs, especially when being with large training sets. The models can also be directly trained without pretraining when the training set is sufficiently large, which results in a considerable relative improvement of 5.82% on word error rates.

Journal Title

Computational Intelligence and Neuroscience

Conference Title
Book Title
Edition
Volume

2015

Issue
Thesis Type
Degree Program
School
Publisher link
Patent number
Funder(s)
Grant identifier(s)
Rights Statement
Rights Statement

© 2015 Chenghao Cai et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Item Access Status
Note
Access the data
Related item(s)
Subject

Neurosciences

Cognitive and computational psychology

Science & Technology

Life Sciences & Biomedicine

Mathematical & Computational Biology

Neurology

Persistent link to this record
Citation

Cai, C; Xu, Y; Ke, D; Su, K, Deep Neural Networks with Multistate Activation Functions, Computational Intelligence and Neuroscience, 2015, 2015, pp. 721367

Collections