A Very Quick Guide to Developing and Training Feed Forward Neural Networks
Jonas Almeida, 1 Oct 99
Among the most important tools deriving from bioinformatics research is the
artificial neural network technique (ANN). An ANN is a massively parallel
distributed processor able to store experience-based knowledge and make it
available for use. It resembles the human brain in two respects: a) knowledge
is acquired by the network through a learning process; b) interneuron
connection strengths known as synaptic weights are used to store knowledge
(Haykin 1994). The ability of ANN to identify mathematical models that closely
fit data without the need for mechanistic assumptions is ideally suited for the
analysis of complex data sets such as ecosystem variables and DNA microarray
hybridization patterns. If properly implemented, the resulting mathematical
formulation distinguishes signal from noise in the associations among an
arbitrary number of parameters. ANN have a wide range of biological
applications, from modeling soil bioremediation (Almeida 1998), to genomic
sequence analysis (Baldi and Brunak 1998). Clustering like samples (Gyllenberg
and Koski, 1995, Noble et al. 1997) and learning from examples without a priori
knowledge of causality (Hinton 1992) are some of the most important uses of
ANN. The reader is referred to Hagan et al. (1996) for applied design and
implementation of artificial neural networks, to Haykin (1994) for the
theoretical foundations, and to Montagne et al. (1994) for a review of
contributions in biotechnology.
The steps involved in developing an ANN are 1) assembling the network of
unit processors; 2) training them by changing the connection weights by
conjugate-gradient minimization of the predictive errors; 3) regularizing the
solution in order to ensure its general nature; and finally, 4) repeating the
previous steps for the domain of ANN topologies in order to find the optimal
extraction of signal versus noise. This procedure is inspired in the process of
natural learning and is detailed bellow for a feed-forward ANN architecture:
1. Assembling parallel processing units in fully connected layers
Schematic architecture of a three
layer feedforward network used to associate
microbial community typing profiles (MCT)
with classification vectors. Symbols
correspond to neuronal nodes (eq. 1).
Each neuronal node is equated to a sigmoidal function, f, of the product of a weight matrix, w, by the input
2. Training ANN with cross-validation as a stop
criterion, i.e. part of the data are excluded from
training and are used to evaluate the generality
3. Bootstrapping each topology evaluated.
The validation subset is re-sampled repetitively and the ANN is retrained with
cross-validation each time. The median performer is selected. This procedure
avoids the possibility that the validation data set may not be representative
of the whole.
4. Optimization of topology
The range of topologies is searched for the one with the best performance.
The complexity of the topology selected is proportional to the complexity of
the signal extracted (Reed and Marks 1999).
- Almeida, J.S., K. Leung, S.J. Macnaughton, C. Flemming, M. Wimpee, G. Davis, D.C.White. 1998.
Mapping changes in soil microbial community composition signaling bioremediation. Bioremediation
- Baldi, P., S. Brunak. 1998. Bioinformatics the machine learning approach. MIT Press
- Gyllenberg, M., T. Koski. 1995. A taxonomic associative memory based on neural computation. Binary
- Hagan, M.T., H.B. Demuth, and M. Beale. 1996. Neural Network Design. Prindle, Weber & Schmidt
Pub., Boston, MA.
- Haykin, S. 1994. Neural Networks-a Comprehensive Foundation, pp.397-443. Macmillan College Pub.,
- Hinton, G.E. 1992. How neural networks learn from experience. Sci. Am. 9:145-151.
- Montague, G., J.N. Morris. 1994. Neural Network Contributions in Biotechnology. Trends Biotech.
- Noble, P.A., K.D. Biddle, M. Fletcher. 1997. Natural microbial community compositions compared by
a back-propagating neural network and cluster analysis of 5S rRNA. Appl. Environ. Microbiol.
- Reed, R.D., R.J. Marks (1999) Neural Smithing supervised learning in feedforward artificial neural
networks. MIT Press, Cambridge MS, p.257-264.
- Bishop, C.M. 1995. Neural networks for pattern recognition. Clarendon Press, Oxford. P. 364-371
- Cheng, B., D.M. Titterington (1994) Neural networks: a review from a statistical perspective. Statistical
Science, 9: 2-54.