ARTIFICIAL NEURAL NETWORKS.ppt on machine learning

AnN machine learning

1. ARTIFICIAL NEURAL
NETWORKS
CLASSIFICATION

2. DEFINITION OF NEURALNETWORK
An artificial neural network (ANN) is a parallel
connection of a set of nodes called neurons.
From the statistical viewpoint, it represents a
function of explanatory variables which is
composed of simple building blocks and which
may be used to provide an approximation of
conditional expectations or, in particular,
probabilities in regression or classification.

3. ANN WITHANINPUT
LAYER, ONEHIDDENLAYERANDANOUTPUTLAYERI1
I2
I3
I4
Sepal.Length
Sepal.Width
Petal.Length
Petal.Width
H1
H2
O1
O2
O3
setosa
versicolor
virginica
B1 B2

4. DEFINITIONOFTHEANN
In this study, we only consider:
A feed-forward net with d + 1 input nodes,
One layer of Hhidden nodes,
Coutput nodes ,
An activation function
The input and hidden layer nodes are connected by
weights W
hj for
and
The hidden and output layers are connected by
weights for
and  Hh,...,1  dj,...,0 ch  Hh,...,1  Cc,...,1 )(x

5. TRAININGTHENETWORK
The connection weights are adjusted through
training.
There exists two training paradigms: Non
supervised and supervised learning.
We discuss and later apply supervised learning.

6. REQUIREMENTS FORSUPERVISEDTRAINING
A sample ofd input vectors, of size
neach,
An associated output vector,
of size n,
The selection of an initial weight set.
A repetitive method to update the current weights to
optimize the input-output map.
A stopping rule.dXXX ,...,
1 CYYY,...,
1

7. A REPETITIVEMETHODTOUPDATETHECURRENT
WEIGHTSTOOPTIMIZETHEINPUT-OUTPUTMAP
Error Function
The maximum likelihood method is used to
determine the error function.
The error function is then used to train a given
network.  
2
1
);();,( 


n
i
ii
XZYXYS 

8. UPDATINGTHEWEIGHTS
This step involves updating the weights until the error
function is minimized.
There are various methods
of minimizing namely:
(1)Backpropagation(BP)
(2)The Quasi-Newton method
(3)Broyden–Fletcher–Goldfarb–Shanno(BFGS)
method
(4)The Simulated Annealing method);,(XYS );,(XYS

9. NUMBEROFHIDDENLAYERS
A neural net can have more than one hidden layer,
see Looney [1] for more details.
However, it is shown in White [2] that one hidden
layer with sufficient number of neurons is enough
to approximate any function of interest.
In practice, however, a network with more than
one layer may provide a more parsimonious
model for the data.
The number of neurodes, H, in the hidden layer can
be determined as in Looney [2] by a rule of thumb: 1)(log*7.1
2
 nH

10. NUMBEROFHIDDENLAYERS
Alternatively, one can use the Black Information
Criterion (BIC) as proposed in Swanson and White
[3] to sequentially determine H.

11. STOPPINGRULES
There are four common stopping rules:
(1)
(2)
(3) a pre-specified lower bound
for the training error
(4) where MAX is the pre-specified
number of iterations.
We note that Rule(4) can be used together with
Rule(1), Rule(2) or Rule(3).0
)()1(


forQQ
rr smallbutforQXYSQXYS
rr
0)
ˆ
;,()
ˆ
;,(
)()1(


 min
)(
)
ˆ
;,( EQXYS
r
 MAXr

12. NNETINR
They are many packages for ANN. We will use the nnet
package that comes with standard R software.
Run the provided Artificial Neural Net R program. Make
sure you understand each and every step.
Fine tune the program based on:
Number of hidden nodes
Important Variables

13. EXERCISE
OPTIMIZE THE NETWORK ON THE NUMBER OF
HIDDEN NODES AS INFORMED BY THE AIC
Y –AXIS AIC
X-AXIS NUMBER OF HIDDEN NODES
Hint: You may use the for loop

14. REFRENCES
(1) Looney, C. G. Pattern recognition using neural
networks: theory and algorithms for engineers and
scientists. Oxford University Press,Newyork, 1997.
(2) White, H. “Some asymptotic results for learning in single
hiddenlayer feedforward network models” [J. Amer.
Statist. Assoc. 84 (1989), no. 408, 1003–1013;
MR1134490 (92e:62119)]. J. Amer. Statist. Assoc.
87, 420 (1992), 1252.

15. REFERENCES
(3) Swanson, N. R., and White, H. A model-selection
approach to assessing the information in the term
structure using linear models and artificial neural
networks. J. Bus. Econom. Statist. 13, 3 (1995),
265 –275.