Functional Feed-forward Neural Networks Part I: Setting it up

This is the first of several posts in which I will go into some subjects concerning artificial neural networks (NN) and their functional implementation. Today, I will set up a classical feed-forward neural network. In future posts I will show how to train and use such a network.
I won't go much into the theoretical details of neural networks as they are covered exhaustingly elsewhere. There are plenty of resources you can check out: books, videos, online material,... whatever you like. Following a list of books I can recommend that cover (not solely) NN:

• Pattern Recognition and Machine Learning (Bishop)
• Pattern Classification (Duda, Hart, Stork)
• Neural Networks for Pattern Recognition (Bishop)
  (I don't own this one myself, but I heard only good things...)
• The Elements of Statistical Learning (Hastie, Tibshirani, Friedman)

Although there is a lot of discussion going on about NN and their widespread use in the fields of Machine Learning, Data Mining, Computational Statistics, Data Analysis and so on I've seldom seen them in conjunction with functional approaches. That's why I was wondering how they would fit.

The Setting

The picture below shows a schematic diagram of a NN, taken from Bishop's PRML book, as I will mostly stick to his nomenclature (the image can be found here):

On the left side is the input of dimension D, in the middle is a so called hidden layer of dimension M and on the right side is the output (of dimension K). In the picture there is only one hidden layer, but there can be any number of them in a network; and they can all be (and usually are) of different dimensions.