I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each node collects inputs from EVERY node in the previous layer and then adding a bias, which is normal - like a normal Feed-Forward network.
But here is my question - for the Recurrence, in addition to inputs from previous layer, does a node:
I'm thinking the first one is the way to go - the second one is what I'm current trying on a number of applications and very simple models just never seem to converge. I imaging the 3rd one, would be even more difficult to train.
OR are all 3 correct - they are just different types of RNNs?
Also, I'm assuming that nodes in the OUTPUT layer do not feedback to themselves - those nodes still do activation and have an array of weights and a bias, but they ONLY collect inputs from the last hidden layer - is that correct?
Thanks
Mike
But here is my question - for the Recurrence, in addition to inputs from previous layer, does a node:
- ONLY get feedback from itself
- get feedback from every node in it's own layer (this is what I'm currently doing - but it may be overkill)
- or can a node get feed back from nodes in layers AFTER it - I've seen architecture diagrams that suggest this.
I'm thinking the first one is the way to go - the second one is what I'm current trying on a number of applications and very simple models just never seem to converge. I imaging the 3rd one, would be even more difficult to train.
OR are all 3 correct - they are just different types of RNNs?
Also, I'm assuming that nodes in the OUTPUT layer do not feedback to themselves - those nodes still do activation and have an array of weights and a bias, but they ONLY collect inputs from the last hidden layer - is that correct?
Thanks
Mike
