Layout questions for a Recurrent Neural Network

Thread Starter

Mike Arsenault

Joined May 11, 2023
I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each node collects inputs from EVERY node in the previous layer and then adding a bias, which is normal - like a normal Feed-Forward network.

But here is my question - for the Recurrence, in addition to inputs from previous layer, does a node:
  1. ONLY get feedback from itself
  2. get feedback from every node in it's own layer (this is what I'm currently doing - but it may be overkill)
  3. or can a node get feed back from nodes in layers AFTER it - I've seen architecture diagrams that suggest this.

I'm thinking the first one is the way to go - the second one is what I'm current trying on a number of applications and very simple models just never seem to converge. I imaging the 3rd one, would be even more difficult to train.

OR are all 3 correct - they are just different types of RNNs?

Also, I'm assuming that nodes in the OUTPUT layer do not feedback to themselves - those nodes still do activation and have an array of weights and a bias, but they ONLY collect inputs from the last hidden layer - is that correct?