That's a bit of a dodgy circuit. The shift registers may have to sink up to 160mA (all segments on), and this could damage them.in this circuit here, the current appears to be for lower power LEDs, I'm powering 20ma 5mm LEDs at 3.1v, and obviously need to use drivers on them.
in the circuit above, I feel I need something like the ULN2803 behind each shift register for extra current sinking, but I'm not sure if that is correct, I feel that it needs to be a PNP, please correct me if I am wrong.
Also, to drive more power, would the Nfets for the rows need to be more powerful?
N-ch fets only need to be used if you are using >20mA LED's or if your microcontroller can't source 20mA per pin and 160mA in total. Most MCU's can.