When you create a very narrow band-pass filter it becomes a resonance circuit. Any input similar to an impulse function will cause the filter to oscillate. Hence you end up with y = sin(ωt) as output.

What you need is a phase-locked loop (PLL) to demodulate the signal.

I am happy to inform that thanks to your nice tip, I made my own software PLL and also found a ready one online, but I'm struggling now to make it work properly. You can see a video of the online one working at http://www.excelunusual.com/a-phase-locked-loop-pll-model-video-preview/ . Seems pretty nice.

The video shows the spreadsheet you find at http://www.excelunusual.com/phase-lock-loop-pll/, but I think there is a mistake there. I recommend using the one at http://www.excelunusual.com/how-to-model-a-phase-locked-loop-pll-in-excel-part4/ instead.

The thing is, I couldn't make it work for frequencies around 5000Hz. Since the video uses 10Hz and my frequency is 500 times higher, it should be enough to decrease the time step by the same ratio or a bit more. I fiddled with the RC and FVCO values, but to no avail. It never locks up. And I have no idea why. That is where I'm stuck right now.

But even when I make it work, it seems clear that, just like my bandpass filter, the PLL takes a few (or many) periods of the input signal to deploy a result. In other words, each bit of information still needs to be stored into 10 or 100 or 1000 consecutive wave periods of a given frequency/amplitude/phase, instead of only 1 as I expected.

 

If I understand your proposed modulation scheme correctly, you are using on-off-keying (OOK) on a 5 kHz carrier. That is, you have a 5 kHz sinusoid and turn it on and off at intervals as fast as once per 5 kHz period. This is multiplication of your "carrier" (5 kHz) by a stream of unfiltered bits (0's and 1's). The bandwidth of your unfiltered bitstream is theoretically infinite, as is the modulated output. In practice, your modulation happens at the sample rate of 44.1 KHz and is thus limited, and the modulated output is also limited in BW to at most 20 kHz by most audio DACs.

The first thing you'll need to do is to slow down the bitstream of 0's and 1's relative to the carrier frequency to reduce your unmodulated bandwidth. Then, filter the bitstream with a pulse shaping filter or run it slowly enough to not care about ISI of the filtered modulated signal. Overall, you should get your modulating signal to be less bandwidth spectrally than your carrier. For example, for a 5 kHz "carrier" tone, you might use 1kHz of signal bandwidth, which means doing OOK at not more than 1 kHz - and even better to be at not more than a few hundred Hz to pass the third harmonic if you don't wan to bother with pulse shaping on the OOK stream. Then, given the bandwidth of the modulating stream of 0's and 1's (or pulse-shaped version of it), you can set your FIR filter BW at the receiver to reject non-signal inputs. For the example above of 1 kHz OOK bandwidth, that means an FIR filter that passes 4.5 to 5.5 kHz at a minimum. If you are too narrow on your receiving FIR, then you will introduce inter-symbol interference (ISI) on the OOK stream, and you'll need to do a de-convolution of the signal (aka Viterbi) - though no deconvolution will be able to undo an overly-harsh receive filter or transmission medium filtering.