How to remove noise from sensor data in python?

Thread Starter

wizardman1313

Joined Oct 23, 2019
5
Hello everyone. I have a sensor that I am continuously reading data from, at a constant rate, using python. Looking at the data there is obviously noise in it (bounces around when it should be zero or a constant value).
After googling for 2 days I've not been able to find a way to remove the noise. All of the examples and questions I've found talk about what to do if you already have the data stored, not if you have the data coming in. They used the scipy package to filter it.
Is there a way to remove the noise continuously? As fast as possible would be great, with reasonable accuracy. Thank you. If this isn't in the right forum feel free to move it, I wasn't sure since it deals with hardware and software.
 

nsaspook

Joined Aug 27, 2009
13,086
Noise is a complex subject. Post a sample capture and picture of the signal if possible. Is the noise source the sensor, daq or environmental in origin?
 

Papabravo

Joined Feb 24, 2006
21,159
Generally speaking, every sampled data system needs an analog front end to limit the spectrum of the noise coming in. If the noise energy is in a band from DC to one-half the sampling frequency you can get rid of it digitally by using a moving average. If the noise is in the band from one-half the sampling frequency to blue light, then you need an analog anti-aliasing filter.
 

Thread Starter

wizardman1313

Joined Oct 23, 2019
5
Noise is a complex subject. Post a sample capture and picture of the signal if possible. Is the noise source the sensor, daq or environmental in origin?
I'm away from the equipment right now for a good while, will share the sample later. The noise is most likely from the sensor. It is the accelerometer from an imu. Doesn't show zero m/s^2 at rest, bounces around by small amounts but never past a certain amplitude.
 

MrSalts

Joined Apr 2, 2020
2,767
The less noise you want, the longer the delay between the movement and the measurement. The so called, analog front end, uses RC filters to average the output of your sensor - if there are any analog components in your accelerometer. Chances are, however, that you are using a modern accelerometer with a serial interface to the registers on the accelerometer and no access to analog circuitry on the accelerometer.

for digital averaging, you'll need to create an averaging system to average out the signals. A simple method in software is to add 4 or 8 or 16 consecutive measurements (or any other multiple of 2) and then divide the sum by the number of measurements you took (make sure your variable can handle the sum total. The speed of the division can be slow if you use actual division operator. Use a bit shift right by one position for each power of two (3 positions for divide by 8, for example).
 

MrChips

Joined Oct 2, 2009
30,720
You can calculate a running average on the fly.
There is no need to store any data besides a double precision sum.
Here are the steps.
1. You need a variable to store the sum. This should be a 32-bit or 64-bit signed integer. Let’s name this Ysum.
2. You will calculate a running average, say Yavg, from Ysum, i.e.
Yavg = Ysum / n, where n is any number of averages you choose.
3. If the instantaneous incoming data is Y, compute Ysum as follows:
Ysum = Ysum + (Y - Yavg)

All variables are assumed to be signed integers.
n values should be powers of 2 so that a division is implemented as right shift operations.
 

WBahn

Joined Mar 31, 2012
29,979
Hello everyone. I have a sensor that I am continuously reading data from, at a constant rate, using python. Looking at the data there is obviously noise in it (bounces around when it should be zero or a constant value).
After googling for 2 days I've not been able to find a way to remove the noise. All of the examples and questions I've found talk about what to do if you already have the data stored, not if you have the data coming in. They used the scipy package to filter it.
Is there a way to remove the noise continuously? As fast as possible would be great, with reasonable accuracy. Thank you. If this isn't in the right forum feel free to move it, I wasn't sure since it deals with hardware and software.
There are a number of approaches you could use, each with pros and cons.

First, make sure that you are not aliasing high-frequency noise down to low frequencies by ensuring that your sensor has suitable anti-aliasing filtering going on.

Next, capture and store a bunch of data so that you have the ability to characterize it and try out various filtering techniques until you find something that works well enough at a speed that is good enough. Then implement that and capture a bunch more data and see whether it really is good enough.

The general field you need to explore is know as digital signal processing. It ranges from ultra-simple to extremely complex.

Among the simplest digital filters to implement is a low-pass infinite impulse response filter of the form:

y[n] = k*y[n-1] + (1-k)*x[n]

To implement this, you simply take the prior output and multiply it by some factor k. Then you add to that the current input data multiplied by (1-k). The value of k is between 0 and 1. If it is 0, you have no filtering and your output equals your input. If it is 1, then you are insensitive to your input and your output remains fixed at whatever the value of y was initialized to (usually zero). The smaller you make k, the faster the response will be, but the less filtering there will be. The larger you make it, the more latency there will be between changes in the input being reflected in the output, but the less noise there will be.

There are games you can play to make the processing blazingly fast on most platforms, at the cost of limiting the allowed values of k.
 

Papabravo

Joined Feb 24, 2006
21,159
If the linear combination of current input x[n] and y[n-1] does not work for you, then an alternate approach is the simple moving average. For example, in an equally weighted SMA at each step you compute the sum of x[n]/4 + x[n-1]/4 + x[n-2]/4 +x[n-3]/4 for a four-sample moving average. If you keep the samples in a ring buffer, you can subtract the oldest value from the running sum, delete it from the ring buffer, divide the new input by 4, add it to the running sum, and place it in the ring buffer. This seems like a good deal of work for four samples, but if you decide on more samples the savings will be manifestly obvious.

If all that turns out to fail, let us know and we'll tell you about other possibilities, like weighting the samples or an exponential moving average. Each method has advantages and drawbacks.
 

ag-123

Joined Apr 28, 2017
276
you need to tell us:
- what is that sensor and what is it measuring?
- what do you perceive that constitute noise?
- you need to present examples of the measured data, say its time series in a graph
anything can be perceived as noise
after all music is 1/f noise
https://www.pnas.org/doi/10.1073/pnas.88.8.3507
and that 1/f noise is music
while there are other 1/f noise that is *not music*
it is *impossible* to remove 1/f noise by the simple means e.g. an RC filter, if the 1/f required signal is part of the 1/f noise high power parts of the spectrum.
https://en.wikipedia.org/wiki/Pink_noise

try this for noise
https://forum.allaboutcircuits.com/...anded-electret-microphone.184747/post-1705927




in all practicality, it is easier to fix it by changing the electret mic
 
Last edited:

nsaspook

Joined Aug 27, 2009
13,086
I'm away from the equipment right now for a good while, will share the sample later. The noise is most likely from the sensor. It is the accelerometer from an imu. Doesn't show zero m/s^2 at rest, bounces around by small amounts but never past a certain amplitude.
Is the accelerometer signal digital or analog? Many of the digital IMU devices have filter and averaging modes that can be enabled to reduce noise. These types of 'cheaper' devices are never going to be very noise free. You will need to
set thresholds for zero or reference points. This device can be configured to do that with an interrupt notification.
PXL_20220812_071520411.jpg

Here is a typical m/s^2 output from a MEMS accelerometer set to 4G. The BMA490L has an internal ASIC for signal processing and movement detection that can be programmed.
PXL_20220812_070909662.jpg

The last number below is the IMU sensor data time-stamp (that generates an interrupt to the controller to get and log the data) that increments every 39.25 usec.
PXL_20220812_065917769.jpg
#define ACCEL_CONFIG 0xa9 // 10101001 200Hz ODR 80Hz filter


PXL_20220812_064117620.jpg
#define ACCEL_CONFIG 0xa5 // 10100101 12.5Hz ODR 5.06Hz filter


https://github.com/nsaspook/wfi32/tree/wfi/firmware/src
 
Top