For my phyphox program of pressure measurements using SensorTag of Texas Instruments I have introduced 'gausssmooth' in order to see averaged pressure value. To the obtained pressure(time) graph I have added a second curve produced by the 'gausssmooth', but after 10 seconds (0.1 sec per point) I got a peak which moves with time. What it could be?
11-15-2020, 10:37 AM (This post was last modified: 11-15-2020, 10:43 AM by solid.)
Ooups, the peak-artifact of the 'gausssmooth' is always here. Now it comes from the offset g=9.8 m/s² of the data. Only here (with the same phyphox program) it is visible with zoom..
If one subtract the offset before applying 'gausssmoth' and then add it, the peak-artifact is not visible. But it is not a good solution, is not it? Anyway I give the "correction" here.
Interesting one. I have a few ideas:
- It could be limit of the numerical stability of our implementation. Even the strongly visible peak is only a deviation of 0.02% and with a sigma of 20 the algorithm is calculating 60 samples to the right and left of the Gauß distribution, summing up all values and normalizing by the raw sum of these samples from the Gauß curve. Especially the small values at the edge will be close to the resolution of the floating point numbers when added to the sum, so something might be happening there...
- If could be that the peak occurs exactly 60 samples before the end of the measurement. As mentioned above, we calculate a width of 3*sigma=60 samples to get numerically close to a true Gauss, which otherwise needs to be calculated across all samples. When we do not have enough samples to the right (or left) we need to change the strategy and instead of using a pre-calculated normalization value, we need to normalize with the values that match the available data. Again, I do not see it right away in the code, but this will also have a numerical limit. Could you check if it is 60 samples to the end?
By the way: Have you tried the LOESS smoothing? Not sure if it is more stable though.
Sebastian, you are right !
The peak appears at the curve center when the number of samples is 120 (approx) and keeps the distance of 60 samples to the end during further evaluation. Peak direction is related to the offset value and for a negative offset is positive. I wonder why there is no peak connected to the beginning of the sampling... It is strange, if you will gauss-smooth a constant value you will get such a peak. Normally nobody see it, it is about 0.00022 from the offset value.
As to the LOESS smoothing, I thought that it is to draw a smooth curve trough rare experimental points.
I used smoothing just to get stable indications of pressure and I have drawn the smoothed curve for a general interest. I could use an averaging of some last points, but I am not so fine with your 'subrange'. It would be nice to have negative counts as in Python...
(11-16-2020, 08:04 PM)solid Wrote: As to the LOESS smoothing, I thought that it is to draw a smooth curve trough rare experimental points.
Au contraire, IIRC. We have introduced the LOESS smoothing for “Walk the Graph” activities and it has given us a smoother experience particularly close to the “end” of the data.
Ah, I think I found the issue and I can not blame it on numerical precision. I use a lookup-table for the Gauss function which I calculate once and which is already normalized. But if I cannot use 60 values to the right and to the left, the normalization of the look-up table is wrong and I need to renormalize the result. And the if that decides whether to do this renormalization is simply of by 1, so the normalization is slightly off if there are exactly 59 values to the right.
I also see your desire for negative indices. However, this would be problematic in other situations when you just calculate the index and want the index to be cut off at the end of the dataset instead of cycling to the other end of the data. However, to get the last n points, there is a simple alternative (unless it is an immense data set and speed is an issue):
Simply create another buffer with a fixed size n of exactly the number of data points that you want to average. Then use append to throw the entire buffer into it from which you want the last n values. The first values will be pushed out until the last n are in the new buffer.
Finally, about the LOESS method: Its use depends on the width (we call it d in the documentation) that you set. In fact, I implemented it to give a smooth representation of noisy data from a range-finder where Gauss would smooth away the curvature. LOESS fits a polynomial to a local window of data points and in principle you can make that window as wide as you like.
Am I correct that the issue with the dip is not too urgent? I will probably quickly fix the code, but if you do not need it right away, I will not push a beta version just for this little change.
(11-16-2020, 08:04 PM)solid Wrote: As to the LOESS smoothing, I thought that it is to draw a smooth curve trough rare experimental points.
Au contraire, IIRC. We have introduced the LOESS smoothing for “Walk the Graph” activities and it has given us a smoother experience particularly close to the “end” of the data.
Exactly. Looks like we are both replying during the same non-working hours
11-17-2020, 12:24 AM (This post was last modified: 11-17-2020, 12:28 AM by solid.)
Thanks for LOESS. I am not sure that I am using it correctly? but somehow it works.
(11-16-2020, 10:50 PM)Sebastian Staacks Wrote: Am I correct that the issue with the dip is not too urgent? I will probably quickly fix the code, but if you do not need it right away, I will not push a beta version just for this little change.
tried something ordinary with 'gausssmooth' - acceleration (y) with g and all is OK, no strange peaks...
Ooups, the peak-artifact of the 'gausssmooth' is always here. Now it comes from the offset g=9.8 m/s² of the data. Only here (with the same phyphox program) it is visible with zoom..
