Weird log data from data sent by ESP32

[Sebastian Staacks]

• When timestamps in Phyphox suggest that 1 sec has passed, only about 120 notifications (equivalent to 600 ms) arrived.This is a stable problem.

Not surprising so far, this is similar to the maximum rate we have seen when trying to use higher rates. This of course can depend on signal quality and the actual devices used.

• While timestamps logged show about 7-8 decimal places (which suggest a high resolution) values don't show a constant increase from entry to entry when one looks to decimals 2-8. Sometimes there is even the same timestamp for sequent entries !
  I know that anything more than 3-4 decimals doesn't make any sense, but if we cannot rely on decimal 2 than everything faster than 10Hz won't work.
  
• Every 2 seconds the notification's value shows an instant increase of about 820 counts (= 820 ms = 160 notifications); so value of 2 sequent notifications differs by 820 counts
  It looks like as ESP32 has paused sending for a while; but at the same time there is NO gap in timestamps logged !!
  

Well, the clock used has indeed a high resolution. Phyphox calls System.nanoTime() (Android) when it receives a value and that is the number you get there (here it is in the code:https://github.com/Staacks/phyphox-android/blob/335ab6bd71ec63a1eb769fcd3a0069310b00f308/app/src/main/java/de/rwth_aachen/phyphox/Bluetooth/BluetoothInput.java#L306) . However, this is not the time the value was measured or the time at which you called "notify". It is the time at which the value has passed the Bluetooth stacks and was handed over to the callback function in phyphox. When multiple values have the same timestamp, I would guess that there is some optimization going on that multiple values are transferred in one "burst", but I have to admit that my understanding of the Bluetooth protocol at that point is very limited.

Good news is: this is not limited to Phyphox app; I also was able to see that behaviour e.g. with nRF connect app or LightBlue when logging data.
So it seems to be a BLE topic in general (may be in combination with Android).

Relieved to read that This adds to my guess that it is an optimization in Bluetooth. BLE is not designed to guarantee the arrival of a value at a given time, but to be suitable for low energy applications. It makes sense that multiple values are stored to be sent in a single burst for efficiency.

1. Any idea where this might come from ? Anybody seen this or something similar before ?
   
2. Is frequency of notifications just to high (didn't test other intervals thoroughly) ?
   
3. Any hint how to proceed or solve ?
   

So, yes, 200 Hz in my experience is a bit much. You might be able to increase the data rate by packing multiple values into a single notification. Simply let your characteristic not hold a single 16bit int (2 byte), but a sequence of multiple, which you can unpack in phyphox by using different offsets and appending the result. Not sure how much it helps, but it should reduce the overhead a bit. If this helps a lot in your case, you can also increase the MTU since the last phyphox version, to pack even more into a single package, although I would expect that there is a limit at which it won't help that much.

The timestamp created by phyphox will always be subject to network lag, so even at lower rates with a weak and unstable connection, you might see some surprising delays in this value. To solve the timestamp issue, meaning to get reliable and precise timestamps, you need to measure the time on your device and either transfer it or make sure that you transfer equidistant (or is "equitemporal" the word?) measurements and create the timestamp in phyphox based on the assumption that you did not miss anything. You can either pack the timestamp with your transferred value (and simply assign them in phyphox using an offset= and the appropriate conversion function) or have a look at what we do for most sensor boxes. They all just send a stream of values and we have to assume that they are created at a given rate. This reduces the required bandwidth and remains precise even on a laggy connection. The downside is that value is not long-term stable if a package is missed. An example can be found in our configuration for the SensorTag: https://github.com/Staacks/phyphox-exper...er.phyphox - we simply create the time axis with a "ramp" module.