Time format and absolute time when exporting data

[turboscholz]

Hello,

first of all, thank you for this awesome app. It's really impressive! As a physician I can only say: I love it!   

My first question here in this forum is the following:

Would it be possible to provide an option in the app where one can activate that the timestamps of the recorded data (of all sensors) will be exported in absolute values following the ISO-8601 standard?

An example timestamp with nanoseconds would be: "2020-09-01T09:00:22.123456789"

Why this would be nice:

1. Sometimes it makes it easier to process the recorded data later on with other tools.
   
2. More importantly, one can compare the recorded data and their timestamps with external tools (which also recorded data with absolute timestamps, on another device) and better align measurements.
   

Best
Uwe

[Jens Noriʇzsɔɥ]

We have just discussed something in that direction (for other reasons) last week. The system clock is not suited for timestamps as we would like to avoid effects of leap seconds – or what could happen more frequently: simple time corrections because the internal oscillator has run off. Moreover, time synchronization could typically be trusted just to a tenth of a second. For us, this would not have been sufficient…

Could you give an example when ISO-8601 might be more convenient for processing data?

-- 
We suppose you mean physicist rather than physician…

[turboscholz]

Well... physicist, of course Sorry for the long text below...

Let me describe my use case: Last month I was reading a very interesting article about the quality of bike lanes in the "Fahrrad Zukunft" magazine. There, they describe how dedicated bikes in the Netherlands are used to measure the surface quality of bike lanes with quite expensive sensors. I than thought that many of those sensors are also available in a standard mobile phone today (with a bit less accuracy of course). And so I got in touch with phyphox: I created an experiment called "Radweganalyse" which measures the acceleration and the GPS coordinates at the same time. Than, I mounted my phone horizontally aligned on my bicycle handlebar, started the phyphox experiment and began to bike along a bike lane in my city. After cycling some minutes I stopped the experiment and exported the data.

The exported data from the acceleration sensor and the GPS sensor have different frequencies. (~200Hz acceleration, 1Hz GPS). With the help of some tools I could manage to resample the GPS data (for example via linear interpolation) to find the interpolated coordinates of every single acceleration measurement record. In this way I could create a resampled Location.csv file which also contains 200 data points per second. This made it finally possible to create a gpx file (with another tool) in which I could use the z-acceleration to be plotted as a function of the GPS coordinates. See the following image as an example result:

   

This was really promising!!! (The data of the hight diagram is actually the z-acceleration in m/s².) Together with a friend I created I script which could even find the positions in the measured data with the highest z-acceleration values, i.e. positions of the bike track where the surface quality is worst. For example when I drove above a manhole cover ("Gullydeckel") I measured quite high z-acceleration values.

Okay, and now I come to the actual issue: When visiting the found positions with high-z values again in real life it turned out that the position on the map is quite often not the real position and that all positions are systematically shifted! For example, the real position of the manhole cover was ~10 meters away from the position in the gpx map. (See the black dots in the image above, these are the measured coordinates with high z acceleration values, but when visiting most of them, the actual street "bump" was always some meters away from that position). Although the data of the GPS and the accelerometer where measured at the same time!

This means that either

1. the GPS measurement of my mobile phone is not that good at all or that
   
2. the times of the GPS and the accelerometer sensors in the data export of phyphox are not well aligned or
   
3. something else which I did not think of up to now happened.
   

So I wondered how can we overcome this misplacement? Well, I also have a GPS device from Garmin which I can mount on my bike. When I record a track with that one and when exporting the track data into a gpx file afterwards it stores the time in the following format as an example: "2020-07-29T20:06:13Z"

My idea would now be to measure the coordinates also with that external GPS device and align that data with the accelerometer data of phybhox. And that's exactly why I would need the data of the phyphox export to contain absolute values of the timestamps. With the current time format I cannot merge both measurements (absolute time <=> relative time).

I hope I made this point clear somehow and I'm interested what you think about the issue. 

[Jens Noriʇzsɔɥ]

This means that either

1. the GPS measurement of my mobile phone is not that good at all or that
   
2. the times of the GPS and the accelerometer sensors in the data export of phyphox are not well aligned or
   
3. something else which I did not think of up to now happened.
   

Reasonable assumptions. Location accuracy of smartphones is (advertised as) down to a few meters.

I would recommend to “sync” your GPS device to a visible event on your smartphone, a certain gully cover or speed bump at a known position – or hit your smartphone at a certain point of interest…

[turboscholz]

I would recommend to “sync” your GPS device to a visible event on your smartphone, a certain gully cover or speed bump at a known position – or hit your smartphone at a certain point of interest…

Yes, I also thought about that. It might be a workaround.

One thing I wondered. In GPS measurements the actual time stamp of the signal is always know, at least I heard about that. Couldn't you use that data and combine it with the Accelerometer sensor data?

[turboscholz]

Well, this workaround requires too much manual intervention in the bike lane analysis.  

It would be even helpful if the file name of the exported data from phyphox would contain the timestamp when the experiment was actually started on the phone. Could that be implemented somehow? (Currently, the timestamp is the current time when actually exporting the data.)

Best
Uwe

[Jens Noriʇzsɔɥ]

One thing I wondered. In GPS measurements the actual time stamp of the signal is always know, at least I heard about that. Couldn't you use that data and combine it with the Accelerometer sensor data?

I have had quite a long discussion with Sebastian about that: yes, there is a time stamp associated to the location, however, it is just an estimate (or vice versa: the location is an estimate for the time stamp). There is definitely an associated uncertainty (IIRC, Android even provides a 1σ interval). This time stamp is in fact already used to sync the location signal to the running time of experiments.

[Jens Noriʇzsɔɥ]

It would be even helpful if the file name of the exported data from phyphox would contain the timestamp when the experiment was actually started on the phone. Could that be implemented somehow? (Currently, the timestamp is the current time when actually exporting the data.)

phyphox and the experiments (in .phyphox file format) are sort-of independent so there is currently no way to achieve this. With an update just released, any changed certainly could not be expected soon…

I just noticed that using absolute times would need some special attention, for instance, what if an experiment is stopped and restarted: such gaps in the timeline need to be sensibly taken care of…

[turboscholz]

yes, there is a time stamp associated to the location, however, it is just an estimate (or vice versa: the location is an estimate for the time stamp). There is definitely an associated uncertainty (IIRC, Android even provides a 1σ interval). This time stamp is in fact already used to sync the location signal to the running time of experiments.

To be on the safe side I am just repeating what you said with my own words: By measuring the GPS coordinates you can retrieve the absolute time stamp of the GPS signal, and with that one you can than align the GPS coordinates to the relative time stamp of the experiment.

Wouldn't it be possible to export the measured absolute time stamp of the GPS signal in a separate column in the Location.csv file? This would be totally sufficient for my special use case. I could compare this absolute time with the time stamp and the coordinates of an external GPS device. This means that there is no need to export all experiments with an absolute time stamp.

Yes, the GPS data also provides the uncertainty for the horizontal and vertical location, that's really nice. Unfortunately, I see a systematic shift of the gully cover locations found in my data analysis compared to their real locations, which makes me wonder if something is not correct in the overall approach: The found location on the map is most of the time like 3-5 meters behind the real position.

Just speculating:

1. Maybe there is an issue in the beginning of the experiment where it always takes some time to retrieve the first data points from enough GPS satellites to calculate the current location of the smartphone. At the same time, the acceleration sensor is already read out from the start. Can the shift I was talking about can come from this time frame in the beginning of the measurement?
   
2. What about time difference (of some seconds) between the absolute time stamp of the GPS signal and the absolute time of the smartphone? Is there a relation which could come into play? I mean, how is the relative time difference between the start of the experiment and the GPS time stamp calculated. I speculate that you somehow must take the absolute time of the smartphone into account, which might be running some seconds off compared to the GPS signal.
   

[Jens Noriʇzsɔɥ]

To be on the safe side I am just repeating what you said with my own words: By measuring the GPS coordinates you can retrieve the absolute time stamp of the GPS signal, and with that one you can than align the GPS coordinates to the relative time stamp of the experiment.

At least three Satellites are required, so there is not one time stamp… the OS –somehow– decides the time stamp that is given with a location.

Wouldn't it be possible to export the measured absolute time stamp of the GPS signal in a separate column in the Location.csv file? This would be totally sufficient for my special use case. I could compare this absolute time with the time stamp and the coordinates of an external GPS device. This means that there is no need to export all experiments with an absolute time stamp.

The absolute time stamp is still not available in experiments at the moment… as we have to change something anyway, we could also do it “right” then…

Unfortunately, I see a systematic shift of the gully cover locations found in my data analysis compared to their real locations, which makes me wonder if something is not correct in the overall approach: The found location on the map is most of the time like 3-5 meters behind the real position.

Ok. Thanks for the feedback. We need to check if there is anything we missed in the .phyphox file.

Just speculating:

1. Maybe there is an issue in the beginning of the experiment where it always takes some time to retrieve the first data points from enough GPS satellites to calculate the current location of the smartphone. At the same time, the acceleration sensor is already read out from the start. Can the shift I was talking about can come from this time frame in the beginning of the measurement?
   
2. What about time difference (of some seconds) between the absolute time stamp of the GPS signal and the absolute time of the smartphone? Is there a relation which could come into play? I mean, how is the relative time difference between the start of the experiment and the GPS time stamp calculated. I speculate that you somehow must take the absolute time of the smartphone into account, which might be running some seconds off compared to the GPS signal.
   

1. No; 2. Location is a black box: we get a place, we get a time, nothing else. :/ As navigation apps in fact work, this appears a non-issue to me.