+- phyphox Forums (https://phyphox.org/forums)
+-- Forum: App-Feedback (https://phyphox.org/forums/forumdisplay.php?fid=6)
+--- Forum: Bugs and Problems (https://phyphox.org/forums/forumdisplay.php?fid=9)
+--- Thread: Can't export > 4 Mo data (/showthread.php?tid=530)
Can't export > 4 Mo data - guillaumeCat - 01-03-2020
Hi,
First of all, thanks for this super useful application.
I have difficulties in exporting large data.
I noticed that for raw accelerometer with g records of durations longer than 5 minutes, i.e. 4 Mo, I cannot export data.
I am using Android 9 on a Motorola One smartphone.
Do you know why ?
RE: Can't export > 4 Mo data - Sebastian Staacks - 01-03-2020
Five minutes is rather short, indeed.
In principle, phyphox has some trouble with exporting large datasets, which is partly due to our implementation but also partly due to the large amount of data. In five minutes your phone generates 60,000 data sets (at least according to our sensor db, the Motorola One acquires data at 200 Hz). Each set consists of a double precision value (8 Byte) for the acceleration along x, y and z, the absolute acceleration and for the time. So, in total the raw data will be 5*60000*8, which is roughly 2.4 MB of raw data without any formatting.
To export it, we need to format it into either a CSV file or an Excel file (which did you use?). Both should in principle be able to handle it, but in case of the Excel file we are using a library which eventually fails if there is too much data in there. Not sure if this is already the limit.
In the case of the CSV file, it will become significantly bigger, because each value is formatted in a text format instead of a binary floating point value, which takes up about twice the amount of memory, resulting in a 5MB text file.
Nevertheless, you usually get trouble to read such large text files in a simple text editor, but exporting should still work - at least in CSV format (not sure about the library for Excel). Phyphox does not yet handle such large files gracefully (there is no warning if we run out of memory and there is no reasonable progress bar during the export). Does it simply crash or do you get no file at all?
There are some things you could try, though:
- You can try to only export a single graph and thereby only exporting two of the five values. To do so, tap the graph you are interested in and then select export from the Tools-Menu of the graph.
- You can try a different format. So, if you used Excel before, try CSV.
- If you do not need to measure at 200 Hz, you can press the "+" button in the main menu and create a simple experiment that reads the accelerometer at 10 Hz to simply have fewer data sets after five minutes.
RE: Can't export > 4 Mo data - guillaumeCat - 01-03-2020
Hi Sebastian,
Following your advices, I succeeded in exporting 1 hour of records directly in .csv format. For 200Hz sampling rate, the generated zip file is 10 MB large and the .csv is 50 MB large. No bug, just some expected latency.
Thank you very much !
RE: Can't export > 4 Mo data - Sebastian Staacks - 01-03-2020
Good to know, thanks. Still, we should improve this part of phyphox to at least communicate the limitations a bit clearer.