Real time data download

[Sebastian Staacks]

Sorry, I somehow missed the notification for your reply.

I did not yet have a chance to test your code, but I think the problem is that you are requesting the channels separately and therefore are requesting faster than the phone can respond. Your request to PP_CHANNELS_def seems to work fine, but your request to PP_CHANNELS_def_x follows without a delay and I would not be surprised if the (single-threaded) web server on the phone is not yet ready.

In any case, this is not a good idea. You should ask for all channels in a single request to keep the network load low, but also to keep the reference time consistent (i.e. what happens if there is one new data point between requesting the y and the z axis?). So, simply request them all in one go:
get?acc_time=42&accX=42|acc_time&accY=42|acc_time&accZ=42|acc_time

[Romaxx]

Hm, we are talking about smartphone sensors, aren't we? There is a difference between the capabilities of built-in sensors and that what the API provides us. The rates in https://phyphox.org/sensordb/ are upper limits and I could not find anything (common) avove 500 Hz…

Thanks, yes, I found this site a few days ago, and the limit seems to be 500Hz overall. Yes, we are talking about smartphone sensors. But the BMI260 from Bosch seems to be capable of 1600 Hz. So perhaps this is somehow restricted by Android or Apple. Did you know about https://developer.android.com/reference/...ectChannel ?

Your code porposal is still not working for me. I have espesially problems with the "...|acc_time..." connection, so the "or" I think.

The Code I tried is:

Code:

#MIDI configuration
M_OUTPUT = "Midi Through:Midi Through Port-0 14:0"
M_CHANNEL = 0
M_CONTROLS = [1, 2, 3, 4] #You can send on different CC channels
# M_CONTROLS = [70]

#phyphox configuration
PP_ADDRESS = "http://192.168.178.29:8080"
PP_CHANNELS = ["accX", "accY", "accZ", "acc_time"] #If using different CC channels, define multiple phyphox buffers
# PP_CHANNELS = ["accY"]

import requests
import time
import numpy

value_temp = 0

time_update = 0
count = 0
while True:
    if count == 0:
        PP_CHANNELS_def = ["acc_time=full","accX=full","accY=full","accZ=full"]
    else:
        PP_CHANNELS_def = ["acc_time=" + str(time_update), "accX=" + str(time_update) + "|acc_time","accY=" + str(time_update) + "|acc_time","accZ=" + str(time_update) + "|acc_time"] #If using different CC channels, define multiple phyphox buffers

    url = PP_ADDRESS + "/get?" + ("&".join(PP_CHANNELS_def))

    data = requests.get(url=url).json()

    time.sleep(0.04)
    count = count + 1
    time_update = count * 0.04;

    # Uncomment to send pitch bend
    time_value = numpy.array(data["buffer"]["acc_time"]["buffer"])
    value_x = numpy.array(data["buffer"]["accX"]["buffer"])
    value_y = numpy.array(data["buffer"]["accY"]["buffer"])
    value_z = numpy.array(data["buffer"]["accZ"]["buffer"])

The error is:

Code:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/roman/ownCloud/Documents/Phyphox/phyphox2.py", line 39, in <module>
    data = requests.get(url=url).json()

  File "/home/roman/anaconda3/envs/phyphox/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)

  File "/home/roman/anaconda3/envs/phyphox/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)

  File "/home/roman/anaconda3/envs/phyphox/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)

  File "/home/roman/anaconda3/envs/phyphox/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)

  File "/home/roman/anaconda3/envs/phyphox/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Thanks again.

[Jens Noriʇzsɔɥ]

Perhaps a stupid question: have you checked the “url” string, does the ampersand require an escape?

[Sebastian Staacks]

I have tracked the problem down to an unexpected (if not buggy) behavior of requests: It form-encodes the pipe "|" no matter how you pass it to get(). The docs (https://docs.python-requests.org/en/mast...t-requests) specifically say that it should not encode it if you pass the url or the parameters as a string, but it still does so and phyphox at the other end does not decode the form encoding (why should it - it's not an endpoint for a HTML form and it still is a valid HTTP request).

So, after searching for ways to circumvent this problem, I ended up using urllib3 instead and this seems to work:

Code:

#MIDI configuration
M_OUTPUT = "Midi Through:Midi Through Port-0 14:0"
M_CHANNEL = 0
M_CONTROLS = [1, 2, 3, 4] #You can send on different CC channels
# M_CONTROLS = [70]

#phyphox configuration
PP_ADDRESS = "http://192.168.2.194:8080"
PP_CHANNELS = ["accX", "accY", "accZ", "acc_time"] #If using different CC channels, define multiple phyphox buffers
# PP_CHANNELS = ["accY"]

from urllib.request import urlopen
import json
import time
import numpy

value_temp = 0

time_update = 0
count = 0
while True:
    if count == 0:
        PP_CHANNELS_def = ["acc_time=full","accX=full","accY=full","accZ=full"]
    else:
        PP_CHANNELS_def = ["acc_time=" + str(time_update), "accX=" + str(time_update) + "|acc_time","accY=" + str(time_update) + "|acc_time","accZ=" + str(time_update) + "|acc_time"] #If using different CC channels, define multiple phyphox buffers

    url = PP_ADDRESS + "/get?" + "&".join(PP_CHANNELS_def)

    response = urlopen(url).read()
    data = json.loads(response)

    time.sleep(0.04)
    count = count + 1
    time_update = count * 0.04;

    # Uncomment to send pitch bend
    time_value = numpy.array(data["buffer"]["acc_time"]["buffer"])
    value_x = numpy.array(data["buffer"]["accX"]["buffer"])
    value_y = numpy.array(data["buffer"]["accY"]["buffer"])
    value_z = numpy.array(data["buffer"]["accZ"]["buffer"])

However, there is one other thing you probably want to change: You should not use "time_update" in your request, but the latest value you received in data["buffer"]["acc_time"]["buffer"] in your previous request (so, the last value in "time_value"). Otherwise, your time_update will eventually get out of sync with phyphox (well, it will probably start out of sync as you cannot start the script and the measurement at exactly the same time) and you will either get duplicates (imagine your time_update is 10 seconds behind and phyphox will send you everything from the last 10 seconds every time) or miss values (imagine it is a few milliseconds ahead and requests before a data point with that timestamp exists). The whole point of this function is to tell phyphox which is the last value you received, so it can send you anything after that.

[Romaxx]

I have tracked the problem down to an unexpected (if not buggy) behavior of requests: It form-encodes the pipe "|" no matter how you pass it to get(). The docs (https://docs.python-requests.org/en/mast...t-requests) specifically say that it should not encode it if you pass the url or the parameters as a string, but it still does so and phyphox at the other end does not decode the form encoding (why should it - it's not an endpoint for a HTML form and it still is a valid HTTP request).

So, after searching for ways to circumvent this problem, I ended up using urllib3 instead and this seems to work:

Code:

#MIDI configuration
M_OUTPUT = "Midi Through:Midi Through Port-0 14:0"
M_CHANNEL = 0
M_CONTROLS = [1, 2, 3, 4] #You can send on different CC channels
# M_CONTROLS = [70]

#phyphox configuration
PP_ADDRESS = "http://192.168.2.194:8080"
PP_CHANNELS = ["accX", "accY", "accZ", "acc_time"] #If using different CC channels, define multiple phyphox buffers
# PP_CHANNELS = ["accY"]

from urllib.request import urlopen
import json
import time
import numpy

value_temp = 0

time_update = 0
count = 0
while True:
    if count == 0:
        PP_CHANNELS_def = ["acc_time=full","accX=full","accY=full","accZ=full"]
    else:
        PP_CHANNELS_def = ["acc_time=" + str(time_update), "accX=" + str(time_update) + "|acc_time","accY=" + str(time_update) + "|acc_time","accZ=" + str(time_update) + "|acc_time"] #If using different CC channels, define multiple phyphox buffers

    url = PP_ADDRESS + "/get?" + "&".join(PP_CHANNELS_def)

    response = urlopen(url).read()
    data = json.loads(response)

    time.sleep(0.04)
    count = count + 1
    time_update = count * 0.04;

    # Uncomment to send pitch bend
    time_value = numpy.array(data["buffer"]["acc_time"]["buffer"])
    value_x = numpy.array(data["buffer"]["accX"]["buffer"])
    value_y = numpy.array(data["buffer"]["accY"]["buffer"])
    value_z = numpy.array(data["buffer"]["accZ"]["buffer"])

However, there is one other thing you probably want to change: You should not use "time_update" in your request, but the latest value you received in data["buffer"]["acc_time"]["buffer"] in your previous request (so, the last value in "time_value"). Otherwise, your time_update will eventually get out of sync with phyphox (well, it will probably start out of sync as you cannot start the script and the measurement at exactly the same time) and you will either get duplicates (imagine your time_update is 10 seconds behind and phyphox will send you everything from the last 10 seconds every time) or miss values (imagine it is a few milliseconds ahead and requests before a data point with that timestamp exists). The whole point of this function is to tell phyphox which is the last value you received, so it can send you anything after that.

Thanks for your answer.

I did not try it so far, but I will (probably tomorrow).

Edit: I now tested your code and it works as expected. Thanks for this!!!

But, can you tell me what you think about my first Question in my last post, the one with the link?

Thanks again!

[Jens Noriʇzsɔɥ]

We are aware of SensorDirectChannel, however, Sebastian has not looked more deeply into it yet. The approach is quite different to the original way and the documentation –as well as information on devices supporting it– does not entirely sound promising, for instance, “actual rate is expected to be between 55% to 220% of nominal rate” – it appears to be optimised for more efficiency rather than higher rates.

It is on an ever-growing list of possible improvements with no plans when it could get be integrated. Certainly, one of the more interesting entries, however, also one of those that require some research to implement it on a reasonable level.

[Romaxx]

We are aware of SensorDirectChannel, however, Sebastian has not looked more deeply into it yet. The approach is quite different to the original way and the documentation –as well as information on devices supporting it– does not entirely sound promising, for instance, “actual rate is expected to be between 55% to 220% of nominal rate” – it appears to be optimised for more efficiency rather than higher rates.

It is on an ever-growing list of possible improvements with no plans when it could get be integrated. Certainly, one of the more interesting entries, however, also one of those that require some research to implement it on a reasonable level.

Thanks for your answer on this.

I would be very interested in such an feature, as the nominal frequency is around 800Hz, and many real world applications require more the 1 kHz, i.e. performing classifications based on FFT.

The standardisation of the sensor samples are easily performed for people who have basic understanding of sensors, so there would be no need to acquire this information if necessary.

Actually I would expect that the method is not hard to implement, there is rather the question how many smartphones fulfill the requirements, but I would say a simple if condition or a query if it is supported would be enough.

Nevertheless, I m very impressed about the Phyphox app anyway .

[Jens Noriʇzsɔɥ]

*inserts “one does not simply …” LOTR meme*

[Romaxx]

Hi guys,

is there a Matlab solution to stream data in real-time?

Thanks!

Romaxx

[Jens Noriʇzsɔɥ]

There are two threads on MatLab in this forum: https://phyphox.org/forums/showthread.php?tid=716 and https://phyphox.org/forums/showthread.php?tid=796 – and a search text field at the top…

You could also find some examples on the net, for instance, at https://uk.mathworks.com/matlabcentral/f...?q=phyphox