My first experiences with the new editor for a larger project

[Trikes]

Hello Phyphox team, in the last couple of days is started to migrate an older experiement (see attachment of this post) from the the old to the new editor.

I believe, I understand the obvious benefits of the new editor. The Analysis tab has become leaner (less crowded) by moving container assignments to individual other tabs. Also the permanent display of the containers is increasing programming efficiency a lot.

Suggested preparation for larger experiments in the old editor before switching to the new editor, because the old editor offers a 2d layout while the new editor requires 1d editing: I believe the new editor respects the horizontal position from the old editor, but it seems that the vertical position is not recognized. Therefore the recommendation would be to "stretch" old projects in the old editor ahead of their migration to the new editor (basically this is resolving the 2d layout into a more linear arrangement of the blocks in the old editor). This way related blocks would be grouped together in the new editor. For my project, which I did not stretch this way ahead of migration, I found that related blocks of my 2d project have been spreaded over a wide range in the new 1d editor. It took quite a while to sort these things out in the new editor to provide in the UI some coding coherence to the user for more easily understanding the coding. I believe, smaller projects and projects with the typical top-left to bottom-right diagonal layout should not suffer much this way when being imported to the new editor.

Grouping blocks in the old editor provided some kind of implicit documentation (proximity of the blocks = "they belong somehow together"). Right from the beginning of working with the new editor I wanted to compensate the missing freedom of the 2d layout of the blocks by adding comments. I really would apreciate some kind of comment block (could be a Logic block) which would do nothing else but displaying some comment in the new editor. No, I don't ask for line numbers, but comments really could help a lot to structure and explain the coding. Yep, I noticed the possibility to add comments to blocks which pop up by clicking on the question mark icon then, but it is an extra click and not directly inside the "coding".

The migration between the editors works pretty well, especailly when taking into account that there is an implicit and enforced upgrade of the file format. From other threads in the forum I understand that attributes are omitted in the XML file created by the new editor in case their value is identical to their default value. But this way a project depends on the fact that the defaults never would change, I would prefer to also keep the default values explicitly in the XML to document them and avoiding the need to look them up each time. In my project the deprecated Clear attribute is essential. I hope that the automatic conversion to the replacing Keep attribute has worked correctly, at least I have not found obvious problems in this regard, but I still need to check things in detail and get used to the Keep thinking.

I noticed that the Empty input block has not been added automatically in all cases to the operation/data/logic blocks with undefined inputs? I like that Empty block because it adds some consistency. I have added it manually in the new editor to not leave originally undefined inputs "uncommented". I admit that I did not care about this in the old editor, which also might be the reason why the Empty input block was missing in the new editor at some of my blocks. I hope that I did not change the logic of the experiment by adding the Empty inputs later on manually, still need to check this. But I found that for some blocks I violated their syntax when adding the Empty input block, which I only noticed in the mobile app after scanning the QR code: 'Line 516: Value-type not allowed for input "length".' has happend for the Const and the Threshold blocks, for example. Hmm, XML line numbers ... maybe I would like them as some kind of mouse-over in the new editor to be able to relate the error message from the app to the actual source code. Or maybe having the XML import check of the app already inside the new editor? This would save some roundtrips between the PC-based browser (design time) and the mobile device (runtime).
 
When switching tabs in the new editor always the vertical center of the new target tab is being displayed. For smaller projects this is not a big thing, but for larger projects (my XML has 1400+ lines) this always means to manually scroll back to the blocks you are currently working on. A lot of enforced scrolling, I can tell. Maybe the browser could remember the position inside the tabs and jump to the last known position when visiting a tab again? This would increase usability a lot.

While grabbing some UI element with the mouse I got another one nearby or even "under" the one I intended to grab. By missing the scroll bar I dragged a block or started creating a new container. There is some general tendency, I feel, in UI elements to become smaller with regard to monitor resolution. It might seem looking cooler, but actually it is getting in the way of efficiently use the UI. In the end it becomes necessary to apply some accessibility functions to not get disturbed by such side effects. Oops, accidentally I touched the Back button on my mouse and lost my experiment in the new editor. The old editor had some emergency popup to ask if leaving the editor is really the thing I want to do. I was lucky to have saved my last changes in the new editor to XML and have not actually lost anything by accidentally leaving the new editor. Speaking of such UI related observations, is it possible to combine blocks in contradiction to their syntax? While not precisely dragging blocks on my large monitor I docked some Subrange blocks inside each others which was not making any syntax sense to me (I have not tried to load such an experiment to the mobile app, maybe such sub-blocks not covered by the original block syntax just would be ignored).

I needed to accept that my browser (Firefox) was not jumping between the occurrences of search terms. It was only possible to highlight them all and scroll through the tabs manually to visit the occurrences (introducing the chance to miss one or the other occurrence). Would be nice to have some search function inside the new editor for container names, which also would help in preparing container renames and deletions.

There was one hard but hidden error when importing my old project to the new editor. I found the symptom of this error only at runtime and the root cause after some debugging in the new editor. The operator of the If block was not recognized at all by the new editor and instead of getting the operator I defined in the old editor I got in all cases the = operator in the new editor. For fixing the If blocks I needed to manually compare all If blocks between the two editors. Better to not switch off the old editor completely, please keep it.

Container rename: The rename command is a little hidding in the container selection popup (scroll down to the bottom, next to the delete command, easy to select the wrong command). After a rename the UI does not update unless the blocks are edited somehow. To avoid manual cleanup in the UI the workaround is to save the experiment after a rename and load that saved file again, then all the dependent blocks in the UI do read the new container name. There also was once a strange error when renaming a certain container which introduced two times the same container definition in the XML file which caused an automatic renaming of one of the identical containers at the next XML load (I needed to edit the XML file to get rid of the duplicate definition).

Container deletion: After removing all usages of a container in the tabs and its final deletion (context menu) from the container list the container remains in the list of the available containers (container selection popup). The deleted container vanishes when saving the XML files and loading the file again. On the other side, if I explicitly delete a container from within the container selection popup it actually vanishes from that popup list (there is not yet confirmation popup when deleting a container?), but its block remains in the permanent list of all containers in the original UI of the editor; I need to delete the block manually from there (just an observation: when opening the container selection popup from this abandoned container block, no container is preselected (checkmark), because that container already is deleted and no longer available). Ah, when deletion a container which is still used somewhere the new editor won't save. Probably there is some check of the experiment against the Phyphox XML schema, but I did not noticed any error message in the UI, took a while to find that last remaining container usage on the View tab. After deleting this last usage I could save again.

Small wish for inside the container selection popup: My project has about 190 containers which is a lot scrolling in that popup. I wonder if there could be some jump in the popup to containers starting with a character by just pressing the character on the keyboard.

Copying the name of a container is possible? Have not seen a simple copy'n'paste option (for deriving dependent container names, for example). To copy the container name I need to open the container selection popup and to start renaming the container. At the rename popup I can copy the container name and cancel the operation. A workaround. Ah, speaking of the renaming popup: the old container name also could appear in the input field as a kind of input help.

Unattached blocks ignored at runtime: blocks at the Analysis tab that happen to be not directly bound to the Analysis block do not through any error at the XML import to the mobile app. Just an observation. I assume such abandoned blocks are just being ignored at runtime.

I understand that the new editor offers an easier access to create (small) experiments. For larger experiments I still love the "data flow connection lines" of the Analysis tab of the old editor which give a nice idea of how the experiment might behave at runtime, in my opinion, this data flow aspect is a little hidden in the new editor (currently I believe I would not have had the idea explained in this posting in the new editor, the "lines" in the old editor have been essential for me). But the new editor supports in a better way debugging an experiments (I could more easily identify semantic container duplicates and improve the experiment). I see why you switch to the new editor and for most users this is the right approach.

New editor rooky errors on my side easily possible, cheers, Trikes.

[Trikes]

Update after some more work with the new editor:

Accidentally and unnoticed I created once more the subrange-subrange block hierarchy, because the second subrange block snapped inside the preceding subrange and not parallel to it. When scanning the QR code this was recognized by the mobile app and I got error message "Line 784: Unknown tag subrange". Snap point proximity might be an issue difficult to solve. But maybe there will be at one point in time the same check in the online editor as at import the experiment to the mobile app.

I saw that abandoned blocks (blocks not snapped in to the dedicated "tab" blocks like "analysis") will not make it into the XML file, isn't it? I was lucky because I kept the browser opened and was not really losing the abandoned blocks; this only would have happened if I would have reloaded the XML file. Well, this is a one time observation which better should be validated by somebody else. Maybe the editor could warn at XML export that there are unsnapped blocks which will be lost at the next XML export/load. There is no guarentee that the browser would survive the next OS update and restart.

With regard to the block comments "(?)"on the Analysis tab: (1) Similar comments are not possible at the other tabs? Would love to use them on the other tabs as well. (2) I started using these comments on the Analysis tab, but after having created 6 comments I found that they don't make it into the XML file. They seem being just lost, better someone is validating this observation as well. Maybe theses comments do not have a related attribute in the XML file, which might be the reason why they are being not offered at the other tabs as well.

Because comments are difficult the only way to introduce some kind of "documentation" seems to be the container names, but they are restricted in the UI to 48 characters? Same for formulas. Well, the blocks hardly fill the horizontal lenght of the screen, increasing that limit to 64+ would be quite nice.

The undo is cool, but the UI does not jump to the block(s) changed far away by the undo = difficult to see if it has worked or not in case the undo was at some part of the Analysis tab that is not being displayed when clicking on undo. Cannot tell how many undos are possible, I found at least 20 are possible.

My experiment as saved by the new editor seems to run from what I can see in the mobile app. The migration was much less effort than expected (to be honest, I had the fear I needed to start from scratch, but this was not the case). Next step is to analyse the exported data and the experiment snapshot to understand if the experiment still behaves the same as before the migration.

[Trikes]

Update after some first executions of my migrated experiment:

A first execution of my experiment as saved by the new editor showed constant NaN for one column in the exported CSV file. I easily was able to find the root cause of this error in the container tab in the new editor. While setting the focus to the browser tab to which the new editor was loaded I had apparently done some wrong mouse click(s) and was not only setting the focus to that tab but also the static tag for one container. After removing that unwanted tag the export data of the experiment looked good again (by values in OpenOffice and visually in QGIS). In the new editor displayed tags can be edited right away, in the old editor changing these tags required some additional navigation step into the container popup. On my screen I have in normal display mode 78 container tag checkboxes displayed at once in which I could accidentally set a checkmark. But on the other side this direct display of so many tags supported me nicely in finding the root cause for the NaN. In the old editor the search for that root cause would have needed significant more time, and probably I would have needed to directly analyze the XML file instead. I assume that with the new editor it is less often necessary to go into the XML file and do edits directly in that file. The "instant edit surface" of the new editor is much bigger, you can accidentally edit something, but at the same time the directly visible "debug surface" is bigger the same way. Support for debugging is of higher value to me than the danger to do a wrong click. The linear block display style of the new editor matches more with the XML format in the Phyphox file than the 2d display of the old editor.

The linear (vertical) sequence of the blocks gives a better idea on the sequence of the commands within an experiment execution cycle. In the old editor I needed to edit the modules horizontal position (pixels) in the XML file to exactly control that execution order (horizontal position from left to right). Taking away the 2d design time freedom helped in better understanding / controlling the runtime behavior. My experiment still seems to be able to use a container ahead of its (value) definition, which is a key feature for me to split and build up buffers as I need them for calculations and for export. If the container has no value defined (in my experiment) in cycle 1 there is no negative impact of this missing value and in cycle 2 the calculations are apparently working fine by accessing the container value computed towards the end of cycle 1. This is special, I believe, with Phyphox, you have got the linear execution sequence of the commands establishing your algorithm plus typically about ~200 (assumption) more of the same cycles before the next GPS data set arrives at the typical frequency of 1 Hz.

What I find a little irritating in fluently reading the coding in the new editor are the two different block styles I noticed. For some math blocks the output container is being displayed at the top left area of the block, whereas in other blocks the output container is being shown at the bottom to the right. While reading through the coding the eyes do some slalom. Don't drink and code, don't get dizzy. Maybe the output channels could get another color scheme than the input channels. This could help nicely in concentrating on the data flow at the Analysis tab in the new editor, which could compensate a little for the no longer available "data flow connection lines" from the old editor. But don't become too fancy with colors, color blind people should still be able to notice some differences (e.g. in brightness). Individual containers should not get their own colors, which my approx. 190 containers I would not be able to notice the differences. Highlighting the output channels with some light orange background color would be perfect for me personally in guiding the eyes through the data flow, just to give an idea.

In the old editor it was possible to select multiple modules at once and duplicate them, have not seen this in the new editor, where you apparently need to duplicate individually one block after the other. Actions on multiple blocks at the same time happen not too often and would contradict somehow the snap-in feature of the new editor. Moving around modules in the old editor was easier, in the new one it is necessary to first break the connection between blocks before rearranging them and connect them again. I believe abandoned modules in the old editor made it into the XML file, while abandoned blocks in the new one won't. Due to the 2d layout in was possible in the old editor to see more coding at one screen, but for large projects and a 30% zoom the actual benefit was limited.

I have not measured it but I estimate that I invested something between 120 and 170 old editor hours in my experiment. The migration to the new editor, reviewing and optimizing the experiment took me about additional 20 new editor hours. The old XML file has about 1500 lines, the optimized new XML file has now about 1400 lines (and approx. Analysis 1030 blocks). The performance and code complexity optimization would have been much more difficult in the old editor than they were with the new one. I'm quite happy that the development invest one year ago was not lost (context to explain the invest: the experiment is recording GPS data and derives certain values from this data, one experiment execution saves me about 6 to 8 hours manual work in OpenOffice). What I avoided like hell was to click on the big trash bins in the new editor's UI. For me, deleting everything would be not the main feature of an editor. If I ever should accidentally click on these icons I hope that the cool undo would save me. ... Ok, I was too curious and I clicked and I was surprised. It does not delete everything? But what is it? The log of the undo buffer?

Next step for me is now to set up a real physical experiment in which I have certain input parameters like distances and velocity under direct control. This way I can calculate myself the expected output of the experiment and I should be able to see if my Phyphox experiment would compute the exact same values as the physical theory. But this is beyond the new editor, which helped in a great way to get there. Thanks!!! Goodbye old editor's module connection lines, hello new editor's development efficiency.

[Jens Noriʇzsɔɥ]

Thanks for the extensive testing. There‘s quite a lot to check and perhaps a bit to fix…

Some first comments:
– The file format is one dimensional, so it is more consistent with the new editor. In the old one, only the horizontal direction is relevant. Just in the rare situations where (rather accidentally) the horizontal positions coincide, the verticality comes into play.
– Default values should not ever change (just out of my head, this never happened…?). For breaking changes the file format version should save you: phyphox is backwards compatible here. We definitely should verify, that the new editor handles these well.
– If you observe something that looks like an import error in the new editor, it would be great if you could (privately) share the file(s) (by mail).

Reading your report, I should have a look at my issue list again. Some topics you raised look familiar…

[Trikes]

I have not noticed a possibility to attach a file to a PN in the forum. I do not want yet to attach the XML file to a posting because the experiment is still work in progress. Later on I plan to have a dedicated thread for that experiment. For the time being I share the related XML code to illustrate the issue with the if block I mentioned before:

Final experiment export from old editor:

Code:

<phyphox xmlns="http://phyphox.org/xml" xmlns:editor="http://phyphox.org/editor/xml" version="1.7" editor:version="1.0" locale="en">

        <if editor:uuid="1544" editor:posx="1460" editor:posy="1780" less="false" equal="false" greater="true">
            <input as="a" clear="false">lat_count</input>
            <input as="b" type="value">2</input>
            <input as="true" clear="false">lat(-2)_append</input>
            <input as="false" type="empty" />
            <output as="result">lat(-2)_if_3+_fixes</output>
        </if>

Then import to new editor and export from new editor without any manual changes in-between:

Code:

<phyphox xmlns="https://phyphox.org/xml" version="1.18">

    <if equal="true">
      <input as="a" keep="true">lat_count</input>
      <input type="value" as="b">2</input>
      <input as="true" keep="true">lat(-2)_append</input>
      <input type="empty" as="false"></input>
      <output>lat(-2)_if_3+_fixes</output>
    </if>

Notice how greater="true" turned into equal="true".

---

And another update on what I mentioned before: it is about accidentally pressing the back button on my mouse, which happened in the meanwhile two more times in front of my screen (I hit this button accidentally with my hand while moving the keyboard around on my desk and colliding with the mouse). Unfortunately there is no software / driver from the manufactuarer to disable these mouse buttons. But is it really necessary to use another mouse without such extra buttons? No, it is not, as I found out only today. Because it is possible to navigate again into the editor. But reloading the experiment again into the editor takes 10 long seconds which is close to forever if you are in fear that you lost your unsaved changes. When this happened to me the first time I was not patient enough to wait this time, I also did not new back then that just navigating back and waiting would solve this problem. The editor, the experiment, and the eternal Internet connection, which never is fast enough. An end user's facepalm experience. I would nevertheless vote for a safety popup when navigating away from the editor. A popup that the old editor was providing and which was indicating to me that I would lose my changes in case I was about to exit the whole browser while having 15+ tabs open.

---

Another delay experience when apparently communicating with the server. But I'm not sure what happened in this case, because I do not know which activities in the new editor are local activities being executed only in my browser on my device or which actually require communication with a server (maybe to generate the QR code). Anyways, a week back maybe and only one time, close to German midnight, there was a freeze situation in the editor in my browser. For about 30 to 60 seconds, I don't remember exactly, there was no reaction by the editor on my inputs, and, of course, I cannot tell what was the actual reason for this freeze, my computer?, the Internet connection?, the server? During the freeze my user input still was being recorded and apparently was applied after the freeze was over, leading to several changes to blocks, which I was not noticing, because I was not expecting such a thing and because after the freeze another section of the Analysis tab was displayed than before (there was some vertical scrolling applied, apparently). It took serval days and some experiment runs to identify these unwanted changes and to correct them again. Would be good to know at which actions in the editor some server call is involved and which other actions are being executed only locally. Is there at all a server side session while I locally edit my experiment? Are there scheduled server restarts introducing downtimes during which I should not use the editor? Can I have my experiment opened in the browser for weeks without losing any possible connection to the server? Or should I save my experiment and close the browser tab each time I turn off my computer?

[Jens Noriʇzsɔɥ]

Final experiment export from old editor:

Code:

<phyphox xmlns="http://phyphox.org/xml" xmlns:editor="http://phyphox.org/editor/xml" version="1.7" editor:version="1.0" locale="en">

        <if editor:uuid="1544" editor:posx="1460" editor:posy="1780" less="false" equal="false" greater="true">
            <input as="a" clear="false">lat_count</input>
            <input as="b" type="value">2</input>
            <input as="true" clear="false">lat(-2)_append</input>
            <input as="false" type="empty" />
            <output as="result">lat(-2)_if_3+_fixes</output>
        </if>

Then import to new editor and export from new editor without any manual changes in-between:

Code:

<phyphox xmlns="https://phyphox.org/xml" version="1.18">

    <if equal="true">
      <input as="a" keep="true">lat_count</input>
      <input type="value" as="b">2</input>
      <input as="true" keep="true">lat(-2)_append</input>
      <input type="empty" as="false"></input>
      <output>lat(-2)_if_3+_fixes</output>
    </if>

Notice how greater="true" turned into equal="true".

I have put a test file together and verified the bug. We have it already locally corrected and the fix should be included in the next release. Thanks for reporting: a really good find (and nasty bug)…

[Jens Noriʇzsɔɥ]

On on- vs. offline: the data is kept in local storage, https://developer.mozilla.org/en-US/docs...calStorage, and is quite save there. You should only lose data if you touch the “reset” button (and if there is an error with local storage some weird things might happen: I need to dig deeper into this case).

You could work entirely offline unless
– you generate an QR code (this requires a file on our server),
– you touch the “reset” button (the editor is entirely reloaded),
– you load default experiments (more or less obviously).

Does this answer your questions in Saturday's post?

[Trikes]

Yes, thanks for explaining how the new editor works in the local browser. If I would encounter again performance bottlenecks while editing, then I will know that it is not a bad Internet connection nor some server thingy at phyphox.org, it will be a problem on my local computer (while having 2 browser instances with 30 tabs open at the same time and the OS is trying to apply some eco mode stuff and the browser itself wants to unload some tabs ... too many smart players involved). This understanding helps in more quickly finding the root cause of such probelms and avoiding frustration because of not knowing what's really going on.

Speaking of browsers ...

• I believe it is not possible to load the same XML file again to the editor. After loading an older experiment I continued to work on it, but then noticed that I would go coding-wise in the wrong direction and I decided to start over (whithout saving my changes) by just loading the same XML file again. The editor was not coming back from the load popup und I needed to close that popup to continue with the editor. While loading the same file again was not working, it is perfectly working to load another file (name) into the editor and overwriting this way the unsaved changes. Maybe there could be some kind of warning when loading a new file while there are unsaved changes to the current experiment in the editor. Maybe there is already such a check for unsaved changes, but only in case the same file name should be loaded again and this freezes the load popup. Ah, I used Edge on Windows with the file selection popup ... but with drag and drop it is the same behavior. My workaround for loading the same file again was:
  
• To skip any changes to the currently loaded experiment and to load the same XML file again I just delete the whole browser cache and reload the editor from its source at https://phyphox.org/editor/neweditor/ ... and because I'm already anticipating your comment on my workaround I was just trying out for the very first time the reset button in the editor. Works nicely, no workaround needed any longer. (But the reset does not reset the old file name at the load popup, which still is being displayed despite the fact that after the reset no experiment is being loaded. Ok, the file name displayed at this popup is not the currently loaded file, but it is just a history record to show the file that has been loaded most recently independent from if it is still being loaded or not.)
  
• To compare two experiments I was using until now two different browsers, Firefox and Edge, in the assumption that their caches would not have any impact on each other. Would it be possible to also load two experiments into the same browser instance? Even if I load in Edge the new editor via "Start the editor" at https://phyphox.org/editor I get the same experiment displayed I already have opened in the other Edge tab. I see only a difference at the load popup: at the old tab the loaded file name is being displayed whereas at the load popup in the new tab I see "No file selected" (or whatever the correct translation into English language would be, I get the text displayed in German language). When I start editing the experiment in one tab the changes are not visible immediately at the other tab, but they become visible at the other tab when I reload the other tab. This indicates that it is not possible to work on two different experiments in the same browser. When loading a new experiment to a second tab and then reload the first tab I get the newly loaded experiment also in the first tab. But the same is not true for the displayed loaded file name at the load popup, indicating that there are possibly some tab specific data stored in the cache whereas other data is shared between tabs. For static experiment comparison it might work using the same browser (without any tab reloads), but when additionally changing one experiment at the same time, no, this would be too risky for me and I would use two different browser brands that don't do cache sharing. Maybe by introducing some tab specific keys at the chaches it would be easily possible to work independently on two experiments in the same browser.
  

I started to edit directly the XML files in a simple text editor and, of course, introduced some error (example: forgot to define a container that was used in the analysis). When loading that invalid Phyphox file to the new editor, there was apparently no error message, only thing I noticed was that the corresponding editor tab was just empty (in this case the Analysis tab). Apparently the editor tab understood that something was wrong, because it was not displaying anything. I just was puzzled because of the empty screen. Some kind of error message would have helped. Took a while to figure out the root cause.

I was wondering about the many empty lines in my experiment's multiline description until I found the pattern behind this: when saving and reloading an experiment the description was getting longer and longer. Since this is about white spaces, kindly select the code examples to see what I want to show: with each load/save cycle an additional line with 2 spaces and a return seems to be introduced (not sure if you can notice the 2 space characters when displaying the code examples in this post, but in the forum editor the extra lines look for me this way:
"  
" = <space><space><return>).

Save experiment with a simple multiline description "Test<return key on your keyboard, whatever actual character this would introduce, newline, linefeed, or carriage return, have not checked>" (in this example the second line of the multiline description is just an empty line).

Code:

  <description>Test

  </description>

Load the saved experiment again (notice new number in file name), you get then:

Code:

  <description>Test

 
  </description>

Repeat this cycle and you get:

Code:

  <description>Test

 
 
  </description>

Same behavior also when the second line is not empty but comes with some text (my original case which I don't show here).
Has not happened for a single line description like this:

Code:

<description>Test</description>