Parse time variables and 'task' definition file

[cstawarz]

Hi Chris,

One more issue. I want to break out the parse-time variable discussion as a lead in to ‘task’ definition files. I know you talked about customizing a new type of selection variable, but I can see parse-time values being useful even outside that - e.g. values/selection fields of trials and blocks and range replicators.

• One option to specify parse-time state is to include parse-time “variables” by using a C-preprocessor-style approach, where lexical substitution is done from another file at parse time. My biggest issue with this approach is I want this to be able to choose different variables per-subject, while not editing the XML file. I could write a script that copies the desired subject’s parameters in place before each experiment load, however.

• Another is to bring the parse-time vars into the variable values XML - the problem is you’d need to work out some UI to specify that file before the experiment is loaded, as it is now required to be loaded afterward.

• I’m most positive about the ‘task definition’ file we discussed. It would be great if the client had a facility to load

0. desired server connection
1. experiment XML
2. python script
3. matlab script, plus variable definitions
4. variable values
5. parse-time variable values
6. Plugin windows/server console opened and their position and state (scroll to bottom; which trees expanded in variables window)

via a single ‘subject/task’ definition file. I’d be happy if this was via a command-line parameter, as I can script it. Then you don’t have to commit to any UI until we’re happy with it (although a menu item to load it, with remembered recent files, would also be great).

No rush on any of this but we should think about it.

Mark

[cstawarz]

Hi Mark,

One option to specify parse-time state is to include parse-time “variables” by using a C-preprocessor-style approach, where lexical substitution is done from another file at parse time. My biggest issue with this approach is I want this to be able to choose different variables per-subject, while not editing the XML file.

I think I have a good solution to this, in two parts.

First, I’ve added support for XInclude directives in MWorks experiment files. The upshot of this is that you can now reference arbitrary XML documents (or portions thereof) in your experiment XML, and the parser will suck them in and process them as if they had been copied and pasted in to the file. The substitution is handled by libxml2 before experiment construction begins, so there aren’t any restrictions on how or where you use XIncude’s, as long as the end result is a valid MWorks experiment; this opens up a variety of code-reuse possibilities. XInclude support is live in the current nightly build.

Second, although it’s not obvious, the variable_assignment directives that you see in setup_variables.xml and saved variable files can also be included in experiment files. When constructing an experiment, the MWorks XML parser first creates all experiment-defined variables, then processes variable_assignment’s, and after that creates all other components (including, as of recent nightlies, selection variables). This means that the result of variable_assignment’s are visible to other components at construction time.

Used together, these features provide a way to get your “parse-time variables”. For a demonstration, have a look at my XInclude tests. The file BaseExp.xml is a normal MWorks experiment containing a selection variable; it makes ten selections on the variable and prints out the value each time. The selection variable’s “values” parameter is an expression that uses another variable in the experiment (“num_values”) . The default value of “num_values” is 10, so if you run BaseExp directly, it will print out the numbers 1-10.

However, there are two additional experiments, FiveValues.xml and TwoValues.xml, which include BaseExp and use variable_assignment to change the value of “num_values” to 5 and 2, respectively. When run, FiveValues prints the numbers 1-5 twice, and TwoValues prints 1-2 five times.

Using this technique, you can create a shared base experiment, and then create a separate “wrapper” experiment for each subject that includes the base experiment and sets the appropriate variable values. (Note that this may eliminate the need to save or load variable sets via MWClient, which would simplify your experiment start-up routine.)

Like I said, this all works with the current nightly build, so feel free to try it out and let me know what you think.

Cheers,
Chris

[cstawarz]

Thanks for this, Chris. The include behavior is nice. And it’s good to know the order in which the parser does things at experiment load.

I’ll have to think this through to see how we can use it. What comes to mind immediately is that different experiments will have their variable sets saved to unique directories, which is probably just fine. Since we use the client UI to save and load variable sets while subjects are running, I think we’ll have to load the same variable set that is getting included, or we’d have to use two files, one for the “parse-time” and one for the dynamic variables.
Let me play around with this to see what the best solution is.

Mar

[cstawarz]

[Follow-up email from Mark]

Hi Chris,

After thinking about your email a bit, it occurs to me that by creating a few more variables we could solve some of my issues in a clean way.

The goal is to be able to start an experimental/training session with minimal client UI interaction, both to avoid errors (user failing to load/save variables files, start Python bridge, set Matlab events/script name) and to reduce time since we run 16-20 subjects per day.

One way to do this is to add a few internal variables set by the client, e.g.:

• #experimentXmlFilename
• #clientVariablesFilename
• #pythonClientScriptFilename
• #matlabWindowScriptFilename
• #mWorksServerName
• etc.

And have the client set them whenever the corresponding UI elements are set, and if set at startup time, do the corresponding load/connect (likely in a defined order).

Then, you would just need to add one commandline switch to the client that loads one XML file on startup (after setup_variables.xml). I would put a bunch of variable_definitions in that file to set the variables above, and the client when done with startup could do the corresponding actions. And the rest of the MWorks code would have access to these, and they would be saved in the data files.

Perhaps this “everything-is-a-variable” approach is too complex, but it seems to be a powerful way to solve this problem.

Mark

[cstawarz]

[Follow-up email from Chris]

Hi Mark,

This seems like a promising approach. Probably it’d be best to add just one variable (e.g. “#clientState”) whose value is a dictionary, just to avoid an explosion of system variables. Let me play around with it and see where it gets us.

Chris

[cstawarz]

[Follow-up email from Mark]

Sounds great.

One comment: I would be completely happy if this dictionary variable could be set only at experiment load time.

For example you said previously:

When constructing an experiment, the MWorks XML parser first creates all experiment-defined variables, then processes variable_assignment’s, (***), and after that creates all other components (including, as of recent nightlies, selection variables). This means that the result ofvariable_assignment’s are visible to other components at construction time.

So for example, I think I’d be happy if the client read the dictionary only at the time marked (***) above. Of course, if you want to make it more dynamic, I won’t complain. (Though it seems like this would be moving towards “scripting MWClient” using variables - if the dictionary was read dynamically, I’d expect that changing the “python bridge script” value would cause the Python bridge to terminate the old script and load a new one, updating the UI, and I can see you might only want to do this at experiment load time.

Mark

[cstawarz]

Hi Mark,

Just a quick update:

After experimenting with the #clientState variable, I have serious doubts about whether it’s a better/cleaner/easier solution than the “task definition” file.

From a client plugin’s perspective, the two approaches seem more or less equivalent. In both cases, the plugin would be watching an event/object for relevant state changes (and probably writing UI-initiated state changes back to said event/object). Thus, the implementation effort required for the two approaches seems comparable.

However, from the user’s perspective, the #clientState approach is probably going to be convoluted and awkward. The most obvious place to put #clientState settings would be in a “wrapper” XML file that includes (via XInclude directives) a base experiment. However, I see two big issues with this:

1. The syntax of variable_assignment directives is verbose and non-obvious, which makes writing them by hand difficult and error prone.
2. It’s not clear how #clientState settings could be updated via the client’s UI. I don’t think you’d want MWClient modifying your experiment XML file by adding/altering variable_assignment’s. However, you can’t add #clientState to a regular saved variables file (since the former should be loading the latter). That means #clientState would have to be written out to a separate file, at which point we’ve basically implemented the task-definition-file approach.

Given all that, my current feeling is that the task definition file probably is the way to go. At moment, I’m still considering how to implement it so that (1) it’s easy for client plugins to read/write the associated “task state” and (2) users can easily read and edit it manually (e.g. in a text editor).

I’d welcome your thoughts on any of this.

Cheers,
Chris

[cstawarz]

Hi Chris,

Thanks for looking into this, what you say makes sense. I’d be happy with the ‘task definition’ file.

A few thoughts:

• From my perspective, I don’t need the client to be able to write to the task definition file - I think I’d be creating it completely by hand in a text editor. It might be good to think about this for future expandability, but it doesn’t need to be in the first pass, I think.
• I’d plan on having one of the task def files for each subject. It would XInclude the variables file for that subject, to allow for range replicator parameters to be specified in a single variables XML file; that should work for experiment XML parsing, correct?
• The list of things I’d like to specify is below – I’m listing only the things I’d use, so you don’t put a lot of effort on something I wouldn’t. I’m assuming you’d implement related parameters if they make sense.
  • Server to connect to
  • Plugins open: Matlab, Python Client bridge, variables window (window position is preserved separately)
  • Matlab state: sync variable, Events list, matlab script name (I currently change the Mac MWClient defaults values to do this, I could keep doing it that way)
  • Python window state: script name (again I could continue to use the defaults system), plus an action to load the script.
  • Plugin variables window state: would be nice to expand a specified tree, but not necessary
  • Experiment to load (I imagine this would achieve parse-time variables by including the subject-specific variable definition XML)
  • Variable definition XML name (Needed so changes will get saved back to it).

Thanks - this would definitely make things easier for us. Let me know if you have questions.
Mark

[cstawarz]

Hi Mark,

I have a first-pass implementation of the task definition file (TDF) that’s probably ready for some testing and feedback.

I decided to use JSON as the file format. Its primary advantages are that it’s widely used and well supported, while also being easy for humans to read and write. In addition, JSON is a proper subset of YAML 1.2, which gives us a straightforward, backwards-compatible upgrade path to YAML if needed.

Here’s an example TDF (based on this test experiment), which should be self explanatory:

{
  "serverURL" : "127.0.0.1",
  "serverPort" : 19989,
  "experimentPath" : "/Users/cstawarz/Downloads/testcase-histed-130728/test.xml",
  "variableSetName" : "myvars",
  "openPluginWindows" : [
    "MWorksMATLABWindow.bundle",
    "MWorksVariablesWindow.bundle",
    "PythonBridgePlugin.bundle"
  ],
  "pluginState" : {
    "MWorksMATLABWindow.bundle" : {
      "filePath" : "/Users/cstawarz/Downloads/testcase-histed-130728/test.m",
      "syncEventName" : "sync",
      "selectedVariables" : [
        "sync"
      ],
    },
    "PythonBridgePlugin.bundle" : {
      "scriptPath" : "/Users/cstawarz/Downloads/testcase-histed-130728/test.py"
    }
  },
}

Currently, a TDF can specify all the items you requested, except for variable-window expansion state. (I tried to implement save/restore for the variable tree some time ago, and the solution proved elusive; I’ll take another crack at it soon.) In addition, MWClient’s “File” menu now has “Load Task” and “Save Task” items for loading and creating TDF’s.

These changes are not yet in the nightly build, and I’m not comfortable with pushing them in until they’ve had more testing. Whenever you’re ready to try it out, I’ll build a custom installer that includes the TDF stuff for you.

Cheers,
Chris

[cstawarz]

Hi Chris,

Sounds great. This will be much more robust than my custom Mac OS X-defaults-poker-scripts.
I’m on board with JSON, far better than XML for hand-editing.
I can probably test this in the next 2-3 weeks. One request before then - can you allow tilde-expansion in the paths? i.e. I want to use “~/Downloads/testcase-histed-130728/test.xml” so that I can keep one TDF under version control per experiment, and usable by multiple users.

Thanks,
Mark

[cstawarz]

Hi Mark,

A few updates:

• Variable window expansion state is now saved and restored, via both TDFs and MWClient’s preferences (support for the latter is in the current nightly build)

• Per your request, TDFs now support tilde expansion in paths. More usefully, perhaps, they also support specifying paths relative to the directory in which the TDF is located.

• MWClient’s “Open Recent” menu now lets you view and load recently-opened TDFs

Here’s an updated example TDF, demonstrating the new features (assuming the TDF is in the same directory as test.xml, etc.):

{
  "serverURL" : "127.0.0.1",
  "serverPort" : 19989,
  "experimentPath" : "test.xml",
  "variableSetName" : "myvars",
  "openPluginWindows" : [
    "MWorksMATLABWindow.bundle",
    "MWorksVariablesWindow.bundle",
    "PythonBridgePlugin.bundle",
  ],
  "pluginState" : {
    "MWorksMATLABWindow.bundle" : {
      "filePath" : "test.m",
      "syncEventName" : "sync",
      "selectedVariables" : [
        "sync",
      ],
    },
    "MWorksVariablesWindow.bundle" : {
      "expandedGroups" : [
        "# ALL VARIABLES",
      ]
    },
    "PythonBridgePlugin.bundle" : {
      "scriptPath" : "test.py",
    },
  },
}

I can probably test this in the next 2-3 weeks.

Great! Let me know when you’re ready, and I’ll do a build for you.

Cheers,
Chris

[cstawarz]

Hi Mark,

I’m interested in testing the TDFs over the next month or so. Want to spin up a build?

Done. The installer is here.

Chris

[cstawarz]

Chris,

I did some testing. Seems to work fine and is a good solution for us.
Whenever it makes it into the nightly, will you let me know?

Thanks,
Mark

[cstawarz]

Hi Mark,

Thanks for the testing. I’ll probably push this stuff in to the nightly shortly after the new release goes out (which I’m hoping will be next week). I’ll let you know when it’s there.

Cheers,
Chris

[cstawarz]

Hi Mark,

The “task definition file” stuff is now in the nightly build, with a few changes:

• Jim suggested changing the name, since “task” already has a specific meaning to MWorks users. The new name is “workspace”, and the relevant MWClient menu items are “Open Workspace” and “Save Workspace”.
• The “Open Workspace” and “Save Workspace” menus items are now assigned the standard “open” and “save” keyboard shortcuts (⌘O and ⌘S, respectively)
• Workspace files are now saved with a “.json” extension
• In workspace files, plugins are now named without the “.bundle” extension. (As a side effect of this change, the first time you run the new nightly, the positions and sizes of plugin windows will be forgotten, but they’ll be remembered again for subsequent sessions; it’s a one-time loss.)
• The list of open plugin windows in now always saved in the workspace file, regardless of MWClient’s “restore open plugin windows” preference

I’m planning to add the ability to set system variables via the workspace file. A DiCarlo Lab user suggested this, and it seems like a good idea. If you have any other suggestions, please let me know.

Cheers,
Chris

[cstawarz]

Setting system variables in the workspace files is a great idea.
Thanks for this!