Next steps with MWorks and Python

Support
1

Hi Chris,

I’m doing some refactoring of our python experiment code now, so I’m thinking a bit more about MWorks and Python.

It’s now been a number of years since I sent the below, and you’ve implemented many of these. 1 and 2 below have turned out to be very helpful for us.

I have one small and one larger point to ask about.

Small: did you ever get to my third bullet below – a Python conduit function that returns the current state of a variable? Right now I register a callback for every variable and record all of their values, in Python. That would probably be better done in C++ on the Python side of the conduit (as I assume the client does with the stream of events from the server). Do you agree?

Large: Before I dig into Python over the next few days, I want to consider my lab’s next steps with MWorks and Python over the next 1-3 years. I see three options.

  1. Keep slogging along along with the Python bridge/conduit.
  2. Put our Python code inside MWServer (using the feature you implemented to let MWServer run Python code).
  3. Think about ways to run experiments, and define and call MWorks objects/actions from Python.

Details, pros/cons:

  1. Keep slogging along along with the Python bridge/conduit.
    Pro: Minimal engineering work for me up front
    Con: Latency between Client and Python is long (50 ms with a sometimes longer tail). Lots of variable callbacks, putting load on the single-threaded Py interpreter. (fixable, see small pt above). Syncing XML and Python is cumbersome and errorprone.
    Still have to deal with non-Python syntax and parser, which slows down our development.

  2. Put our Python code inside MWServer (using the feature you implemented to let MWServer run Python code).
    Pro: Hopefully Python call latency is much shorter (but needs testing). Variable callback processing load goes away.
    Con: Engineering up front for me. Need to figure out system python packages, install and sync across machines. Might Python need to be called at parse time? If so need to work with Chris. Have to debug timing - will all Python dispatch and simple statements be as fast as the XML?
    Still have to deal with non-Python syntax in XML and parser.

  3. Think about ways to run expts and call MWorks objects from Python
    Con: Engineering up front for me. Chris and Jim don’t really want to do this. Even if they agree, takes potentially a lot of Chris’s time (my lab can deal with a few months delay, though). Python syntax not a great fit for state machine. Have to debug timing - will all Python dispatch and simple statements be as fast as the XML?
    Pro: Can probably use anaconda, which is newer and easier to sync across machines. Can use Python syntax to do conditionals, variable assignment, loops, subfunctions, numpy – makes our experiments much more maintainable and readable. Latency issues go away (but timing issues same as (2)).

Any comments? Happy to talk. Arash if you’d like to be part of the convo please let us know.

Mark

2

Hi Mark,

did you ever get to my third bullet below – a Python conduit function that returns the current state of a variable? Right now I register a callback for every variable and record all of their values, in Python. That would probably be better done in C++ on the Python side of the conduit (as I assume the client does with the stream of events from the server). Do you agree?

No, I haven’t addressed this yet. I still agree that it would be useful.

That said, I’ve begun to think we should expose the core Client class in Python. That would give you everything you have now with the conduits, plus variable-value caching, plus full control to load/start/stop/unload/etc. the experiment. For client-side Python scripts, it would also eliminate the intermediate hop through MWClient, since the script could communicate directly with MWServer.

Before I dig into Python over the next few days, I want to consider my lab’s next steps with MWorks and Python over the next 1-3 years. I see three options.

As a first step, I think it’s worth reviewing what you’re currently using Python for and whether some or all of it can now be handled directly by MWorks.

I know you don’t want to use MWEditor or edit XML by hand. I fully empathize and agree with that position, which is why I’ve been working hard on an alternative experiment format/DSL. (I know I’ve mentioned this before, and I swear I’m going to share details about it very soon!)

But beyond that, what do you currently do in Python that you can’t or don’t want to do directly in your experiment XML? As I recall, the lack of array and dictionary support was an issue for you, but I’ve addressed many of MWorks’ previous limitations in that area. Are you using Python to communicate with I/O devices? Something else?

Chris

3

Hi Chris,

You’re asking about why a DSL wouldn’t be sufficient, and why I’m asking
for the experiment to be specified in python rather than a DSL/XML. First,
let me say I propose Python would replace the XML fully transparently from
a client/server UI perspective. So there would be a button to “Load
experiment”, mw.report() in python would display on the server console, and
events generated by mw.setVariable() would get passed to event streams just
as currently. No plotting would be allowed from the experiment.

But beyond that, what do you currently do in Python that you can’t or don’t
want to do directly in your experiment XML? As I recall, the lack of array
and dictionary support was an issue for you, but I’ve addressed many of
MWorks’ previous limitations in that area. Are you using Python to
communicate with I/O devices? Something else?

I looked back over some of our past emails about this, and the issues
python would now improve over a DSL are:

  • subroutines (i.e. code to send a set of digital events, that needs to be
    called from multiple states, where flexibility is required that can’t be
    done in C++)

  • sharing code across experiments: Python modules do this already. You and
    I would need to work together to implement and test includes in the DSL.
    That requires some hard thinking about scoping. Already solved and
    documented in Python.

  • local variables. Our experiments have variables like “debugMs” “debugUs”
    “tempUs” all over the place to put values into reports. Python already has
    a way to declare such variables in local scope. MWorks may do it too, but
    it would take some time for me to ask you about that and possibly for you
    to change it.

  • report actions. Limited in MWorks, but the Python string processor is
    sophisticated, and well documented and could be used to flexibly specify
    strings for reporting.

  • operators. You and I spent time discussing truncating integer division
    in MWorks and the change in identity of the ‘/’ operator in MW 0.6. That
    operation for all types is fully specified in Python, and that would have
    saved us time over discussing and reimplementing in MWorks.

  • More generally, object types and syntax are fully specified in Python. I
    see a lot of pain for you and also myself fixing all the corner cases of a
    DSL syntax, when this work has already been done for extant languages. You
    and I have spent time on, from the best of my memory:

  • if/else

  • variable literals and how their parsing depends on the “type” attribute

  • operators, including / and // (I would also prefer to use bitshift
    operators but it’s kind of not worth my time now to work with you on that
    since I have workaround)

  • list/dict access (in the future we might want .append(); already
    specified and tested in Python)

  • variable types and casting

  • trial selection / selection variables which I finally moved to Python
    because it’s just more flexible and fits our needs

Some of the above are solved, but I think switching to Python would prevent
dealing with these kind of corner cases in the future and ultimately save
us time. For example, in the future I could see wanting for loops (esp.
useful for arrays/dicts) or transitions that depend on both timers and a
condition (currently worked around with ‘&& timestamp >= now()’ in our
transitions. I guess I just see a DSL or future XML as a long treadmill of
continuing work by you and me to work out syntax that is now solved by
Python.

What do you see as the major downsides of specifying experiments in
Python? I know you mentioned state machine syntax. Agree it would be
unwieldy, but the upsides outweigh this. And I noticed recently the
package “transitions” on github has solved some of these problems.

best,
Mark

4

Hi Chris and Arash,

Arash and I spoke recently about the future of MWorks (and I recently
chatted with John Maunsell as well).
I’m planning to get in touch with Jim about charting our labs’ plans with
MWorks.
Chris, are you ok with me forwarding this conversation along?

thanks,
Mark

5

Hi Mark,

I’m happy to open this discussion up to anyone who’s interested. I’ve made the thread public, so you can just share the URL.

However, I would appreciate it if, before making any decisions or jumping to conclusions, you’d give me a chance to present my position more clearly and to respond directly to your arguments. I’m working on some examples to demonstrate why I think the DSL I’ve developed provides a better user experience than Python-based experiments. I will try to finish those and get back to you early next week.

In the meantime, I’ll just make a few quick points:

  1. The DSL I keep mentioning isn’t just an idea. It’s done – as in, I’m currently writing full experiments in it and running them via MWClient and MWServer. The implementation is still in a private branch, but only because I have a few loose ends to tie up before I perform integration testing and move it in to the nightly build.

    Of course, by “done”, I don’t mean that I’m not open to feedback, or that I’m unwilling to make changes to it. I hope for lots of feedback, and I’m sure things will change. My point is simply that we’re not talking about a hypothetical DSL. It’s a real thing, with a real, working implementation.

    Also, from here on out I’ll refer to it by its name, MWEL, which is an acronym for “MWorks Experiment Language”.

  2. When run, MWEL experiments are first converted into MWorks XML and are then parsed and executed like any other experiment. This has significant advantages over a Python-based approach. Specifically, (1) it doesn’t introduce a new, independent codepath for implementing experiments that needs to be developed and maintained, and (2) it ensures that XML-based experiments (by which I mean every MWorks experiment ever written until now) remain fully supported, first-class citizens in the MWorks ecosystem.

  3. The first two items in your list of Python’s advantages (subroutines and sharing code across experiments) are addressed by MWEL. Specifically, it supports function-like macros and “include” statements. Again, I will try to provide examples very soon.

Cheers,
Chris

6

Hi Chris,

Yes, I think this will be a conversation that will take a bit of time
before we come to any conclusion one way or the other. I appreciate your
expertise in implementing MWorks and I’m sensitive to your concerns about
maintainability and how much effort any Python changes might be. But since
we’re talking about a proposal that’s reasonably long-term, I thought it
would make sense to bring others who use MWorks into the conversation. I
suspect that I have relatively specific needs (perhaps also shared by
Lindsey) since we deploy MWorks code across more than a dozen training
rigs, and thus we perhaps need more automation than most. So I think it’s
worth me hearing more about other users’ needs and what you see as the best
path forward. I do think that the Python proposal is largely independent
of the DSL/XML - sounds like MWEL is close to done, and we would need to
depend on backwards-compat of a lot of old XML code.

More next week.

thanks,
Mark

7

Hi Mark,

Here are the promised examples.

To produce these, I started with an existing, XML-based MWorks experiment (analog_io.xml). This is actually a test of MWorks’ NIDAQ interface, which gets run as part of the nightly test suite. The details of what it does aren’t important, but note that it includes

  1. an I/O device with channels;
  2. a stimulus that updates dynamically, based on input from the I/O device; and
  3. a task system with multiple states.

Hence, it’s short but somewhat sophisticated.

I then rewrote the example in both Python (analog_io.py) and MWEL (analog_io.mwel). The Python version is my best guess at what a Python-based MWorks experiment would look like. You may have other ideas about how it should look; if so, I’d like to hear about them. The MWEL version is working code; I can load and run it just like the XML version, and it performs identically.

In the Python code, I’ve included comments pointing out various issues I see. The MWEL version has none of these issues.

Please have a look and let me know what you think.

Chris

8

Attachments:

9

Hi Chris,

I think the MWEL seems like a big step forward over XML. I don’t have any major comments, though I guess I’d prefer ‘transition’ over ‘goto’ in some cases.

To move forward the discussion of having a parallel Python approach to defining experiments (supported along with the XML/MWEL), I wanted to also provide an example of what a ‘MWEPy’ experiment might look like.
I put together something that can serve as a start for discussion (attached).

I’d think the next step (no rush) would be for you to make comments on the python syntax and then perhaps we can discuss.

-M

10

Attachment: analog_io.py (3.29 KB)

11

Hi Mark,

A few updates:

  • MWEL support is now in the nightly build
  • I’ve published an in-progress snapshot of the MWorks 0.8 docs, which include an MWEL reference

I’ll share some comments on your MWorks/Python example soon.

Cheers,
Chris

12

Hi Mark,

I’ve attached a copy of your “MWPL” example with some comments added (all prefaced with “CJS”). Some additional questions:

  • How would you implement an experiment like this, where the position of the stimulus is determined by expressions that are evaluated every time it’s drawn? Would you just write those expressions as strings in Python?

  • How do you imagine stopping or pausing an experiment inside a state? Stopping might be implemented by the MWorks’ action wrappers throwing an exception if the experiment is stopped. However, I don’t know how we’d pause and restart a Python function in the middle, short of delving deep into the guts of the Python interpreter and adding support for it, somehow.

  • Is this something you might someday want to use on iOS? If so, we’d need to bundle Python with MWorks on that platform and deal with whatever porting issues come up. (MWEL neatly avoids this by converting to XML on the client side.)

I think most of the work in implementing support for Python-based experiments is in exposing the basic elements of experiment construction to Python. Specifically, I mean providing the means to create, connect, and control components via Python. The biggest issue is that most of the details of experiment construction are handled in the XML parser, and breaking them out of there would need to be handled with great care (and may require some substantial rethinking/reworking of MWorks’ internals). Another issue, which I noted in my Python mockup, is that the “guts” of actions aren’t really accessible as simple functions, so they’d need to be reworked.

Once that work is done, then it should be relatively straightforward to implement MWPL on top of it. And it could be useful for other things, too. For example, the MWEL processor could construct the experiment directly, instead of converting to XML and handing it off to the XML parser.

I don’t know how good a time estimate I can put on this work – maybe a couple months? However, mucking with the XML parser is definitely dangerous territory, with high potential for things getting busted in subtle ways. Again, great care (and much testing) would be needed.

Chris

13

Attachment: analog_io_cjs_comments.py (4.77 KB)

14

Hi Chris,

I revised the Python a bit based on your comments. New proposal is attached.
Notable:

  • I proposed a new way to keep track of MW variables and stimuli inside their own namespace. Initialization is now just creating a Python object and assigning to a field a namespace/mapping object. Is something like this possible?
  • Shared python data across state system States, like local vars (not MW vars) can live in the top-level namespace.
  • I’d propose using the python Exception/Warning/Logging mechanism and displaying those messages in the Server console.
    (Perhaps: embed a Python interpreter. import user script inside a try: except, and call a particular method to run the experiment.)

Additional comment:
I now minimally use the “live update” mechanism, in part because it’s been unclear to us when various MW elements read
the changed values. I could have gone through with you or with the code, but we haven’t done this.
As I understand, the usualy way JS/Python deal with ‘live updates’ is invoking callbacks at particular well-defined times.
So an alternative to the live update mechanism would be to add a callback to each stimulus:

def cbFrameRect(stim, curr_ms, time_to_next_frame_ms): # or whatever other vars are useful; called each frame
stim.set_position(x_position=curr_ms*10, y_position=cursor_y)
mw.stim.rect = mw.stimulus.Rectangle(x_size=cursor_size, y_size=cursor_size,
x_position=cursor_x, y_position=cursor_y, frame_callback=cbFrameRect)

Responses to your questions:

Stopping/pausing: We don’t use pause now, but I understand the desire to implement it if others use it. I guess I don’t fully understand what people use it for. Stopping by throwing an exception sounds good. The question here is how to preserve state across stop/start as the naive thing is to reload the script on each start.

iOS: we haven’t even looked at iOS, and as we discussed Apple seems to be signalling they care at least a little about MacOS and Mac hardware. Again I can appreciate other users may want this.

Once that work is done, then it should be relatively straightforward to implement MWPL on top of it. And it could be useful for other things, too. For example, the MWEL processor could construct the experiment directly, instead of converting to XML and handing it off to the XML parser.

I don’t know how good a time estimate I can put on this work – maybe a couple months? However, mucking with the XML parser is definitely dangerous territory, with high potential for things getting busted in subtle ways. Again, great care (and much testing) would be needed.

It seems like these changes could have other benefits, but I take your point that this is a major decision. Let’s see if we can agree about the syntax details before committing to anything.

My guess is the live-update / callback question is a bit too big for email only and best via phone - perhaps sometime next week?

thanks,
Mark

15

Attachment: analog_io_mh2.py (7.85 KB)

16

Hi Chris,

• How would you implement an experiment like this, where the position of the stimulus is determined by expressions that are evaluated every time it’s drawn? Would you just write those expressions as strings in Python?

I’ve thought a bit more about how to update stimulus attributes during an experiment. My last was a little discombobulated.

We’re talking about this type of definition from your python example, I believe:

target = mw.stimulus.fixation_point(‘target’,
y_size = target_size,
x_position = ‘-amplitude’)

amplitude is a mworks variable

I see a few ways to handle this. I agree eval’ing strings is not a great approach.
First, passing a callable as a parameter value. That callable would be called anytime the value is needed. (Pros and cons at bottom of email):

target = mw.stimulus.fixation_point(‘target’,
y_size = 10,
x_position = lambda: -mw.vars.amplitude)

3 lines

mw.vars is the variable dictionary exposed by the mworks module

Second, using a callback function passed to the stimulus object:

def fn(self, frame_ms):
self.x_position = -mw.vars.amplitude
target = mw.stimulus.fixation_point(‘target’,
y_size = 10,
x_position = None,
frame_callback = fn)

6 lines

Third, a class that inherits from the stimulus class to override a callback method:

class xMovingFp (mw.Stimulus.FixationPoint):
def frame_fn(self, frame_ms):
self.x_position = -mw.vars.amplitude
mw.stim.target = xMovingFp( # uses my preferred stim def syntax, assigning instance to mw.stim attribute
y_size = 10,
x_position = None)

6 lines

scoping of mw.vars should be fine, I think.

Pro: easy to document the various callbacks that can change instance attributes

Pros for 2, 3

  • (vs 1 and also passing a mw.Variable and letting the stimulus read it when needed): makes the update time where param values are read explicit.
  • (vs 1): Fewer function calls, as the callable parameter needs to be called anytime the value is used; the callback is called at defined times, like once a frame.
    Cons for 2, 3:
  • code is more verbose
    Cons for all:
  • perhaps requires fundamental changes in mworks?

Of the three I think I prefer #3, as I think it’s least surprising for python users, and would be easy to document the different functions: “All stimulus types have a frame_fn and a trial_start_fn method that can be used to…”

Also, I believe with this approach (any of 1,2,3) we can drop Variable.value and just say that a mw.Variable always evaluates to a numeric value (float/int?) when evaluated in a numeric context. And when MW classes (stimuli, variables, etc.) are instantiated, they evaluate any Variables. Then, the only way to change stimulus attributes is explicit — a callable.

I think this is a slight mismatch with the current MWorks architecture. But I’m not sure it’s a big mismatch. With MWorks today, the stimulus doesn’t actually use the new variable value till redraw, so the real-world action is temporally decoupled from the variable change that generates an event. If we log (or throw off an event) on redraw, I think we capture the same conceptual functionality as today.

Comments?

Mark