Hi Chris,
I’m working on the code now that will test/rely on array editing in the variables window.
I see that list initializers use the same syntax as selection variables, with attribute:
default_value=“0,1:4”
etc.
Since variables now autodetect the types of constant initializers (float,int, etc.), one suggestion is that you may want to enforce a list syntax: e.g. default_value=“[0,1:4]”
Then next time someone asks you for dicts, you won’t have to do something fancy with colon autodetection to know whether
“0:5” means “[0:5]” or “{0:5}”. And parsing back and forth into strings may also be easier.
That’s just a suggestion – you know more about the consequences than me.
Mark
Hi Mark,
Are you still thinking about the JSON-string solution for #1 below?
- Support for displaying and editing lists and dicts in the Client. We talked about turning each into a JSON string representation, and on edit parsing the edited string back from JSON. That would be fine for us - once lists get to have more than 10 elements or so we can edit XML variables files.
Upon further reflection, I think that using JSON in the variables window might be a mistake, as it would introduce a new expression syntax that would be similar to but not quite the same as the one used in MWorks experiments. Instead, I think the variables window should use exactly the same expression parser (i.e. STX) as experiments. Here’s how I see this working:
-
A string entered in the variables window gets run through the normal MWorks expression parser and evaluated. If the string is a comma-separated list of expressions, then the value of the variable becomes a list containing the values of the expressions; otherwise, the variable gets a single, non-list value.
-
If the parsing or evaluation of the string fails, then the value of the variable becomes a string containing the text that was entered. This preserves the existing behavior of the variables window, e.g. entering the text “foo” causes the variable to be assigned the string value "foo".
-
To make it evident when a variable has a string value, the variables window will display such values with enclosing quotes. That way, if you want to enter “1.2” but instead type “1…2”, the variables window will display the value as "1..2", making it clear that you set a string value, not a number.
One neat feature of this is that, since your string is processed by the expression parser, you can enter an expression and have it evaluated, e.g. entering “1+2” will set the corresponding variable to 3. That’s something you wouldn’t get if we parsed the input as JSON.
On the other hand, a drawback of this design is that there’s no way to create a list with 0 or 1 elements, though I’m not sure how often you’d want to do that in practice. Also, we haven’t addressed the issue of how to display and edit dicts.
Regarding this:
one suggestion is that you may want to enforce a list syntax: e.g. default_value=“[0,1:4]”. Then next time someone asks you for dicts, you won’t have to do something fancy with colon autodetection to know whether “0:5” means “[0:5]” or “{0:5}”.
I see your point, but I think that if and when “{0:5}” describes a dictionary, it’ll be fine for “0:5” to still mean a list (and be equivalent to “[0:5]”).
Anyway, I have the expression-parser-backed variables window implemented on a branch, and I’m pretty happy with how it works. What are your thoughts on this approach?
Cheers,
Chris
Anyway, I have the expression-parser-backed variables window implemented on a branch, and I’m pretty happy with how it works. What are your thoughts on this approach?
I’m fine with this approach.
However, I think that to specify a list, a bracket syntax is likely to be nicer than using comma auto-detection. One corner case is the one you pointed out:
On the other hand, a drawback of this design is that there’s no way to create a list with 0 or 1 elements, though I’m not sure how often you’d want to do that in practice. Also, we haven’t addressed the issue of how to display and edit dicts.
We would use 1-element lists and would want to set them in the variables window. Our main use-case for a list is to specify the number of trials in a block, split out by condition, so that they can be selected from without replacement. We will frequently want to use just one condition or trial type, for example in early training before the subject is performing the full task.
I’d vote for adding a bracket syntax for list literals to the STX parser which would fix this. The simplest case is to allow “[1,2,3]” to be the same as “1,2,3”. Another possibility is, like python, to evaluate “1,” to a one-element list. Another is the full Matlab list syntax so the following is all the same list:
[1,2,3]
[1:2 3]
1:3
I realize that making the parser more complicated is not ideal, so let me know what you think is best.
Mark
Hi Mark,
After giving this stuff yet more thought, I’ve decided it’s time to bite the bullet and add proper support for list (and, eventually, dictionary) literals to the expression parser.
The biggest obstacle here is that stx::AnyScalar doesn’t support any aggregate data types. Rather than extending it to do so, I decided that the best approach would be to replace all usage of AnyScalar with mw::Datum, which (obviously) does support lists and dicts. This turned out to be a relatively small job: I now have a branch in which AnyScalar has been completed eliminated, and the expression parser uses Datum exclusively.
At this point, I’m working on improving the error reporting in various Datum methods and expanding its unit test suite. Basically, I want to ensure that Datum is up to the task of its new duties.
Once that’s done, I’ll add the actual list-parsing support to the expression parser. That should be quite straightforward, as most of the required pieces are already there. After that (although maybe not immediately), I can tackle dict support.
My plan for the variables window remains mostly unchanged. The only difference now is that lists won’t be inferred from the presence of commas. Instead, they’ll need to be explicitly delimited by square brackets. This eliminates the problem with recognizing 0- and 1-element lists.
I’m hoping to get this stuff done soon – ideally within the next couple weeks. I’ll let you know when it’s available in the nightly.
Cheers,
Chris
Chris, this sounds great. Thank you!
Mark
Hi Mark,
Expression parser support for list literals is now implemented and in the current nightly build. Some examples:
[] # Empty list
[1] # Single-element list
[1, 2.0, 'three'] # Multiple-element list
[1:10] # Range expression list (includes both endpoints)
[1,2,3:6,7,8:10] # Same as above
In addition, the following changes are also in the current nightly:
- Variables can now have type “any”, in which case the default_value string is passed directly to the expression parser, which evaluates it like any other expression, and the result is assigned to the variable as is, without any type casting. (In the editor, new variables now have type “any” by default.)
- Division expressions now always produce a floating-point result
- Boolean values can now participate in all arithmetic operations (they’re simply promoted to integers)
- The variables window now uses MWorks’ expression parser to parse input values (meaning it supports list literals, too)
- The variables window now displays string values in enclosing quotes
Enjoy!
Chris
This is a big advance for MWorks. The changes provide largely complete support for arrays.
Hi Mark,
More progress on this front: The expression parser now supports dictionary literals, too, using Python-like syntax, e.g. {'a': 1, 2: 'b'}. In addition,
-
The variables window now displays and parses dictionary values using this syntax.
-
The expression parser now supports “generalized” subscripting operations. Specifically, the subscripting target is no longer limited to being a variable name, and subscripting operations can be chained. For example, [1,2,[3,[4,5]]][2][1][0] evaluates to 4, as you’d expect.
One casualty of the latter change is selection variable subscripting, which is no longer supported. However, there’s a new function, selection(), that replaces it. Where previously you would write sel_var[2] to get the third non-accepted selection on sel_var, now you’d write selection("sel_var", 2).
This is all in the currently nightly build.
Cheers,
Chris
Chris, this is a big advance for us, with dicts and the ability to access 2-d lists. Thanks. I expect to be testing this in the next few weeks.
Mark
Chris,
These additions are really great- we had no real work around for editing
lists and dicts online so this all enables new functionality for us.
I think that cleaning up the type handling and the indexing and initializer
syntax is also a big step forward which will reduce bugs- thanks.
The only big item remaining I can think of now is the ability to call
python code and block on its return. We have a semi-functional workaround
for this now, but native functionality would reduce our XML complexity a
lot.
Happy 2015,
Mark
Hi Mark,
As of the current nightly build, the final piece of list and dictionary support is implemented: It’s now possible to assign to indices/keys in list/dictionary-valued variables. To enable this, the variable parameter of an assignment action can now include one or more square-bracket-enclosed index/key expressions after the variable name, e.g.
<action type="assignment" tag="Set x[i][j]" variable="x[i][j]" value="0"></action>
The index/key expressions can be arbitrarily complex. Like other expressions, they are evaluated both at load time (in an attempt to catch obvious errors) and every time the action executes. However, the actual assignment is done only at run time.
Examples: list, dictionary
Cheers,
Chris
Excellent! Thank you.
Full support for arrays is a huge plus for us.
Mark