Hey Chris,
so I’m trying to replace our importing function that brings mwk variables in and saves them to a mat file using Python. Much of the process was actually pretty easy (we can load/save mat files in Python with scipy, much of the matlab code is very similar to Python format), however I’m hung up on saving the data because the f_getevents brings the data in as a funky data structure. When I bring the import the mwk data into a variable with f.get_events(code…) it will create an ‘eventwrapper’ that looks like a dictionary within a dictionary, called ‘<mworks._mworks.EventWrapper at 0x7faa26d2bed0>’ If I put that into X, I can go to X[0].data, X[1].data … and get information, but if I try to save X to a mat file (scipi.io.savemat) it is empty.
If you have any sample Python scripts for importing mwk data, perhaps that would be the easiest way to see what works for handling these even wrappers.
I've continued looking into this and I'm wondering if the confusion is based on the new mwk2 file formatting (as I read in the Current Event File Format (MWK2)) section of the manual. I was expecting a nested dictionary format similar to numpy, however this is in SQLite (which I'm less familiar with). I'm wondering if it's this new formatting that is making the porting of the output of the f_getevents then importing to a dictionary for the .mat file to such funky formatting.
I'll look up how to handle SQlite data, and hopefully there's a simple way to move it to numpy (then .mat via scipi).
Hi Travis,
MWKFile’s get_events method returns a sequence of event objects. The event objects (and, prior to MWorks 0.10, the sequence itself) are a custom type, so scipi.io.savemat isn’t going to know what to do with them. In any case, the docs for savemat say that you need to provide a dictionary mapping names to variable values, so passing in any sort of non-mapping type isn’t going to work.
The closest I can get to successfully using savemat is this function:
def events_to_mwk(infile, outfile):
with MWKFile(infile) as fp:
evts = [{'event_code': e.code, 'time_us': e.time, 'data': e.data}
for e in fp.get_events_iter()
if e.data is not None]
scipy.io.savemat(outfile, {'events': evts}, long_field_names=True)
However, it isn’t fully successful, because savemat seems unable to save Python dictionaries that can’t be converted to MATLAB struct’s. In particular, MWorks codec events can’t be saved, as they map integers to dictionaries, and integers aren’t valid struct keys in MATLAB. (MWorks’ own MATLAB data tools will convert dictionary values that can’t be MATLAB structs into Map instances.)
So, I don’t think your desired approach is going to work. But if you’re looking to store MWorks events in MAT files, why wouldn’t you just use MATLAB to do so? And if you want your events in a format that can be read in both Python and MATLAB, why not just stick with MWK2? If you can explain what you’re trying to accomplish, I may be able to suggest a more fruitful approach.
Cheers,
Chris
Hi Chris,
Yeah sorry I didn’t give much context before. We’d like to start to move
away from Matlab to Python, Mworks looked like a good place to start since
it supports both; So I started rewriting our importing function (which
currently uses Matlab) to Python, but this is going to be a transition, so
I’d love to make a temporary bridge so we can still use all of our Matlab
functions (redundancy always seems to be the best failsafe 
Your method might be good enough for the data we extract so I’ll give that
a try. I was trying to avoid bringing the mworks code into Python, then
rebuilding the dictionaries from scratch before porting it over with scipi
- but that might be the inevitable next step. One trick that might come up
is there are some System dictionaries that are nested, like
#stimDisplayUpdate.
To answer your question: why not just keep it in mwk2 - From what I’ve seen
so far it’s kind of cumbersome to extract data from these files because it
seems to be in serialization. I’m used to extracting data straight from
databases into arrays, not serialized data formats so it’s a less intuitive
process. I’m not a computer science major so perhaps there’s a gap in my
knowledge that would make this more intuitive that I just haven’t come
across In fact the MWK2 being in SQlite now might be my savior, but I’ve
got a queue of SQlite python youtube tutorials now to watch haha.
Travis
Hi Chris,
Your code was very helpful, I think I’m getting close! I’m going to include
my python code I’m working on to help give an idea of the issue, but I
think it’s a resolvable issue!
In the code I create a list of codecs I want to extract from the mwk2 file
(‘called list). I then create a codecs array that includes all codecs in
the file with codecs = f.codec. I scan through the codecs list and if a
member equals a member in my list, I try to extract it into a variable. if
I try extracting the data like the manual says ‘data =
f.get_events(codes=[codecs[i]])’ then the data is in a weird wrapper,
however if I do it like in your code it saves it as a list of dictionaries;
’ data = [{‘event_code’: e.code, ‘time_us’: e.time, ‘data’: e.data} for e
in f.get_events_iter() if e.code == i]’
it’s a bit clunky because the iter will run through each event so I’m sure
there’s a more efficient way to code this up, but I’d be fine with a
working code. What I’m trying to do now is to automate the variable ‘data’
to equal the event code by evaluating a string, where the variable changes,
name = [codecs[i]]
st_out = str(name[0] + “”" = [{‘event_code’: e.code, ‘time_us’: e.time,
‘data’: e.data} for e in f.get_events_iter() if e.code == i]“”")
then just eval(st_out). It’s not working because of a syntax error
(pointing to the equal sign) but I’ll work on figuring this out. I just
figured providing this info would give you a better idea of how I’m trying
to extract the variable info from the mwk2 and port it. My thinking is I’ll
create a list for each event code, then just save all of them with scipy
and then it should be in the same format for matlab.
T
Attachment: import.py (1.63 KB)
Hi Chris,
Success! Sort of…
Ok, figured some stuff out. Using your code I can extract the Events in the
appropriate way! The: sort of… scipy doesn’t allow appending more than
one dictionary to a present dictionary, Yikes! So at this point I think I
have the solution, but I’ll just have to come up with a change that we’re
going to use, maybe I’ll just store all of our Event lists as one nested
dictionary that In Matlab we can just extract to test whether it will work
with our current code. I thought I’d upload the current code, which runs
(with an mwk2 file called velvet) but the saved mat file is
registered corrupt in Matlab, which many people online have had the exact
some problem with this appending method. So it’s sort of a fix, but unless
you see an obvious solution, you can close the case. Thanks for your help!
T
Attachment: import.py (1.51 KB)
Hi Travis,
Try this:
data = {}
with MWKFile('velvet.mwk2') as f:
rc = f.reverse_codec
for name in getEventList():
code = rc[name]
if name.startswith('#'):
name = name[1:]
data[name] = [{'event_code': e.code, 'time_us': e.time, 'data': e.data}
for e in f.get_events_iter(codes=[code])]
io.savemat('velvet_mwk.mat', data)
I haven’t run that code, so there may be typos, but it should do what you want and get everything in one file.
Cheers,
Chris
Hi Chris,
Your intuition is spot on! I just tried that yesterday and it worked, so
scipy doesn’t save the dictionary as one nested dictionary, it unwraps it
saving it as all of the internal dictionaries, which is perfect!
It’s a bit slow so it’d be nice not to have to iterate with f.get_events_iter
I wonder why the 'f.get_events(codec) spits out something different;
because it’d be more efficient to use that I’d think.
Matlab can bring it in and it’s all whereas using f.get_events
saves some things as so that’s the last thing I’m working on and
then we’ll have a Python Matlab Bridge woo hoo!!
T
Hi Chris,
I’m making a ton of headway on this, I’m able to bring in the mwk2 file using the MWK2Raeader class, and moving, saving the data in a numpy array, then saving into a matlab file! Better yet, it’s OS independent, which clears out an old post I had on here years ago, woo hoo!
one quick question - using the MWK2Reader, is there a way to get the dictionary of all the codes and their corresponding variables? In the old way, a call to the codec gave this list, ‘f.codec’ and I assume the sqlite table should have this information somewhere in the mwk2 file.
Hi Travis,
using the MWK2Reader, is there a way to get the dictionary of all the codes and their corresponding variables?
Yes. See this post.
Better yet, it’s OS independent, which clears out an old post I had on here years ago, woo hoo!
Yeah, MWK2Reader is proving to be popular, mostly for that reason. I’m planning to make something like it part of MWorks’ official data tools. Not sure when I’ll get to it, but hopefully it’ll be relatively soon.
Cheers,
Chris