Hi Chris,
So trying to get around this, I’m still not able to find the image filename or its hash.
Below is a piece of code where I try to filter an “stimulus_presneted” event (code 71) and get its associated ‘#stimDisplayUpdate’ (code 7) which should reflect the data I’m looking for.
Kindly advise.
Here is the mwk2 file just in case.
Thank you!
Guy
Hi Guy,
To extract all the events for stimulus_presented and #stimDisplayUpdate:
with MWKFile(filename) as fp:
sp_events = fp.get_events(codes=['stimulus_presented'])
sdu_events = fp.get_events(codes=['#stimDisplayUpdate'])
You now have two sequences of Event objects, each of which has properties code, time, and data:
>>> evt = sp_events[0]
>>> evt.code, evt.time, evt.data
(71, 3375706433, -1)
You’ll need to use the time property to associate stimulus_presented and #stimDisplayUpdate events. Note that the times will not be equal. Instead, for a stimulus_presented event with time t, you probably want the #stimDisplayUpdate event whose time is closest to t. You might want to talk with Yoon or someone else with experience analyzing MWorks event files, as they may have better suggestions.
Once you find the #stimDisplayUpdate events you want, you can get the image filename and hash from the data property:
>>> evt = sdu_events[100]
>>> img_params = [d for d in evt.data if d.get('type') == 'image']
>>> for info in img_params:
... print(info['filename'])
... print(info['file_hash'])
...
/var/folders/_6/d6xsx0bj6vggxrmhsg_zxvrh0000gn/T/MWorks/Experiment Cache/_Users_rig1_Desktop_Ani_RSVP-normalizers-v3_normalizersV3.mwel/tmp/images/NormalizerV3_15.png
2e94f82e721a32eab22f10bdbd78eba3fd4052e1
Chris
Helpful, thanks!
you probably want the #stimDisplayUpdate event whose time is closest to t .
Can I trust that it would robustly be just the following (preceding) #stimDisplayUpdate event?
Guy
Can I trust that it would robustly be just the following (preceding)
#stimDisplayUpdateevent?
No, not really. Stimulus display updates happen on their own thread, on their own schedule. If the assignment to stimulus_presented happens before the update_stimulus_display (aka update_display) action, then the corresponding #stimDisplayUpdate event will definitely happen later. However, if the assignment happens after update_display, then the #stimDisplayUpdate event will still almost certainly be later.
If you have a way to convert a stimulus_presented value in to an image filename, then you could find the corresponding SDU event by matching the filename (or rather, matching the end of the filename). For example, if a stimulus_presented value of 15 corresponds to file NormalizerV3_15.png, then that could be a simpler way to find the SDU events you want.
Chris
Wouldn’t this approach amount to fooling ourselves?
I could just as well implement such a check using the stimulus_presented value alone on the user end.
If we cannot make a robust association between stimulus_presented and the hash then there is no add to implementing hash based checks at all, no?
Wouldn’t this approach amount to fooling ourselves?
I could just as well implement such a check using the stimulus_presented value alone on the user end.
Yes, of course you’re right. Dumb suggestion on my part.
But why are you trying to associate #stimDisplayUpdate with stimulus_presented? If you want determine which images were presented and when, all that information is recorded, robustly, in #stimDisplayUpdate alone. Why look at stimulus_presented at all?
Chris
Looking at our preprocessing code, it appears that the stimulus_presented was traditionally used as the field to lock onto. Isn’t the #stimDisplayUpdate just a GUI event whose timing may not accurately reflect the actual stimulus presentation for neural recordings processing purposes? And that the stimulus_presented is the precise backend event for those purposes?
You have that exactly backward 
.
The time stamp on a #stimDisplayUpdate event is the most accurate estimate MWorks has of when the physical display actually updates. It has to be an estimate, because the actual update happens after the SDU event is generated. For more info, see Understanding Display Updates in the MWorks manual.
On the other hand, stimulus_presented gets set when the experiment author chooses to set it. Probably the experiment will set it immediately before or after the update_display action, but as the docs note, that is not when the display update takes place. Hence, the time stamp on stimulus_presented events is going to be arbitrary and unreliable for determining when a stimulus presentation actually happens.
If you want to know when the display really does update (as opposed to when it’s predicted to update), you have to measure it physically with a photodiode. The photodiode signal would typically be fed in to the neural recording system. In the MWK2 file you shared with me, I see that the #stimDisplayUpdate events include a stimulus named “photodiode_image”, which is presumably positioned under the photodiode on the display. I don’t know if the photodiode signal gets used in the data analysis pipeline, but I assume it’s stored and available somewhere.
Cheers,
Chris
Also, in case it wasn’t clear already, I have never seen any of the DiCarlo Lab’s data processing code, so I’m only making guesses about what it does. My job is developing and maintaining MWorks, so I’m just trying to help you understand what MWorks does. That’s why I recommended checking with Yoon or someone else who is familiar with the lab’s post-MWorks data processing.
Chris
Thank you!
Photodiode signal is used and is part of the preprocessing pipeline. I’m not certain if in a way that remedies inaccuracies of time reports in stimulus_presented or that this error propagates and corrupts our PSTHs (btw do you have order of magnitude of the time mismatch between stimulus_presented and #stimDisplayUpdate?).
In any case, am I safe to completely revise the pipeline such that stimulus_presented is discarded and just using #stimDisplayUpdate time reports and info instead?
Photodiode signal is used and is part of the preprocessing pipeline. I’m not certain if in a way that remedies inaccuracies of time reports in
stimulus_presentedor that this error propagates and corrupts our PSTHs
I assume it’s used to get accurate stimulus-presentation times, but I don’t know.
btw do you have order of magnitude of the time mismatch between
stimulus_presentedand#stimDisplayUpdate?
If stimulus_presented is set immediately after update_display, then the actual display update should normally happen within one display refresh cycle (so about 17ms for a 60Hz display). It may actually happen several refresh cycles later (probably a maximum of 3), if the graphics hardware is well ahead in the drawing loop.
In any case, am I safe to completely revise the pipeline such that
stimulus_presentedis discarded and just using#stimDisplayUpdatetime reports and info instead?
I’d say so, unless you want to hold on to stimulus_presented just for the additional redundancy.
Chris
Hi Guy,
I took a look at the code you shared (spike_tools/mwk_ultra.py).
It looks like the assignment of times to each stimulus presentation is handled well. Both stimulus_presented_df['time'] and correct_fixation_df['time'] are set to the value of photodiode_on. This variable contains the times (on the Intan clock) of the peaks in the photodiode signal, which is the most robust measure of actual stimulus presentation time.
Also, most of the usage of stimulus_presented is fine, since it’s just determining a general ordering of stimulus presentations (i.e. the order of a given stimulus in its trial) and isn’t concerned with specific stimulus identity or the real-world time of the presentation.
To get the relevant #stimDisplayUpdate (SDU) events for specific stimulus identification, you need to extract events where an image was presented. Here’s some Python code to do this:
with MWKFile(filename) as fp:
sdu_events = []
file_paths = []
file_hashes = []
for e in fp.get_events_iter(codes=['#stimDisplayUpdate']):
for d in e.data:
if d.get('type') == 'image':
sdu_events.append(e)
file_paths.append(d['filename'])
file_hashes.append(d['file_hash'])
break
The number of SDU events extracted this way should equal the number of stimulus_presented events whose value is not -1. More specifically, borrowing some code from the section “Extract stimulus presentation order and fixation information”:
stimulus_presented_df = data[data.name == 'stimulus_presented'].reset_index(drop=True)
correct_fixation_df = data[data.name == 'correct_fixation'].reset_index(drop=True)
stimulus_presented_df = stimulus_presented_df[:len(correct_fixation_df)] # If you have one extra stimulus event but not fixation, use this
assert len(stimulus_presented_df) == len(correct_fixation_df)
# Drop `empty` data (i.e. -1) before the experiment actually began and after it had already ended
correct_fixation_df = correct_fixation_df[stimulus_presented_df.data != -1].reset_index(drop=True)
stimulus_presented_df = stimulus_presented_df[stimulus_presented_df.data != -1].reset_index(drop=True)
sdu_events = sdu_events[:len(correct_fixation_df)]
assert len(sdu_events) == len(stimulus_presented_df)
One area where I think it would be better to use SDU instead of stimulus_presented is in the “Get eye data” section. Since the point there is to find the eye data coincident with the stimulus presentation, SDU’s more accurate timestamp makes it the better choice.
Finally, in the “Save output” section, I think it would be easiest to save file_paths and file_hashes, rather than the raw SDU events. There’s really no reason to save stimulus_presented, except perhaps as a redundant means of image identification.
I hope that helps!
Cheers,
Chris
Very helpful Chris! Thanks for looking into this.
sdu_df as you suggested as a substitute of the stimulus_presented_df everywhere in the code seamlessly?0).Guy
Hi Guy,
So I can just create an
sdu_dfas you suggested as a substitute of thestimulus_presented_dfeverywhere in the code seamlessly?
No, I would keep using stimulus_presented_df where it’s currently used (except in the “Get eye data” section, as I mentioned). For lining things up with other MWorks variables (e.g. trial_start_line, correct_fixation), it’s a better choice than SDU. What stimulus_presented is not good for is establishing stimulus identity (use SDU for that) or stimulus presentation time (use the photodiode peaks for that).
But certainly do extract the file paths and file hashes from SDU and include those in the output.
Chris
OK, great. I think I have successfully implemented this.
Just for clarification, what do you mean by “For lining things up with other MWorks variables (e.g. trial_start_line , correct_fixation ), it’s a better choice than SDU”. If I got this correctly, once timing is in any case overridden by photodiode data, then the SDU and stimulus_presented whose value is not -1 series of events are completely equivalent and I am safe to use any one of them arbitrarily (maybe up to eye data as you suggested), isn’t it so? This held up in a specific .mwk2 analysis I made.
Just for clarification, what do you mean by “For lining things up with other MWorks variables (e.g.
trial_start_line,correct_fixation), it’s a better choice than SDU” .
The issue here is one I mentioned previously: Stimulus display updates (and their associated SDU events) happen on a separate thread from the main experiment logic. This means that you don’t know precisely when the SDU event associated with a particular display update is going to occur relative to variable assignments performed in the experiment code.
For example, suppose you have four MWorks variables, a, b, c, and d, and your experiment includes the following code:
a = 1
b = 2
update_display ()
c = 3
d = 4
Because the four variable assignments are made on the same thread (the one executing the experiment logic), you know that, in the event file, the events corresponding to those assignments are going to appear in exactly the order given in the MWEL code (1, 2, 3, 4). However, although the display update is triggered between the assignments to b and c, the corresponding SDU event is almost certainly not going to fall between those assignments in the event file; more likely, it will happen after all four assignments. If you’re using event order to try to associate SDU events with other events in the trial – say, looking for the SDU event that lies between a particular pair of b and c events – you’re going to get the wrong answer.
Although what I said above is true in general, it may also be true that, in the case of mwk_ultra.py and the experiments it’s used with, using SDU instead of stimulus_presented would work correctly. Looking through the code, I think that may be the case. However, I’m not 100% sure, which is why I recommended mostly keeping things as they are.
Chris
Amazing! Thank you for the detailed walk through!
Guy