Making movies out of tasks

Support
1

Hi Mahdi,

Support for capturing stimulus display frames is now in the MWorks nightly build.

You can enable frame capture by including a Stimulus Display device in your experiment and setting its capture_format parameter to a supported value. For example:

var capture_enabled = false

stimulus_display (
    capture_format = PNG
    capture_height_pixels = 400  // Set to zero or omit for full display resolution
    capture_enabled = capture_enabled
    )

In general, PNG is probably the best format, as it uses lossless compression.

If the parameter capture_height_pixels is omitted or set to zero, frames are captured at the full resolution of the display. Otherwise, frames are scaled down to the specified height, with the width chosen to match the display’s aspect ratio.

By default, every display update will be captured. However, if you set the parameter capture_enabled to the name of a variable, then frames will be captured only when the variable’s value is true/1. This lets you capture only the frames you care about (e.g. while a trial is active).

Each captured frame is assigned to the system variable #stimDisplayCapture. The value of the variable is binary data (bytes in Python, uint8 array in MATLAB). The data is an image file in the chosen format (e.g. PNG). If you write the data to a file, you should be able to open it in Preview or any standard image viewer. It’s possible to preview the captured images while your experiment is running by using MWClient’s Image Viewer window with “Image data variable” set to #stimDisplayCapture. (Note that the image viewer wasn’t designed to be updated every frame and can be quite laggy in this role. I will attempt to make it more performant in the future.)

Once you’ve run your experiment and have an event file full of captured frames, the next step is to extract and combine them into a video, with added eye traces and other desired adornments. I will work on an example of this and try to get it to you soon. You use MATLAB, right? If so, the VideoWriter class looks like a useful tool for this job, so I’ll probably try that first.

If you have any questions, please let me know!

Cheers,
Chris

2

Hi Mahdi,

Some updates on the task movie stuff:

First, I made a few improvements to the stimulus display frame capture mechanism itself. MWorks now limits the number of “in flight” frames, i.e. frames that have been captured but are still in the process of being converted to image files and written to the event file. This prevents MWServer from running out of memory due to an ever-growing backlog of in-flight frames. Also, I eliminated a potential bottleneck in writing the captured frames to the event file. These changes are in the current nightly build.

Second, I now have an example of how to extract the captured frames from the event file and turn them in to a movie using MATLAB. As I suggested previously, the example uses VideoWriter to create the movie. Basically, for each frame of the video, it loads and displays the appropriate stimulus display frame, renders any additional data/info using standard MATLAB plotting commands, and writes the completed figure to the video.

The attached ZIP file includes an example experiment and a MATLAB script for creating movies from event files generated by the experiment. The experiment is a very lightly-modified version of the standard RSVPDemo experiment. The movie-creation script adds the elapsed time (as text) and an eye trace over each stimulus display frame. The ZIP file also includes an example movie generated by the script.

Using the example script as a starting point, you should be able to generate movies using any data available in your event file. Since you create each frame using standard MATLAB commands, you can make use of all your existing MATLAB plotting skills (which I’m sure are much greater than mine).

I hope this is helpful. If you have any questions or need more info, please let me know!

Cheers,
Chris

3

Attachment: task_movie.zip (1.78 MB)

4

Hi Chris!

This is great, thank you so much!! I will take a look and try to run the
code!

I will share this with all of Jazlab, I am sure they will find this very
helpful!

Mahdi