Guidance on Integrating VR Task and Camera Setup with XBI

Support
1

Hi Chris,

My name is Simona Vaitekunaite and I am setting up an XBI device in Fribourg, Switzerland in collaboration with Michael Schmid’s group.

We have three main goals:

  • Developing new tasks in mworks including running Unity-based Virtual Reality tasks
  • Set-up video-recordings of arm movements (side cameras)
  • Set-up a face recognition procedure for animal identification (front camera)

At this stage we managed to have Unity talking to the XBI touchscreen and reward system and we are brainstorming the type of cameras we should be using.

We have three specific questions:

  • How can we best integrate (if possible at all) our VR task into mworks?
  • In humans we use Basler cameras for arm tracking (model acA1920-155uc)—do you think they’d be suitable for mworks & XBI?
  • Do you have a list of recommended Mac-compatible cameras for the face-recognition procedure? We have seen a XIMEA interface described in the MWorks documentation (XIMEA Camera Device — MWorks 0.14.dev documentation), but we are aware that any Mac-compatible camera could work (as you noted previously – Integrating video camera recording into MWorks - #9 by cstawarz). Could you recommend specific camera models, whether XIMEA or otherwise, compatible with mworks & XBI and suitable for reliable real-time recording?

I’d be happy to schedule a meeting to discuss this further.

Thanks in advance for your advice!

Sincerely,
Simona

2

Hi Simona,

How can we best integrate (if possible at all) our VR task into mworks?

I’m not sure exactly what kind of integration you have in mind.

Does the application running the VR task expose some kind of API or network interface for external control? If so, you could probably use MWorks’ Python actions to communicate with and/or control it.

If you want to use MWorks’ fixation points or EyeLink interface with the VR task, you could use a transparent stimulus display window. You can enable this by going to MWServer → Preferences → Display → Advanced and unchecking “Make stimulus display window opaque”. Then, in your experiment, include a stimulus display device whose background color components and background alpha multiplier are all set to zero. This will make MWorks’ stimulus display window 100% transparent, so anything beneath it (such as the VR task window) will be visible.

In humans we use Basler cameras for arm tracking (model acA1920-155uc)—do you think they’d be suitable for mworks & XBI?

To be usable with MWorks’ face recognizer, a camera must be recognized by macOS and usable via the standard system API’s. I suspect the Basler cameras are only usable via their own software, but I don’t know. Have you used them with macOS in the past?

At present, MWorks doesn’t support video recording at all. If that’s what you need, I would need to implement a new I/O device plugin for it. This shouldn’t be too hard, but it will take some time.

For questions specific to XBI, I recommend contacting Ralf Brockhausen at DPZ. I have never worked with it and know very little about it.

Could you recommend specific camera models, whether XIMEA or otherwise, compatible with mworks & XBI and suitable for reliable real-time recording?

I don’t have any specific recommendations. As I said in one of the discussions you referenced, any camera that claims to be UVC compliant and/or work on macOS without drivers should be fine for the face recognizer. This list probably has some good options.

I think MWork’s XIMEA interface has only even been tested with one camera. (Sorry, I don’t remember the model, although I know it was an infrared camera.) But it should work with any camera supported by xiAPI. At the moment, it captures only 8-bit, grayscale images, but it should be straightforward to add support for other image formats.

Cheers,
Chris

3

Thank you, Chris, for your prompt and detailed response. We will review the various sections shortly.

Regarding your first question, yes, Unity does run a network interface for external control. We will explore the Python actions you suggested to see if they are compatible.

While this approach might work for ‘simple’ behavioral recordings in the XBI, we are concerned that it might lack the necessary control and precision when we transition to ephys recordings or microstimulation.

We have been brainstorming an idea similar to your transparent stimulus display window, but our requirements are a bit more complex than just simple fixation.

Here is our idea in bullet points:

  1. Instead of running an application with its network interface, we record a video of the VR scene and the exact the spatial locations of the items the monkey needs to touch or fixate on.
  2. We then play the video using MWorks.
  3. For each video frame, we inform MWorks about the relevant locations to be touched or looked at.

Would something like this be feasible within MWorks?

Sincerely,
Simona

4

Hi Simona,

Would something like this be feasible within MWorks?

Yes, I think so. You could play the video using MWorks’ video stimulus, and you could use render actions to update the fixation/touch targets on a frame-by-frame basis. As long as you don’t need the VR scene to be interactive, that sounds like a good solution.

Cheers,
Chris

5

Hi Chris,

I hope you’re well.

Following up on your previous advice regarding camera compatibility, I wanted to discuss the best approach for setting up video recordings of monkeys during tasks using Basler cameras (model acA1920-155uc). Specifically, we want to start and stop recordings as the monkeys perform tasks and synchronize them with MWorks.

We’ve seen that Basler cameras are macOS-compatible through their pylon Camera Software Suite, but I understand from your response that MWorks would need the cameras to be natively recognized by macOS for seamless integration. We’ll test this to see if they work directly via macOS APIs, but if they aren’t compatible, we’ll look into the alternative UVC-compliant options from the list you shared.

Additionally, I came across Campy (GitHub - ksseverson57/campy: Python package for streaming video from multiple cameras to disk. Features real-time compression and debayering using FFmpeg.), an open-source library designed for high-performance video recording with Basler cameras. Do you think this could be a viable solution for our needs, either as a standalone tool or integrated with MWorks?

Lastly, you mentioned the possibility of developing an I/O device plugin for video recording within MWorks. If we were to pursue this, regardless of camera compatibility, how long might this integration take?

Thank you again for your help!

Best regards,
Simona Vaitekunaite