I would like to start making video recordings of the monkeys face and possibly body and synch them to the recordings. Jim really liked the idea and suggested we might already get something going with the animal we are currently recording. There may be a coarse way to do this without MWorks (I will look into it), but Jim also asked me to contact you about this so this can be done with MWorks. There will be many advantages to integration probably, like not disturbing the setup by doing it manually, risk of forgetting to start or stop, etc. It should be treated like a piece of hardware that can be turned on or off I guess. Happy to also zoom about it if you have the time!
Apologies for the delayed reply. I was out on vacation last week.
We can definitely add video recording control to MWorks. I already did most of the required work when implementing face recognition support, so it shouldn’t be a big job. Do you have a timeframe in mind for when you’d be ready to use this?
If Jim wants it implemented for the monkey that we are already recording from, it should be done in the next few weeks. I realize that you have a queue so if that is not feasible from your side, I might look into a temporary solution.
The most important thing will be to synch the camera recordings to the neural recordings.
Note that we may need to be able to record from several cameras (depending on field of view choices etc)
I will have a meeting with Jim on Wednesday where I will present some camera options, and maybe we go ahead and order right away. (Delivery time depends a bit if we want/need infrared cameras or if we can buy something standard that will be delivered quickly).
Do you have ideas what would be more or less easy from your side to integrate?
Do you have ideas what would be more or less easy from your side to integrate?
It’s easy to integrate any camera that macOS recognizes automatically. If you can plug in the camera and immediately use it with FaceTime or Photo Booth, then MWorks can control it using the infrastructure I created for doing face recognition.
On the other hand, if macOS doesn’t see the device as a standard camera, then the amount of work involved in integrating it with MWorks will depend on the interfaces that are available to control it. Be careful not to buy anything that depends on Windows-only drivers, since that likely won’t work at all on a Mac.
Note that we may need to be able to record from several cameras (depending on field of view choices etc)
That shouldn’t be a problem.
The most important thing will be to synch the camera recordings to the neural recordings.
We can easily record the start and stop times on MWorks’ clock of each recording. How you align MWorks timestamps with those of the recording hardware will depend on the system you’re using. Typically, folks have MWorks send periodic TTL signals to a digital input on the recording system and use the timestamps for those events to compute a clock offset between MWorks and the neural recordings.
If Jim wants it implemented for the monkey that we are already recording from, it should be done in the next few weeks. I realize that you have a queue so if that is not feasible from your side, I might look into a temporary solution.
I may be able to get things ready in that time frame. As I said, the amount of work required on my end mainly depends on how MWorks will control the cameras. If you have some specific cameras in mind, I’d be happy to check out the specs and try to assess how easily we could integrate them with MWorks.
This is all good to know. I learned yesterday that Ko may have a camera from a side project sitting around that we might be able to use, he’s going to dig it up tomorrow. I will check if it is compatible (likely) and of sufficient quality and tell you the model asap. If this one somehow does not work I will send you some other model suggestions in the next few days.
Regarding the synch of video to neural recordings and MWorks, what I am worried about is possible frame skips if we just have one long recording. If I have 2h of task recording and 100 missing frames, when did they happen? So ideally the camera sends regular TTL pulses to the recording system (I know some mouse researchers who do this but their camera has a mouse-size field of view… this is also where my fear of frame skips comes from). I am not sure if we can get the average camera (typically usb) to do such a thing?
I played around a bit with the camera that Ko already had, but the manual focus is either not working or hard to use. Also, the model has been discontinued so I looked into buying an updated version of the one he had.
Could you take a look at the following ones to see which ones you think may be most suitable?
https://www.amazon.com/dp/B07N182MP6 %mac-compatible, it says manual focus, fps good, fov unclear but says good for family chat. Facetime is advertised. 3m cable
I also looked at some that explicitly mention TTL. However, the combination IR+TTL triggers +mac compatible, + decent resolution and sampling rate, is hard to find. What they mean when mentioning TTL tends to be for everything, not just triggers I fear. So I am hoping we can make a standard camera synch well.
What do you think?
Alina
Addendum
Some other cameras that are more notes for me unless you get enthusiastic
There are smart home cameras that typically work with sd cards. Usually viewing angle is not what we need and sd card is undesirable manual chore for every day.
This one looks great if suddenly we don’t need infrared:
or this
does simultaneous natural light and infrared. No focus?
This is Ko’s camera: https://www.amazon.com/Vision-Webcam-Camera-Windows-Android/dp/B01M9DM4R6
It says manual focus, I am used to rotating things to focus but it seems stuck. Either I overlook something trivial or it is at least not easy. I will still try some things but they involve taking it apart, which we might not want to do every time even if this turns out to be the solution.
https://www.amazon.com/dp/B07N182MP6 %mac-compatible, it says manual focus, fps good, fov unclear but says good for family chat. Facetime is advertised. 3m cable
Both of these should be compatible with MWorks’ existing camera interface code, as should any other camera that claims to be UVC compliant and/or work on macOS without drivers.
These are an entirely different kind of device. The “TTL” in this case is the TTL serial interface used to control the camera. It’s not a TTL trigger to initiate image capture or signal the beginning of an exposure, which I think is what you want.
However, the combination IR+TTL triggers +mac compatible, + decent resolution and sampling rate, is hard to find.
Yes, I tried some searching myself, and I see what you mean. Places like Thorlabs and Teledyne Photometrics seem to sell cameras with the type of TTL triggering you want. However, they all appear to depend on Windows-only software.
Regarding the synch of video to neural recordings and MWorks, what I am worried about is possible frame skips if we just have one long recording. If I have 2h of task recording and 100 missing frames, when did they happen?
Well, one option would to be to record the “video” as a sequence of still images, instead of in an actual video file format. Each image could be saved with the associated operating system timestamp (converted to MWorks’ clock) for when it was captured. This approach would require more storage space than a video file format. Also, it’s possible that processing 30-100 images per second might be more than MWorks or the OS can handle, though this might not be an issue.
Perhaps a more significant question is how accurate those OS-provided timestamps are. Would they reflect the lag between when the camera started the exposure and when the OS received the image over the USB connection? If so, do we have any way to characterize that lag? My guess is no, which is probably why folks who perform timing-critical imaging rely on special cameras with TTL signals that indicate the start of each exposure. I don’t imagine any consumer-grade webcam would provide that sort of detailed timing info.
The companies that you mentioned match the ones I know from people doing mouse recordings. I waited for a while for a former colleague to get back to me on further American-based (ie more likely mac compatible) options but did not hear from him anymore.
I suggest we have a zoom meeting about this, sometime this week if that suits you, to see what solution to settle on?
