CoreML Object Detector model as iodevice

Support
1

Hello Chris, I am Ayuno, a postdoc at the DPZ. I’ve been using mWorks as a task controller for freely moving animals (Vladyslav and Ralf have been awesome help to me).

I’d like to use an Object Detector CoreML model to detect the position and identity of animals, and use its output to trigger events/states in mWorks. I tried using Face Recognizer by feeding in an Object Detector model, which didn’t work because it expects an Image Classifier model’s output. Is there a plan for integrating an Object Detector model? If not (or rather, in the meantime), do you have any suggestions on how to go about it (e.g., use Python Bridge with coremltools)?

Thank you in advance for your support,

Ayuno

2

Hi Ayuno,

I’m currently away on vacation. I can help you with this when I return the week of August 17.

Cheers,
Chris

3

Hi Ayuno,

I think it would be pretty straightforward to add support for object-detection models to MWorks’ face recognizer. The only real change would be to handle the different output type, which shouldn’t be too difficult. If you’re still interested, this is something I can work on.

I’m not sure about alternatives. You certainly could access the captured images (via the variable assigned to the image_data parameter) and process them using MWorks’ Python actions. However, I’m not sure coremltools would help, as its purpose seems to be creating/converting models, not applying them. Maybe there are other Python libraries that would be more useful?

Cheers,
Chris

4

Hello Chris,

I hope you had a nice vacation!

Yes, I am still very much interested in using the object-detection models with the face recognizer plugin (or as a separate plugin, whatever works the best). Please work on it. And it would be great if it can output more than one detected object with all the relevant info (i.e., xywh, label and confidence).

I thought about alternatives too, like using PyTorch, openCV etc. I still think that adapting the face recognizer plugin is the most straight forward solution to my situation. The architecture is basically already there, and it’s tested and proven (based from what I heard from Ralf and the face recognizer users).

Sincerely,

Ayuno