I still think there must be memory corruption somewhere, but I don’t know where (or why I don’t see it in my testing). From the crash report, I see that the only other non-standard plugin you have installed is OrientedNoise. Did you recompile that plugin for MWorks 0.12.2? And are you using it in your experiment?
Actually, can you also send me the experiment you’re using, so that I can test it on my system? To be clear, this crash happens after you run the experiment and then try to unload it, right?
It could be the OriNoise plugin if it’s not been compiled for that MWorks version. The experiment does include it as a stimulus though probably Nina is not displaying it. Mark
I recompiled the OrientedNoise plugin, which was last updated a couple of years ago. I am using OrientedNoise in my experiment, and then unloading the experiment, which is when I get the crash. I’ll attach the experiment here.
We should just loop you into our highly— and probably over complicated — experiment repository.
It’s been on my list for a while to ask you to help us improve it. It works fine, but there’s a lot of historical stuff in there that could be simplified.
We should just loop you into our highly— and probably over complicated — experiment repository.
That would also be fine.
It’s been on my list for a while to ask you to help us improve it. It works fine, but there’s a lot of historical stuff in there that could be simplified.
Sure, I could take a look. I’d probably need some guidance from you on what everything is, but I might be able to help you re-organize.
I added you to our ExperimentXML repo which has a whole bunch of experiments in it. Nina is working with the HoldAndDetectConstant19 expt.
Please don’t judge me for all the crazy baroque code. There are a bunch of messy things there.
You’ll probably want to start with the Workspace json in that directory. It loads a client python, a matlab script for plotting, and an mwel.
Let us know if you have questions.
I used the copy of HoldAndDetectConstant19.mwel that Nina sent me, as it differed slightly from what was on GitHub
I had to disable use of the pytoolsMH Python package, since I couldn’t find that
In PythonBridgeCode/Expt/constants.py, I modified getMachineConsts to act like the hostname was histedrigtest
With those changes, I can load and run the experiment, including the Python bridge code. So far, everything seems to be working smoothly, with no issues when unloading the experiment or closing MWServer. The only exception: Once, after the experiment had been running for maybe 20-30 minutes, I pressed the stop button, but the experiment wouldn’t stop. After a minute or so, I got a message from macOS saying my system was out of application memory, and I had to force quit MWServer. Unfortunately, I didn’t get a crash report, so I don’t know where things got stuck. I haven’t been able to reproduce this problem.
Nina: I’ve attached a very simple experiment that includes your LabJack device definition and nothing else. All it does is start the LabJack, wait for 10 seconds, and end. Can you try running this and see if you still need to force quit MWServer afterward?
I wasn’t able to load your mwel because of the following error:
00:00:29: ERROR: Can’t start iodevice (LabJackU6) and no alt tag specified
Extended information:
parser_context: mw_finalize
I was not exactly sure how to fix this but I attached the crash report. I figured maybe it is helpful to see if a different simpler mwel unloads. I created an mwel based on HoldAndDetectConstant19 with a short protocol, which loads, but crashes during the unload. I’ve attached that mwel and the associated crash report too.
I wasn’t able to load your mwel because of the following error
00:00:29 : ERROR: Can’t start iodevice (LabJackU6) and no alt tag specified
Oh, no. I think this means that there isn’t actually any memory corruption, and the problems you’re seeing have a much simpler (and sillier) explanation, namely: The LabJack is failing to initialize, and your experiment is falling back on the FakeMonkey device.
This would explain the crash on exit: If the LabJack device never initializes, the connected flag is never set to true, and the previously-mentioned assertion in detachPhysicalDevice fails and crashes MWServer. (FYI, that’s a really silly assertion that should be removed.) It also explains why I wasn’t seeing the crash: The LabJack device did initialize for me.
Here’s a quick test to verify this conclusion: In MWELSharedCode/includeExptDevices.mwel, comment out or remove the alt attribute of the LabJack device, i.e.
Then, try to load one of your experiments. If it fails with “ERROR: Can’t start iodevice (LabJackU6) and no alt tag specified”, my conclusion is correct.
Is the LabJack connected to the Mac running MWServer via a USB cable?
Is the LabJack’s yellow power LED on?
If the answer to both questions is “yes”, the next step is to see if the LabJack library code is giving any useful error messages. To do this, open Terminal, and launch MWServer with the following command:
Then, try loading an experiment with a LabJack, and see if any messages appear in the Terminal window. For example, if I try to load simple_labjack_test.mwel with no LabJack connected, I see the following message:
Open error: No U6 devices could be found
Hopefully, you’ll see some messages that provide a hint as to what’s going wrong. If you do, please send them to me, and we’ll figure out the next step.
It sounds like the issue is an assert that is not needed in our lab jack driver. (Probably added by me at some point I take responsibility. )
So let’s just take that out yes?
Chris,
Our medium term plan is to switch to the T series and your new more general labJack driver.
But in the short term, we’re still using U6 devices.
So Nina and I just did some cleanup and simplification of our U6 driver. Making this assert change on top of that would be totally fine.
It sounds like the issue is an assert that is not needed in our lab jack driver.
Well, that’s why MWServer is crashing. But I’d say the bigger issues are
Why is the LabJack device not initializing?
How was Nina able to run her experiment without noticing that the LabJack wasn’t functional?
If the answer to (2) is that Nina doesn’t need the LabJack in her experiment, then this whole discussion has just been an insane waste of time.
So Nina and I just did some cleanup and simplification of our U6 driver.
I don’t see any recent changes (other than my own) on GitHub. If you’ve modified the code, then (1) that would have been very helpful for me to know, since I’ve been assuming that Nina is using the same version of the plugin as me, and (2) perhaps you introduced a bug that causes the device to fail to load.
First, thank you for all your time and yes this is helpful to us – we often write MWEL code and test it on laptops without the LabJack device, using ‘fake subject’ code to impersonate the hardware, and test more quickly than we can in front of a rig where we have to flip levers to test. Then we move the code to the rigs.
That’s the answer to this:
How was Nina able to run her experiment without noticing that the LabJack wasn’t functional?
She’s currently in a testing/development stage where she (and I) are running the experiments on our laptops without a functional labjack. We have gone through a few cycles of write-code/review/test and are now just about to start moving this to a rig for final testing. The crash has been annoying us because during our testing sessions we’ve had to force quit the Server repeatedly.
So we’d like this to work both with, and without, the labjack getting initialized.
If I understand all this correctly, sounds like the issue is that assert. Yes?
Re: new code: we just made the first cut at these changes this last Fri and it hasn’t been tested with live Labjack hardware yet either. I don’t know if the code is perfect, but you can see it on the 2310/imaginggate-laseroff branch. In sum, we removed our output ‘reward’ variable/FIO line and all its timing code; we now use a pump rather than a solenoid, we renamed a few variables, and removed some dead code. We haven’t actually used these changes yet, so it won’t affect the debugging here, and I also don’t think it has touched the assert issue you seem to have found (thanks!).