Hi Chris,
We set up a new rig with a Mac Pro trash can running 10.9.5 (downgraded) with a NiDAQ 6251 PCIe card in a thunderbolt conversion housing. I’m getting the following error that I’m not seeing in our other rig:
00:13:58: ERROR: NIDAQ error: Specified operation did not complete, because the specified timeout expired. (-200474)
00:13:58: WARNING: Scheduled task (/Users/mwdev/Documents/mworks_buildbot/slave/build_all/build/plugins/core/NIDAQ/NIDAQPlugin/NIDAQDevice.cpp:372) falling behind, dropping 3 scheduled executions (priority = 95, interval = 3000, task = 0x6000001c5dc0)
Do you have any idea of what could be causing this? Our other rigs are running 10.8, so I’m thinking the issue could be either Mavericks or Thunderbolt.
Thanks,
Evan
Hi Evan,
My first guess is that it’s a Thunderbolt issue. Have you confirmed with someone at NI that your DAQ is supposed to work over Thunderbolt? (I’ve investigated it in the past but never found a definitive answer.)
Also, is this an intermittent error, or are all I/O operations failing? What specific I/O type(s) have you tried (i.e. analog/digital, input/output)?
Thanks,
Chris
Hi Chris,
I spoke with an NI sales guy who said the adapter should work, so I guess I’ll open up a service request with NI to see if they can figure out what the issue is. Are these errors enough for them to work with or do you have a more detailed explanation for what the problem is for MWorks?
Thanks,
Evan
Hi Evan,
It looks like some read or write operation is timing out. More specifically, one of the following functions is returning error code DAQmxErrorOperationTimedOut (-200474):
DAQmxBaseReadAnalogF64
DAQmxBaseWriteAnalogF64
DAQmxBaseReadDigitalU32
DAQmxBaseWriteDigitalU32
DAQmxBaseReadCounterScalarU32
If all you’re doing is writing a digital output, then the culprit must be DAQmxBaseWriteDigitalU32.
That’s probably the most I can say without running your experiment in the debugger and observing the error there.
Cheers,
Chris
Hi Chris,
I’ve spoken with Apple, NI, and mLOGIC (the PCIe to Thunderbolt expansion chassis manufacturer) about the issue. After having me run several tests, the NI guy thinks the problem is with the converter. An engineer at mLOGIC pointed me to this document from Apple:
Which mentions that PCI drivers may need to be modified to account for the latency added by the thunderbolt connection. There could be additional latency from the converter, so the NI drivers might need to be modified. I was wondering though, are there any timers on the MWorks side that could be modified?
Thanks,
Evan
Hi Evan,
I was wondering though, are there any timers on the MWorks side that could be modified?
Yes. MWorks specifies the timeout for the read/write functions. Currently, it’s set to update_interval (the reasoning being that the current operation must complete before we need to start the next one), but we can change it to whatever we want. Do you want me to make the timeout value a parameter that can be set by your experiment?
Chris
Oh, in that case no. I was thinking we could extend the MWorks timeout if it was on the order of microseconds.
Evan
Hi Chris,
After some additional testing it seems that the Error depends on the update_interval parameter. Testing an older PCIe rig, the lower limit is between 0.5 and 0.75 ms for my particular channel configuration, whereas in the new Thunderbolt rig it’s somewhere north of 3ms, the original setting. The official line I’m getting now from NI is that the PCIe-Thunderbolt adapter hasn’t been tested and so isn’t supported.
Evan
Hi Chris,
Using your USB-6212, I’m getting the following error intermittently:
ERROR: NIDAQ error: <err>Some or all of the samples requested have not yet been acquired.
To wait for the samples to become available use a longer read timeout or read later in your program. To make the samples available sooner, increase the sample rate. (-200284)
I’ve only tested update intervals of 2 and 3 ms and it happened for both. Surprisingly, when I moved the unit, I started getting the error in rapid succession.
Hi Evan,
I see that error sometimes, too. It seems like it’s been showing up in my automated test results more frequently since I upgraded the test machine to NI-DAQmx Base 14.0. When you’re done with the device, I’ll run some tests and see what I can figure out.
Surprisingly, when I moved the unit, I started getting the error in rapid succession.
You mean, you saw lots of errors while you had the device in your hands and were physically moving it? That’s really strange (and suggests it may be a hardware issue, not a software one).
Chris
Hi Evan,
I’ve run some analog I/O tests with the USB-6212 on several different machines. Unfortunately, they haven’t revealed much.
On my primary test machine (late-2009 Mac mini running OS X 10.8), I can easily induce the “samples requested have not yet been acquired” error by loading up the CPU cores (e.g. making a couple Python processes evaluate infinite loops) or doing UI-intensive tasks in other applications (e.g. mucking around with Google Maps in Safari). This isn’t too surprising, as that Mac is old and underpowered, and it’s not hard to imagine things falling behind when the system is taxed.
On a secondary test machine (late-2013 iMac running OS X 10.10), I can occasionally induce the error with similar activities. On my main development machine (mid-2010 Mac Pro running OS X 10.10), I can’t induce the error at all.
In addition to testing with MWorks, I tried a standalone analog I/O test (based on the example programs distributed with NI-DAQmx Base) and got the same results. I also tried downgrading the Mac mini to NI-DAQmx Base 3.7, but the results were unchanged.
None of this suggests why you might be seeing these errors on your new Mac Pro. If the issue comes down to system load, then, if anything, your machine ought to be outperforming all the ones I tried.
Would it possible for me to run some of my tests on your system? In particular, I’d like to try the non-MWorks test program. If I can reliably induce errors running that, then I think we’d have cause for opening another support request with NI.
Thanks,
Chris
Hi Chris,
I tried to reproduce the error on my older Mac Pro and so far haven’t been able to see it unless I set the update interval to 1ms. Even then, it doesn’t just start at some random delay then snowball out of control - it starts immediately.
It’s curious to me that the errors from the two different devices are different. The PCIe devices (Thunderbolt or no) give a “timeout” error, whereas the USB devices give the “samples have not been acquired error.” This is constant across the computer being tested.
Thanks,
Evan
Hi Evan,
I tried to reproduce the error on my older Mac Pro and so far haven’t been able to see it unless I set the update interval to 1ms. Even then, it doesn’t just start at some random delay then snowball out of control - it starts immediately.
Hmm. I suppose it’s possible that the issue is unique to the new Mac Pro – maybe a driver issue? However, it’s not immediately clear how we could test that hypothesis. Let me think about it a bit.
It’s curious to me that the errors from the two different devices are different. The PCIe devices (Thunderbolt or no) give a “timeout” error, whereas the USB devices give the “samples have not been acquired error.”
That could just reflect that different driver pathways for PCIe and USB devices.
Also, it’s possible that the Thunderbolt problem is a different issue entirely. Have you tried the PCIe card + Thunderbolt chassis on any other Macs? I can try it on an iMac or Mac mini, if you like. (The old Mac Pro doesn’t support Thunderbolt, so that’s not an option.)
Chris
Hi Chris,
Just to update from the last post - I let the program run for a day on the old mac pro using the USB DAQ and a 2 ms update interval and got a total of 3 errors in 24 hours (no consecutive errors). I also ran with the Thunderbolt expansion PCIe DAQ on a Macbook Air and only got one warning. You don’t have a new Mac Pro in the lab that I could test do you?
Evan
Hi Evan,
I also ran with the Thunderbolt expansion PCIe DAQ on a Macbook Air and only got one warning.
That’s really interesting. Maybe it is an issue with the new Mac Pro.
You don’t have a new Mac Pro in the lab that I could test do you?
No, I don’t think anyone in our lab has one yet.
Chris
I suppose it’s possible that the issue is unique to the new Mac Pro – maybe a driver issue? However, it’s not immediately clear how we could test that hypothesis.
Maybe the right way to proceed is for me to develop a simple, standalone application that performs the exact same I/O tasks as your MWorks experiment. If we can get that application to produce the errors you’ve been seeing with the USB DAQ, then you can open a support request with NI and ask them to check it out. The latest NI-DAQmx Base is supposed to support the USB-6212 on OS X 10.9, so if there are problems using it with the new Mac Pro, then that’s definitely something they should address.
Does this sound OK to you? If so, I’ll pick up the DAQ sometime today.
Chris
I opened a service request back in January and I believe NI already tested the specific scenario directly after I sent them a copy of the experiment I’m using to test the issue. I never had them test it with a 6212 though, so maybe I can pick up that old thread with NI.
I’d like to try and track down another Mac Pro here to see whether I can reproduce the problem. If not, I’ll try a clean install on the computer and if that doesn’t fix it, then it must be some hardware issue with our particular computer.
Hi Evan,
As we’ve discussed offline, I’ve been able to reproduce the “samples requested have not yet been acquired” error with the USB-6212, using both your MWorks experiment and a simple (non-MWorks) test application. I’ve seen the errors on a late 2012 Mac mini (running OS X 10.9.5) and a late 2013 iMac (running OS X 10.10.2). However, I still haven’t seen them on my mid-2010 Mac Pro.
I’ve submitted a support request to NI. (If you want to reference it in your communications with them, it’s Service Request #7441619.) I’ll let you know what I hear.
Cheers,
Chris
Thanks for the update. I also got the same error on a mid-2014 Macbook Pro, but still haven’t seen the error on our early 2014 Macbook air or mid 2012 Mac Pros.
Hi Evan,
Following up on our earlier (offline) discussion: I’ve updated and regenerated the MWorks nightly build so that the NIDAQ interface is once again built against NI-DAQmx Base 3.7. It’d be great if you could use the new build to re-test your PCIe card over Thunderbolt.
FYI, I found that installing NI-DAQmx Base 3.7 over 14.0 left things in a bad state (e.g. my test app crashed almost immediately after starting), so you may want to remove all the 14.0 files before installing 3.7. To do this, open Terminal and run the following commands:
sudo launchctl unload /Library/LaunchDaemons/nilxid.plist
sudo rm -rf /Applications/National\ Instruments/ /Library/Frameworks/NiSpyLog.framework /Library/Frameworks/LabVIEW\ * /Library/Frameworks/lib* /Library/Application\ Support/National\ Instruments/ /Library/Frameworks/nidaqmxbase* /Library/Extensions/niusb9162k.kext/ /Library/Preferences/NIvisa/ /Library/Frameworks/VISA.framework/ /Library/LaunchDaemons/nilxid.plist /Library/Extensions/NiViPciK.kext/ /Library/Extensions/nipalk.kext/ /Library/StartupItems/nipal/ /usr/sbin/nipalsm /usr/sbin/palModuleMgr.sh /Library/Frameworks/NI-RPC.framework/ /Library/Frameworks/nisysapi.framework/
Also, on systems running Yosemite, you’ll need to disable the signing requirement for kernel extensions. In Terminal, run the command
sudo nvram boot-args="kext-dev-mode=1"
and then restart. (This isn’t necessary on Mavericks or Mountain Lion.)
Chris