MWorks client crash on remote connection

[hokysung]

Hi Chris, the issues that I generally described here are happening again:

MWorks client error

Hi Chris, I’ve been observing an error where MWClient crashes as soon as I open Eye Window when I’m connected to a remote MWServer. This started to happen a few days ago, before that the Eye Window would open fine. The following is the full error message: ‘’’ Translated Report (Full Report Below) Process: MWClient [41626] Path: /Applications/MWClient.app/Contents/MacOS/MWClient Identifier: org.mworks-project.MWClient Version: 0.13 (20…

Both MWorks on my Mac and the rig are up-to-date versions 0.13. I can connect to the experiment remotely, and can view the variables and console in real time. But whenever I open the eye window, the client immediately crashes. The experiment running in the rig is unaffected. I haven’t used this functionality for a while, so I’m not sure what changed since then.

Here is the error message I’m seeing:

---

Translated Report (Full Report Below)

Process: MWClient [46407]
Path: /Applications/MWClient.app/Contents/MacOS/MWClient
Identifier: org.mworks-project.MWClient
Version: 0.13 (2024.05.01)
Code Type: ARM-64 (Native)
Parent Process: launchd [1]
User ID: 501

Date/Time: 2025-03-19 16:08:07.5328 -0400
OS Version: macOS 15.3.2 (24D81)
Report Version: 12
Anonymous UUID: 0E7AEC52-7295-2F8F-029F-21D04837733A

Sleep/Wake UUID: 156C429A-5CD3-42C8-BCEF-03DE9006750D

Time Awake Since Boot: 200000 seconds
Time Since Wake: 48 seconds

System Integrity Protection: enabled

Crashed Thread: 0

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Termination Reason: Namespace SIGNAL, Code 6 Abort trap: 6
Terminating Process: MWClient [46407]

Application Specific Information:
abort() called

Thread 0 Crashed:
0 libsystem_kernel.dylib 0x18853b720 __pthread_kill + 8
1 libsystem_pthread.dylib 0x188573f70 pthread_kill + 288
2 libsystem_c.dylib 0x188480908 abort + 128
3 libc++abi.dylib 0x18852a44c abort_message + 132
4 libc++abi.dylib 0x188518a40 demangling_terminate_handler() + 348
5 libobjc.A.dylib 0x1881c13e4 _objc_terminate() + 156
6 libc++abi.dylib 0x188529710 std::__terminate(void ()()) + 16
7 libc++abi.dylib 0x1885296b4 std::terminate() + 108
8 libdispatch.dylib 0x1883c15c8 _dispatch_client_callout + 40
9 libdispatch.dylib 0x1883d0e08 _dispatch_lane_barrier_sync_invoke_and_complete + 56
10 MWorksEyeWindow 0x106560930 -[MWTimePlotView drawRect:] + 300
11 AppKit 0x18c2e6770 _NSViewDrawRect + 160
12 AppKit 0x18ccbddf4 -[NSView _recursive:displayRectIgnoringOpacity:inContext:stopAtLayerBackedViews:] + 1084
13 AppKit 0x18c2e6164 -[NSView(NSLayerKitGlue) _drawViewBackingLayer:inContext:drawingHandler:] + 536
14 AppKit 0x18c9707ac -[NSViewBackingLayer drawInContext:] + 56
15 AppKit 0x18c5ee5e0 0x18c197000 + 4552160
16 AppKit 0x18c5e9420 0x18c197000 + 4531232
17 AppKit 0x18c5e4c78 0x18c197000 + 4512888
18 AppKit 0x18c5e9000 0x18c197000 + 4530176
19 AppKit 0x18c5e6b70 0x18c197000 + 4520816
20 AppKit 0x18c65f18c 0x18c197000 + 5013900
21 AppKit 0x18c65f790 0x18c197000 + 5015440
22 AppKit 0x18c9703d8 -[NSViewBackingLayer display] + 1272
23 QuartzCore 0x1911bc1b8 CA::Layer::display_if_needed(CA::Transaction) + 784
24 QuartzCore 0x19134b0e4 CA::Context::commit_transaction(CA::Transaction*, double, double*) + 528
25 QuartzCore 0x19119e780 CA::Transaction::commit() + 648
26 AppKit 0x18c2f5a9c __62+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayLink]_block_invoke + 272
27 AppKit 0x18cd008f4 ___NSRunLoopObserverCreateWithHandler_block_invoke + 64
28 CoreFoundation 0x18865bbe8 CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION + 36
29 CoreFoundation 0x18865bad4 __CFRunLoopDoObservers + 552
30 CoreFoundation 0x18865b104 __CFRunLoopRun + 788
31 CoreFoundation 0x18865a734 CFRunLoopRunSpecific + 588
32 HIToolbox 0x193bc9530 RunCurrentEventLoopInMode + 292
33 HIToolbox 0x193bcf17c ReceiveNextEventCommon + 216
34 HIToolbox 0x193bcf508 _BlockUntilNextEventMatchingListInModeWithFilter + 76
35 AppKit 0x18c1d2848 _DPSNextEvent + 660
36 AppKit 0x18cb38c24 -[NSApplication(NSEventRouting) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 688
37 AppKit 0x18c1c5874 -[NSApplication run] + 480
38 AppKit 0x18c19c068 NSApplicationMain + 888
39 MWClient 0x10026a440 main + 112
40 dyld 0x1881f4274 start + 2840

Thread 1:
0 libsystem_pthread.dylib 0x18856f0e8 start_wqthread + 0

Thread 2:
0 libsystem_pthread.dylib 0x18856f0e8 start_wqthread + 0

Thread 3:
0 libsystem_kernel.dylib 0x1885366ec __psynch_cvwait + 8
1 libsystem_pthread.dylib 0x188574894 _pthread_cond_wait + 1204
2 libc++.1.dylib 0x1884aa60c std::__1::condition_variable::__do_timed_wait(std::__1::unique_lockstd::__1::mutex&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>) + 100
3 MWorksCore 0x101504fc0 mw::Event::getNextEvent(long long) const + 148
4 MWorksCore 0x10157ac7c mw::EventBufferReader::getNextEvent(long long) + 32
5 MWorksCore 0x1015d7ee0 mw::EventListener::service() + 100
6 MWorksCore 0x1015d8028 void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, mw::EventListener::startListener()::$_0>>(void*) + 52
7 libsystem_pthread.dylib 0x1885742e4 _pthread_start + 136
8 libsystem_pthread.dylib 0x18856f0fc thread_start + 8

Thread 4:: com.apple.NSEventThread
0 libsystem_kernel.dylib 0x188532f54 mach_msg2_trap + 8
1 libsystem_kernel.dylib 0x188545604 mach_msg2_internal + 80
2 libsystem_kernel.dylib 0x18853baf8 mach_msg_overwrite + 480
3 libsystem_kernel.dylib 0x18853329c mach_msg + 24
4 CoreFoundation 0x18865ca4c __CFRunLoopServiceMachPort + 160
5 CoreFoundation 0x18865b2ac __CFRunLoopRun + 1212
6 CoreFoundation 0x18865a734 CFRunLoopRunSpecific + 588
7 AppKit 0x18c2f7278 _NSEventThread + 148
8 libsystem_pthread.dylib 0x1885742e4 _pthread_start + 136
9 libsystem_pthread.dylib 0x18856f0fc thread_start + 8

Thread 5:
0 libsystem_pthread.dylib 0x18856f0e8 start_wqthread + 0

Thread 6:: /Reaper
0 libsystem_kernel.dylib 0x18853901c kevent + 8
1 MWorksCore 0x1017f4c40 zmq::kqueue_t::loop() + 228 (kqueue.cpp:163)
2 MWorksCore 0x101820e60 thread_routine(void) + 76 (thread.cpp:229)
3 libsystem_pthread.dylib 0x1885742e4 _pthread_start + 136
4 libsystem_pthread.dylib 0x18856f0fc thread_start + 8

Thread 7:: /0
0 libsystem_kernel.dylib 0x18853901c kevent + 8
1 MWorksCore 0x1017f4c40 zmq::kqueue_t::loop() + 228 (kqueue.cpp:163)
2 MWorksCore 0x101820e60 thread_routine(void) + 76 (thread.cpp:229)
3 libsystem_pthread.dylib 0x1885742e4 _pthread_start + 136
4 libsystem_pthread.dylib 0x18856f0fc thread_start + 8

Thread 8:
0 libsystem_kernel.dylib 0x18853b830 poll + 8
1 MWorksCore 0x10180edcc zmq::signaler_t::wait(int) const + 76 (signaler.cpp:219)
2 MWorksCore 0x1017f5aa4 zmq::mailbox_t::recv(zmq::command_t*, int) + 80 (mailbox.cpp:54)
3 MWorksCore 0x101810e4c zmq::socket_base_t::process_commands(int, bool) + 76 (socket_base.cpp:1478)
4 MWorksCore 0x101813b90 zmq::socket_base_t::recv(zmq::msg_t*, int) + 348 (socket_base.cpp:1363)
5 MWorksCore 0x10182e294 s_recvmsg(zmq::socket_base_t*, zmq_msg_t*, int) + 16 (zmq.cpp:471) [inlined]
6 MWorksCore 0x10182e294 zmq_msg_recv + 60 (zmq.cpp:622)
7 MWorksCore 0x1016588cc mw::ZeroMQSocket::Message::recv(mw::ZeroMQSocket&, int) + 48
8 MWorksCore 0x101659578 mw::ZeroMQSocket::recv(mworks_boost::shared_ptrmw::Event&) + 100
9 MWorksCore 0x101644a50 mw::ZeroMQIncomingConnectionBase::handleNextEvent() + 36
10 MWorksCore 0x101644fc8 void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, mw::ZeroMQIncomingConnectionBase::startEventHandlerThread()::$_0>>(void*) + 68
11 libsystem_pthread.dylib 0x1885742e4 _pthread_start + 136
12 libsystem_pthread.dylib 0x18856f0fc thread_start + 8

Thread 9:
0 libsystem_kernel.dylib 0x1885366ec __psynch_cvwait + 8
1 libsystem_pthread.dylib 0x188574894 _pthread_cond_wait + 1204
2 libc++.1.dylib 0x1884aa60c std::__1::condition_variable::__do_timed_wait(std::__1::unique_lockstd::__1::mutex&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>) + 100
3 MWorksCore 0x101504fc0 mw::Event::getNextEvent(long long) const + 148
4 MWorksCore 0x10157ac7c mw::EventBufferReader::getNextEvent(long long) + 32
5 MWorksCore 0x101644ddc mw::ZeroMQOutgoingConnectionBase::handleEvents() + 104
6 MWorksCore 0x101645098 void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, mw::ZeroMQOutgoingConnectionBase::startEventHandlerThread()::$_1>>(void*) + 52
7 libsystem_pthread.dylib 0x1885742e4 _pthread_start + 136
8 libsystem_pthread.dylib 0x18856f0fc thread_start + 8

Thread 10:
0 libsystem_pthread.dylib 0x18856f0e8 start_wqthread + 0

Thread 11:
0 libsystem_pthread.dylib 0x18856f0e8 start_wqthread + 0

Thread 12:: HIE: M_ 8c056f2c192dd847 2025-03-19 16:08:07.525
0 libsystem_kernel.dylib 0x188532f54 mach_msg2_trap + 8
1 libsystem_kernel.dylib 0x188545604 mach_msg2_internal + 80
2 libsystem_kernel.dylib 0x188562458 thread_suspend + 108
3 HIServices 0x18efec164 SOME_OTHER_THREAD_SWALLOWED_AT_LEAST_ONE_EXCEPTION + 20
4 Foundation 0x189825444 NSThread__start + 724
5 libsystem_pthread.dylib 0x1885742e4 _pthread_start + 136
6 libsystem_pthread.dylib 0x18856f0fc thread_start + 8

Thread 0 crashed with ARM Thread State (64-bit):
x0: 0x0000000000000000 x1: 0x0000000000000000 x2: 0x0000000000000000 x3: 0x0000000000000000
x4: 0x000000018852ee1b x5: 0x000000016fb939a0 x6: 0x000000000000006e x7: 0x00000000000001b0
x8: 0xba3d8b03703f8e8a x9: 0xba3d8b0281d1c6ca x10: 0x0000000000000051 x11: 0x000000000000000b
x12: 0x000000000000000b x13: 0x0000000188a3aca2 x14: 0x00000000001ff800 x15: 0x00000000000007fb
x16: 0x0000000000000148 x17: 0x00000001fa5262c0 x18: 0x0000000000000000 x19: 0x0000000000000006
x20: 0x0000000000000103 x21: 0x00000001f1ee4920 x22: 0x00000001f1ee6188 x23: 0x00006000000abc90
x24: 0x000000011a63d4c0 x25: 0x0000000000000000 x26: 0x0000000000000000 x27: 0x00000001ebfed000
x28: 0x0000000000000098 fp: 0x000000016fb93910 lr: 0x0000000188573f70
sp: 0x000000016fb938f0 pc: 0x000000018853b720 cpsr: 0x40000000
far: 0x0000000000000000 esr: 0x56000080 Address size fault
… (more)

Thank you for your help!

[cstawarz]

Hi Hokyung,

Based on the crash report, it looks like this is indeed the same issue as before. Back in August, I made a change that should prevent the client crash, so it’s in the nightly build but not 0.13.

The fix handles the case where the eye window receives non-numeric or invalid values (e.g. NaN) for some of its parameters. Maybe it’d be worth going through one of your event files and looking for invalid eye position data? If you find some, maybe we can figure out where’s it coming from and thereby track down the root cause of the issue.

Also, are you once again seeing messages of the form Invalid persistent value for variable "x"? I’m guessing you’re not, since you’re running MWorks 0.13 for both client and server, but I just wanted to check.

Thanks,
Chris

[hokysung]

Hi Chris,

I see, I will try to track down the event file and figure out what the issue is. I’m not seeing the errors you mentioned.

Thank you