iPadOS: loading multiple images; frame lag bug is back

Support
1

Hey Chris,

Loading multiple images:

  • I attempted to run a task involving 3584 image assets (256x256, grayscale PNGs) on the iPadOS version of Mworks.
  • The client seems to hang at around 3100 images loaded. The server does not display the default (gray) background; it remains on the screen showing the IP address.
  • So far, we have been successfully running MWEL code which is identical (except for the image loading lines) using 300 images.
  • I imagine this has something to do with MWorks attempting to buffer all the images simultaneously in memory (see code below).
  • Can I work around this issue somehow using existing functionality?

stimulus_group image_data {
    range_replicator (
        from = 0
        to = size(image_path_list) - 1  // End point depends on size of "image_paths" list
        step = 1
        variable = rr_index
        ) {
        image_file 'im${rr_index}' (
                path = image_path_list[${rr_index}]  // Index in to "image_paths" list
                x_size = image_draw_size
                y_size = image_draw_size
                x_position = image_draw_x
                y_position = image_draw_y
            )
        }
}

Frame lag bug is back:

  • There was an issue some number of versions ago where (for unknown reasons) the iPadOS server would have a time delay when responding to touches (this would be accompanied by a warning in the client console window). I think you diagnosed this as being related to the image assets failing to be buffered; and you recommended starting and stopping the task using the client window as a workaround. Then, I believe you patched this bug altogether.
  • After updating iPadOS (to the latest version; around 3 weeks ago?), this issue has begun to pop up again. The original workaround still seems to work.

Michael

2

Hi Michael,

I imagine this has something to do with MWorks attempting to buffer all the images simultaneously in memory (see code below).

Yes, that’s almost certainly the issue.

Can I work around this issue somehow using existing functionality?

Your only options are to reduce the number of images or make the image files smaller. Something like pngcrush or ImageOptim might give you smaller files without reducing resolution.

Sorry, this is a dumb limitation, but I haven’t had a chance to fix it yet.

I think you diagnosed this as being related to the image assets failing to be buffered; and you recommended starting and stopping the task using the client window as a workaround. Then, I believe you patched this bug altogether.

You’re referring to this issue, right? I think it’d be more correct to say that I implemented a mitigation, and I’m not super surprised it isn’t helping anymore. I believe the real fix lies in transitioning from OpenGL to Metal. I hope to get started on that any time now.

The original workaround still seems to work.

That’s good, at least.

Cheers,
Chris

3

Thanks for the suggestion, I’ll look into reducing the image sizes as much as possible without affecting their rendered appearance on the screen.

So just to confirm - there is no way to effectively load and “deload” image assets after the experiment load has concluded? Maybe through a Python bridge streaming from server disk or client connection?

You’re referring to this issue, right? I think it’d be more correct to say that I implemented a mitigation, and I’m not super surprised it isn’t helping anymore. I believe the real fix lies in transitioning from OpenGL to Metal. I hope to get started on that any time now.

Yes

4

So just to confirm - there is no way to effectively load and “deload” image assets after the experiment load has concluded?

You can do that, but it won’t help.

MWorks is running out of memory during the process of copying the image files to the iPad. Basically, the experiment and all related files (including images) get transferred in a single message sent from MWClient on the Mac to MWorks on the iPad. That single message needs to fit in to RAM in order for MWorks to decode it. In your case, it doesn’t fit, and things fail badly.

Like I said, this is a dumb problem, as there’s no reason we couldn’t send each image or other resource in a separate message. I’m planning to make that change, but I haven’t yet.

Chris