Memory leak in MEX functions

[cstawarz]

Hi Chris!

For some time now I am trying to find the cause for the massive memory leak in the getEvents MEX. But I am not having much luck with that.

The problem is that once I start loading a data file with Matlab, it literally takes several hours to complete reading a 1Gb data file, even with 16Gb of RAM. Matlab immediately starts allocating all of my RAM and then swaps even more. I’ve seen memory usage of up to 25Gb. Worst of all, this memory is never released again, you have to quit Matlab to release it.

I tried to modify getEvents.cpp to make it deallocate all the std::vectors that were allocated outside of the mx-scope. That didn’t help much since I didn’t find a way to completely delete the EventWrapper vector without making the whole program crash. And I don’t know whether clearing it is enough.
Next I’ve looked into the EventWrapper methods, but I cannot find anything suspicious there. Then there is the data file index which presumably also stays alive (I don’t think Matlab will release any memory that wasn’t allocated using the mx* methods).

Would you have time to look into this? Right now I have to run my data files (1-2 Gb, since they include a lot of IO data) over night, since running them stalls my machine for 2-5 hours. This is a very unpleasant situation for me and unfortunately it seems that I am not able to fix it myself.

Thank you !
Philipp

[cstawarz]

That didn’t help much since I didn’t find a way to completely delete the EventWrapper vector without making the whole program crash

My best guess is btw the DataFileIndexer. Just initializing the data file indexer with dfindex(mwk_file) , without doing anything else causes the memory to get lost… For what it’s worth, I’ve also recognized that there is no destructor for dfindex()…

[cstawarz]

Hi Philipp,

Thanks for the detailed problem description. I’ll try to look in to this soon.

Cheers,
Chris

[cstawarz]

Hi Philipp,

Just a quick update: I’m working on a fix for this issue. I hope to have it done by the end of this week, but it should be in the nightly build by early next week at the latest. I’ll keep you posted.

Chris

[cstawarz]

Hi Chris,

I see you are working hard on this (already lots of changes to the code!). Although the current nightly still shows the issue, I am very grateful for your effort and I hope to hear from you soon.

Thanks!
Philipp

[cstawarz]

Hi Philipp,

Yeah, this is taking longer than I expected – the more I look at the data file code, the more issues I find. Hopefully I’ll have something you can test in another day or two. Thanks for your patience!

Chris

[cstawarz]

Hi Philipp,

OK, here’s a rundown of what I’ve learned about this issue:

The biggest problem with the MATLAB getEvents function was that it basically doubled its memory usage by keeping two copies of each event in memory. (One of the copies was properly deleted when the function returned, so while it certainly wasted memory, it didn’t actually leak any.) This is fixed in the current nightly build. The function now keeps only one copy of each event, so it only needs (approximately) as much RAM as is needed to hold the returned events array.

A more fundamental problem is the fact that getEvents returns all the events at once, in a single array. Regardless of how efficiently the function uses memory, it’s still going to be possible to exceed available RAM simply by reading a huge number of events. In the Python bindings, I’ve added a path around this issue by providing a method (get_events_iter) that returns an iterator over the requested events, which reads them into memory one at a time. While we can do something similar in MATLAB, it will take a bit more work – probably I’ll need to create a MATLAB equivalent to the Python MWKFile class, with methods to start iteration and to return events one by one.

The question of whether the current approach (where all the events must be in RAM simultaneously) will work for you depends on how many events you want to read and how large the typical event’s payload is. In my testing (using 64-bit MATLAB R2010a), I’ve found that a 500MB file containing ~1,000,000 events (most of which are random dots draw announcements, without the per-frame dot positions) consumes about 3.8GB of RAM when read in to MATLAB. A 1GB file with twice as many events would presumably consume about twice as much memory, which would be murder on my Mac Pro with 8GB of RAM but would probably run pretty smoothly with 16GB.

I’d appreciate it if you could download the latest nightly and try loading your 1GB file. Hopefully it will load smoothly, but if you still end up in swapping hell, then I’ll need to implement event-by-event iteration sooner rather than later.

One final note: I believe your observation about the memory used by the events array not being released is actually the result of some quirky memory-management behavior by MATLAB, rather than a memory leak. Specifically, I’ve noticed that if I load a large number of events into the array events, and then I assign a new value to the variable, e.g.

events = 0;

then MATLAB’s memory usage does not decrease. However, if I use the clear command on the variable, i.e.

clear events

then the memory is released. My guess is that MATLAB is being conservative about destroying arrays that are themselves embedded in structure or cell arrays, deferring any action until it can do a thorough scan of allocated memory to ensure that all references to such arrays are gone. I could be completely wrong, of course, but you might want to see if clear does the trick for you, too.

Cheers,
Chris

[cstawarz]

While we can do something similar in MATLAB

I am doing something like this to avoid the problem of the MEX function reserving too much memory and to parallelize the process (since it also needs a lot of computational power)
eventBuffer = cell(time_steps,read_steps);
for i=1:read_steps
parfor a=1:time_steps
eventBuffer{a,i} = getEvents(filename,codes(i),min_time+((a-1)100000000),min_time+(a100000000)-1);
end
end
This reads the file codec by codec in chunks of 100 seconds each and stores the result in eventBuffer.

I believe your observation about the memory used by the events array not being released is actually the result of some quirky memory-management behavior by MATLAB, rather than a memory leak

Well, the fact that clear all did not release the ~25GB of RAM allocated to Matlab after running the MEX made me believe that we are dealing with a memory leak. The good news is, with your latest nightly the memory is released after clear all so what ever it was, it’s gone now. Thank you so much for that!

About the memory issues, however, I am not sure what to respond. As I said, I am recording a lot of IOs and that for several hours. A datafile can then easily be above 1.5GB, so that even if I exclude most of the IOs (e.g. eye position) and read only the rest (including display updates), 16GB of RAM is not enough to hold the resulting eventBuffer. Since this is surely a Matlab issue and somehow unrelated to the way the data comes in, I have to think about possible ways to read the data in chunks (e.g. read display updates separately…)

Anyhow, your changes seem to have solved my issue, so thank you again! If possible, could you provide me with a no_gl_fence installer that includes the fix?

Cheers,
Philipp

[cstawarz]

Hi Philipp,

If possible, could you provide me with a no_gl_fence installer that includes the fix?

Done. The new installer is here.

the fact that clear all did not release the ~25GB of RAM allocated to Matlab after running the MEX made me believe that we are dealing with a memory leak.

I see. I tackled the two-copies-of-each-event issue before anything else; in doing so, I may have fixed a memory leak without realizing it. In any case, I’m glad it’s gone!

A datafile can then easily be above 1.5GB, so that even if I exclude most of the IOs (e.g. eye position) and read only the rest (including display updates), 16GB of RAM is not enough to hold the resulting eventBuffer.

Yeah, we definitely need an easy way to read data files incrementally. My current idea is to add a MWKFile class that would let you do something like this:

f = MWKFile(filename);
f.selectEvents(codes, minTime, maxTime);   % Prepare for iteration
while true
    event = f.nextEvent();  % Load a single matching event
    if isempty(event)
        break;
    end
    processEvent(event);
end

What do you think?

Chris

[cstawarz]

Hi Chris,

is this functionality not already included? I can do something like this: getEvents(filename,codec,minTime,maxTime) is that already what you mean?

Otherwise I don’t know if incrementally reading the file would solve the problem on the Matlab side. Our current way to process the data requires that at some point, sooner or later, everything is stored in a single Matlab structure and this structure would need a massive amount of RAM. Incrementally reading could speed up the process of filling that structure, but in the end I still need to store it somewhere…

Or am I misunderstanding something here?

[cstawarz]

is this functionality not already included? I can do something like this: getEvents(filename,codec,minTime,maxTime) is that already what you mean?

The existing getEvents finds every event that matches the given parameters and returns them all at once, in a single array. What I’m proposing breaks this process into several discrete steps. Here’s the procedure in words:

1. Specify which events you want (*without* reading any into memory)
2. While at least one matching event is available, repeat the following:
   a. Read one (and only one) event into memory
   b. Process the event

Does that make sense?

If your data processing task requires you to have a large number of events in memory simultaneously, then what I’m proposing doesn’t necessarily help you. It might, though. For example, your “process the event” step could entail grabbing only the relevant data from the event (maybe display update time and dot positions, for instance) and sticking those into an array. Then you wouldn’t waste memory storing stuff you don’t need (e.g. stimulus type names).

Of course, you can do the same thing using you current approach of partitioning your reads into time slices. My proposal might be a little simpler, though.

In any case, it’s your call as to how you want to process your data. I’m just trying to find ways to make it less painful for you

Chris

[cstawarz]

I’m just trying to find ways to make it less painful for you

I really appreciate that!

Does that make sense?

Yes, I think I understand what you want to do. This is a very elegant way to solve the problem but it would require a lot of work on my side (since the whole workgroup is using a different approach right now). Currently, partitioning the data into event groups and saving those to disk in separate mat files seems to be the easiest approach to avoid too much memory usage.
However, having the possibility to read incrementally would be great! I am just not sure of how quickly I will be able to make use of this once it is available. All those major changes need major testing and that needs time…

What do you think of putting this on a (low-priority) wish list since the biggest problems (I am finally able to read the data within 1-2h !!) are solved for now?

Best,
Philipp

[cstawarz]

What do you think of putting this on a (low-priority) wish list since the biggest problems (I am finally able to read the data within 1-2h !!) are solved for now?

OK, that sounds like a good plan.

Thanks,
Chris