Memory leak identified (maybe)

[toby]

I had noticed that the memory usage reported by Union Admin
was on a steady upward path, occasionally falling back when
a garbage collect happened, but to higher levels than earlier.
In other words, it was a sawtooth graph like this:
15 20 25 ... 18 23 28 ... 21 26 31 ... and so on.

After a few days, it would have climbed into the 50's, 60's and higher.
I'm not sure how high it could get, because for various other reasons,
there would be periodic restarts, starting the process over.

After some experiments, I concluded that letting rooms DIE ON EMPTY,
and then re-creating them was the cause of the problem, and after changing
my app so that I leave empty rooms around and re-use them, it appears
that the memory usage remains more or less steady, topping out so far
(after just a few days) around 30-35 MB, then falling below 20 again sometimes.

The question then is: which side of the API does the leak lie on?
i.e. Is my code written incorrectly, or is the leak inside Union?

Here is what I do to create a new room:
Create a roomdef:
var roomDef = new RoomDef();

Set up about 8 attributes like this (each using a new AttributeDef):
var attr = new AttributeDef();
attr.setName(nm);
attr.setValue(val);
attr.setFlags(Attribute.FLAG_SHARED | Attribute.FLAG_SERVER_ONLY);
roomDef.addAttribute(attr);

var md = new ModuleDef(); (again a new one each time)
md.setType(ModuleDef.SCRIPT);
md.setSource("blah.js"); // same thing every time too
roomDef.addModule(md);

server.createRoom(roomDef);

This might occur dozens of times per day, amounting to hundreds in a few days.

When I reduced those hundreds to under a dozen (total), the perpetual increases
in memory use stopped. Just now, with the instance running for 3d 20h,
and having had 1200 client connections, and probably (but no longer known exactly)
400-500 room re-uses, the memory reported was just 18 MB, not much more than
when the server is first started.

[derek]

Hi toby,

I ran a test with the following profile:

- rooms die on empty
- rooms have a few attributes
- rooms have a javascript module attached

I ran the test for ~90 minutes and created over ~20000 rooms with a steady state of 1500-1700 concurrent rooms. Once the test hit the steady state the memory was steady so I was unable to reproduce your issue.

If you encounter the issue again or if you go back to your prior configuration could you please take a jmap snapshot of the memory and then send me the output so I can take a look.

Thanks!

Derek

[colin]

solving until we get a memory dump.

[toby]

(Marking unsolved pending answer to this follow-up question.)

Could this make any difference?

I create rooms on the server side by sending a room module message
to a "lobby" (which is itself a permanent room).
From ITS (javascript) module code, the other rooms are created.

Within the lobby room's module code it says

Code: Select all

(the code from earlier in this thread for the attributes and module definitions),
followed by
server.createRoom(roomDef);


While surely a client-side room create and a server side room create
eventually end up executing the same code, it's possible that some difference
in the paths they follow to get there is causing a problem.

I might also note that I am reusing room names,
so even though hundreds of rooms may be created,
only a limited number of names are used, such as
theRooms.001, theRooms.002, theRooms.003, etc. up to about 30 of them.
I would then "round robin" these names taking the next available each time,
even if a lower number were available because it had been emptied and died.

It's also possible that having users join the rooms and execute module code
causes a problem. For example, there might be a difference between just
having module code attached to a room, and actually using it.

I don't know if any of those things could make any difference, but since
your test did not reveal a problem, I am trying to identify anything that might
have been different.

One thing is certain: the behavior changed once I put in this change.
Right now, my system has been running for 9 days + and the memory
report says 19.16 MB. I would never see it that low after the first few
hours under the old method of creating rooms.

[toby]

Another update:

It seems that the memory usage can still creep up.
The server has been running for 15+ days now, and I'm seeing
Memory: over 40 MB, and when it goes back down,
it's only into the 30's.

QUESTION for followup:
When running jmap, which option should I use?