Memory-hungry Ruby daemons

We’ve had a perplexing issue with our Ruby daemons at OneSpot: they seem to grow to 300-400MB each within about 30 minutes, at which point our Monit scripts restart them. We suspected a memory leak and so upgraded from stock Ruby 1.8.5 shipped with CentOS to the latest REE 1.8.6 but nothing changed. I also saw a very similar issue at FiveRuns. Why is this problem seemingly endemic, even with completely different source code? After some thought and research I think I understand the root clause of the problem: it’s part of Ruby’s history and design.

Memory Management in Ruby

Ruby uses 5 constants to control how it manages an application’s heap, 3 of which are important to this discussion. From the REE user’s guide:

    This specifies the initial number of heap slots. The default is 10000.
    The number of additional heap slots to allocate when Ruby needs to allocate new heap slots for the first time. The default is 10000.

    For example, suppose that the default GC settings are in effect, and 10000 Ruby objects exist on the heap (= 10000 used heap slots). When the program creates another object, Ruby will allocate a new heap with 10000 heap slots in it. There are now 20000 heap slots in total, of which 10001 are used and 9999 are unused.

    Multiplicator used for calculating the number of new heaps slots to allocate next time Ruby needs new heap slots. The default is 1.8.

    Take the program in the last example. Suppose that the program creates 10000 more objects. Upon creating the 10000th object, Ruby needs to allocate another heap. This heap will have 10000 * 1.8 = 18000 heap slots. There are now 20000 + 18000 = 38000 heap slots in total, of which 20001 are used and 17999 are unused.

    The next time Ruby needs to allocate a new heap, that heap will have 18000 * 1.8 = 32400 heap slots.

So MRI will initially allocate the application RUBY_HEAP_MIN_SLOTS or 10,000 slots. Let’s assume for ease of math that this corresponds to 1MB of memory. Now Rails and our application code can’t fit into anything less than 50MB so Ruby will need to allocate additional heaps for the necessary objects. It does this by using RUBY_HEAP_SLOTS_INCREMENT and RUBY_HEAP_SLOTS_GROWTH_FACTOR each time. So we allocate 1.8MB, 3.24, 5.83, 10.5, 18.9, 34, 61, 110, 198, … where the size of the newest heap is expanded by 1.8x each time. As you can see, just to get us to our 50MB minimum, we’re now allocating 34MB for the latest heap. Once the app starts actually processing data, we’ll allocate 61 and then 110 MB!

This is the core of the problem: loading Rails expands the Ruby process so much that additional memory allocation grows much larger than we actually need, due to the exponential growth factor. And since MRI never gives back unused memory, our daemon can easily be taking 300-400MB when it’s only using 100-200.

It’s important to note that this is essentially by design. Ruby’s history is mostly as a command line tool for text processing and therefore it values quick startup and a small memory footprint. It was not designed for long-running daemon/server processes. Java makes a similar tradeoff in its client and server VMs.

Our solution was to move to Ruby Enterprise Edition. It allows those constants to be modified via environment variables, so that you can greatly increase MIN_SLOTS and greatly reduce GROWTH_FACTOR. Our settings:

export RUBY_HEAP_MIN_SLOTS=800000

That gives our daemon ~80MB to start and each heap is a fixed 10MB. Our daemon stabilizes at ~120MB and the memory usage doesn’t change, even after hours of processing. My takeaway: if you own a Ruby daemon, you need to tune the heap to ensure it does not take too much memory!

7 thoughts on “Memory-hungry Ruby daemons”

  1. You could also require less of Rails by just including the absolute minimum (ActiveRecord/ActiveResource and ActiveSupport in most cases) if you’re not already, which would save some space. You normally don’t need to include what’s in ActionController et al.

  2. Oh I got the impression that he was speaking about the application processes, which probably DO require ActionPack. Matt: sounds like you’re talking about long-running ruby job queue processes, which don’t need ActionPack

  3. My daemon happens to be an SQS queue processing daemon but has to load much of Rails because it uses ActionMailer. I’ll probably move to merb-mailer so I can remove ActionPack.

  4. “MRI never gives back unused memory” is not entirely true. Ruby will free unused struct heaps_slot when free_unused_heaps() is called at the end of gc_sweep() and after rb_gc_finalize_deferred().

    You should be able to see the resident set size go down (RSS) from time to time as well, but this depends in part on how your OS’s malloc(3) is implemented and how lucky you are in your allocation patterns.

    Only VALUEs are stored on the ruby’s heap, not the objects themselves [hand wave over Fixnum]. Arrays and Strings are allocated on the C heap [hand wave over 1.9 behavior for small Arrays and Strings].

    Here’s a thought experiment:

    Some objects are allocated that cause a new struct heaps_slot to be allocated, then a bunch of objects Strings or Arrays are allocated on the C heap. Memory will look like this:

    [old stuff, struct heaps_slot, Array, String, Array]

    Then there is a GC pass and one of those items are appended to an object that ends up in an earlier heaps_slot (the object is long lived). Finally the recently allocated objects and their struct heaps_slot are freed. Memory will look like this:

    [old stuff, free, free, free, Array]

    The free space will still count against you in your processes size. Until the OS swaps out the free pages it may even count against you in RSS.

    The problem is that some memory allocation patterns cause fragmentation both in the process address space and the struct heaps_slot address space.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>