Signal Handling with Ruby 2.0

A user recently ran Sidekiq with Ruby 2.0 and found that the signal handling did not work well at all. Ctrl-C and other signals resulted in some ominous stack traces.

It turns out that Ruby 2.0 locks down what you can do in a signal handler in order to prevent unsafe or possibly non-deterministic behavior. You can’t take a Mutex within a signal handler anymore as this could result in a thread context switch or even deadlock. In fact you can’t even write to a Logger because it tries to use a Mutex internally.

I rewrote the signal handling to conform with the new restrictions: all handlers now just push the name of the signal onto a global array and the main thread polls once per second for unhandled signals. This isn’t perfect, polling is something to be avoided where possible, but I don’t know of a better solution and it’s a lot better than the previous “fat” handlers that did a lot of work.

4 thoughts on “Signal Handling with Ruby 2.0”

  1. Hi.

    Certainly when writing C programs which employ signal handling, it’s generally recommended that you do as little as possible in a signal handler, so I can see the logic here. I think that the chief problem that’s being referred to here is that signals can be re-entrant–that is, a signal handler can be interrupted by another signal (or sometimes the same signal).

    With regard to threads, then , I’d suspect that the problem is that normally, the mutex locking functions (pthread_mutex_lock &c) depend on being able to run to completion. If you catch a signal halfway through calling a mutex function, then there’s a good chance that the internal state of the mutex won’t be consistent, and hence fail to behave properly (ie: possibly deadlock the calling thread mysteriously).

    A compounding problem is that signal handlers need to be re-entrant-safe, so if another signal gets delivered whilst in a signal handler, the first signal handler will be interrupted, thus increasing the probability of poking a mutex whilst in an invalid state.

    So I can understand the decision taken by the ruby core team, even if it is surprising. Mixing posix threads and signal handling usually is a bit of a nightmare.

    Hope that’s useful.

  2. Back in the days I used a pipe(2) with the read side in the main loop and a global write side to push a single character (possibly the signal number) from the signal handler to the main loop. This way you can use your current multiplexer, avoid polling and still have a nice list of all the triggered signals.

  3. I ran into this problem while making a unicorn-style forking server for rabbitMQ consumer processes – I’ve been digging into the Unicorn source code and noticed they also do the selfpipe trick mentioned by dubek and Victor above.

    It works well to encapsulate the entire thing in a class that acts like a blocking Queue (I guess similar to Thread::Queue but using instead of a mutex). The signal handlers PUSH into the queue and the primary thread does a blocking POP. It’s working quite well (so far) for my project

    The relevant class I’m using is here:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>