Advanced Sidekiq: Host-specific Queues

This is the first in a series of posts offering neat tricks to get the most out of Sidekiq.

Recently we rewrote part of The Clymb to process images asynchronously using Sidekiq. The user uploads the image file, it is saved to disk and a job created to process the file. Almost immediately we saw a bunch of retries with the error “Unable to find file xyz.jpg”. We just uploaded the file, how could it not be there?

The problem is that we have multiple app servers and they all run Unicorn and Sidekiq. This means the file can be uploaded to a Unicorn on app-1 and the job processed by a Sidekiq on app-2. The job queue is global to the cluster but the filesystem is local. The solution is a cool hack: use a queue which is processed only by Sidekiq processes on that server.

First we need to tell each Sidekiq process to listen to a queue named after the machine’s hostname. In your config/sidekiq.yml, do this:

---
:verbose: false
:concurrency: 25
:queues:
  - default
  - <%= `hostname`.strip %>

Sidekiq runs the YAML file through ERB automatically so you can easily add the queue dynamically.

Second, we need to configure the jobs to use the queue:

class ImageUploadProcessor
  include Sidekiq::Worker
  sidekiq_options queue: `hostname`.strip

  def perform(filename)
    # process image
  end
end

Now when we create an ImageUploadProcessor job, it will be saved to a queue named after the machine’s hostname and processed by a Sidekiq worker on that machine. Easy!

3 thoughts on “Advanced Sidekiq: Host-specific Queues”

  1. That is a sweet hack. I could see how this is super useful for many people.

    Though I would probably use Socket.gethostname over the shell since I don’t trust shell commands in ruby.

    Really love it though I’ll have to use this sometime.

  2. This is precisely why I wrote Fragmenter, a gem that managed storing image parts in redis and then rebuilding them as a background job. Multiple servers can accept uploaded image fragments and when the last section is completed a sidekiq job will pull all the fragments down and rebuild it.

    Works wonderfully for slow mobile clients.

    https://github.com/dscout/fragmenter

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>