Sidekiq and Upstart
The best and most reliable way to manage multiple Sidekiq processes is with Upstart. Many developers know little to nothing about Upstart so I wanted to write up how to integrate Sidekiq with Upstart. With Upstart doing the hard work, it becomes easy to manage deployments with Capistrano or another similar tool.
The Sidekiq repo has example .conf files you can use as a template to create your
own services. Customize the .conf files as necessary and place them in
they tell Upstart when and how to start the associated processes.
workers service is a “fake” service which knows how to start/stop N sidekiq
processes. Upon machine boot, Upstart will start
workers which will start those N
workers.conf we’ve declared how many processes we want to start:
If you want to quickly shut down all Sidekiq processes, run
stop workers. Start
them back up with
start workers. Of course you can do both with
It literally can’t be any easier!
sidekiq service is an “instance” service, allowing you to create N processes.
It requires an index parameter to define which instance you are controlling:
$ start sidekiq index=0
$ start sidekiq index=1
$ stop sidekiq index=2
Deployment should do two things: quiet Sidekiq as early as possible and restart as late as possible.
During a deployment, we want to signal Sidekiq to stop processing jobs as early as possible so Sidekiq has as much time as possible to finish any jobs in progress. We do this by sending each process the USR1 signal, here’s a Capistrano task which does that:
task :quiet do
on roles(:worker) do
puts capture("sudo pgrep -f 'sidekiq' | xargs kill -USR1")
workers does not support reload since it doesn’t map to a single process so we have to
use that pgrep hack.
You can use Upstart’s
reload command to quiet a specific instance:
$ reload sidekiq index=X
Restarting is easy:
restart workers. This will actually stop and then start the processes.
task :restart do
on roles(:worker) do
puts capture("sudo restart workers")
- We don’t need to daemonize. Modern daemons should never daemonize themselves.
- We don’t need PID files. PID files are legacy from years ago and their use should signal that something is wrong.
- We don’t need to specify our own log files. Sidekiq will output to stdout; Upstart will direct stdout to
a file within
/var/log/upstart/and automatically rotate those log files for you - no logrotate setup necessary!
In other words, stop reinventing the wheel and let the operating system do the hard work for you! I hope this makes your Sidekiq deployment cleaner and more stable.