Sidekiq for Crystal

2016-05-25

Sidekiq is a popular job framework for Ruby. Now we’re bringing it to Crystal!

Why Crystal?

I wanted to use a language that was a good complement to Ruby. Its syntax is similar enough to Ruby that I can reuse a lot of code but it adds a huge leap in performance. In summary:

• very similiar syntax to Ruby
• at least 3-5x faster than Ruby 2.3 on most code
• at least 3x smaller in memory footprint
• statically typed
• compiles to a single, 1MB binary! Deployment is easy.
• comes with a large, useful standard library

In other words, Ruby is friendly, flexible and works for most usecases while Crystal is fast and efficient for those usecases where performance is paramount. Use each where appropriate.

How productive is it? I started this project from scratch not knowing the language at all a week ago and had the core job processor running in 3 days.

Gimme!

The project resides at mperham/sidekiq.cr. To get started:

brew update
brew install crystal-lang
# brew install redis
git clone git://github.com/mperham/sidekiq.cr.git
cd sidekiq.cr
crystal deps
make

If you have Redis and the sidekiq gem installed, you can run the benchmarks:

brew install redis
gem install sidekiq
make bench

This is the result for me: with zero optimization on my part, Crystal is 3.6x faster and 3x smaller. To process 100,000 noop jobs:

The codebase is a trainwreck though, right?

The code is shockingly similar to Ruby in many cases. Take a gander at this testcase:

require "./spec_helper"

class MyWorker
  include Sidekiq::Worker

  def perform(a : Int64, b : Int64, c : String)
    #puts "hello world!"
  end
end

describe Sidekiq::Worker do
  describe "client-side" do
    it "can create a basic job" do
      jid = MyWorker.async.perform(1_i64, 2_i64, "3")
      jid.should match /[a-f0-9]{24}/
      pool = Sidekiq::Pool.new
      pool.redis { |c| c.lpop("queue:default") }
    end

    it "can schedule a basic job" do
      jid = MyWorker.async.perform_in(60.seconds, 1_i64, 2_i64, "3")
      jid.should match /[a-f0-9]{24}/
    end

With the exception of a few type hints, that’s identical to Ruby.

Cool, just gonna push this to production…

Whoa, this project is alpha. Hold off porting your nuclear reactor controller code for another week or two, ok? Major functionality is missing, (notably the data API and Web UI), the test suite is still baking, etc. Take it for a test drive and let me know how it goes for you.

Looking for other libraries written in Crystal? Check out the CrystalShards listing. AwesomeCrystal is another curated list of resources. You can find database drivers, web frameworks, etc.

What about Sidekiq Enterprise?

Based on demand, I will port the Sidekiq Pro and Enterprise functionality to Crystal. If you are interested, email me.

Conclusion

Let’s make Sidekiq.cr amazing, try it out and help us improve it!