Sidekiq 3.0!

After tons of work on Sidekiq 2.x for the last 18 months, I decided it was time for some deeper refactoring and cleanup necessitating a major version bump.

Sidekiq 3.0 is the result of three months of hacking, cleanup and community suggestions. There’s an huge amount of stuff in here so hang on to your hats…
Continue reading

Dipping a Toe into Open Source

This is an excerpt from the foreword I wrote for Brandon Hilkert’s new e-book, Build a Ruby Gem.

I was a junior in college when I published my first open source program. It was Fall 1995 and Windows NT 3.5 had this fat and slow interface for launching applications. Windows 95 had an awesome new Start bar and so the answer was obvious: I decided I wanted to learn how to program the new Win32 API and solve my own problem at the same time. I set to write a lighter-weight, fast application launcher in the vein of the Start bar called AppBar.

Continue reading

Happy 2nd Birthday Sidekiq!

Let’s review some numbers, from the 1st birthday:

  • 214,300 downloads
  • 2144 stars
  • 662 closed issues
  • 266 forks
  • 228 closed pull requests
  • 44 versions released
  • 25 Sidekiq Pro customers

Now, on the 2nd birthday:

  • 1,192,259 downloads (wow, huge uptake!)
  • 3535 stars
  • 1420 closed issues
  • 563 forks
  • 380 closed pull requests
  • 74 versions released
  • Over 200 Sidekiq Pro customers

At this point I believe I’ve achieved my goals: build the best background job framework, bar none. With Sidekiq I try to have it all: good performance, easy setup, deep integration with an application framework like Rails and a rich set of functionality. I hope you think I was successful in my efforts.

As always, thank you to my users and keep ‘kiqing!

Don't Forget What's Important

Technologies come and go. We learn and grow as engineers over time but some things are eternal: knowing what is truly important to you is critical in differentiating between a path to misery versus fulfillment. Like your coworkers and your environment, the technology you work with day to day can make a big difference in your job satisfaction.

Continue reading

Advanced Sidekiq: Host-specific Queues

This is the first in a series of posts offering neat tricks to get the most out of Sidekiq.

Recently we rewrote part of The Clymb to process images asynchronously using Sidekiq. The user uploads the image file, it is saved to disk and a job created to process the file. Almost immediately we saw a bunch of retries with the error “Unable to find file xyz.jpg”. We just uploaded the file, how could it not be there?

The problem is that we have multiple app servers and they all run Unicorn and Sidekiq. This means the file can be uploaded to a Unicorn on app-1 and the job processed by a Sidekiq on app-2. The job queue is global to the cluster but the filesystem is local. The solution is a cool hack: use a queue which is processed only by Sidekiq processes on that server.

First we need to tell each Sidekiq process to listen to a queue named after the machine’s hostname. In your config/sidekiq.yml, do this:

---
:verbose: false
:concurrency: 25
:queues:
  - default
  - <%= `hostname`.strip %>

Sidekiq runs the YAML file through ERB automatically so you can easily add the queue dynamically.

Second, we need to configure the jobs to use the queue:

class ImageUploadProcessor
  include Sidekiq::Worker
  sidekiq_options queue: `hostname`.strip

  def perform(filename)
    # process image
  end
end

Now when we create an ImageUploadProcessor job, it will be saved to a queue named after the machine’s hostname and processed by a Sidekiq worker on that machine. Easy!

The Emperor has no Clothes

“In theory, theory and practice are the same. In practice, they are not.” — Albert Einstein

The original Dynamo paper created a wave of interest in the CAP theorem and gave rise to the recent crop of distributed databases: Cassandra, Riak, et al. These systems are generally AP where C can be tuned to provide some guarantee of consistency, i.e. they do their best to provide CAP according to the application’s needs. For instance, you might have a cluster of 5 nodes where a write to the cluster will return success if 3 of the nodes acknowledge the write. The cluster will still be available even if two of the machines fail.

In theory they are a great way to ensure availability to your application in the face of network failures. In practice, I believe these databases are so complex that they often provide less availability than a simpler CP system like a SQL database.
Continue reading

The Three Best Debugging Tools

1. Your Coworker

It’s happened to me over and over: I’ll spend hours trying to track down a problem. In frustration, I’ll ask a coworker to look at the code and frequently they will point out the problem in seconds. It’s called situational blindness and it means that you will often overlook the bug right in front of your eyes because you’ve looked at it so much, your mind has started to ignore it.

Your coworker has another valuable trait: they are different than you. They think differently and have a different set of knowledge. A hard bug for you to find might be easy for them. Next time you’re frustrated, call over a coworker and see what happens.

2. Your Creative Side

It’s happened to me over and over: I’ll build something with a flaw that doesn’t quite work right. I’ll go to sleep frustrated and wake up the next morning knowing exactly what the problem is.

It’s well known that we think in two different manners: creatively and logically. When focused on details (such as debugging a problem), we’re 100% logical as we jump from thought A to B to C. But it’s your creative side that can jump from thought A to thought Z via intuition. Like your coworker, your creative brain thinks differently from your logical brain. By going to sleep, you’re calling over that “creative coworker” and asking for help.

3. Your Mental Model of the System

If you don’t know the software tools and APIs you are using to solve problems really well, you’ll find yourself endlessly debugging. When learning a system like Ruby on Rails for the first few years, debugging is often really just learning how the system works. Software is insanely complex and every level of the stack is insanely complex. Abstractions hide much of the complexity and understanding those abstractions well allows you to fit it all within your head.

Malcom Gladwell’s maxim that expertise comes with 10,000 hours of practice holds here too. After 5 years of full-time experience (40hrs/wk * 50 wk/yr * 5), you’ll have a very strong understanding of your system of choice. That knowledge is very valuable.

Pro tip: read the documentation for a library that you are going to use, implement your first attempt at using that library, and then read the documentation again. The first time will give you an overview and things to look for but frequently you won’t comprehend certain aspects due to your inexperience with the library. You almost certainly will comprehend more when you read it the second time.

Conclusion

All of these things have one thing in common: thought. Debugging is thinking through the execution of code in your head. By improving your knowledge and looking at the problem from different angles, you can much more effectively debug those hard problems that can be so frustrating.

On Ruby, Software and the Internet