Signal Handling with Ruby 2.0

A user recently ran Sidekiq with Ruby 2.0 and found that the signal handling did not work well at all. Ctrl-C and other signals resulted in some ominous stack traces.

It turns out that Ruby 2.0 locks down what you can do in a signal handler in order to prevent unsafe or possibly non-deterministic behavior. You can’t take a Mutex within a signal handler anymore as this could result in a thread context switch or even deadlock. In fact you can’t even write to a Logger because it tries to use a Mutex internally.

I rewrote the signal handling to conform with the new restrictions: all handlers now just push the name of the signal onto a global array and the main thread polls once per second for unhandled signals. This isn’t perfect, polling is something to be avoided where possible, but I don’t know of a better solution and it’s a lot better than the previous “fat” handlers that did a lot of work.

Bill Gates

Some back of the envelope numbers I was pondering this morning:

40 hours / week * 50 weeks / year = 2000 hr / yr

Bill Gates is worth about $66 billion. He’s worked for about 33 years now.

66 billion / 33 years = $2 billion/year / 2000 = $1 million / hour.

Bill Gates has made roughly $1 million per hour over the length of his career. Most people would be happy to make that amount in a decade. No moral or judgment here, just an astonishing number to think about.

Happy Birthday Sidekiq!

This day last year I released Sidekiq 0.5.0 to the public. I left Carbon Five in mid-January and was moving to Portland to join The Clymb with wife and child in tow. For the previous few months, I’d been thinking about background processing and felt there was opportunity for a library that was more efficient and also provided more functionality. I spent my two weeks of downtime building what became Sidekiq.
Continue reading

One Great Year at The Clymb

I joined The Clymb one year ago today and what an amazing year it’s been.

Some context for you: I was employee 33 and the third engineer to join. The test suite had hundreds of failures and had been running red for months. We were on REE, Rails 2.3 and Delayed Job – a stack that was current two years before – and had no error service to know what errors were happening in production. The site was deployed with fabric, a Python-based tool. The first two developers were overworked and fought fires all day, every day.
Continue reading

12 Gems of Christmas #1 – puma

I’ve spent the last few years working to advance and improve Ruby’s efficiency through concurrency, first with EventMachine and fibers and now with Actors and multithreading so it shouldn’t surprise you that my #1 pick is puma. It’s my belief that puma and sidekiq are a new breed of Ruby infrastructure that can dramatically improve your application’s efficiency — should you decide to take advantage of them.

puma is a pure Ruby, Rack-based web server and drops right in as a replacement for thin or unicorn. Unlike unicorn or thin, puma is designed to run multithreaded by default so you get far better memory efficiency. A typical single threaded Rails unicorn process takes 250MB. puma defaults to a maximum of 16 threads per process so one puma process can replace 16 unicorn processes taking 4GB of RAM! puma, like all multithreaded libraries, works best in a truly concurrent Ruby VM like JRuby or Rubinius but you’ll still get a big win running on MRI.

To test this, I ran 50 concurrent requests 20 times for a total of 1000 requests against a non-trivial endpoint on Rails application. config.threadsafe! was enabled, a database pool size of 10 and puma’s default of 16 threads. Each request makes two database queries and renders a slim-based template.

Unicorn/MRI 1.9.3 is the baseline: single-threaded, it runs the 1000 requests in 19 seconds. Puma/MRI manages to speed up a bit but is still hampered by the GIL and runs in 15 seconds. Puma/JRuby unlocks the second core on my MacBook Air and runs in under 9 seconds!

What this means is simple: threading with puma will get you better performance than Unicorn, even on MRI, and jumping to JRuby gets you a significantly bigger boost by giving you truly parallel threads. It took me about one hour to get our Rails app, which has always run on MRI, working with JRuby. Give JRuby a try some weekend and you might be surprised how well it works!

I hope you enjoyed my 12 Gems of Christmas series and found a few gems that were worthy of further study.

12 Gems of Christmas #2 – bullet

ActiveRecord has been a huge boon for web development in promoting conventions in databases. Every new company I joined we had to determine how to name tables, primary keys and indexes. Standardizing id, automatic timestamps, pluralizing nouns, all of it makes development easier and more friendly to developers who just want to build stuff, not worry about every mundane detail.

All is not wine and roses though, ActiveRecord suffers from a common ORM performance issue: the N+1 query problem. Find all shopping carts created in the last month and print out the number of items in those carts. This is what you might see in your terminal:

select * from shopping_carts where created_at > 1.month.ago
select count(*) from shopping_cart_items where shopping_cart_id = 111
select count(*) from shopping_cart_items where shopping_cart_id = 222
select count(*) from shopping_cart_items where shopping_cart_id = 333
select count(*) from shopping_cart_items where shopping_cart_id = 444
select count(*) from shopping_cart_items where shopping_cart_id = 555

This is because the initial query loads the data associated with the shopping_carts but does not load the associated items. As you iterate through each cart, Rails lazy loads the item count; unfortunately it does it one cart at a time.

Enter bullet which aims to help you find and fix any N+1 query issues in your ActiveRecord code. Add it to your Gemfile and activate it in your Rails configuration, soon you’ll see warnings like this:

Unused Eager Loading detected
  Brand => [:products]
  Remove from your finder: :include => [:products]

As we learned when we were younger, knowing is half the problem. These performance problems should be easy to track down and tune now that you know about the problem; you’ll find your page rendering times plummet since the number of database queries performed will drop dramatically.

Tomorrow we’ll discuss my favorite subject, concurrency, and unveil the #1 gem!

12 Gems of Christmas #3 – thor

Ever try to write a rake task that took one or more arguments? How about calling a rake task from another task? Rake makes basic invocation simple but everything else obtuse. Enter thor, courtesy of the unstoppable Yehuda Katz. Thor aims to make command line script development simpler by making your scripts standard Ruby objects and providing good documentation in its wiki for many common use cases. Consider a simple example, create a task to process a CSV file:

# inventory.thor
class Inventory < Thor
  include Thor::Actions

  desc "process_csv FILE", "process the nightly inventory update"
  method_option :delete, :aliases => "-d", :desc => "Delete the file after parsing it"
  def process_csv(file)
    # do something, maybe like:
    #require 'inventory_file'!
    remove_file(file) if options[:delete]

You invoke the task via thor inventory:process_csv some_file.csv. Tasks are just public methods within a subclass of Thor. The desc and method_option methods describe those tasks and any expected options on the command line. Thor::Actions gives us some filesystem helper methods, like remove_file. The wiki goes into further detail on things like inter-task dependencies, task grouping, namespaces, etc. Next time you need to write a command line script, give Thor a try and see if it doesn’t make your life easier.

Tomorrow, I’ll show you a tool for detecting and fixing a common Rails performance problem.

12 Gems of Christmas #4 – mailcatcher and mail_view

Surprise, today’s blog posting is a double header!

Testing email can be painful, verifying delivery and look and feel quickly becomes tedious. mailcatcher and mail_view are my go-to tools for dealing with email.


With mailcatcher, you install the gem and then run the mailcatcher binary to start the mailcatcher daemon in the background. Now you have an SMTP server running on port 1025 and an inbox available at http://localhost:1080. I configured The Clymb Rails app to automatically use mailcatcher if it’s running:

# config/initializers/email.rb
  sock ="localhost", 1025)
  catcher = true
  catcher = false

ActionMailer::Base.smtp_settings = if Rails.env.development? && catcher
  { :host => "localhost", :port => '1025', }
  { } # normal SMTP config

Now you can test email via http://localhost:1080 and not have to use gmail, hotmail, etc.


mail_view is awesome for rapidly testing UI changes in email. The trick is to render the email as a webpage directly in your Rails app so you can see changes immediately with Cmd+R just like any normal webpage. You just need to set up the email to be rendered with some data; in our case we just pull random data from the database since we all work with production database snapshots.

class Preview < MailView
  def order_receipt
    cart = Cart.where(:state => 'pushed').order(:id).last

Of course, we need to mount the mail_view engine itself:

# config/routes.rb
mount MailPreview => 'mail_view' if Rails.env.development?

Now we can go to http://localhost:3000/mail_view/order_receipt and immediately see what our order_receipt email looks like, no email required!

With mailcatcher and mail_view at your side, email development and testing goes from tedious to stupendous! Tomorrow we’ll cover a gem designed to build better command line scripts in Ruby…

12 Gems of Christmas #5 – lograge

Rails has a problem: its production logging is somewhat lacking. By default, Rails will emit a bunch of lines for each request, which makes grepping the output tougher than it should be. Look at this, it’s nice to look at but painful to aggregate:

Started GET "/" for at 2012-03-10 14:28:14 +0100
Processing by HomeController#index as HTML
  Rendered text template within layouts/application (0.0ms)
  Rendered layouts/_assets.html.erb (2.0ms)
  Rendered layouts/_top.html.erb (2.6ms)
  Rendered layouts/_about.html.erb (0.3ms)
  Rendered layouts/_google_analytics.html.erb (0.4ms)
Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

Lograge ‘tames’ Rails production logging setup to emit one line per request, a la Apache or nginx. The output is key-value pairs; parsing it into a hash is trivial. Here’s some output from The Clymb production logging with lograge enabled:

Nov 24 13:31:27 app-1 rails[22622]: method=GET path=/brand-event/2818/show-product/183537?f=mi format=html controller=presentations action=show_product status=200.00 duration=237.54 view=206.14 db=17.12
Nov 24 13:31:28 app-1 rails[22630]: method=GET path=/brand-event/2818/show-product/183539?f=mi format=html controller=presentations action=show_product status=200.00 duration=125.37 view=94.05 db=17.59
Nov 24 13:31:28 app-1 rails[22622]: method=GET path=/brand-event/2818/show-product/183541?f=mi format=html controller=presentations action=show_product status=200.00 duration=117.47 view=86.00 db=16.79
Nov 24 13:31:28 app-1 rails[22630]: method=GET path=/brand-event/2818/show-product/183548?f=mi format=html controller=presentations action=show_product status=200.00 duration=132.09 view=97.69 db=20.69
Nov 24 13:31:28 app-1 rails[22622]: method=GET path=/brand-event/2818/show-product/183554?f=mi format=html controller=presentations action=show_product status=200.00 duration=119.36 view=84.44 db=18.87
Nov 24 13:31:29 app-1 rails[22622]: method=GET path=/basket/tracking_pixels format=html controller=baskets action=tracking_pixels status=200.00 duration=71.47 view=41.00 db=23.65

Ah, much cleaner! (and notice the HTTP status value is a float. :-) This format gives us the critical request information required for performance tuning and is easily digestible by simple scripts. Enabling it is trivial:

# config/environments/production.rb
MyApp::Application.configure do
  config.lograge.enabled = true

Much happier logs lead to much happier developers. Happy logging! Tomorrow I’ll cover a gem (or two!?) designed to make email development happier…