12 Gems of Christmas #3 – thor

Ever try to write a rake task that took one or more arguments? How about calling a rake task from another task? Rake makes basic invocation simple but everything else obtuse. Enter thor, courtesy of the unstoppable Yehuda Katz. Thor aims to make command line script development simpler by making your scripts standard Ruby objects and providing good documentation in its wiki for many common use cases. Consider a simple example, create a task to process a CSV file:

# inventory.thor
class Inventory < Thor
  include Thor::Actions

  desc "process_csv FILE", "process the nightly inventory update"
  method_option :delete, :aliases => "-d", :desc => "Delete the file after parsing it"
  def process_csv(file)
    # do something, maybe like:
    #require 'inventory_file'
    remove_file(file) if options[:delete]

You invoke the task via thor inventory:process_csv some_file.csv. Tasks are just public methods within a subclass of Thor. The desc and method_option methods describe those tasks and any expected options on the command line. Thor::Actions gives us some filesystem helper methods, like remove_file. The wiki goes into further detail on things like inter-task dependencies, task grouping, namespaces, etc. Next time you need to write a command line script, give Thor a try and see if it doesn’t make your life easier.

Tomorrow, I’ll show you a tool for detecting and fixing a common Rails performance problem.

12 Gems of Christmas #4 – mailcatcher and mail_view

Surprise, today’s blog posting is a double header!

Testing email can be painful, verifying delivery and look and feel quickly becomes tedious. mailcatcher and mail_view are my go-to tools for dealing with email.


With mailcatcher, you install the gem and then run the mailcatcher binary to start the mailcatcher daemon in the background. Now you have an SMTP server running on port 1025 and an inbox available at http://localhost:1080. I configured The Clymb Rails app to automatically use mailcatcher if it’s running:

# config/initializers/email.rb
  sock = TCPSocket.new("localhost", 1025)
  catcher = true
  catcher = false

ActionMailer::Base.smtp_settings = if Rails.env.development? && catcher
  { :host => "localhost", :port => '1025', }
  { } # normal SMTP config

Now you can test email via http://localhost:1080 and not have to use gmail, hotmail, etc.


mail_view is awesome for rapidly testing UI changes in email. The trick is to render the email as a webpage directly in your Rails app so you can see changes immediately with Cmd+R just like any normal webpage. You just need to set up the email to be rendered with some data; in our case we just pull random data from the database since we all work with production database snapshots.

class Preview < MailView
  def order_receipt
    cart = Cart.where(:state => 'pushed').order(:id).last

Of course, we need to mount the mail_view engine itself:

# config/routes.rb
mount MailPreview => 'mail_view' if Rails.env.development?

Now we can go to http://localhost:3000/mail_view/order_receipt and immediately see what our order_receipt email looks like, no email required!

With mailcatcher and mail_view at your side, email development and testing goes from tedious to stupendous! Tomorrow we’ll cover a gem designed to build better command line scripts in Ruby…

12 Gems of Christmas #5 – lograge

Rails has a problem: its production logging is somewhat lacking. By default, Rails will emit a bunch of lines for each request, which makes grepping the output tougher than it should be. Look at this, it’s nice to look at but painful to aggregate:

Started GET "/" for at 2012-03-10 14:28:14 +0100
Processing by HomeController#index as HTML
  Rendered text template within layouts/application (0.0ms)
  Rendered layouts/_assets.html.erb (2.0ms)
  Rendered layouts/_top.html.erb (2.6ms)
  Rendered layouts/_about.html.erb (0.3ms)
  Rendered layouts/_google_analytics.html.erb (0.4ms)
Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

Lograge ‘tames’ Rails production logging setup to emit one line per request, a la Apache or nginx. The output is key-value pairs; parsing it into a hash is trivial. Here’s some output from The Clymb production logging with lograge enabled:

Nov 24 13:31:27 app-1 rails[22622]: method=GET path=/brand-event/2818/show-product/183537?f=mi format=html controller=presentations action=show_product status=200.00 duration=237.54 view=206.14 db=17.12
Nov 24 13:31:28 app-1 rails[22630]: method=GET path=/brand-event/2818/show-product/183539?f=mi format=html controller=presentations action=show_product status=200.00 duration=125.37 view=94.05 db=17.59
Nov 24 13:31:28 app-1 rails[22622]: method=GET path=/brand-event/2818/show-product/183541?f=mi format=html controller=presentations action=show_product status=200.00 duration=117.47 view=86.00 db=16.79
Nov 24 13:31:28 app-1 rails[22630]: method=GET path=/brand-event/2818/show-product/183548?f=mi format=html controller=presentations action=show_product status=200.00 duration=132.09 view=97.69 db=20.69
Nov 24 13:31:28 app-1 rails[22622]: method=GET path=/brand-event/2818/show-product/183554?f=mi format=html controller=presentations action=show_product status=200.00 duration=119.36 view=84.44 db=18.87
Nov 24 13:31:29 app-1 rails[22622]: method=GET path=/basket/tracking_pixels format=html controller=baskets action=tracking_pixels status=200.00 duration=71.47 view=41.00 db=23.65

Ah, much cleaner! (and notice the HTTP status value is a float. :-) This format gives us the critical request information required for performance tuning and is easily digestible by simple scripts. Enabling it is trivial:

# config/environments/production.rb
MyApp::Application.configure do
  config.lograge.enabled = true

Much happier logs lead to much happier developers. Happy logging! Tomorrow I’ll cover a gem (or two!?) designed to make email development happier…

12 Gems of Christmas #6 – childprocess

Ruby has many different gems and APIs for spawning child processes but none I like more than childprocess. posix-spawn and open4 are popular alternatives but childprocess has a clean Ruby API and aims to work on all Ruby platforms.

Here we have an example setting up a child process pipeline, performing the equivalent of ps aux | grep -E 'redis|memcached'.

require 'childprocess'

listing = ChildProcess.build("ps", "aux")

search = ChildProcess.build("grep", '-E', %w(redis memcached).join('|'))
search.duplex = true
search.io.stdout = STDOUT

listing.io.stdout = search.io.stdin


With subprocesses, the tricky bit is usually hooking up the IO streams correctly. In this case, the duplex flag signals that you want to write to STDIN of the process (i.e. use it as the RHS of a pipe). Everything else is pretty straightforward; childprocess makes multi-processing with Ruby simple.

Tomorrow I’ll dive into a little gem which aims to improve Rails logging in production.

12 Gems of Christmas #7 – lunchy

I couldn’t resist covering one of my own gems in the 12 Gems of Christmas. While I’m best known for dalli and sidekiq, lunchy has little place in my heart because it solves a minor annoyance all OSX-based developers have: starting and stopping system daemons.

Here’s how everyone does it with standard OSX:

launchctl load [oh, what's the blasted filename?]
cd ~/Library/LaunchAgents
launchctl load ~/Library/LaunchAgents/io.redis.redis-server.plist

That’s a lot of typing and annoyance every time you want to start Redis. Here’s how you do it in lunchy:

lunchy start redis

No need to remember the filename of the plist file, lunchy just scans for a plist with the given substring in it. lunchy ls will show you all the agents you can control. See the README for the full command set available and happy l(a)unching!

Tomorrow I’ll show you an elegant gem for starting and managing your own child processes!

12 Gems of Christmas #8 – wicked_pdf

We recently rolled out gift certificates in time for the holiday shopping season at The Clymb and one of our major implementation hurdles was a very innocent feature request: send the certificate via email as a PDF.

How on earth do I create a PDF? I’m a web developer – HTML/CSS is no problem – but PDF is a binary format and making a PDF with a “nice” design is way out of my bailiwick. wicked_pdf to the rescue! wicked_pdf wraps the wkhtmltopdf command line tool which can convert an HTML file into a PDF file for offline access or printing purposes. Our email code is simple: we render the HTML content and then pass it to WickedPdf for conversion.

pdf_str = render_to_string(:template => 'emails/gift_certificate',
                           :layout => false,
                           :locals => { :amount => number_to_currency(amount),
                                        :product_code => pc.code } )
pdfs << WickedPdf.new.pdf_from_string(pdf_str)

With wicked_pdf, we didn't have to sacrifice the tools and technologies we know best in order to support PDF files. We rolled out the feature on time and no web developers were harmed in the process.

I couldn't resist this platform to plug one of my own gems; tomorrow I'll cover my own favorite, under-utilized creation.

12 Gems of Christmas #9 – pundit

When it comes to authorization, Ryan Bates’ CanCan gem is the 800 lb gorilla that most Rails apps use. pundit is the latest gem from Jonas Nicklas (author of Carrierwave and Capybara) with some interesting ideas that you might like. Pundit uses code conventions along with a plain old Ruby API to make for a very simple implementation. First, write a policy for each type of model you wish to authorize:

class PostPolicy < Struct.new(:user, :post)
  def create?
    user.admin? or not post.published?

Then in the corresponding controller, use authorize to verify permissions:

def create
  @post = Post.new(params[:post])
  authorize @post
  if @post.save
    redirect_to @post
    render :new

Pundit assumes the current user is available via current_user within the controller and passes it to your policy along with the model instance.

There's a few more features to be discovered over on the GitHub README but the entire library is less than 200 lines of code; it's beautifully succinct.

Next up, I'll cover a handy little gem for creating nice PDF files without having to dive into the complexities of the PDF format.

12 Gems of Christmas #10 – konacha

It’s got a crazy name but solves a serious problem that continues to vex the Rails community: javascript testing. There are a number of libraries out there that try to solve the problem but konacha is the first I’m aware of that integrates tightly with Rails 3 and Sprockets to make JavaScript testing as easy as possible within your Rails app. For instance, you might have spec/javascripts/array_sum_spec.js:

//= require array_sum

describe("Array#sum", function() {
  it("returns 0 when the Array is empty", function() {

  it("returns the sum of numeric elements", function() {

You can load HTML template fixtures via Sprocket’s JS template support. For example, in spec/javascripts/templates/hello.jst.ejs:

Hello Konacha!

And your spec:

//= require_tree ./templates

describe("templating", function() {
  it("is built in to Sprockets", function() {
    $('body h1').text().should.equal('Hello Konacha!');

I’m really impressed by the quality of code and documentation here. Kudos to John Firebaugh and his contributors! Next time you write some JavaScript, take a look at Konacha and see if it can help you test that code automatically.

Tomorrow we’ll take a look at a brand new gem with a fresh take on authorization.

12 Gems of Christmas #11 – statsd-ruby

As software developers, it’s often our job to monitor the systems we build. Step one is to collect the data necessary for monitoring.

Statsd is a nice metrics aggregation server open sourced by Etsy. We use Statsd at The Clymb to collect and aggregate various technical and business metrics before uploading them to Librato Metrics for storage and display on our dashboards.

The statsd-ruby client sends metrics from all our Unicorn and Sidekiq processes to statsd. One thing I really like about statsd is that it uses UDP for network transport so sending a metric doesn’t incur much overhead at all and I don’t have to have statsd installed locally or in staging – the metric send will just silently fail. We set up a global METRICS variable with a Statsd client instance (instances are thread-safe so this is ok, even with Sidekiq) so our code can send metrics trivially:

statsd_host = Rails.env.production? ? '' : ''
METRICS = Statsd.new(statsd_host, 8125)
METRICS.namespace = (Sidekiq.server? ? 'sidekiq' : 'web')

Now we just sprinkle metrics throughout our code. Want to track the number and value of orders that your e-commerce site gets?

METRICS.count('orders.total', @basket.total) if @basket.total > 0

Remember that we might have hundreds of Unicorn processes so statsd will collect and aggregate all the individual metric packets from each process into a single metric value to upload to your metrics service. With statsd and statsd-ruby working for you, you can monitor the metrics important to your business and verify the behavior of your system at runtime.

Tomorrow we’ll discuss a bane of the Rails world: testing JavaScript!

12 Gems of Christmas #12 – awesome_nested_set

Category trees are standard e-commerce functionality, allowing the user to filter their results when searching for products. Here’s a category tree from Amazon.com:

So if you have a table which holds those categories, how do you write a SQL query which loads a given category PLUS all child categories below it? The answer is that it’s extremely difficult/impossible with standard SQL. Oracle can do it with their SQL “connect by” extension or you can change your tree structure into a nested set.

The awesome_nested_set gem adds really useful tree functionality to your category table. Need to query for all products in a given category or below in the category tree?

class Category < ActiveRecord::Base
  has_many :products
  belongs_to :parent, :class_name => 'Category'
  attr_accessible :name, :parent_id, :parent

class Product < ActiveRecord::Base
  belongs_to :category
  attr_accessible :category_id, :name, :category

cat = Category.find_by_name("Electronics")
# look up all children in one query
subcats = cat.self_and_descendants

# Find all products within Electronics subtree with one query.
# A bit messy.
products = Product.active.joins(:category).where('categories.lft > ? and categories.lft <= ?', cat.lft, cat.rgt)

awesome_nested_set is missing some critical documentation, the README explains how to set it up but doesn't cover queries and scopes at all. I couldn't find a way to do something cleaner like so without rolling my own scope:

cat = Category.find_by_name("Electronics")
products = Product.active.within_category(cat)

Is this possible? Could the category association provide some built-in nested set scopes? Nevertheless, adding a few custom scopes is a small price to pay: nested sets turn a very hard SQL problem into something easily solved.

UPDATE: I learned of two nice alternatives to awesome_nested_set: ancestry and closure_tree.

Tomorrow I'll show you a great little gem for adding metrics to your Ruby code.

On Ruby, Software and the Internet