Two years ago today I released Sidekiq Pro, my commercial extension to Sidekiq, as an experiment to see if I could make OSS development financially viable for individual developers. I had no idea if anyone would trust me and buy it. Can you think of anyone else selling a Rubygem?
Sidekiq Pro sales for the last three months of 2012 were $7500. In 2013 they totalled $85,000. This year sales should top $175,000. Open Source development is now my full-time job, with no need for a corporate patron!
I’m being financially transparent here because I want my fellow developers to know: you can build incredibly valuable software and make real money from it. A small percentage of your users are willing to pay for extra features and the assurance of ongoing support over the years. It does take time, experience and a little vision. Here’s what I did:
A Path to Success
- Find a tool/library that is non-trivial and an important component to your current system or workflow.
- Plan out how you can make it better: simplify it, discard superfluous functionality, add useful functionality.
- Divide the functionality into open source and commercial parts. Use a GNU license and a commercial license for the respective parts.
- Build the open source part, document it.
- If it takes off, build the commercial part and the infrastructure necessary to sell it.
This path will take many months and none of these steps are easy! Be patient and work part-time on it for as long as possible. Remember that the open source version itself needs to offer plenty of value for users or there’s no reason for them to use your creation. The most important aspect is that you are your own best customer: you know the current tool and the pain it causes. So as the old saying goes, build that better mousetrap!
For me, I started with Resque. It was a major component of so many Ruby on Rails websites, and yet it was barely maintained and missing a lot of important features. I knew I could build something better. From this “business plan”, I created Sidekiq and Sidekiq Pro by following the steps above.
Now I want to do it again.
Three months ago I quit my job to work on Sidekiq and build a brand new OSS project and commercial product. Tomorrow I want to introduce it to you.
For years developers have followed the same arcane dozen steps to create a long-lived daemon process on Unix-based systems. These steps were state of the art in 2000 but they are no longer best practice today. Jake Gordon’s recent blog post on daemonizing Ruby processes is 100% factual but his approach is not recommended these days. Your application code should not be dealing with PID files, log redirection or other low-level concerns.
It’s time for our annual Semantic Versioning argument/gripefest! This time it was kicked off by Jeremy Ashkenas’s post why he believes Semantic Versioning is wishful thinking. Olivier Lacan chipped in further thoughts on the importance of a changelog.
Yes, Semantic Versioning is wishful thinking. Change cannot be compressed into three version numbers to guarantee safe upgrades. Developers get things wrong and forget changes such that versioning often isn’t correct, even if they wanted to follow SemVer exactly1. I thought I would write down my own versioning policies as another example for people to consider.
If you are building a system to run in the cloud, be prepared to spend much of your time building a resilient system.
Not a fast system. Not a very efficient system. Not a system full of fun, quirky features that users love. A resilient system because you will see performance and network issues at every connection point in your system. I hope that’s what you want.
I’ve been exploring a few new (to me!) technologies recently and runit is one that I’ve come away really impressed with. Linux distros have a few competing init services available: Upstart, systemd, runit or creaky old sysvinit. Having researched all of them and having built lots of server-side systems over the last two decades, I can firmly recommend runit if you want a server-focused, reliable init system based on the traditional Unix philosophy.
After 2.5 years I’ve decided to move on from The Clymb. I’m incredibly proud of our accomplishments during my time there: the site has dramatically increased in stability and scalability, the GitHub development workflow really improved code quality and we increased the size of the development team from 3 to 15. On the technical side, we moved from manually-configured cloud-based servers to Chef-managed dedicated servers, switched email providers twice, moved warehouses and rewrote our inventory and fulfillment process, integrated our logistics and fulfillment processes with an ERP system, and much more. I’m really proud to have helped grow the business while they helped me grow in management skill.
What am I doing next?
Since the Sidekiq 3.0 release, I’ve been slowly chipping away at some new features in Sidekiq Pro. What’s new and upcoming?
A lot of people ask me “How can I guarantee that a batch of jobs finished successfully?” Here’s the sad fact: you can’t. 99% of the time things go perfectly but there will always be some small percentage that fail for a myriad of reasons: hardware failure, software bug, thunderstorms in the cloud.
I’ve been noticing a theme in certain Rubygems recently that I like: opinionated designs which explicitly don’t allow the user to do certain things. I call these bounded libraries because they draw a functional boundary and won’t go beyond that point.
Starting in MySQL 5.6.5, datetime columns can have an actual useful default of CURRENT_TIMESTAMP and MySQL will auto-populate the columns as necessary. This is incredibly handy if you ever do bulk updates in SQL, now you don’t need to remember to set updated_at! Inserting records manually will auto-populate those columns too. Let’s try it:
create_table :rows do |t|
t.datetime :created_at, null: false, default: "CURRENT_TIMESTAMP"
t.datetime :updated_at, null: false, default: "CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP"
Run that and we’ll see this:
ActiveRecord::StatementInvalid: Mysql2::Error: Invalid default value for 'created_at': CREATE TABLE `rows` (`id` int(11) DEFAULT NULL auto_increment PRIMARY KEY, `value` int(11) NULL, `created_at` datetime DEFAULT 'CURRENT_TIMESTAMP' NOT NULL, `updated_at` datetime DEFAULT 'CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP' NOT NULL) ENGINE=InnoDB
Notice that Rails quotes the default value, making it invalid. We can bypass this by using a custom type to define all the special logic we need and use the generic
column definition method:
CREATE_TIMESTAMP = 'DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP'
UPDATE_TIMESTAMP = 'DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP'
create_table :rows do |t|
t.column :created_at, CREATE_TIMESTAMP
t.column :updated_at, UPDATE_TIMESTAMP
Big Caveat: you must make sure your database’s timezone is set correctly. MySQL defaults to the system’s timezone and we set our system timezone to Pacific so everything should work fine for us.
mysql> select @@time_zone;
| @@time_zone |
| SYSTEM |
Defined like that, those columns will be populated and updated any time rows are touched, not just when Rails does it.