Mike Perham

I needed to create a simple, but IO-intensive, thumbnailing service for OneSpot last week. This service acts as a proxy to S3 and so a blocking implementation would not scale well, even if threaded. I wanted to use EventMachine instead. Lessons learned:

• The programming model is a mind twist and takes a long time to understand. The sprinkling of implementation at various layers makes it harder. I’ve spent several days now reading through EventMachine, Thin, Rack and em-http-request source code.
• There’s no non-trivial examples out there. It seems like every example is 10 lines of “Hello World” code with no samples of how to integrate multiple pieces. Ok, here’s a 10 line async web server. Now how do I integrate an async call to the DB? How do I make an async 3rd party web service call?
• There’s no testing support. No libraries for doing async testing and no best practices or suggestions on how to test.

So how can we make things better rather than just complain? I’m going to show you a non-trivial example. In return, I want you to send me more examples. Evented is my new Github repository for EventMachine examples. The first example is that big chunk of code that I puzzled over for the last few days which implements the thumbnailing service using Thin, S3, image_science and em-http-request. But I want more examples and I’d love to hear ideas on how to test this type of code. Leave a comment, send a pull request, and help me help you!

After talking about document-oriented databases in general in Part 1, for Part 2 I’ve written some code comparing MongDB 1.1.1, CouchDBX 0.9.1 and Tokyo Tyrant 1.4.32 in an apples to apples test.

The shootout code is on Github. I welcome patches and improvements as long as they don’t bias the tests in favor of any one system.

Results

========== Running Tokyo Tyrant tests
Using rufus-tokyo 1.0.0
user system total real
init 0.000000 0.000000 0.000000 ( 0.013781)
create 19.770000 4.260000 24.030000 ( 39.982273)
query 0.160000 0.030000 0.190000 ( 0.318070)
delete 0.000000 0.000000 0.000000 ( 0.421201)

========== Running CouchDB tests
Using jchris-couchrest 0.23
user system total real
init 0.000000 0.000000 0.000000 ( 0.000007)
create 9.290000 0.560000 9.850000 ( 51.177824)


init is the time required to initialize the database and create any necessary indices. In practice, this number isn’t terribly relevant as this is usually an infrequent operation.

The create operation shows how long it takes for the system to bulk load 200,000 documents. You can see that Tokyo is quite fast while the Mongo client hits the CPU pretty hard. The couchrest client seems more efficient than the other two but the task still takes longer than Tokyo.

The query operation shows how long it takes to perform a non-trivial query against those 200k documents. Both Mongo and Tokyo perform about the same speed although Mongo lazy fetches the results in order to minimize network traffic when used with pagination. Tokyo returns the entire result at once AFAIK. I was not able to complete this test in a weekend using CouchDB because its view layer is so alien to me. I’d welcome help with this task.

The delete operation tests the time required to delete a subset of documents within our set of 200,000. Again, Tokyo comes out on top. Since I couldn’t perform the query in CouchDB I couldn’t delete anything either.

Conclusions? Tokyo has a reputation for being very fast and it appears to be well-founded. Couch is fast for what I could get working – I would be much more concerned about developer training and learning curve with Couch. Mongo is by no means slow but someone has to finish last. I like Mongo as an interesting mix of RDBMS and document technologies – it’s not quite as conventional as Tokyo but not as unconventional as CouchDB with its unique view layer and Erlang underpinnings. What do you think? Leave a comment and let me know!

We’re looking for a Ph.D-level machine learning specialist who will maintain and improve our content scoring algorithms and codebase at OneSpot. Our current system is based on technologies like Hadoop, Cascading and EC2. The position is full-time in Austin, TX. Please contact me if you or someone you know is looking for this type of job.

With the closing of FiveRuns, the blog post that was the main page for DataFabric has disappeared. I’m creating this blog post as a replacement.

DataFabric is a Rails plugin/gem that adds sharding support to ActiveRecord. It supports Rails 2.x and is theoretically database independent, although I haven’t tested it personally with anything but MySQL and SQLite. Please see the github home page for the latest code, examples and a more in-depth explanation.

RubyConf 2009 is taking place in San Francisco November 19-21. I’ll be there and have most of the 18th free if anyone is near SFO and wants to join me in some coffeeshop coding.

MongoDB is a relatively new “schema-free, document-oriented database.” The closest competitor to MongoDB is probably CouchDB or Tokyo Cabinet’s Table database but all three differ in significant ways:

• CouchDB guarantees the ACID properties when saving documents through an MVCC mechanism like postgresql. Tokyo Cabinet provides ACID support via locking, like mysql.  Mongo updates documents in place with no real support for concurrency (e.g. optimistic or pessimistic locking). This means Mongo will be much faster for writing and scale horizontally very easily at the expense of guaranteed data consistency. This is a very common tradeoff.
• Couch and Mongo support datatypes for the values in documents but Mongo uses a binary JSON representation and protocol which makes it faster over the wire. Tokyo Cabinet does not support datatypes, except for string and number types in indexes. It does not have native boolean and date types which means you can’t efficiently do queries like “created_at < 1 week ago” although you could store dates and booleans as numbers to work around this limitation.
• Couch and Mongo support more complex structures (arrays, hashes) as values.  Tokyo only supports basic datatypes.
• Couch requires you to instantiate views for the queries required by your application. Couch will then auto-index the data required to fetch the view. Tokyo Cabinet and Mongo have a more traditional RDBMS notion of indexes, which are maintained separately from the table.
• CouchDB is written in Erlang while MongoDB is written in C++ and Tokyo Cabinet in C. I’m inclined to trust Erlang more for distributed infrastructure, given its long history in telecom. That said, I have no evidence that the other two are anything but rock solid.

All projects are interesting takes on the traditional RDBMS datastore.  CouchDB would be useful where you absolutely must keep ACID and transactions to ensure data integrity but want to avoid the hard-coded schema that a traditional database requires.  Semantic web applications come to mind where your objects are just a bag of attributes.

MongoDB would seem to be more designed for applications which need dynamic query functionality with high performance and can sacrifice data integrity to get it – metrics and operational data come to mind.  As the MongoDB website says: “High volume, low value data”.

Tokyo Cabinet feels a little more traditional and lower-level, like a layer on top of BerkeleyDB.  It’s similar in design in that they are both C libraries and not designed to run as standalone daemons themselves.  It would be great for embedded applications.

In my next post, I’ll try out each with their latest Ruby driver and see how they perform in basic usecases.  Did I get anything wrong?  Leave a comment and let me know!

Update: Tokyo Tyrant does not appear to support transactions and so Tokyo Cabinet cannot guarantee ACID when used as a service.

Here’s my slides for LSRC covering the new Engines feature in Rails 2.3. Basically I cover Rails’ history from monolithic application to supporting code reuse via plugins to supporting MVC reuse via engines.

Rails Engines (PPT, 2.9MB)
Rails Engines (Keynote, 1.8MB)

We had some performance issues recently and discovered we were using memcache-client 1.5.0. Our daemon was blocked by memcached socket operations hanging forever, which caused monit to kill and restart the daemon after 5 minutes.

I performed the brain surgery to use 1.7.4 (since we are using Rails 2.1) and deployed. See if you can guess when. Graph shows operations per hour (more is better).

Many people don’t know this but the latest memcached release (1.2.8 right now) can be about 15% more efficient in its memory usage than older releases. If you have a 600MB memcached server, upgrading will magically “gain” you 100MB of RAM. Why is this?

When you ask memcached to store a value, it looks up the “slab” associated with that value. A slab holds values within a particular size range. Slabs are composed of 1MB pages, which are broken into chunks of the slab’s size. Let’s say your value is 1001 bytes; memcached will look up the slab which holds values between 1000 and 2000 bytes. It then finds a page with an empty chunk and inserts the value into that chunk. Note that a chunk is fixed in size – it must be 2000 bytes in order to store the largest value for the slab.

Now you know why memcached limits values to one megabyte: the value must be stored in a chunk and a page needs to hold the chunk. Since a page is hardcoded as 1MB, it follows that a chunk must also be limited to 1MB.

So we understand the “object model” for memcached memory allocation: a slab has many pages which has many chunks. Each chunk is a fixed size, based on the maximum size for the slab so e.g. the 2000 byte slab will hold values between 1001 and 2000 bytes. Older versions of memcached used slabs sized based on powers of two, so you’d have a 1KB slab, 2KB slab, 4KB slab, …, all the way to 1MB. If your memcached server was full of 1001 byte values, your memory efficiency would be 50% (1001 / 2000) in the worst case. Assuming you have an even distribution of value sizes, you’ll get 75% efficiency (1500 / 2000). Your 600MB memcached server will only hold 450MB of actual data!

In this image, we see a single slab with two pages. Each page has several chunks, the green chunks are empty and some have orange values. The yellow area is the waste we are talking about.

One of the improvements Facebook made to memcached last year was moving to a smaller exponential so there is not as much waste in storing values in chunks. Instead of 2^n for the slab allocation, the latest versions of memcached use a much smaller growth exponential, 1.25^n, so you will see slabs with sizes 1KB, 1.25KB, 1.56KB, etc… This means that instead of 25% waste on average, you should see closer to 10%. Effectively you regain 15% of your memcached memory just by installing the latest version!

memcached is Evan Weaver’s Ruby wrapper around the libmemcached C library and widely regarded as quite fast. After an hour of trying, I finally got a build of memcached to actually compile and install on my machine (the trick: you need to download the custom packages Evan links on his blog, nothing else seems to work). Here’s the results:

== memcached 0.13 + libmemcached 0.25.4 versus memcache-client 1.7.4

                                     user     system      total        real
set:plain:noblock:memcached      0.090000   0.030000   0.120000 (  0.277929)
set:plain:memcached              0.220000   0.270000   0.490000 (  1.251547)
set:plain:memcache-client        0.610000   0.270000   0.880000 (  1.670718)
set:ruby:noblock:memcached       0.150000   0.020000   0.170000 (  0.309201)
set:ruby:memcached               0.300000   0.290000   0.590000 (  1.390354)
set:ruby:memcache-client         0.670000   0.270000   0.940000 (  1.713558)
get:plain:memcached              0.240000   0.270000   0.510000 (  1.169909)
get:plain:memcache-client        0.850000   0.270000   1.120000 (  1.885270)
get:ruby:memcached               0.270000   0.280000   0.550000 (  1.229705)
get:ruby:memcache-client         0.890000   0.260000   1.150000 (  1.861660)
multiget:ruby:memcached          0.190000   0.090000   0.280000 (  0.396264)
multiget:ruby:memcache-client    0.530000   0.100000   0.630000 (  0.901016)
missing:ruby:memcached           0.280000   0.290000   0.570000 (  1.254400)
missing:ruby:memcached:inline    0.300000   0.290000   0.590000 (  1.235122)
missing:ruby:memcache-client     0.570000   0.250000   0.820000 (  1.461293)
mixed:ruby:noblock:memcached     0.540000   0.620000   1.160000 (  2.429200)
mixed:ruby:memcached             0.580000   0.570000   1.150000 (  2.610819)
mixed:ruby:memcache-client       1.580000   0.540000   2.120000 (  3.632775)

In most cases, memcache-client is within 33-50% of the performance of memcached. This is amazing for a (mostly) pure Ruby library performing a lot of network IO against a C library which has been tuned for speed! I hope that puts to bed any lingering doubts that memcache-client is slow.

Remember: if you are using Rails 2.3, just “gem install memcache-client” and Rails will pick up the latest version with all these performance improvements.