Tokyo Cabinet vs Memcached

March 8th, 2009 · 5 Comments

Tokyo Cabinet bills itself as a next-generation DBM. Like BerkeleyDB, TC gives you low-level database operations and allows you to build your own very fast data access operations. Unlike BDB, TC has several functional layers on top of TC which provide network access, schema-less document-oriented storage (a la CouchDB) and supported bindings for many languages. This project is an infrastructure nerd’s dream!

One interesting feature of its network access layer, Tokyo Tyrant, is support for the memcached protocol, which means we can access it via memcache-client! I downloaded tokyocabinet and tokyotyrant and built and installed them. I then started a tyrant server like so:

ttserver -port 45000 data.tch

Let’s try some simple benchmarks:

require 'rubygems'
require 'benchmark'
 
gem 'memcache-client', '1.7.0'
require 'memcache'
 
memcached = MemCache.new(['localhost:11211'], :timeout => nil)
tokyo = MemCache.new(['localhost:45000'], :timeout => nil)
 
Benchmark.bm(20) do |b|
  b.report 'memcached-write' do
    10000.times do |idx|
      memcached.set idx.to_s, idx.to_s*100, 120, true
    end
  end
  b.report 'tokyo-write' do
    10000.times do |idx|
      tokyo.set idx.to_s, idx.to_s*100
    end
  end
  b.report 'memcached-read' do
    10_000.times do
      memcached.get(rand(100).to_s, true)
    end
  end
  b.report 'tokyo-read' do
    10_000.times do
      tokyo.get(rand(100).to_s)
    end
  end    
end

Results:

                          user     system      total        real
memcached-write       0.380000   0.200000   0.580000 (  1.308354)
tokyo-write           0.470000   0.200000   0.670000 (  1.663922)
memcached-read        0.410000   0.170000   0.580000 (  1.354334)
tokyo-read            0.470000   0.180000   0.650000 (  1.688558)

Not too shabby at all! Keep in mind that TC doesn’t support time-based key expiration and is persistent. Memcached is designed to be a memory-only cache, so while the underlying semantics of the two are somewhat different, the same basic operations are available to memcache-client. In fact, you can even theoretically use TC as your Rails cache by configuring the mem_cache_store to point to your Tokyo Tyrant server instead of memcached!

Tags: Ruby

5 responses so far ↓

1 Ilya Grigorik // Mar 8, 2009 at 5:09 pm

Mike, another project to check out if you’re comparing TC / Memcache is tokyo-cache-cow by joshbuddy. While TC doesn’t have native TTL support, it does allow for doing substring matching, which is pretty nice.

Also, it is possible to implement TTL’s in TC with a simple Lua script (I wish I could find my bookmark for that, but can’t… It already exists though).
2 Joshua Hull // Mar 15, 2009 at 7:34 pm

Hey Mike. tokyo-cache-cow does indeed support all of the memcache protocol, it does seem much slower than I would expect. I’m not sure if the problem lies in the memcache client or something with event machine (because when I’ve benchmarked tc itself, everything is fine), but I’m vowing to get to the bottom of this.

The other part of the memcache protocol to implement is the UDP side, and thats on the books for tokyo cache cow as well.
3 Mitchell Hashimoto // Apr 3, 2009 at 6:00 pm

@Ilya:

Here is a little paragraph on expiring records with TT:

http://tokyocabinet.sourceforge.net/tyrantdoc/#luaext
4 Bucket o’ Links / April 2009 < Code Monkey Island // Apr 13, 2009 at 9:13 pm

[...] (respectively). This came out of Japanese social network mixi.jp. Tokyo is file-backed but is nearly as fast as memcached for most needs. We’ve been using this for the last couple weeks to store long-lived cache [...]
5 tokyocabinet 使用 « Neo的技术笔记 // Feb 26, 2011 at 7:31 am

[...] 参照 :Tokyo Cabinet vs Memcached [...]

Mike Perham

On Ruby, software and the Internet