Mike Perham

On Ruby, software and the Internet

← →

Dalli – memcached for Ruby

August 30th, 2010 · 24 Comments

Dalli is my brand new memcached client for Ruby. I’ve maintained Ruby’s memcache-client for two years now and been dissatisfied with the codebase for a while.

Coincidentally, NorthScale approached me recently about building a pure Ruby memcached client which used the new binary protocol defined in memcached 1.4. We worked out an arrangement to sponsor the OSS project which became Dalli.

My goals for Dalli were threefold:

  1. Clean sheet codebase using the binary protocol
  2. Drop-in replacement for memcache-client in Rails for a very simple upgrade path for Rails developers
  3. Equivalent or faster performance than memcache-client

I’m happy to say that Dalli meets all those goals. For one, the Dalli core is almost half the size of the memcache-client core, 700 vs 1250 LOC! But wait, there’s more! Using Rails 3? Dalli drops right in! Using Heroku? Dalli works without any additional configuration! Take a look at the README for more details.

Please file an issue if you find a bug or have a feature you’d like to see. In the meantime, happy caching!

Tags: Ruby

24 responses so far ↓

  • 1 Carl Mercier // Aug 30, 2010 at 10:42 am

    You forgot one thing… benchmarks :)

  • 2 Stan // Aug 30, 2010 at 10:59 am

    Thumbs up! It sounds really interesting.
    Have you done any benchmarks?

  • 3 Mike Perham // Aug 30, 2010 at 11:34 am

    Benchmarks are right here:

    http://github.com/mperham/dalli/blob/master/Performance.md

    They run as part of the test suite.

  • 4 Brian Mitchell // Aug 30, 2010 at 1:11 pm

    Very good to see other memcached libraries spring up with real binary protocol support for the 1.4 series. I originally wrote my remix-stash client (now maintained by Arron Quint/quirkey) for API reasons (which I still find to be my biggest issue with other libraries) but the protocol actually exposes a lot more that isn’t possible in the old text based protocol. Stats specifically have much more to pull down now.

    I’m in the process of rewriting it to support UDP and async setups, though I’ve been using CoffeeScript and node more these days so my ruby project time is scaled back… I’ll probably release it once I really need it.

  • 5 Mike Perham // Aug 30, 2010 at 1:15 pm

    Brian, your remix-stash work was very helpful to me as a reference when implementing the actual binary protocol. I need to update the README to give you a shoutout. Thanks for reminding me!

  • 6 James Phillips // Aug 30, 2010 at 2:34 pm

    Can’t wait to put this to use. Thanks!!

  • 7 Kristof // Aug 30, 2010 at 2:51 pm

    no benchmarks against libmemcached?

  • 8 Mike Perham // Aug 30, 2010 at 3:25 pm

    You are welcome to run the benchmarks with libmemcached. The code is in test_benchmark.rb. Keep in mind that libmemcached does not support Ruby 1.9.

  • 9 SugarCRM: memcache.connect « Php Bugs // Aug 30, 2010 at 11:57 pm

    [...] Dalli – memcached for Ruby (mikeperham.com) [...]

  • 10 Joel Watson // Aug 31, 2010 at 3:04 pm

    Just out of curiosity, was there a reason why you decided to publish this as a completely separate library rather than just releasing it as memcache-client 2.0? That would have made things a lot simpler in terms of keeping track of libraries and so forth, IMO.

  • 11 Mike Perham // Aug 31, 2010 at 3:32 pm

    The Dalli API is not 100% backwards-compatible with memcache-client. Dalli only supports the binary protocol so it requires memcached 1.4.0+. Several features have been removed and the codebase is completely new. For these reasons, I elected to start a completely new gem.

  • 12 Pierre // Sep 13, 2010 at 4:42 pm

    Hi Mike,

    Dalli looks really exciting! Do you by chance know whether it could be used instead of memcache-client with Starling?

    Thanks!
    Pierre

  • 13 Mike Perham // Sep 13, 2010 at 5:19 pm

    Pierre, it cannot. Dalli uses the binary protocol in memcached 1.4. Starling uses the old text protocol and so you should continue to use memcache-client with it.

  • 14 Evan Weaver // Sep 19, 2010 at 7:58 pm

    Memcached.gem has been compatible with Ruby 1.9 and supported the binary protocol for over a year, although the version of libmemcached it uses is somewhat out-of-date.

    Benchmarks are here: http://blog.evanweaver.com/files/doc/fauna/memcached/files/BENCHMARKS.html

    Pierre, you should probably use http://github.com/freels/kestrel-client and http://github.com/robey/kestrel these days, instead of Starling.

  • 15 Mike Perham // Sep 19, 2010 at 8:39 pm

    Thanks, Evan, I just verified that memcached does install in 1.9.2. Sorry for the misinfo and thanks for including Dalli in your benchmarks!

    Odd versioning scheme though, memcached v1? What was the reasoning behind a single digit version?

  • 16 Evan Weaver // Sep 20, 2010 at 12:22 pm

    Well, it’s the same as 1.0.0…after fixing only one bug in a year I figured it was time for a stable release. :-)

  • 17 Joost Schuur // Oct 24, 2010 at 7:26 pm

    It didn’t quite work without any further configuration on heroku. I had to make sure to specify the specific memcached servers I was using on my production.rb:

    config.cache_store = :dalli_store, ENV['MEMCACHE_SERVERS']

    I’m not sure if this was always needed or it’s a recent development, but this info was hard to come upon for me. Heroku’s own docs page on memcached doesn’t mention it, and I couldn’t get it to work until the anonymous hero behind http://arailsdemo.com clarified it.

  • 18 Mike Perham // Oct 24, 2010 at 10:50 pm

    Joost, you’re hacking around a Dalli bug that has been fixed. Use 0.10.1 and you shouldn’t need to specify the servers anymore.

  • 19 Joost Schuur // Oct 25, 2010 at 3:39 pm

    Good to know, Mike. I haven’t seen 0.10.1 kick in for me when I do the usual gem update, and it looks like it just came out a few days ago. Is there a built in waiting period before places like rubygems.org push it out when you update your gems?

    I know I can specify a specific version too, just curious about the process for new gem updates in general.

  • 20 Mike Perham // Oct 25, 2010 at 4:19 pm

    There’s no waiting period but perhaps there was a foulup somewhere. This shows that people have been downloading it for several days now:

    http://rubygems.org/gems/dalli/stats

  • 21 Joost Schuur // Oct 25, 2010 at 5:46 pm

    User error. Running ‘bundle update’ worked fine for me know. I was doing another ‘bundle install’, not realizing it doesn’t update automatically.

    However, with this update, dalli never read from the cache. My Heroku log was full of repeated ‘write fragment’ statements for the same fragment. On top of that, the time to write a fragment skyrocketed to 500 ms, even for a few hundreds bytes of text. The stats method output from the client was still reporting data though, so some connection must have been established.

    I rolled back to 0.10.0 and it was able to read fragments just fine again, and write time went back to a few milliseconds.

  • 22 Evan Weaver // Dec 21, 2010 at 4:41 pm

    URL changed for the benchmark: http://fauna.github.com/fauna/memcached/files/BENCHMARKS.html

    Feel free to just update the original comment; thanks.

  • 23 Bill Harding // Feb 27, 2011 at 4:14 pm

    Great work, always exciting to see progress on the Ruby memcaching front.

    I notice that when Dalli tries to connect to a memcached server that is less than 1.4, it seems to hang indefinitely. Any idea why this scenario doesn’t get caught by the timeout mechanism and result as an error rather than a hang?

  • 24 Replacing Memcached with Dalli // Apr 4, 2011 at 6:27 am

    [...] http://www.mikeperham.com/2010/08/30/dalli-memcached-for-ruby/ [...]

Leave a Comment