Obviously we can’t use something like an MD5 or SHA1 fingerprint – we have to create a fingerprint based on the content of the image, not the exact bytes. This is what the pHash library does. A “perceptual hash” is a 64-bit value based on the discrete cosine transform of the image’s frequency spectrum data. Similar images will have hashes that are close in terms of Hamming distance. That is, a binary hash value of 1000 is closer to 0000 than 0011 because it only has one bit different whereas the latter value has two bits different. The duplicate threshold defines how many bits must be different between two hashes for the two associated images to be considered different images. Our testing showed that 15 bits is a good value to start with, it detected all duplicates with a minimum of false positives.

Phashion is my new Ruby wrapper for the pHash library and wraps just enough of the pHash API to implement the described functionality. Here’s the test in the test suite which verifies that Phashion considers the images to be duplicates:

  def assert_duplicate(a, b)
    assert a.duplicate?(b), "#{a.filename} not dupe of #{b.filename}"
  end
  def test_duplicate_detection
    files = %w(86x86-0a1e.jpeg 86x86-83d6.jpeg 86x86-a855.jpeg)
    images = files.map {|f| Phashion::Image.new("#{File.dirname(__FILE__) + '/../test/'}#{f}")}
    assert_duplicate images[0], images[1]
    assert_duplicate images[1], images[2]
    assert_duplicate images[0], images[2]
  end

pHash does have much more functionality, including video and audio support and persistent MVP tree support for similarity searches based on previously processed files, but I have not wrapped any of those APIs. Try it out and let me know what you think!

An Analogy

Risk is something that you either purposefully manage or you roll the dice with your life, sometimes literally. I ride/race a motorcycle as my main hobby away from the computer. Riding a moto is a risky activity and I do several things to manage that risk:

• Always wear a helmet, gloves and jacket
• Ride a relatively low power bike
• Taken every MSF training course available
• Refuse to ride in groups

Do these guarantee I won’t crash? Certainly not but I hope they will lessen the odds and minimize any damage if I do.

Managing Risks

As engineers, what are the risks of working at a startup? The main risk is the company failing and going bankrupt. A second, related risk is being laid off. In both cases, your job and paycheck are at risk. How do we manage those risks? I have three tactics to manage the risk of working at a startup.

1) Make it as easy as possible to find a job

You could make yourself essential to the operation of the company; that helps with layoffs but does not help with bankruptcy and has the drawback that you will start from square one at the next startup. My strategy has been to make myself a valuable developer, independent of any one startup, by working on open source software and maintaining a high quality blog that evangelizes myself and my work. This is a last resort strategy: if anything happens to make my job disappear, ideally I can interview and find another job within days. This recently proved successful when I announced my upcoming move to San Francisco and had 20-30 inquiries over the next few days.

2) Exercise common sense and your math skills

Do you know your startup’s monthly burn rate, cash reserves and revenue? I’d bet that the majority of people at startups do not. Get those numbers and figure out how many months the company has before it has no money. Just a few months left? Would it be difficult to raise more money? Are you part of a “layer of fat” that could be laid off to cut the burn rate? Is revenue rising or dropping? Are you getting more customers? These are questions you should be asking yourself every month to evaluate the health of your startup. At some point you will need to leave on your own terms, before you are forced out by bankruptcy or layoffs. I left FiveRuns last year when these questions made bankruptcy look unavoidable. Leaving on my own terms meant I could take a few weeks to interview around to find the right job.

3) Stick with Success

They say failure is the best way to learn but in my experience nothing breeds success more than previous success. I try to stick with entrepreneurs that have past successes. As developers, we want to work with smart developers, yes, but you also want to work with great business guys who have a network of contacts, know how to raise funding and can navigate the company to a successful exit. I can interview a person to learn if they are a good developer but I can’t interview a CEO to learn if they are a good CEO. I have only two metrics:

• do they have a reasonable business plan with a way to make money?
• have they had previous startup successes?

The “halo” effect is very real. VCs are more willing to talk to someone who has previous success and knows the funding process. People are more willing to work at a company run by someone with previous success. Press is easier to get and customers are easier to talk to if they already know the company as the latest effort by a successful entrepreneur.

4) Educate yo’self (Extra bonus tip!)

You may know computer science but how much do you know about management or finance? Read a management book. I recommend anything by Peter Drucker – he literally invented the science of management and his writing really opened my eyes. Read a book on business finance. You’re not trying to become an expert in these fields but when you learn a little bit about the other major roles in a startup, you’ll be able to evaluate your startup’s current situation more accurately.

Even with all this, you will fail often. I’ve been part of two moderately successful exits and several bankruptcies. I’ve only been caught flat-footed once and tried to learn as much as I could from that experience. No matter what happens the startup experience is rewarding but with a little foresight you can minimize the inevitable risk to yourself and your livelihood.

Now that I have RabbitMQ, Cassandra, Solr and the Amazon AWS services evented, the only holdup was ActiveRecord. Some people may advocate using another ORM layer but when you have 2-3 other Rails apps, all sharing 100+ models, you can’t afford to maintain two separate ORM layers. Plus, frankly I like the Rails stack: it works pretty well, is thoroughly documented and every Ruby developer is familiar with it.

So what do we need to do to get AR working event-style? At a high level, there’s two things required:

• The database driver itself must be modified to send SQL asynchronously. The postgresql driver, for instance, calls the exec(sql) method for all traffic to the database. So we just need to provide an exec method which uses Fibers under the covers to work asynchronously.
• AR’s connection pooling needs to be Fiber-safe. Out of the box, it is Thread-safe. Since we are using an execution model based on a single Thread with multiple Fibers, all the Fibers would try to use the same connection, with disastrous consequences.

These are the things that em_postgresql does.

• postgres_connection is a basic, EM-aware Postgres driver. It provides the Fibered exec() method which makes the whole thing asynchronous.
• em_postgresql_adapter.rb wraps postgres_connection to make it a proper ActiveRecord driver.
• patches.rb overrides a bunch of AR’s internal connection pooling to make it Fiber-friendly.

Unfortunately the latter makes one hack necessary – we have to have a list of current Fibers to release any lingering connections associated with those Fibers. The Threaded version can use Thread.list but Ruby does not provide an equivalent method for Fibers. Instead I require the application to register a FiberPool with AR to clear stale connections.

So what does it all mean? Well, here’s a Sinatra application that uses plain old ActiveRecord and is completely asynchronous! Try ab -n 100 -c 20 http://localhost:9292/test to hit the app with 20 concurrent connections; it will process them all in parallel, without any painful threading issues (autoloading, misbehaving extensions, etc). Awesome!

You should guess what’s next. Coming soon: the whole Rails stack, running asynchronously…

Gossip

Cassandra is a cluster of individual nodes – there’s no “master” node or single point of failure – so each node must actively verify the state of the other cluster members. They do this with a mechanism known as gossip. Each node ‘gossips’ to 1-3 other nodes every second about the state of each node in the cluster. The gossip data is versioned so that any change for a node will quickly propagate throughout the entire cluster. In this way, every node will know the current state of every other node: whether it is bootstrapping, running normally, etc.

Hinted Handoff

In writing, I mentioned that Cassandra stores a copy of the data on N nodes. The client can select a consistency level for a write based on the importance of the data – for example, ConsistencyLevel.QUORUM means that a majority of those N nodes must reply success for the write to be considered successful.

What happens if one of those nodes goes down? How do those writes propagate to that node later? Cassandra uses a technique known as hinted handoff, where the data is written to anther random node X to be stored and replayed for node Y when it comes back online (remember that gossip will quickly tell X when Y comes online). Hinted handoff ensures that node Y will quickly match the rest of the cluster. Note that read repair would still eventually “fix” the old data if hinted handoff did not work for some reason but only once the client asked for that data.

Hinted writes are not readable (since node X is not officially one of the N copies) so they don’t count toward write consistency. If Cassandra is configured for three copies and two of those nodes are down, it would be impossible to fulfill a ConsistencyLevel.QUORUM write.

Anti-Entropy

The final trick up Cassandra’s proverbial sleeve is anti-entropy. AE explicitly ensures that the nodes in the cluster agree on the current data. If read repair or hinted handoff don’t work due to some set of circumstances, the AE service will ensure that nodes reach eventual consistency. The AE service runs during “major compactions” (the equivalent of rebuilding a table in an RDBMS) so it is a relatively heavyweight process that runs infrequently. AE uses a Merkle Tree to determine where within the tree of column family data the nodes disagree and then repairs each of those branches.

This is the last post in my series on Cassandra. I hope you enjoyed them! Please leave a comment if you have questions or if I’ve made an error above.

In my previous post, I discussed how writes happen in Cassandra and why they are so fast. Now we’ll look at reads and learn why they are slow.

Reading and Consistency

One of the fundamental thereoms in distributed systems is Brewer’s CAP theorem: distributed systems can have Consistency, Availability and Partition-tolerance properties but can only guarantee two. In the case of Cassandra, they guarantee AP and loosen consistency to what is known as eventual consistency. Consider a write and a read that are very close together in time. Let’s say you have a key “A” with a value of “123″ in your cluster. Now you update “A” to be “456″. The write is sent to N different nodes, each of which takes some time to write the value. Now you ask for a read of “A”. Some of those nodes might still have “123″ for the value while others have “456″. They will all eventually return “456″ but it is not guaranteed when (in practice, usually just a few milliseconds). You’ll see why this is important in a second.

Reads are similar to writes in that your client makes a read request to a single random node in the Cassandra cluster (aka the Storage Proxy). The proxy determines the nodes in the ring (based on the replica placement strategy) that hold the N copies of the data to be read and makes a read request to each node. Because of the eventual consistency limitations, Cassandra allows the client select the strength of the read consistency:

• Single read – the proxy returns the first response it gets. Can easily return stale data.
• Quorum read – the proxy waits for a majority to respond with the same value. This makes it much more difficult to get stale data (nodes would have to go down) but slower.

In the background, the proxy also performs read repair on any inconsistent responses. The proxy will send a write request to any nodes returning older values to ensure that the nodes return the latest value in the future. There are a number of edge cases here that I’m not clear how Cassandra deals with:

• What if an even number of nodes reply, with half returning a value of “X” and the other half returning a value of “Y”? Since each column value is timestamped, presumably it will use the timestamp as a tie breaker.
• What if two nodes return “X” with an old timestamp and one node returns “Y” with a newer timestamp? Does quorum override the clock?
• What if the clocks on the cluster nodes are out of sync?

Scanning ranges

Cassandra works fine as a key/value store: you give it the key and it will return the value for that key. But this is often not enough to answer critical questions: what if I want to read all users whose last name starts with Z? Or read all orders placed between 2010-02-01 and 2010-03-01? To do this, Cassandra must know how to determine the nodes which hold the corresponding values. This is done with a partitioner. By default, Cassandra uses a RandomPartitioner which is guaranteed to spread the load evenly across your cluster but cannot be used for range scanning. Instead a ColumnFamily can be configured to use an OrderPreservingPartitioner, which knows how to map a range of keys directly onto one or more nodes. In essence, it knows which node(s) hold the data for your alphabetically-challanged users and for February’s orders.

Reading on an Individual Node

So all of that distributed system nonsense aside, what does each node do when performing a read? Recall that Cassandra has two levels of storage: Memtable and SSTable. The Memtable read is relatively painless – we are operating in memory so the data is relatively small and iterating through the contents is fast as possible. To scan the SSTable, Cassandra uses a row-level column index and bloom filter to find the necessary blocks on disk, deserializes them and determines the actual data to return. There’s a lot of disk IO here which ultimately makes the read latency higher than a similar DBMS. Cassandra does provide some row caching which solves much of that latency.

That’s a whirlwind tour of Cassandra’s read path. Take a look at the StorageConfiguration wiki page for much more content on this subject. Next up, I’ll discuss some of the various “tricks” Cassandra uses to solve the myriad of edge cases inherent in distributed systems.

We’ve started using Cassandra as our next-generation data storage engine at OneSpot (replacing a very large Postgresql machine with a cluster of EC2 machines) and so I’ve been using it for the last few weeks. As I’m an infrastructure nerd and a big believer in understanding the various layers in the stack, I’ve been reading up a bit on how Cassandra works and wanted to write a summary for others to benefit from. Since Cassandra is known to have very good write performance, I thought I would cover that first.

First thing to understand is that Cassandra wants to run on many machines. From what I’ve heard, Twitter uses a cluster of 45 machines. It doesn’t make a lot of sense to run Cassandra on a single machine as you are losing the benefits of a system with no single point of failure.

Your client sends a write request to a single, random Cassandra node. This node acts as a proxy and writes the data to the cluster. The cluster of nodes is stored as a “ring” of nodes and writes are replicated to N nodes using a replication placement strategy. With the RackAwareStrategy, Cassandra will determine the “distance” from the current node for reliability and availability purposes where “distance” is broken into three buckets: same rack as current node, same data center as current node, or a different data center. You configure Cassandra to write data to N nodes for redundancy and it will write the first copy to the primary node for that data, the second copy to the next node in the ring in another data center, and the rest of the copies to machines in the same data center as the proxy. This ensures that a single failure does not take down the entire cluster and the cluster will be available even if an entire data center goes offline.

So the write request goes from your client to a single random node, which sends the write to N different nodes according to the replication placement strategy. There are many edge cases here (nodes are down, nodes being added to the cluster, etc) which I won’t go into but the node waits for the N successes and then returns success to the client.

Each of those N nodes gets that write request in the form of a “RowMutation” message. The node performs two actions for this message:

• Append the mutation to the commit log for transactional purposes
• Update an in-memory Memtable structure with the change

And it’s done. This is why Cassandra is so fast for writes: the slowest part is appending to a file. Unlike a database, Cassandra does not update data in-place on disk, nor update indices, so there’s no intensive synchronous disk operations to block the write.

There are several asynchronous operations which occur regularly:

• A “full” Memtable structure is written to a disk-based structure called an SSTable so we don’t get too much data in-memory only.
• The set of temporary SSTables which exist for a given ColumnFamily are merged into one large SSTable. At this point the temporary SSTables are old and can be garbage collected at some point in the future.

There are lots of edge cases and complications beyond what I’ve talked about so far. I highly recommend reading the Cassandra wiki pages for ArchitectureInternals and Operations at the very least. Distributed systems are hard and Cassandra is no different.

Please leave a comment if you have a correction or want to add detail – I’m not a Cassandra developer so I’m sure there’s a mistake or two hidden up there.

Here’s a few examples of good changelogs: memcache-client, Java, Nokogiri, Resque, Redis.

Personally I consider a changelog one of the best indicators of a well run OSS project. If you run an OSS project, please consider supplying release notes or a changelog so that other developers can follow your project with ease!

Update: looks like I just missed the changelog for RabbitMQ. Alexis was kind enough to point me to the release notes in the comments.

Executive summary: Varnish really, really wants to run on a 64-bit system. Don’t run it on 32-bit systems if possible.

Varnish wants to memory map the entire cache. This means the entire cache needs to be able to fit into virtual memory. On a 64-bit system, VM is virtually unlimited. On a 32-bit system, processes usually have access to a maximum of 3GB of virtual memory. Since you also need to allocate stack space and other standard process requirements, in practice people don’t recommend more than 2GB of cache space for Varnish on 32-bit systems. Pretty small for a web content cache. If you want Varnish to use an entire disk for a cache, it must run on a 64-bit system.

We had a few minutes of outage recently due to this architecture. We read some Varnish tuning tips and decided to modify our default configuration. Specifically we raised the minimum thread count from 1 to 500. Because, after all, “ idle threads are cheap“. But they are only cheap on 64-bit systems where allocating hundreds of MB for extra stack space is a no brainer! When we rolled out this change, the process ran out of memory and couldn’t allocate the extra threads. Klaxons went off and I rolled back the changes. Over the next few months, we’ll be upgrading our caches to 64 bit so that we don’t need to worry about sizing issues moving forward.

For these reasons, I would argue that IO-intensive applications need to either use an event-driven application model or a language designed for concurrency like Clojure. Since I like to work with Ruby, the former is the route to follow.

This overview is important to understand because the main deployment pattern with Rails apps is to instantiate 5-10 Rails processes, which can each handle one request at a time. If a request takes 5-10 seconds to process (maybe it is calling Amazon S3 or SimpleDB), that entire Rails process is stuck waiting for the data. Even a multi-threaded Rails application is limited due to the GIL. For this reason, people use a message queue to handle long-running tasks but often that just passes the buck: now the message queue processor is the one stuck for 5-10 seconds instead. You don’t have a user waiting for a response but you still are limited in how fast you can process the queue based on the amount of memory you have and the number of daemon processes you can start.

This is where an event-driven model would help immensely. The fundamental tools at your disposal are NeverBlock and EventMachine. EventMachine provides the reactor, the fundamental “switch” in your application which decides what code is ready to run now, and NeverBlock provides various drop-in replacements for the common Ruby code used for network and IO: mysql and postgres database drivers, tcp sockets, etc. Using these, the message queue processor can process many messages at the same time: there’s never any concurrent execution but as one message performs some IO request, eventmachine and neverblock will seamlessly switch to handle another message while waiting for the IO response. That’s the fundamental difference with threaded code: instead of switching threads at a non-deterministic point in the future, event-driven code only switches when the code tries to perform IO. Your code does not need to be thread-safe because your code will not be interrupted while modifying variables and data structures in memory.

Sounds good, right? Well, a few caveats:

• CPU-intensive processes won’t gain much. There’s still only a single actual thread of execution under the covers so event-driven applications will only take advantage of a single processor/core.
• Your application should run on Ruby 1.9 to take advantage of Fibers. Fibers have been backported to Ruby 1.8 but I encourage you to try Ruby 1.9. Most extensions are Ruby 1.9 safe now and Rails is fully supported on Ruby 1.9. Without Fibers, your application code needs to change dramatically to work as success/error callbacks. With Fibers, your code needs little change and can be written in the more familiar procedural style.
• Application exception handling becomes tricky, just as with threads. It’s easy to lose an exception.

Next time, we’ll take a deeper look into some event-driven code and how it works.