*/5 * * * * osascript -e 'tell application "System Events" to if (processes whose name is "iChat") exists then tell application "iChat" to log in'

This uses AppleScript to tell iChat to log in every 5 minutes. Now if the network drops, you’ll only be disconnected a few minutes.

Take a simple use case: “I want to send a welcome email when a user registers”. Commonly you want to send this email in the background so it does not impact the user’s experience. Do you need to install ActiveMQ, RabbitMQ or Resque to do this? Certainly not.

Message queueing is a fundamental architectural pattern when building complex systems. Your various system components might be written by different teams but they communicate through messages sent via queues. One component can send a message to another component, saying “please send this email”. But message queueing systems have their cost: they are complex because they are designed to be the foundation of your distributed system. They must be deployed and monitored like the rest of your infrastructure; they must be reliable and highly available.

I think that a lot of people install a message queue to perform simple background processing; it doesn’t need to be that complicated. The fundamental question to me is, “Am I communicating between different subsystems or just trying to spin off some work?” The registration email use case comes up almost immediately when building nearly every website. Consider also the case where you want to perform some action that might take 30-60 seconds and have the user’s browser poll for the result. Spinning off a separate thread to perform this work is entirely sufficient and much simpler. This is the reasoning behind my girl_friday project. I want a simple and reliable way to perform background processing without needing the complexity of an MQ system. Let’s examine a few characteristics of girl_friday:

• In-process – your background processor is part of your Ruby application and has access to the exact same codebase as your webapp. No need to share ActiveRecord models across projects via git or filesystem trickery. No need to deploy or monitor a separate set of processes.
• Threaded – huge memory savings because you don’t have to spin up other processes which load the exact same code. Threads are notoriously tricky to get correct so girl_friday uses Actors for the equivalent behavior in a simpler and safer API.

I have issues with the other contenders in the space:

• delayed_job – stores jobs in your RDBMS and polls for jobs which is a terribly unscalable idea. Spins off processes instead of threads.
• resque – forks a new process for every message. Safe but memory hungry.

The biggest caveat with girl_friday is threading, of course. Typical Ruby deployments aren’t thread-friendly but I’d like to help change that. Rainbows! is thread-friendly, as are all the JRuby app servers. The girl_friday wiki gives more specifics about features and usage. Are there any other dimensions to the problem that I’m missing? Any other projects that solve a similar problem? Post a comment and let me know!

• Node.js, Part I: Overview
• Node.js, Part II: Spelunking in the Code
• Node.js, Part III: Full Stack Application

• In MongoDB’s case, they don’t provide true durability so writes can be batched into a large set of writes when actually persisting to disk. This gets them great performance but means they don’t provide true ACID transactions. Side note: the latest release has a new --dur flag which gives true durability with the resultant loss in write performance.
• For Node.js, the trade-off is in programming style: everything is done asynchronously so you have to learn an entirely new style of programming. Great performance but great developer learning curve.
• With Telehash, UDP is a more efficient network protocol than TCP by design. TCP is essentially UDP with reliable delivery baked on top, so it suffers from round trip latency and the state required to track the current network packets in flight in order to ensure delivery. You can use UDP but if a router drops a UDP packet, your application will never know.

When you are looking at a new system that promises better performance or scalability than existing systems, ask yourself “what shortcuts did they take to get that performance or scalability?” Sometimes those shortcuts are worth it but it is completely dependent on your own situation. If you are writing a small, high-traffic network service, Node.js makes sense. Writing a high volume of low-priority logging data with MongoDB makes sense. I would argue there are very few instances where UDP is a good idea, realtime data streaming is the best case I can think of, off hand. Part of being an engineer is learning when these shortcuts are unreasonable and what you are paying for that shortcut.

In my case, my slice was freezing. I didn’t know what the problem was until I ran top and saw this. Yikes.

The problem was the Apache is configured by default to allow up to 150 Apache processes. Each process took 5-10MB of real memory so my slice’s 512MB was quickly overwhelmed. But why was it creating 150 processes in the first place? Shouldn’t WP-SuperCache respond very quickly, such that the process can serve many requests per second? Yes, but…

Keep-Alives

Keep-Alives try to help client performance. This is a performance tweak that will kill you. By default, Apache is configured to hold the process locked for a given socket for 15 seconds (!!?) in case that socket makes another request. That’s a terrible, terrible default: you should never lock resources waiting for human input. So in 15 seconds, Hacker News delivered me 50-100 requests. These requests all generated their own process, quickly overwhelming my RAM and swap and effectively freezing my slice.

I lowered the maximum number of processes (MaxClients) to 20 and the keep-alive timeout from 15 to 2 seconds. Before I was seeing load averages in the 100s and since reconfiguration, my slice’s load average has been under 1 all afternoon. Here’s the config I changed:

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 20
</IfModule>

As an engineer it makes me feel good to know my efforts are improving the system. Working on a complex ticket can take days to reproduce and fix the issue, often with little noticeable payoff in the end. So I grabbed two lower priority issues and fixed them – both were UI cleanups that led to a nicer user experience. I left the office that day with a spring in my step and smile on my face, ready to tackle the complex ticket again the next morning.

Rule #1 of Network Security: Don’t expose services which are not designed to be exposed.

Web and app servers will usually have 2-3 ports open to the public: ssh, http and https. All others should be vetted to determine if they should be public or not. Here’s the current state of mikeperham.com:


mike@perham:~$ netstat -a | grep LIST
tcp 0 0 localhost:mysql *:* LISTEN
tcp 0 0 *:www *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 localhost:smtp *:* LISTEN


There’s two types of ports in this list. ‘localhost’ means that my database is just listening locally:

localhost:mysql

whereas the star indicates my web server is listening on all network interfaces, including the public:

*:www

In the case of memcached, you want to configure it to listen locally only if you just have a single memcached instance. In Ubuntu/Debian, you would edit /etc/memcached.conf and ensure that:

-l 127.0.0.1


is in the file. Otherwise memcached will by default listen on all interfaces and be exposed publicly.

Firewall configuration brings another dimension of variability into the mix but I prefer to configure my services to listen correctly first and then determine any additional firewall rules necessary based on the network topology. Using Memcached servers on multiple machine might require some fancy firewall rules to ensure that they can talk to each other while not being exposed publicly. One nice thing about Amazon’s EC2 service is that it forces you to explicitly open ports to the public via firewall rules, everything else is internal by default.

In summary, I always perform a quick port audit of all machines after I’m done configuring them to ensure that they are as secure as possible before putting them in production. A quick netstat command can go a long way to ensure a sound night’s sleep.

Obviously we can’t use something like an MD5 or SHA1 fingerprint – we have to create a fingerprint based on the content of the image, not the exact bytes. This is what the pHash library does. A “perceptual hash” is a 64-bit value based on the discrete cosine transform of the image’s frequency spectrum data. Similar images will have hashes that are close in terms of Hamming distance. That is, a binary hash value of 1000 is closer to 0000 than 0011 because it only has one bit different whereas the latter value has two bits different. The duplicate threshold defines how many bits must be different between two hashes for the two associated images to be considered different images. Our testing showed that 15 bits is a good value to start with, it detected all duplicates with a minimum of false positives.

Phashion is my new Ruby wrapper for the pHash library and wraps just enough of the pHash API to implement the described functionality. Here’s the test in the test suite which verifies that Phashion considers the images to be duplicates:

  def assert_duplicate(a, b)
    assert a.duplicate?(b), "#{a.filename} not dupe of #{b.filename}"
  end
  def test_duplicate_detection
    files = %w(86x86-0a1e.jpeg 86x86-83d6.jpeg 86x86-a855.jpeg)
    images = files.map {|f| Phashion::Image.new("#{File.dirname(__FILE__) + '/../test/'}#{f}")}
    assert_duplicate images[0], images[1]
    assert_duplicate images[1], images[2]
    assert_duplicate images[0], images[2]
  end

pHash does have much more functionality, including video and audio support and persistent MVP tree support for similarity searches based on previously processed files, but I have not wrapped any of those APIs. Try it out and let me know what you think!

An Analogy

Risk is something that you either purposefully manage or you roll the dice with your life, sometimes literally. I ride/race a motorcycle as my main hobby away from the computer. Riding a moto is a risky activity and I do several things to manage that risk:

• Always wear a helmet, gloves and jacket
• Ride a relatively low power bike
• Taken every MSF training course available
• Refuse to ride in groups

Do these guarantee I won’t crash? Certainly not but I hope they will lessen the odds and minimize any damage if I do.

Managing Risks

As engineers, what are the risks of working at a startup? The main risk is the company failing and going bankrupt. A second, related risk is being laid off. In both cases, your job and paycheck are at risk. How do we manage those risks? I have three tactics to manage the risk of working at a startup.

1) Make it as easy as possible to find a job

You could make yourself essential to the operation of the company; that helps with layoffs but does not help with bankruptcy and has the drawback that you will start from square one at the next startup. My strategy has been to make myself a valuable developer, independent of any one startup, by working on open source software and maintaining a high quality blog that evangelizes myself and my work. This is a last resort strategy: if anything happens to make my job disappear, ideally I can interview and find another job within days. This recently proved successful when I announced my upcoming move to San Francisco and had 20-30 inquiries over the next few days.

2) Exercise common sense and your math skills

Do you know your startup’s monthly burn rate, cash reserves and revenue? I’d bet that the majority of people at startups do not. Get those numbers and figure out how many months the company has before it has no money. Just a few months left? Would it be difficult to raise more money? Are you part of a “layer of fat” that could be laid off to cut the burn rate? Is revenue rising or dropping? Are you getting more customers? These are questions you should be asking yourself every month to evaluate the health of your startup. At some point you will need to leave on your own terms, before you are forced out by bankruptcy or layoffs. I left FiveRuns last year when these questions made bankruptcy look unavoidable. Leaving on my own terms meant I could take a few weeks to interview around to find the right job.

3) Stick with Success

They say failure is the best way to learn but in my experience nothing breeds success more than previous success. I try to stick with entrepreneurs that have past successes. As developers, we want to work with smart developers, yes, but you also want to work with great business guys who have a network of contacts, know how to raise funding and can navigate the company to a successful exit. I can interview a person to learn if they are a good developer but I can’t interview a CEO to learn if they are a good CEO. I have only two metrics:

• do they have a reasonable business plan with a way to make money?
• have they had previous startup successes?

The “halo” effect is very real. VCs are more willing to talk to someone who has previous success and knows the funding process. People are more willing to work at a company run by someone with previous success. Press is easier to get and customers are easier to talk to if they already know the company as the latest effort by a successful entrepreneur.

4) Educate yo’self (Extra bonus tip!)

You may know computer science but how much do you know about management or finance? Read a management book. I recommend anything by Peter Drucker – he literally invented the science of management and his writing really opened my eyes. Read a book on business finance. You’re not trying to become an expert in these fields but when you learn a little bit about the other major roles in a startup, you’ll be able to evaluate your startup’s current situation more accurately.

Even with all this, you will fail often. I’ve been part of two moderately successful exits and several bankruptcies. I’ve only been caught flat-footed once and tried to learn as much as I could from that experience. No matter what happens the startup experience is rewarding but with a little foresight you can minimize the inevitable risk to yourself and your livelihood.