Ted Dziuba

Dan Lyons vs. The Valley

2012-02-12T00:00:00-08:00

Evidently Dan Lyons is on the warpath now, against MG Siegler and Mike Arrington, formerly of TechCrunch, now of CrunchFund. You remember Dan Lyons, he was entertaining for about five minutes a couple of years ago as Fake Steve Jobs? This sort of depraved cannibalism is common in Silicon Valley tech media, so it should surprise nobody.

But I missed it all, I was too busy building a product.

See, when people spend all day making absolutely nothing any more valuable, they leave fingerprint smudges all over shit that used to be nice. That's a general observation: tech journalists, the government, anywhere that people are comfortable.

Who Needs Process?

2011-12-27T00:00:00-08:00

Software development methodology is organizational Valtrex. Sure, it treats a symptom, but the only cure for the underlying disease is to never have contracted it in the first place. This is not to say that process and methodology are bad. They are means to an end. But the ability of your team to execute on a goal is inversely proportional to the amount of process you have in place. It's not a direct correlation, though. The underlying cause is that the variance of developer skill on your team is too high, which means your team can't execute well, and you need process to wrangle the laggards.

Software development processes exist to manage the bell curve of ability in developers. It's simple mathematics. The more people you collect on a team, the more likely it is that the team's average skill is the average skill of software developers as a whole. This is the Strong Law of Large Numbers, and it is non-negotiable. There's not a meeting you can hold to make it go away. Most organizations simply accept their fate, and design policies and procedures to keep the back half of the bell curve from causing damage. Unfortunately, policies and procedures irritate top performers, and are just more grievances on their list that will some day metastasize into a resignation letter. (Side note: if someone keeps such a list, there's a high probability they are a top performer.)

So, how do you produce good software?

Rule 1: Resist Process from the Start

Anti-process needs to be in your blood from the beginning. Growing the team causes problems, but adding people who like to make process is catastrophic. Busybodies are toxic. When we were making Milo, the development process was, for a long time, loosely coordinated chaos. Of course this caused problems, and there are a few bodies in the codebase to prove it. When there was some big fuckup, a bunch of us would sit in a room (mistake 1), try to "trace the root" of the problem, which is code for "place blame" (mistake 2), and then figure out what process we need to put in place to make sure it never happens again (mistake 3).

Every time we tried to force process on the team, it failed, because process was not part of the corporate culture. Here's a concrete example:

Around the time we were raising our Series A financing, I was doing some maintenance work on our one and only database machine, Zeus. Because I had overconfigured Nagios, it was going apeshit as I was working, so I disabled all alerts for that machine. Sure enough, I forgot to re-enable alerts when I was done, and sure enough, that night, Zeus's RAID controller decided he'd had enough of our bullshit and up and died.

The site was down for 6 hours and nobody noticed because we were all asleep. Do all the root cause analysis you want on that one, I fucked up, and everyone knew it. We tried policy: don't ever disable Nagios alerts, just tell people when you're working on something so they know to ignore the alerts. But nobody followed the policy because homie don't play dat.

It turned out that nobody ever made that mistake ever again. Maybe it was just by chance, but maybe it's because of the Old Testament type stomping that the next person to do that would get.

Learn from your mistakes, but don't flagellate yourself with them.

Rule 2: Grow Headcount as a Last Resort

Headcount is not a virtue. Functionality is a virtue. Ability to execute is a virtue. But having a lot of people working on the project? No. It's a liability.

The Strong Law of Large Numbers burns you when the team grows, so be reluctant to grow the team. When you think there is too much work for your current team to handle, do what any good programmer does: profile. Quantify the amount of time your team spends on every task, then figure out what you can optimize.

When you do hire, do it carefully. Really carefully. ... More careful than that. At Milo, we do "trial periods", where we invite a candidate to work with us for a few days so we can judge their work. Here's a really simple trial period task: make this thing better. Use your judgment and your programming skills, just make it better. This will keep a common vision in your team.

Rule 3: Use The GitHub Workflow (And Other Good Stuff)

For brevity:

versus:

Seriously, what the fuck is going on in that branch model? Policy and procedure, that's what. We used the latter branch model at Milo, as GitHub was not yet ready enough to support their awesome workflow. That complicated but popular workflow is vicious, there is just too much state to track in your head as a developer. It's not something you need to spend your time on.

It's not specific to the GitHub workflow: your tooling should have the same attitude toward policy that your team does. GitHub's (the software, not the company) opinion is that process is unnecessary, and having the tools to support process will only beget process.

If you leave enough handguns hanging around, eventually someone's going to get shot.

If a tool is forcing process on you, or even encourages you to follow some process, ditch the tool and find another. Ticket tracker has a lot of fields on the ticket form? Dump it. Code review tool has a lot of shit going on? Find a new one. You get the idea. Don't let your tools infect you.

Rule 4: Just Let Go

This is the hardest one, and, if you're serious about doing away with process, is the one from which all others follow. Just stop having process. Cancel your weekly planning meetings. Get rid of your prioritized list of tasks. Stop having your daily stand-ups. No more status report e-mails.

Just stop doing that stuff, and get rid of anyone who can't cope.

Nothing terrible will happen.

My current project, which will be launching in the first half of 2012, is as close to anarchy as a project can get. And it's working. I trust my teammates to get their work done and do it well. Sure enough, they do.

Dirty Words

2011-11-09T00:00:00-08:00

Anyone who ever told you that swear words have no place in technical discussion is right. They're right, and sadly, they're part of the problem because they miss the point. The sterile word placement that's supposed to support an argument makes any true motivation indistinguishable from all the hired bullshit.

Objective technical discussion is a God damned lie, and it's the most rotten kind of lie because it's a way to stick your nose in the air, disguised as altruism. Every time you post benchmarks, you're not moving any discussion forward. When you compare web frameworks on one useless dimension or another, you don't bring any value to the world. What you've done is relieve me of a task that's kind of a pain in the ass, but by no means insurmountable. Ass scratching is not value. You want to be heralded as a great visionary for your work, and you think that getting on the front page of Hacker News or Reddit means that people respect your opinion.

No. Your link is just space between the ads, and fuck-all if mine aren't too.

The skill it takes to write objectively about technology can be automated, and publishing it yourself is disingenuous because it lacks passion.

However, when someone starts swearing in technical discussion, showing emotion, that's a strong indicator that I'm about to receive wisdom. Wisdom is earned the hard way, and it is permanent, not like some statistically shaky performance benchmark that we'll all forget about next week.

Anyone who has ever told you that swear words are a cheap way to get an audience is right, too. I've been on both the amateur and professional side of technical journalism, and I'll tell you this: every way to get an audience is cheap. Let's take Paul Graham, for example. Any emotion you detect in his essays is purely by accident, but he conveys a message, and has a following. He would not have that following if he were some guy off the street, so rattling off any damn thing and putting a Paul Graham byline on it is a cheap way to get an audience.

But admit it, somewhere in the back of your gut is a rebellious nerve that wonders what happens when Paul gets pissed off.

People like me, Zed Shaw, and Zach Holman will give you a brutally honest answer if you ask for it. People like Paul won't. You will get a response, but it's in newspaper words. The same newspaper words, that, by the way, with their self-imposed emotional blockade, allow the nicest haircut to slither into the White House every couple of years.

That's not to say that Paul is so proper in private. I don't know ﬁrst hand, but I have met other leaders in technology who are more than willing to give you an earful over cocktails. In public, however, they cultivate a persona.

Just like I do.

Straight Talk on Event Loops

2011-10-04T00:00:00-07:00

Two days ago, I pointed out how Node.js, an event-driven web framework, will eat it hard if it's given any nontrivial amount of CPU work to do in its request handler. After I published that, it seemed that the point of the article went sailing right past the Node.js camp, who proceeded to see how fast they can make a Fibonacci number generator.

The Fibonacci function was arbitrary. It was inefficient on purpose. I needed a function that would use CPU time, and chose that because it's familiar and easy to implement. So, now I offer a more formal analysis of what CPU usage does to the throughput of an event-driven system as compared to a threaded system.

Since it's now clear that reading comprehension and critical thinking are not strong suits of the Node.js programmer, I would suggest that all Noders reading this article read it aloud, slowly and loudly, like an American tourist trying to find a train station in Tokyo. Furthermore, to assist the Node camp, I will highlight the important parts in large lettering, like this:

When the weather is threatening rain, bring an umbrella with you.

A Math Model of Throughput

Assume we've got a request handler that processes an HTTP request and sends back the result. Let's see how threads and event loops differ on processing these requests. Note that we're measuring throughput here, not latency. That's an article for another day.

This is an analysis of Queries Per Second (QPS) only.

Let's start with some definitions:

Let C be the amount of CPU time used by the handler, in milliseconds.
Let I be the amount of I/O time used by the handler, in milliseconds.
Let W be the wall clock time it takes for a handler to execute. By definition, W = I + C
Let N be the number of threads running in the threaded system.
Let E be the throughput of the event driven system.
Let T be the throughput of the threaded system.

Given that the times are measured in milliseconds, we can define and .

Since the wall time W is expressable in terms of CPU time C and I/O time I, and considering that both C and I are positive, nonzero, it is helpful define , with the factor k expressing the relationship between C and I.

It follows then that and .

THEOREM 1. When a handler takes more CPU time than I/O time, an event-driven system has greater throughput than a threaded system if and only if the threaded system has exactly one thread.

PROOF (partial). (note: for brevity, I will only prove one direction. The other direction is an exercise left for the reader.)

Suppose

Simplifying the inequality,

Given that , we can bound the inner term

Further simplifying

Since N is integral and nonzero, it follows that .

If you do more CPU than I/O, use threads.

THEOREM 2.When the handler takes more I/O time than CPU time, an event-driven system has greater throughput than a threaded system if and only if .

PROOF (partial).(note: again for brevity, I will prove one direction).

Given our previous construction,

and the alternate expression

it follows that .

If you do more I/O than CPU, use more threads.

A Practical Example

Let's suppose you have a request handler that does 10 milliseconds of CPU work and 50 milliseconds of database I/O. Would you choose threads or events?

I this case, the theoretical maximum throughput of the event driven system is 1000/10 = 100 QPS, where as a threaded system with 50 threads has a theoretical maximum throughput of 50,000/60 = 833.33 QPS. Of course, in the threaded case, you need to worry about being bound by the CPU, but given the number of cores on modern hardware, threads seems like a winner here.

Multiple Event Workers

The Noders got really into this one: forking "workers" from your event loop to do the heavy CPU work, and having them call back to the event loop when they're done. One parent process coordinates work among many children? Where have I heard that before?

Anyhow, let's extend the model to that case. Just for funsies.

Since your asynchronous processes do not block on I/O, at full utilization, they will theoretically take 100% of the CPU. Therfore, the number of worker processes to spawn must be equal to the number of CPUs in the system to avoid oversubscribing the machine. Let's introduce a new variable, M, to represent the number of CPUs in the computer.

The throughput formula for the event driven system therefore becomes

Now, with threads, we also need to avoid oversubscribing the CPU. Considering that during a single handler execution, only C milliseconds of CPU are used, it follows that the number of threads that will achieve theoretical maximum utilization is .

Our formula for the threaded system's throughput is therefore

...but look at this:

At full utilization, threads and events have the same theoretical throughput.

This makes intuitive sense, as if the CPUs are working as hard as they can, all else equal, they should yield the same performance regardless of the framework used.

Hold up, this does not prove that Node is good.

Of course, in a practical setting, threads have a greater memory overhead, and programming an event loop with multiple workers just seems silly, as if you're doing that much CPU work in an event looped system, you've already fucked up somewhere, so why add to it?

Node.js Is Still Cancer

So, let's review.

Suppose you're a less-than-expert programmer, which Node seems to attract in droves for some reason. You are using Node for the supposed "scalability" of it, but as we have just seen, threaded programming, which is easier to understand than callback driven programming, meets or exceeds the asynchronous model in the vast majority of cases. Chances are, you're not going to be forking worker processes to do CPU jobs, what with the less-than-expert and all.

Therefore, the reason you're using Node is not a lack of technical ability, it's because all the cool kids are doing it.

Node.js is a danger to novice programmers.

Next, suppose you're an expert programmer, and you've got some CPU bound work that you fork off to child processes to keep your event loop trucking. OK man, how complicated do you want to make this thing? At full capacity, you're at par with threads, provided it's not memory bound. At this point, you are less focused on solving the problem at hand than you are on coming up with something you can blog about and get on programming Reddit.

If you're forking workers in Node, you've got bigger problems.

Plus, it's fucking JavaScript ... on the server.

Node.js is Cancer

2011-10-01T00:00:00-07:00

If there's one thing web developers love, it's knowing better than conventional wisdom, but conventional wisdom is conventional for a reason: that shit works. Something's been bothering me for a while about this node.js nonsense, but I never took the time to figure it out until I read this butthurt post from Ryan Dahl, Node's creator. I was going to shrug it off as just another jackass who whines because Unix is hard. But, like a police officer who senses that something isn't quite right about the family in a minivan he just pulled over and discovers fifty kilos of black horse heroin in the back, I thought that something wasn't quite right about this guy's aw-shucks sob story, and that maybe, just maybe, he has no idea what he is doing, and has been writing code unchecked for years.

Since you're reading about it here, you probably know how my hunch turned out.

Node.js is a tumor on the programming community, in that not only is it completely braindead, but the people who use it go on to infect other people who can't think for themselves, until eventually, every asshole I run into wants to tell me the gospel of event loops. Have you accepted epoll into your heart?

A Scalability Disaster Waiting to Happen

Let's start with the most horrifying lie: that node.js is scalable because it "never blocks" (Radiation is good for you! We'll put it in your toothpaste!). On the Node home page, they say this:

Almost no function in Node directly performs I/O, so the process never blocks. Because nothing blocks, less-than-expert programmers are able to develop fast systems.

This statement is enticing, encouraging, and completely fucking wrong.

Let's start with a definition, because you Reddit know-it-alls keep your specifics in the pedantry. A function call is said to block when the current thread of execution's flow waits until that function is finished before continuing. Typically, we think of I/O as "blocking", for example, if you are calling socket.read(), the program will wait for that call to finish before continuing, as you need to do something with the return value.

Here's a fun fact: every function call that does CPU work also blocks. This function, which calculates the n'th Fibonacci number, will block the current thread of execution because it's using the CPU.

function fibonacci(n) {
  if (n < 2)
    return 1;
  else
    return fibonacci(n-2) + fibonacci(n-1);
}

(Yes, I know there's a closed form solution. Shouldn't you be in front of a mirror somewhere, figuring out how to introduce yourself to her?.)

Let's see what happens to a node.js program that has this little gem as its request handler:

http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end(fibonacci(40));
}).listen(1337, "127.0.0.1");

On my older laptop, this is the result:

ted@lorenz:~$ time curl http://localhost:1337/
165580141
real	0m5.676s
user	0m0.010s
sys	0m0.000s

5 second response time. Cool. So we all know JavaScript isn't a terribly fast language, but why is this such an indictment? It's because Node's evented model and brain damaged fanboys make you think everything is OK. In really abusive pseudocode, this is how an event loop works:

while(1) {
  ready_file_descriptor = event_library->poll();
  handle_request(ready_file_descriptor);
}

That's all well and good if you know what you're doing, but when you apply this to a server problem, you've pluralized that shit. If this loop is running in the same thread that handle_request is in, any programmer with a pulse will notice that the request handler can hold up the event loop, no matter how asynchronous your library is.

So, given that, let's see how my little node server behaves under the most modest load, 10 requests, 5 concurrent:

ted@lorenz:~$ ab -n 10 -c 5 http://localhost:1337/
...
Requests per second:    0.17 [#/sec] (mean)
...

0.17 queries per second. Diesel. Sure, Node allows you to fork child processes, but at that point your threading/event model is so tightly coupled that you've got bigger problems than scalability.

Considering Node's original selling point, I'm God Damned terrified of any "fast systems" that "less-than-expert programmers" bring into this world.

Node Punishes Developers Because it Disobeys the Unix Way

A long time ago, the original neckbeards decided that it was a good idea to chain together small programs that each performed a specific task, and that the universal interface between them should be text.

If you develop on a Unix platform and you abide by this principle, the operating system will reward you with simplicity and prosperity. As an example, when web applications first began, the web application was just a program that printed text to standard output. The web server was responsible for taking incoming requests, executing this program, and returning the result to the requester. We called this CGI, and it was a good way to do business until the micro-optimizers sank their grubby meathooks into it.

Conceptually, this is how any web application architecture that's not cancer still works today: you have a web server program that's job is to accept incoming requests, parse them, and figure out the appropriate action to take. That can be either serving a static file, running a CGI script, proxying the connection somewhere else, whatever. The point is that the HTTP server isn't the same entity doing the application work. Developers who have been around the block call this separation of responsibility, and it exists for a reason: loosely coupled architectures are very easy to maintain.

And yet, Node seems oblivious to this. Node has (and don't laugh, I am not making this shit up) its own HTTP server, and that's what you're supposed use to serve production traffic. Yeah, that example above when I called http.createServer(), that's the preferred setup.

If you search around for "node.js deployment", you find a bunch of people putting Nginx in front of Node, and some people use a thing called Fugue, which is another JavaScript HTTP server that forks a bunch of processes to handle incoming requests, as if somebody maybe thought that this "nonblocking" snake oil might have an issue with CPU-bound performance.

If you're using Node, there's a 99% probability that you are both the developer and the system administrator, because any system administrator would have talked you out of using Node in the first place. So you, the developer, must face the punishment of setting up this HTTP proxying orgy if you want to put a real web server in front of Node for things like serving statics, query rewriting, rate limiting, load balancing, SSL, or any of the other futuristic things that modern HTTP servers can do. That, and it's another layer of health checks that your system will need.

Although, let's be honest with ourselves here, if you're a Node developer, you are probably serving the application directly from Node, running in a screen session under your account.

It's Fucking JavaScript

This is probably the worst thing any server-side framework can do: be written in JavaScript.

if (typeof my_var !== "undefined" && my_var !== null) {
  // you idiots put Rasmus Lerdorf to shame
}

What is this I don't even...

tl;dr

Node.js is an unpleasant software library and I will not use it.

The Craigslist Reverse Programmer Troll

2011-07-13T00:00:00-07:00

Stop me if you have heard this before. I'm a business guy, not so good on the technical side, and I've got a great idea that I need a programmer to develop for me. I don't have any funding yet, but I've got a really nebulous connection to the venture capital world. That being said, I'll start paying you once we get funding or we start making a lot of money from the project! All you need to do is write a Facebook clone in 2 weeks. For a smart programmer like you, that should be easy, right? I'll also cut you in on a little bit of equity. Let's get started!

This kind of shit lands on Craigslist so often that it makes you wonder what they actually teach at business schools (Side note: I recently learned that if you earn $1, you get to multiply that by about 20, the price/earnings ratio, meaning that the $1 you've earned is actually $20 in value. Makes me wonder how MBAs do differential equations.). It's time for we programmers to take revenge. So, a couple of months ago, I did a reverse-programmer troll on Craigslist. It went something like this:

Title:(computer gigs) Looking for Tech Idea Person

I am a computer programmer looking for a top-notch idea person to help build the next great internet company. Being a good programmer, I don't have many business ideas of my own. That's where you come in.

The perfect idea person to work with me will have:

A great business idea!

At least a passing knowledge of computers and the internet.

A vague reference to knowing somebody in the venture capital industry.

I have whipped up peoples' ideas very quickly in the past. Here is a list of some of the things I've built, and how long it's taken:

A facebook clone (4 days)

A flickr-like photo sharing web site (3 days)

A Google-like seach engine (2 weeks - longer because you have to stop spam!)

And hopefully, I can add YOUR idea to the list!

Note that I have a structured settlement from a lawsuit that I covers my basic expenses, so I don't need to be paid, but of course once we start making money, I'd like to be paid. Since I put the code together for these web sites so quickly, it's not fair to do a 50/50 split of ownership, but I would like to have at least a couple of percent.

Too obvious, right? This can't possibly generate any responses, I thought. Nope. 31 replies in about 2 hours, before Craigslist pulled the post. Here are some of the highlights:

Hi there,
I have been looking for a Man like you for years now.
i have a few superb ideas which have big potential.
please share you phone number.
we will discuss more about it.
thanks,
SP.

Swingers have a term, 'unicorn'. Look it up. It's called a unicorn for a reason.

We are introducing a similar site to Groupon.com. We are currently speaking with [VC I have never heard of] of [fund I have never heard of], who is interested in funding our project after beta is up. We businesses people who have been extremely successful in our past positions, looking for someone like you. We to get our site up and running, I already have one programmer. I would love to speak with you more, can we arrange a meeting at my Fremont office?

[Business Dude]

How refreshingly original. Do go on.

Hi,
I'm in the middle of a business plan and looking for co-founders to develop a web appliance for the medical industry. Do you know how to develop on a LAMP platform? Also, please let me know where you are located so we can arrange a meeting.
Cheers
--j

LAMP? Web appliance? Medical industry? Excuse me, I'm getting flustered by how awesome this idea probably is. Either that or it's the taste of bile. Can't tell.

People are afraid of lonely but dont't want to go out to street. People want to share and want communicate. That why social network is sucess such as facebook, youtube ...

Has anyone really been far even as decided to use even go want to do look more like?

Hello ,
I am happy to contact you with a set of fresh offerings.
- Saas
- Web 2.0
- Enterprise 2.0
- Open source consulting and Implementation

We are sure you will find it compelling. We have full time web developers and designers working with us in various technology stacks & moreover you can also hire them according to your need and get your stuff completed from your virtual team working dedicatedly for you fulltime (We can arrange for a telephonic Interview also with these designers & developers). We do have a global presence catering clients in US, UK, Australia and Canada. We have tons of expertise in developing Web 2.0, Action Script, AJAX, CSS, C, C++, JavaScript, XML, PHP, Joomla,Drupal,Wordpress, E commerce, SEO and .NET Framework, Mobile application based projects.

We can help with designing team who as expertise on designing tools such as ADOBE Suites,CS3,CS4,Flash,HTML etc

It will be great if you can have a short call/chat for a better understanding. As we work round the clock, time zone will not be a problem. Please let me know your time of convenience and a number / Skype id through which I can reach you..

For Web Application development we work for $12/hr and for Mobile application development we work for $15/hr.

Price will never be a deal breaker its always negotiable depending upon the project requirement.

Looking forward to hear from you for a win win business relationship.

Dammit, I knew I should have listed reading comprehension as a requirement.

You Can't Be Serious.
You sound like what every CL poster's dream, a poster dream
You should have added links.

This was from a fellow programmer who apparently got the joke. Jim F, keep on keepin' on, my brother.

Not long after that, Craigslist pulled the post, or, flagged and removed, which is the jargon for troll detected. Oh well, it was fun for a while.

The Most Important Concept in Systems Design

2011-06-30T00:00:00-07:00

About a year ago, I hit a turning point in my career as a programmer after reading The Art of Unix Programming by Eric S. Raymond. It was written before web apps became popular, but everything in it still applies. ESR tries to communicate the Unix Way to programmers, and if you can read and understand that book, you'll have an A-HA! moment where, for just for a second, everything will make sense. It's kind of like the first time you smoke pot and realize that everyone and everything are made out of atoms, or the first time you figure out how fuckin' magnets work.

Then, of course, the understanding is gone, and you spend a long time trying to get it back.

If you don't feel like reading it, the most important take-away for web programmers is the Single Point of Truth Rule, that is, "every piece of knowledge must have a single, unambiguous, authoritative representation within a system". If you design a system that violates this rule, you are setting yourself up for endless headaches and disasters.

Stuff You Need to Stop Doing

There are a couple of really common SPOT violations I've seen in the web world, and they almost all revolve around misconceptions about databases. No, wait, strike that. Almost every SPOT violation I've seen stems, in one way or another, from MySQL's failure at being a database. Actually, most of the NoSQL catastrophe is a product of MySQL's perverse representation of the SQL spec, but that's not the drum I'm here to beat.

1. Using Solr to Search a Database

I know you have all gone through this same thought process at least once. I've gone through it twice. You need to add search to your database-backed website, and decide to use Apache Solr. You set up an "indexing pipeline" that either sucks information from your database regularly and throws it into Solr, or you have a single entry point for inserts/updates that updates the DB as well as Solr.

The problem is that your application doesn't know what the single point of truth for records is. If results come back from Solr, do you display the data that came from the search index? Do you take those row IDs, query the database, and then display the DB-backed data? Any way you try to solve this problem, it gets painful. And, oh Jesus, what happens if your indexing bridge program fails? How long will it go on failing before you notice that something is wrong?

The Solution: Don't do that. PostgreSQL actually has pretty awesome full-text search capabilities. You can layer your query tuning on top of it if you like; that doesn't violate SPOT. Hell, you could even use Solr as your database, if it will satisfy the requirements.

2. Precaching SQL Results in a NoSQL System

This is somewhat less common, but still happens. You've got all your data in MySQL with a great object-relational model, but querying it from your web app involves a 3 or 4 way JOIN, which causes MySQL to choke, as it hasn't yet learn how to open its throat all the way. So, to fix it, you precompute some data structures and store them in something like Memcached or Redis.

Durr, what if something changes? Do you recompute the data structure on the fly? Update the NoSQL and queue the SQL write for later? You can rig up something so that it appears to work, but as soon as your "sync" script stops running, you're proper fucked, because you can't be sure which representation of the data is authoritative.

The Solution: Don't do that. If your authoritative DB is too slow, either re-jigger your data model or change databases to one that will work. If you truly want to add a caching layer to your application, do it at the outer-most point, that is, in front of your web servers. In my experience, any attempt to cache below the presentation layer just leads to consistency disasters. Plus, caching at the edge is easy: you can use Squid or a commercial content delivery network.

3. Frankendatabases

OK, I've spit enough truth about MySQL. This is a problem that I've recently had to design around: you have information from multiple databases that you want to query in one database. If you work with a legacy system, you know what I'm talking about. There's such a sweaty-palmed temptation to write a sync script to pull data from database A, database B, and update records in your database.

You get yourself into logic like this:

my_record = query("SELECT * FROM records WHERE id = x")
if not my_record:
  insert_into_my_db(their_record)
elif their_record != my_record:
  update_my_db(their_record)

This is a consistency nightmare, among other things. If the sync script breaks, blah blah blah, if you need to write new records to your DB, you're just making the problem worse for the next guy who needs to do something similar, and of course, this will turn into a shit-ton of queries at update-o-clock.

The Solution: You guessed it: Don't do that. Do your best to canonically consolidate the databases. Sometimes this isn't practical in legacy systems, so avoid the aggregation DB if you can - query both A and B directly. If you can't that, then hide your shame somewhere out of the way.

A Sign That You're Doing it Wrong

In general, if you find yourself writing a lot of plumbing code, or "updater" scripts, or if your crontab is longer than like 10 lines, chances are you've fucked up and have a SPOT violation somewhere in your architecture. Writing code like that that is tedious and painful, and pain is generally a sign that you should stop doing something.

So stop doing that.

Come to OSCON Data

2011-06-15T00:00:00-07:00

Hey, what are you doing July 25th at around 1:30PM? I know what — you're watching my live-action troll at OSCON Data in Portland, Oregon: deep inside bat country.

The formal title of my talk is "What Every Programmer Needs to Know About Disks", an overview of why everything you know about disk I/O is wrong, how vendors lie to you, and how a little knowledge of how disks work down to the hardware will make EC2 customers think Jesus, this neighborhood's really gone to shit. I've got to get out of here before those fucking brutes throw a flower pot through the window and make off with my TV.

It's going to be geared toward Linux platform programmers, as Linux will do most of your job for you if you point it in the right direction. Bad systems administrators may learn a thing or two. If you want to show up to troll me, that's fine too, just remember I'm the one with the microphone.

You can buy tickets here, and use the code os11fos for 20% off. They're not paying me, I'm in it for the luls.

Amazon — The Purpose of Pain

2011-04-23T00:00:00-07:00

Pain is nature's way of telling you that you have just fucked up. It's a hint to your future self that maybe you should never do that again. Yet, you dumbasses continue to host things full-bore in Amazon. Since its inception, EC2 has gone down, S3 has dropped off the face of the earth, and Amazon's Elastic Block Store bludgeons Reddit to death every few weeks, but yet every time, the apologists line up. (Since it's foolish to waste a perfectly good crisis, some jackass is even selling an eBook about how to design your service around EC2 failures).

There's a point where "I told you so" doesn't quite fit, so in light of Amazon's most recent aristocrats joke, let's explore some common myths about Cloud Computing that developers actually believe.

Myth 1: SLAs Are Meaningful

Amazon EC2 has a stated service level agreement of 99.95% uptime, yearly. As of right now, EC2's uptime is 99.23%, well below the SLA. Since computer programmers like to take a pathologically literal interpretation of the law and contracts, they usually don't understand the reality of such matters.

"But, but, EC2 is violating their SLA! That can't happen!"
"It just did."
"But, but...Segmentation Fault (core dumped)"

The trouble with SLAs is that shit happens is not yet in the vernacular of modern jurisprudence. You should never try to compare hosts based on SLA, compare them based on how they respond to downtime, because it will happen everywhere you go, without fail. For example, the machine that is serving you this web page is a physical box hosted by SoftLayer at a data center in Seattle. Last week, I had about an hour worth of downtime because of some networking problems in their data center. Whatever, like I said, shit happens. What I'm really looking for is communication. I logged a ticket with support, and in six minutes they updated me about the situation, how widespread it was, and an ETA on the fix. The tech also asked if there was anything else he could do for me. They restored connectivity quickly, but did not keep me in the dark about what was going on.

Try that with Amazon. There's a thread on the AWS forum where some genius decided to host safety critical software on EC2, and can't get his data up. The thread was posted on Friday, it's now Saturday, and with Sunday coming afterward, I'm pretty sure that nobody whose safety depends on EC2 is lookin' forward to the weekend. Now, maybe it's a troll, but not even a "we're working on it" reply?

Myth 2: Architecture Will Save You from Cloud Failures

Fault-tolerant architecture a centerpiece of the NoSQL dog and pony show, but by and large, the programmers using it don't understand that the software depends on hardware. Note, I said hardware not virtual machines. The trouble with using virtual machines is that your visibility into the actual metal of the device ends at the hypervisor. There are certain things that software packages must hold sacrosanct, for example, the fsync() system call, that instructs the kernel to make sure that data is written to physical disk. In virtual machine land, whether or not fsync() does what it should is a bit of a mystery. This gets even more entertaining with Amazon Elastic Block Store, which, as the Reddit administrators have found, will happily accept calls to fsync(), and lie to your face, saying that the data has been written to disk, when it may not have been.

No amount of architecture is going to save you from lying virtual hardware. Applications, especially databases, are built on the assumption that there is an atomic way to commit data to disk. Sure, there are problems with disk writeback caches sometimes, but anybody who knows what they are doing can check to see if it's actually going to be an issue. If you're running on Amazon's virtual machines, take a guess; it's turtles all the way down.

Myth 3: A Virtual Machine is an Appropriate Gift for All Occasions

This, perhaps, is the cause of such widespread service downtime — developers who are hosting entire services full-bore on virtual machines. VMs have their place, sure, but they are by no means the solution to every hardware problem. In my experience, you should use VMs for:

Web application servers
Offline data processing
Squid/Memcache servers
One-off utility computing

The general rule is that if the machine eats shit, nothing of value will be lost. Remember what I said about pain in the beginning? If you're hosting a database on a VM, well, at some point it will become abundantly obvious.

The trouble with "commodity" computing is that servers are not really something that should be commoditized. There is so much variability in these devices that a "six sizes fit all" offering is insulting. The things you can do with disk controllers alone are more than worth the effort to colocate hardware in a data center. Personally, I prefer to solve problems with hardware than software, for example, throwing SSD drives in a database machine that is dogging it on disk I/O. It's much more business-efficient to throw money at performance problems than it is to throw code at them, but I guess some of you guys just really like to type.

Personal preference, I suppose.

MacOS X is an Unsuitable Platform for Web Development

2011-03-27T00:00:00-07:00

Part of the process of becoming a new eBay employee is selecting your company laptop. I was offered a choice: Lenovo Thinkpad or MacBook Pro. Coming from a Linux development world, I picked the Mac, thinking it would be closer to what I am used to.

Man, did I fuck up.

Thankfully, I still have my Ubuntu workstation to get real work done on, but the Mac does it duty — running Outlook, maybe Firefox or Google Chrome every now and then. Oh, I also have VMWare installed on it so I can boot Windows to browser test in Internet Explorer. I should have picked the PC, at least then I would save myself the step of booting VMWare.

So what's wrong with using the Mac as a development machine for Milo, a Python application backed by PostgreSQL and Redis (or any web project, for that matter)? Well, sacred cow, here come the spears.

Horrific package management

I've only poked around a little, but so far I've found three separate package managers for OS X: Fink, MacPorts & Homebrew. Each is heinous in its own special way, but the fact that you have three competing package managers, that don't talk to each other has convinced me that Mac users, in the typical hipster fashion, brutally raped the Unix culture, throwing away everything that made it unique because they did not understand it. These Visigoths are the single best case for government mandated licensing for computer programmers.

What's wrong with having disparate package managers? Being completely comfortable with the risk of sounding like your grandfather, the problem is I'll tell you when we get to production. No, really. Installing your packages in production is going to be a pain in the balls if you are using any or all of these OS X package managers. If it's not, then you haven't been doing this long enough. I realize that if you're a Mac web developer, your deployments probably consist of ssh and git pull, but when you are older, you will understand the value of automated version dependency satisfaction. Better not tell you now, it would spoil the surprise.

The scary part about having many general use package managers is that it pushes programmers toward using programming language specific package managers like gem and pip, which only serve to metastasize the problem. I generally make Debian packages for everything, even if it means repackaging a pip package, but thankfully, there are scripts for that.

One of the unfortunate trends in OS X package management is the idea that the user should be compiling everything. This is being perpetrated mostly by the Homebrew package manager, whose basic building block is the formula, basically a Ruby script that tells it how to download, compile, and install the package. Well congratulations, dipshit, you've reinvented dpkg, poorly. I am simply trying to develop an application, is there a good reason why I am compiling libxml2 and all of its dependencies? What is this shit, Gentoo?

You don't deploy to BSD

So why the hell are you developing on it? You can't possibly expect software packages to behave similarly across MacOS X and Linux, which is probably your production environment. OS X and Linux have different kernels, which means different I/O & process schedulers, different file systems, and a whole host of other implementation details that you'll write off as having been abstracted away until you have your first serious encounter with It Works On My Machine (confrontations with this beast that were resolved by installing packages are nothing compared to the unwelcome violation you'll get when you discover a real operating system difference). These problems won't come often, but when they do, you'll conveniently forget that I ever warned you.

This is one of those lessons that it will take you a few catastrophic failures to learn. Your development platform should be as close as possible to your production platform.

Textmate sucks

Sooner or later, you have to face facts. Man up and learn Emacs.

The hardware is overpriced

A basic 15" MacBook Pro will run you $1,800 while you can buy a comparable model from Lenovo for $1,200. An extra $600 for the privilege of running a shitty operating system, where do I sign up? You could put that extra cash into time saving hardware like SSD drives, time-altering hardware like tequila, or perhaps to pay a few months worth of project hosting.

While the flesh has certainly rotted off this horse's bones by now, the price issue for Mac developers is more of an indicator of other problems: you're spending too much money on a device that looks nice, but ultimately makes your job harder. If you're a startup, this is called dick swinging and doesn't serve anybody.

'Lost' apologists are almost always Mac users. Scientists are baffled.

I used to be a big time Mac fanboy. In fact, I even had a letter published in Macworld magazine when I was 15. However, at some point, I started writing code to put food on my table, and found that the Mac just does not cut it. I'm generally all for developers using the tools that they want to use and feel the most productive with, but tools like MacOS X cause more problems for the rest of the development team, and are a net negative. Bring your pet Rhesus monkey to work if you want, but the first time that little fucker throws feces at me, he's going to the hot dog factory.

But really, Mac developers, stay out of the command line. You'll hurt yourselves.

Devops Is a Poorly Executed Scam

2011-03-20T00:00:00-07:00

I've got to hand it to the Agile development guys — they were really good at liberating money out of organizations that all had trouble with something inherently difficult. The geniuses who developed Scrum and Extreme Programming executed masterfully; selling books and training; and they made some serious bank doing it. If you hang around Silicon Valley long enough, you know to applaud the hustle. It's the classic Rainmaker scam. You pay a man to make it rain on your crops, and when it rains, he takes the credit. If it doesn't rain, he comes up with an excuse that involves you paying more money.

So, given that, I'm befuddled by the Devops movement. It's got the potential to make a handful of people a lot of money in the same way that Agile did, but nobody is really executing on it. It's proper snake oil with all the trimmings: prescription of "culture change", few and vague concrete steps for implementation, and most of all, the promise to solve an age old problem.

How do you implement Devops?

Point one. Nobody seems to know. At least with Scrum, you could buy the book and take the course. From what I have gathered by reading blogs, if you want to apply Devops to organization, you do any or all of these things:

Automate configuration with Puppet or whatever. You should be doing this anyway. Not earth shattering.
One-step build and deploy. I'm still waiting for you tell me how these steps will solve my problems.
Culture of respect & trust, good attitude toward failure. How about "culture of stop fucking up"? This is one of those obvious happy-horseshit type statements that makes you believe the salesman is benevolent. A developer who consistently ships broken code or a sysadmin who consistently pushes out broken configuration aren't going to get any better with respect or trust.
Have cross-functional team members to facilitate communication Every technical person you hire should be cross functional enough that you don't need this.

These things are all the basics you pick up by reading Learn How Not to be a Complete Failure at Software Development in 24 Hours. None of it will make your developers any less prone to do stupid shit, and none of it will prevent your systems administrators from roadblocking developers just for funsies.

One of the things I read frequently is that Devops is about building bridges and communication. What the shit does that even mean? Cute, but not useful. Clearly everybody deserves to be treated with respect in the workplace, but you can't make two different groups work together just by telling them to, or even by having cross-functional team members to coordinate. If you've hired people explicitly as peacemakers between development and ops, you fucked up somewhere in your hiring process; it's fixing a self-inflicted problem.

If you are going to pimp this stuff as "the new way of doing things", at least try to sell me a book.

What is the problem you want to solve?

The main issue with the Devops movement is that it treats symptoms, not problems. Yes, everybody wants to ship new code frequently and keep it stable, but the dev vs. ops feud is as old as the phrase "it's 98% done, I just have to test it". The symptoms of the problem are these:

Developers write code on their workstations and it doesn't work in production.
Systems administrators are slow and reluctant to change production configurations.

As a result, it takes longer to ship features than it should.

The underlying problem, however, is that dev and ops have different goals, and each group's problem solving skills is a product of those goals. The Devops movement does try to cultivate some kind of understanding that developers and systems administrators are both working toward the same end, to put food on the table, but you will never be able to effect cultural change just by saying so. Surly's only looking out for one person, and that's Surly.

What condition your condition is in

You will always have problems between development and operations if the two groups think so differently about technical problems. So, I offer a test. One technical question that will show you how different development and operations really are:

Devise a caching infrastructure for responses from the Google Maps Geocoder API.

The end. Gather dev in one room, and ops in the other, and have them each come up with an answer. If they come up with the same answer, there is hope for your organization. If they don't, put them in one room and have them work it out until there is a unanimous solution, and everyone agrees that it is the best. If they can't agree on a solution, you have problems that no methodology can fix. (For bonus points, make this a universal interview question.)

After that exercise, development and operations should reasonably be on the same page. Next, you need to implement policy that will force convergence between development and operations. This is my prescription:

Developers are in the on-call rotation If you ship a feature, you help support it. This one is first because it's the most important. If you architect a system, you write the Nagios alerts for it, and they page your phone. Believe me, you will get a crystal clear understanding of why ops throws up so many roadblocks after doing this for a few months.
Developers develop in the same environment production runs in If you deploy to Linux, you develop on Linux. No more of this coding on your Macbook Pro and deploying to Ubuntu: that is why you can't have nice things.
Downtime never happens twice After problems are fixed short-term, you make it first priority to ensure that the same failure does not happen again.

That's it. When developers are woken up during downtime, they will adjust their attitude toward operations in a hurry. Yes, the site is down because your architecture sucks. No, cosmic rays did not flip bits in RAM. Clearly, you want development and ops to solve problems collaboratively, and it just won't work if the two groups are too different.

Since no methodology peddler ever wants to say this, I will: there's a point where you're simply fucked. Meaning, you can't solve the problem with the tools available. Sometimes, you have to fire people who aren't working out. Sometimes, you're too deep in technical debt and too pressed for time to do it the "right way". And sometimes, projects fail. It is what it is. This isn't defeatist, it's realist.

I am not trying to sell you a book, I am just being honest about the problems you face. None of this amounts to a methodology, as the Devops people would have you believe. If your developers and your sys admins are so culturally different that they can't agree on a solution to a simple technical problem, then your organization will not be fixed by some sunshine-up-your-ass methodology you read about in a blog or hear about at a conference. You need to change the culture the hard way, or replace people as necessary until the culture works.

The Devops movement smells of a scam in the making, not that I have any problem with that, after all, don't knock the hustle. However, I'd rather not see people with real problems get roped in, thinking that there's a magical 12-step program that will solve deep rooted problems. It just doesn't work that way. Even so, the Devops people have a bit of traction, and they're failing to capitalize on it. You've got a good thing going here, profit from that shit. Books, training, conferences, the whole bit. Get down to it.

Monitoring Theory

2011-03-11T00:00:00-08:00

Around the time in my life when I stopped ordering drinks made with more than one ingredient, I was woken up for the last time by a hypochondriac Nagios monitoring installation. If you are on-call long enough, you cultivate a violent reaction to the sound of your cell phone's text message alert. If your monitoring is overconfigured, that reaction boils over hastily, as it will interrupt you during meals, sex, sleep — all of the basics — with the excruciating operational details of your web site.

I've since developed, with the help of some noble systems administrators, a theory around service monitoring: monitors can be informative, actionable, or both. By informative, I mean that the alert must tell you categorically that there is a problem. By actionable I mean that receiving the alert must prompt some kind of immediate response. Alerts can therefore break down like this:

Neither Informative nor Actionable

I call these types of alerts Cool Story, Bro for short. These are bits of information that do not indicate any sort of problem state, and do not prompt any action. Cool Stories are things that you should not even have alerts for. They waste your time and make you paranoid. If you want to track a metric whose state is neither informative nor actionable, make it a graph, not an alert. Cool Story Bro alerts are things like:

The load average on a server is above 20. This doesn't actually indicate a problem. In Linux, the load average is simply the number of processes in the kernel's run queue, and as long as the CPUs, disk channels, network interfaces, and memory are less than 100% capacity, the machine is not busy enough. A high load is nothing to worry about. I don't even have alerts on load average. I have production machines with load averages well over 100 that are working just fine.
A job queue has more than X work units in it. Congratulations, dipshit, your queue is doing exactly what it is supposed to do. It's a subjective decision whether or not this is a failure state. One of the reasons that I hate queues.
Some metric is greater than an empirically determined mean. I get personally offended by shit like this. "Have it alert if it's 10 more than the average"; first of all, read Zed Shaw's Programmers Need To Learn Statistics Or I Will Kill Them All, and second, alerts based on empirical data will frequently give false positives, as the measurements the alert takes are new empirical data you haven't seen before. Furthermore, any up/down check program that needs to keep state about the return results of previous checks is a recipe for tears.

Informative, but not Actionable

Informative but unactionable alerts are ones that indicate an abnormal state, but are not things that you need to handle immediately, so they should be e-mail alerts, not pager messages. They are things that you can handle during the workday, and don't need your surly, undivided attention at four o'clock in the morning. For example:

The primary disk on your database server is at 90% capacity. Is the site down? No? Then fuck off, I'll get to it.
Memory on your MongoDB server is at 80% capacity. For those of you working at Foursquare, when a critical server is approaching its maximum physical memory capacity, you should be aware, as it means bad shit may happen soon if it keeps growing.
One out of three load-balanced web servers is down. Good to know, but I don't have to get off my ass just yet.

Informative and Actionable

This is your meat and potatoes. When one of these fuckers goes off, it's battlestations. Drop your cocks and grab your socks, we got shit to fix. Yes, these are the alerts that should be waking you up in the middle of the night. Milo's production Nagios config has roughly 10 of these, double that in e-mail only alerts, and quite a few cool stories to satisfy some paranoid delusions. Some example alerts that should get your lazy ass out of bed:

The search handler of your public site serves HTTP 500. Your users probably weren't looking for that.
Your API's latency is outside of SLA. Welcome to losing-moneyville. Population: you.
The CSS statics on your home page fail to load. It's a really obscure HTTP status code, you probably haven't heard of it.

Systems Design Considerations

When you design a new system, design it to be monitorable. The basic criterion is this: there must be a stateless, deterministic way to check the system's health.

stateless: The check run at time t must not depend on the outcome of the check run at time t - 1.
deterministic: There is no random variability or subjective judgment to determine whether or not the system is healthy. Health is a binary value.

This seems obvious but I've seen a lot of smart people fuck it up. I keep harping on message queues, but they are a good example of poor system design that is unmonitorable. Consider this system:

Producer → Queue → Blocking Consumers

End to end, how do you check that this system is OK? The logical entry point for a monitor is the producer, send a job through the system and check that it gets processed by a consumer, but the asynchronous queue makes that determination a judgment call. What if it takes a minute? What if it takes an hour? Is the system still OK if the time from job production to job completion is a day?

If you need the asynchronous model, what you generally want is a spool, where you say that it is not a requirement that work be processed as soon as possible, but rather, in an offline batch job. In this case, you can simply monitor that the spool size has not overflowed the capacity of the physical device it's on, and that your processing batch job has run successfully.

If you want work processed as it comes in, think about this:

Producer → Load Balancer → Consumers

You still have a fixed number of consumers, except work is being done synchronously, and the load balancer decides which of the consumers gets the work. If all consumers are busy, the load balancer refuses incoming work, because your system is over capacity. This is a failure state, and can be deterministically measured.

But Ted you say, what if there's a lot of traffic and the consumers get backed up? Aye, welcome to the world of capacity planning. In this case, using an asynchronous queue is a crutch that helps you avoid thinking about your system's actual resource utilization, and it will come back to burn you because it us unmonitorable. In the face of increased traffic, if you're so confident that you can "spin up more queue workers", then you can sure as hell "spin up more synchronous workers".

Stupid Unix Tricks: Workflow Control with GNU Make

2011-02-26T00:00:00-08:00

I was hacking around on part of a project last week that looked something like this:

Fetch an API Response → Validate response → Munge response → Write response to a DB

If any of the steps fail, then the whole production should stop or else Bad Things happen. Furthermore, I would like to be able to pick up where I left off in the process, should one of the steps fail - the validate step was somewhat CPU intensive, so I'd rather not lose that work if it succeeds. This is a pretty common workflow, so I wanted to apply as much of the Unix Way to it as I could, in hopes that my solution would be easier and more robust. That turned out to be true.

Makefile Abuse

As chance would have it, GNU Make solved this problem for me without a whole lot of effort. Here's what my makefile looked like:

api_response.json:
    curl -o api_response.json http://api.company.com/endpoint

validated_response.json: api_response.json
    validate_response -o validated_response.json api_response.json

munged_response.json: validated_response.json
    munge_response -o munged_response.json validated_response.json

update_database: munged_response.json
    copy_response_to_db munged_response.json

clean:
    rm -f munged_response.json
    rm -f validated_response.json
    rm -f api_response.json

.PHONY: update_database clean

To execute the workflow, I invoke make -f workflow.mk update_database, which will do the following:

Compute the dependency tree: munged_response.json depends on validated_response.json which depends on api_response.json.
If any of these files is not on disk, it will recursively execute the make targets to build the missing ones.
Update the database
If any of the recursive execution fails (command returns nonzero), freak the fuck out and print an error message.

The .PHONY line tells make that the clean and update_database targets are always out of date, and need to be run every time.

Graceful and Robust

There are a couple of things that I really like about this gadget:

Fail-fast execution: if any of the steps before update_database fail, the database doesn't get updated.
Pick up where you left off: if munge_response fails after the fetch and validate steps succeed, the next time it executes, it won't fetch & validate again unless I make clean.
Small programs that do small things: instead of one monolithic program, there are 4 independent programs that perform the work: curl, validate_response, munge_response, and copy_response_to_db. This modular system is more debuggable and and robust than a single program that does everything.
Free parallelization where available: since the workflow is a linear dependency chain, make can't parallelize it. However, if there were another step that only depended on munged_response.json, say, publish_munged_response, make would be able to parallelize publish_munged_response and update_database, as they are not linearly dependent on one another.

I could have used a pretty standard pipeline to solve this problem, for example:

curl http://api.company.com/endpoint | validate_response | munge_response | copy_response_to_db

But that would not satisfy the "pick up where you left off" requirement without some Aristocrats joke within each of the processing programs to track state, and pipelines are linear - it's hard to get the free parallelization without doing some shameful things.

Feels good, man.

The Case Against Queues

2011-02-06T00:00:00-08:00

Some people, when confronted with a problem, think "I know, I'll use a queue." Now they have an unbounded number of problems.

Networked message queues like ActiveMQ, RabbitMQ, ZeroMQ, and a host of other Java inspired software tumors are crutches of systems design. I love asynchronous stuff as much as the next guy, but think of a queue like Java: it encourages large swaths of mediocre programmers to overengineer shit, and it keeps systems administrators in business by giving them something to respond to at 4AM.

Here is some of the dumb stuff that queues enable:

The Blocking Consumer

You have some work that is sometimes produced faster than it can be done; a common problem. One of the commonly problematic solutions to it is to stick the work in a message queue, and have one or more consumers that block on the queue, picking work off as soon as it's available and doing it.

What's wrong with this? First of all, it blurs your mental model of what's going on. You end up expecting synchronous behavior out of a system that's asynchronous. One of the concrete outcomes of that is the question how do you determinstically monitor this system? If the queue size is greater than zero, is this a failure state? If the queue size is greater than zero, it means that your system is over capacity, but what is your response to that? Spin more workers or let it ride? If your answer is "spin more workers", then you should be doing the work synchronously because the implication is that you care about the amount of time it takes for a worker to get to the work. If your answer is "let it ride", then how do you know when your system is in trouble: if there are ten jobs in the queue? Ten thousand?

If you are designing a system that relies on a blocking queue consumer, you should likely be doing the work synchronously, without the queue. System gets overloaded? I've got a solution for that, too: capacity planning.

Collecting Data for Offline Processing

Say you've got some events that you want to record, and then process offline in a batch job. Using a message queue for this will only lead to tears.

In such a system, you've usually got multiple data producers, and you want the data aggregated in a single place. As chance would have it, UNIX ships with a facility that can do this consistently and reliably. We call it syslog.

Depending on your queue implementation, when you pop a message off, it's gone. The consumer acknowledges receipt of it, and the queue forgets about it. So, if your data processor fails, you've got data loss. Collecting messages syslog, your processor program is just processing a text file, and can process it again if something goes wrong. Throw in some split and xargs, and you've got parallel processing. Event messages aren't text? You fucked up. Go buy a subscription to the Microsoft Developer Network.

Everybody Loves System Complexity

Obviously I have been generalizing thus far. There are host of situations where you need to separate the production of work from the consumption of work, and fully understand the consequences. I'm not hating on that asynchronous pattern, I'm hating on introducing more software into your stack unnecessarily. Can you use a database table for it? Can you use files on disk or a named pipe? Syslog? (modern syslog implementations will write to a database) Bringing new services in should be the absolute last resort, because every new service is an unknown that needs to be configured and maintained. Adding a queue to your stack isn't just adding a service. It's ancillary maintenance code, libraries, monitoring scripts - all more things that can and will fail.

Liabilities, as it were.

Multiple Concurrent Linux Distributions

2011-01-01T00:00:00-08:00

You can run multiple Linux distributions at the same time, on the same computer, without a virtual machine. Milo's production environment is a mix of Ubuntu Hardy and Lucid, while eBay's production Linux is Red Hat. Eventually, this will all converge on one environment, but in the mean time while we port, we need a way to rapidly iterate changes on a handful of Linux distributions. A virtual machine seems like the obvious answer, but that's overkill for this situation.

A Linux distribution is a kernel, some libraries, binaries, and a package manager. The kernel is the lowest level of abstraction over the hardware; everything else is fairly interchangeable. Apache running on Red Hat will make the same system calls as Apache running on Ubuntu. In theory, as long as shared library paths are managed correctly and the package managers don't trample on each other, you can have multiple distributions "running" under one kernel, no virtual machine needed.

Unix provides the chroot mechanism to keep all of the distribution files in order. There are some tools that build on chroot to support these virtual environments so that you don't have to do any bookkeeping or stupid shit with /dev or /proc. I am using Ubuntu as my "host" operating system, and a Debian tool called schroot to manage it all.

Ubuntu In Ubuntu

I use Ubuntu Maverick as my desktop operating system, while production uses either Hardy or Lucid because they are long-term support releases. Here's how you install one Ubuntu distribution within another:

sudo apt-get install debootstrap schroot
sudo mkdir -p /var/chroot/lucid
sudo debootstrap --variant=buildd --arch [amd64|i386] lucid /var/chroot/lucid http://archive.ubuntu.com/ubuntu/

Now open up /etc/schroot/schroot.conf in a text editor and add a configuration stanza that looks like this:

[lucid]
directory=/var/chroot/lucid
description=Ubuntu Lucid
root-users=root,ted
users=ted
type=directory

In practice this can be any Ubuntu release you want, just change the argument to debootstrap and your configurations accordingly. I have one stanza for Lucid and one for Hardy, but you can have as many as you've got disk space for.

Now you can use your virtual environment with schroot -c lucid:

ted@duke:~$ lsb_release -dc
Description:	Ubuntu 10.10
Codename:	maverick
ted@duke:~$ schroot -c lucid
(lucid)ted@duke:~$ lsb_release -dc
Description:	Ubuntu 10.04 LTS
Codename:	lucid
(lucid)ted@duke:~$

At this point you should probably apt-get install ubuntu-minimal as debootstrap doesn't install a whole lot.

Neat.

Red Hat in Ubuntu

Now that we've got ourselves bothered with schroot, let's make it jump. Installing Red Hat in a chroot environment is going to be tricker because we don't have debootstrap, which is a Debian tool for bootstrapping an installation. Fortunately, some enterprising programmer has unfucked the Red Hat bootstrap system with a utility called rinse.

sudo apt-get install rinse
sudo mkdir -p /var/chroot/centos
sudo rinse --distribution centos-5 --arch [amd64|i386] --directory /var/chroot/centos

If CentOS isn't your game, there are others to choose from. Run rinse --list-distributions to see what is available.

Add a schroot.conf stanza for Red Hat like so:

[centos]
description=CentOS 5 amd64
directory=/var/chroot/centos
root-users=root,ted
users=ted
type=directory

And now I can jump into CentOS 5 with schroot -c centos, and I'll prove it, too:

ted@duke:~$ schroot -c centos
(centos)ted@duke:~$ lsb_release -dc
Description:	CentOS release 5.5 (Final)
Codename:	Final

Of course, CentOS uses RPM and some derelict dependency manager called yum. You will have to yum install lsb to get the Linux Standard Base.

This is Not Virtualization

If you're asking yourself how you can get a private network interface for a chroot environment, then you've got to take a little more time to understand the problem we are solving. When I first started digging into this, my goal was to be able to run flymake in a hermetic environment. Just the tip, just for a little bit, just to see how it feels. And look where it lead.

This doesn't solve a whole lot of production problems, in fact, it will probably create more than it solves. If you're interested in chrooting your way out of a jam in production, check out FreeBSD jails. This trick is useful for development, when you need to target more than one Linux distribution, or you want a clean environment to test in without booting up a VM.

Ain't Unix grand?

The 3 Basic Tools of Systems Engineering

2010-12-07T00:00:00-08:00

One of the most important things I learned when programming for a startup is how to design reliable systems. A startup programmer needs to understand the business economics of systems design: that the goal is to create the desired functionality, not to write code. Code is only incidental, and it should be the last tool you use to solve a problem.

There are three basic tools you can use to solve a technical problem: money, time, and code. This seems obvious, but the critical point is that you must try them in that order. Out-of-order execution of these tools leads to Very Bad Things, which we will discuss later.

Money

Money is by far the best way to solve a problem because it saves time and helps you avoid writing code. You can usually use money to solve performance and scalability problems, either by buying more hardware or faster hardware. My favorite example is how solid state disk drives make disk I/O problems go away because there is no penalty for disk seek.

At Milo, we did this when we had a database performance problem: read queries were running slowly, so we spent the money to buy a really powerful server for our database: 24 cores, 64 gigabytes of RAM, and solid state disk drives. This solved the problem for the life of the company until we were acquired. It was absolutely worth the money because we then had more time to spend building the product, and no liabilities that would come from re-architecting the data model.

It is rare that money can completely solve the problem (or that you have enough money), but it is an easy tool to try first.

Time

If money doesn't work, invest time to research existing pieces of functionality that do. As I have said before, basic Unix literacy can help you know what tools are available to solve a given problem. For systems design, it helps to know what larger services are available for different classes of problems. To name a few:

Load balancing/redundancy: HAProxy
Caching: Squid, Varnish (not Memcache because it forces you to write too much code)
Database: PostgreSQL or Oracle if you can afford it. If you're using a NoSQL database, you fucked up somewhere.
Database replication: Slony-I
Full-text search: PostgreSQL, Solr (warning: if you use Solr the way I think you will, you will have multiple points of truth in your system)
Queueing: if you're using a queue, again, you fucked up somewhere.
Logging: syslog, and nothing else. Ever.

It seems obvious, but sometimes it needs to be stated: use other peoples' work to accomplish your goals. Well-known open source packages are very high quality, and are far more reliable than anything you could build yourself. Even a 90% solution off the shelf is worth it because of the time you save in maintenance.

Code

Writing code is the last resort for solving a problem. Code is a versatile enough tool that you can make it solve just about any problem, but every line is a liability. It's design, future maintenance, monitoring, testing and profiling. Write code only when you have proven categorically that money and third party software don't work.

As a side note, when you are testing the code, I have found that unit testing is a losing investment. Acceptance tests, however, are the most cost effective way to manage the risk that new code introduces, in terms of time spent developing.

Using the Tools Out of Order

The worst thing you can do is to try the code tool first, without considering money or time. This is called technical masturbation and it can sink a project in a hurry.

When the first thing you do is dive into code, you are dooming yourself to either designing an unmaintainable system, or to reinvent existing tools poorly. This may be acceptable in an academic or research setting, but in a startup, it's downright foolish. You may be able to deploy your system faster if you code it all yourself, but it will be a monkey on your back for its entire lifetime. PostgreSQL has never woken me up in the middle of the night with a segmentation fault or NullPointerException, but databases I've written myself have.

Functionality is an asset, but code is a liability. I will say this until you like it.

Taco Bell Programming

2010-10-21T00:00:00-07:00

Every item on the menu at Taco Bell is just a different configuration of roughly eight ingredients. With this simple periodic table of meat and produce, the company pulled down $1.9 billion last year.

The more I write code and design systems, the more I understand that many times, you can achieve the desired functionality simply with clever reconfigurations of the basic Unix tool set. After all, functionality is an asset, but code is a liability. This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them - Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions.

Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.

The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script.

Moving on, once you have these millions of pages (or even tens of millions), how do you process them? Surely, Hadoop MapReduce is necessary, after all, that's what Google uses to parse the web, right?

Pfft, fuck that noise:

find crawl_dir/ -type f -print0 | xargs -n1 -0 -P32 ./process

32 concurrent parallel parsing processes and zero bullshit to manage. Requirement satisfied.

Every time you write code or introduce third-party services, you are introducing the possibility of failure into your system. I have far more faith in xargs than I do in Hadoop. Hell, I trust xargs more than I trust myself to write a simple multithreaded processor. I trust syslog to handle asynchronous message recording far more than I trust a message queue service.

Taco Bell programming is one of the steps on the path to Unix Zen. This is a path that I am personally just beginning, but it's already starting to pay dividends. To really get into it, you need to throw away a lot of your ideas about how systems are designed: I made most of a SOAP server using static files and Apache's mod_rewrite. I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python.

If you don't want to think of it from a Zen perspective, be capitalist: you are writing software to put food on the table. You can minimize risk by using the well-proven tool set, or you can step into the land of the unknown. It may not get you invited to speak at conferences, but it will get the job done, and help keep your pager from going off at night.

Programming Things I Wish I Knew Earlier

2010-09-04T00:00:00-07:00

Programming in a startup is much different than programming at a big company. At a startup, not only are you the developer, but you are also the systems administrator for the most part. I've been startupping for three years now, and have had my ass kicked enough times to step back and think that maybe I should learn how to do things the right way rather than try to bludgeon my way through with raw intellect.

These are the things I wish I had known in the beginning, or at least I wish I hadn't been too subborn to learn.

How To Avoid Over Complicating

Like most software people, I have a natural tendency to over-engineer things. To help fight this urge, I've come up with two simple rules to help myself avoid most of it:

If you are writing a program that touches more than two persistent data stores, it is too complicated.

And the the hard disk counts as a persistent data store. This is kind of a reinterpretation of the Unix way of having small tools with a single input and single output. Tracking state across many different persistent stores and handling failure cases is just too much for one program to do.

If Linux can do it, you shouldn't.

Don't use Hadoop MapReduce until you have a solid reason why xargs won't solve your problem. Don't implement your own lockservice when Linux's advisory file locking works just fine. Don't do image processing work with PIL unless you have proven that command-line ImageMagick won't do the job. Modern Linux distributions are capable of a lot, and most hard problems are already solved for you. You just need to know where to look.

Parallelize When You Have To, Not When You Want To

I know it seems obvious, but sometimes I need to tell myself explicity: if the physical machine is not the bottleneck, do not split the work to multiple physical machines. It is usually pretty apparent when you have to parallelize a CPU bound job, but for I/O bound stuff, you have to do some more in-depth measurement.

For example, if you are doing web crawling, and you have not saturated the pipe to the internet, then it is not worth your time to use more servers. This guy got me thinking about it, he's doing "Large-scale HTTP fetching" in Clojure. He talks about parallelizing with some queueing silliness, but never mentions how much data is moving down the pipe on any one machine. If you have a 100 megabit connection to the internet, and your fetcher is using 700 kilobits, then figure out why your fetcher sucks. (As a side note, I was talking about that post with Milo's prolific systems administrator, and we could not figure out whether the author was an incredibly elaborate troll or just a run-of-the-mill idiot.)

This tidbit also goes for data storage. I know Cassandra is all neat and whiz-bang, but I can pretty much guarantee that you don't need it. Multi-terabyte drives are cheap, and PostgreSQL is a known quantity. It's just not worth the risk.

How to Babysit a Process

This is one I only recently learned. If you have a process running and you want it to be restarted automatically if it crashes, use Upstart. Upstart is a replacement for the init daemon that can do a lot of cool things, one of which is restart a process if it crashes. An example Upstart config to do this would look like this:

respawn              # Respawn this process if it dies

respawn limit 10 600 # If you have to respawn 10 times 
                     # in 10 minutes, give up

exec python /path/to/my/program.py

NoSQL is NotWorthIt

I've talked a lot of shit on NoSQL in the past, but I recently decided to see if I had been living a lie. I tried to use Redis for some not-as-mission-critical systems at Milo - applications where data loss is not that big of a deal. Redis, even though it's an in-memory database, has a virtual memory feature, where you can cap the amount of RAM it uses and have it spill the data over to disk. So, I threw 75GB of data at it, giving it a healthy amount of physical memory to keep hot keys in.

For the most part things went smoothly, until it hit the wall. The Redis server would hang, accepting new sockets, but not servicing any requests. Replication stopped working. When re-starting the server, it would take forty-five minutes (!) from invocation time before it was ready to serve requests.

To nobody's surprise, I was right. Redis was an unknown quantity, both in how much data it could store reliably and how performance degraded. Yes, maybe things could have been different if I used Cassandra or MongoDB, but the point is the same: newfangled stuff is not worth the risk, especially if something like PostgreSQL can do the same job.

Event Loops are Just Okay

One of the most positively retarded things I've ever read comes out of the node.js home page, describing why nonblocking I/O is so great:

Almost no function in Node directly performs I/O, so the process never blocks. Because nothing blocks, less-than-expert programmers are able to develop fast systems.

Statements like this give me The Fear. Nothing ever blocks, huh? What about the callback that Node runs for new requests? If that does any CPU work, it sure as hell blocks. If each callback does 100 milliseconds of CPU work, then the Node server will only be able to handle 10 requests/sec as a theoretical maximum, because the event loop doesn't pick up new requests until the callback is complete. Scalability indeed.

Nothing is more dangerous than a programmer who doesn't know what he doesn't know. Event loops work well if your server is heavily I/O bound, whereas if the server needs to do some nontrivial CPU work, you may be better off with threads. Hell, you can even use both, like Nginx does (well, worker processes, at least), to hold lots of sockets open but still do CPU work asynchronously.

The point is, evented I/O is not magic scalability pixie dust, and like anything, there is a tradeoff.

Hardware Matters

Cloud computing was built for suckers by hustlers. The physical machine your programs run on can make all the difference in the world when it comes to performance and reliability. I am not talking about using an Extra Large EC2 instance for your database because it's "beefier", I am talking about understanding down-to-the-metal, what the performance characteristics of a system are.

For example, in a heavy write throughput application, you want fsync() to return as quickly as possible. (For those of you using MongoDB, fsync() is the system call a program makes to synchronize writes to disk). To the software, fsync is a black box that you can't muddle with, that is, unless you have your shit together in the hardware. If you spend the extra money on a battery-backed RAID controller, fsync can return almost immediately, because the controller can hold writes in battery-backed memory and guarantee that they will be flushed to disk, even in the event of a power failure. On a database machine, you will see a significant write performance increase if you use the correct hardware.

Suppose again that you are serving data from disk in a heavy read throughput application. Data is accessed randomly, so to service those requests, the disk's read heads are scurrying about the platters constantly. With EC2, when Amazon says "I/O performance: High", what does that even mean? Is that suitable for a heavy random read scenario? Again, knowing your shit when it comes to hardware is valuable here. Solid-state hard disks, while expensive, have unbelievable random read performance. (Their sequential read performance isn't amazingly better than spinning disks, though) Spending the extra money on SSD drives is almost always a win, if you have an disk bound problem.

Hardware is one of the best places to put capital to work. It is far more efficient to buy your way out of a performance problem than it is to rewrite software. When running your app on commodity hardware, don't expect anything better than commodity performance.

Too Smart for Git

2010-08-01T00:00:00-07:00

The worst part about Git is the "Git-SVN Crash Course" tutorial, because it will convince a newcomer that their understanding of how source control works transfers smoothly to Git. If you combine that with the fact that most smart people simply refuse to read documentation and will just mash on the keyboard until it appears to work, it means that teams who are starting out using Git are on the yellow brick road to a world of hurt.

Git follows Linux's philosophy of refusing to protect you from yourself. Much like Linux, Git will sit back and watch you fuck your shit right up, and then laugh at you as you try to get your world back to a state where up is up and down is down. As far as source control goes, not a lot of people are used to this kind of free love.

Milo has been using Git from the get-go, and the git pull/git push with a central repository works well enough when you only have a handful of developers.

This fits with the Subversion worldview that a benevolent central server will never lead you astray. The first time you get some error about not being able to push non-fast-forward commits to origin, you kind of gloss over it, and a git pull fixes you right up. If you understand Git as you understand SVN, you can easily mistake this error for something like "oh, I just have to do a merge because someone else's commits beat me to it". Well, technically that's right, but for the wrong reasons. Using this model, if you think of the developers as different threads sharing the origin repository resource, Git is putting lock on the whole repository, as opposed to individual files. With a lot of threads sharing a coarsely-locked resource, there's always going to be contention. However, in a small enough team, it's workable to think of it this way.

This is where smart people run into trouble. We know in the backs of our minds that something is just a little off about that non-fast-forward-commit error message, but everything looks fine, so why go read about what's going on? I am generally not a fan of reading documentation until I am debugging a problem, because I expect software to behave as it would if I wrote it. Good documentation is condescendingly terse, and Git's documentation is like an art critic who giggles at you. (The description of git-rebase is "Forward-port local commits to the updated upstream head". Oh, fuck off.)

In the last couple of weeks, we started using Gerrit for code reviews, and anyone who was using Git like a translation table for SVN commands just heard the bell ring at the School of Hard Knocks.

Gerrit acts as an intermediary between developers and the origin repository. You send commits to Gerrit and it holds them in purgatory until they are signed off on by another developer. Then, if the commit applies cleanly to the branch, Gerrit applies it. Otherwise, it asks you to upload a merge commit, which is where the fun really starts.

The upside to this system is that it prevents other peoples' git-fuckups from becoming your git-fuckups. The downside is that because commits are held in this transient state, it's very easy to lay a path of destruction through your local repository with rebase and merge if you don't completely grok what's going on. Like old skid marks on a highway, our codebase is peppered with merge commits and cherry-picks that are the only living records of a series of oh-shit moments.

The problem isn't that Git is to hard, it's that smart developers are impatient and have exactly zero tolerance for unexpected behavior in their tools. While Git is the trendy thing right now, perhaps some day you will come across a grizzled developer who is using SVN, and when you ask him why, his answer won't make sense, because it's a Zen thing.

SEO Is Mostly Quack Science

2010-06-12T00:00:00-07:00

Most college mathematics or computer science departments have a "crank bin": a box with collected papers that people have sent in for review. There are all sorts of gems in there: a two-page proof of the Riemann hypothesis, a drawing that demonstrates P = NP, and of course, a draft of a patent application for a free energy machine. Many professors just throw this crap out, but some collect it because it makes for a good read when you're feeling discouraged by a hard problem.

Me, whenever I need a pick me up, I go read some of the latest new techniques for SEO. There are a handful of fundamentals about page design and other nitty things like URL structure that are generally accepted as good SEO, and you can derive all of this from the principles of not completely failing at web design. Non-brain-damaged web design and link building are 100% of SEO.

Anyone who tells you different is a quack that is only trying to separate you from your money.

Quackery in medicine is pretty easy to spot, and quackery in computing is pretty similar:

"Research" performed by people with slim technical backgrounds
Suspect experimental controls or no experimental controls
Little investigation of alternative explanations for phenomena
Little to no data reported from findings

Let's look at my favorite example: SEOMoz. It's a wealth of collected information about SEO, almost completely anecdotal, and of course you can get access to more "professional" information for a fee. There probably a place on the site where you can buy Acai berries, but I haven't found it yet.

SEOMoz recently published a correlation analysis of ranking factors for Google and Bing. First of all, out of all the factors they measured ranking correlation for, nothing was correlated above .35. In most science, correlations this low are not even worth publishing. As a warm-up, the author explains a graph that shows negative correlation with rank for URL length and .com TLD extension, meaning that longer URLs were less high up in the search results as were URLs that came from .com domains:

This is the explanation, verbatim:

The data for URL length shows that longer URLs are negatively correlated with ranking well. This isn't particularly shocking, and it probably iswise to limit the length of our URLs if we want to perform well in the engines. However, the second data point on .com TLD extensions shouldn't necessarily suggest that using .com as your top-level domain extension will actually negatively affect your rankings, but merely that all other things being equal, .com domains didn't perform as well in the dataset we observed as other domain extensions.

That is not how science works. You can't discount data just because you feel like it. Also notice that the most negative correlation metric they found was -.18. A correlation of zero suggests that the two variables are completely independent of one another. Such a small correlation on such a small data set, again, is not even worth publishing.

There is no hypothesis being tested here. It's just graphs, and misleading graphs at that. The sad part is, SEOMoz is as close as the SEO industry comes to real science. They may be presenting specious results in hopes of looking like they know what they're talking about, but at least they are collecting some sort of data.

Everything else in the field is either anecdotal hocus-pocus or a decree from Matt Cutts. When you hire an SEO consultant, what you are really paying for is domain experience in the not-failing-at-web-design field. It's fine to pay for this kind of service, but beware of anyone who claims to have studied the effects of different techniques. They might give you skin failure.

Update: It looks like I am not the first one to notice this. Here is a good article with more statistical formalism on SEOMoz quackery.

The Future of Apple's Curated Computing

2010-05-15T00:00:00-07:00

Here’s to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They’re not fond of rules. And they have no respect for the status quo. You can praise them, disagree with them, quote them, disbelieve them, glorify or vilify them. About the only thing you can’t do is ignore them. Because they change things. They invent. They imagine. They heal. They explore. They create. They inspire. They push the human race forward. -- Apple ad, 1997.

My first computer was a 33MHz Macintosh Performa 637-CD. It had 8 megabytes of memory. It was one of the newer Macs that featured a CD-ROM drive. We bought Apple's 2400 baud modem as an accessory, and signed up for eWorld.

The one thing that got me into exploring computers in the first place was learning how to hack the game Escape Velocity using ResEdit, Apple's pistol-without-a-safety tool that was supposed to be for developers but available to anyone. (I became an expert at fresh-installing MacOS 7.5)

At the time, I was about as fanboy as they came. This was around the era when Apple was long considered the walking dead, only kept standing by the mercy of Adobe continuing to release Photoshop, and Microsoft, possibly to keep the antitrust litigators out of their ass, publishing Office for the Mac. Being a Mac user in the late nineties, you had this feeling that you were on the side of right — that the competition wasn't about megahertz or gigabytes, but that the counterculture was the spark that would give way to the natural order of things. We thought we were on the ground floor of the inevitable.

Apple now stands as a monument to the failure of that free-thinking counterculture. We thought that freedom from the tie-wearing, meeting-holding, memo-dictating corporate world was going to be the catalyst for utopian computing, and that Steve Jobs had the vision for how it was all going to work. Maybe he did, maybe he didn't, but that utopian endgame is quickly de-evolving into a dictatorship.

It started with Apple's tight control on the iPhone app market, the approvals process, and the well-manicured app store. Now, Apple is not only dictating what applications may or may not run on the iPhone or iPad, but they are also dictating the language in which apps must be written. Their justification for all of this is "for the good of the user", but it might just be the capstone delusion of an aging hippie who never got a chance to run for Congress. I predict that within five years, Apple will begin telling development shops what kinds of apps they should make. Why? Because it will be "good for the user", and you know, Mr. iPad developer, apps that are good for users usually sail right through the approvals process. Apple's iPhone/iPad department will be renamed Central Planning, and may God help you if you cross them.

I could be wrong, though. The backlash had been pretty severe, to the point where it may be getting to Steve Jobs. Take, for example, a recent e-mail exchange he had with a Gawker reporter, in which Jobs took a shot:

By the way, what have you done that’s so great? Do you create anything, or just criticize others work and belittle their motivations?

Back when I was writing Uncov, I would see this particular flavor of ballache pretty frequently, and it was a very good indicator of the person's nerves. The thing is, being a CEO, you need to be able to let the critics roll off your back. We talk shit, it's our job, and the bigger the shit we talk, the more we get paid. Most executives know this, and don't respond to us trolls. It's only when they're starting to wear down do they bust out the Teddy Roosevelt man-in-the-arena speech. (By the way, Theodore Roosevelt was shot in the chest once, and proceeded to deliver a speech with the bullet still in him. He left the bullet in his body until his death seven years later. Executives: That shit's hard core...you are not TR.)

I will still probably buy an iPhone some day because they are very cool. However, I will never develop for it, because I'm a crazy one. A misfit. And I'm not fond of rules.

Now where did I pick up that idea?

Why Engineers Hop Jobs

2010-05-01T00:00:00-07:00

What's with all the hate on my generation? It started when somebody quit Jason Calacanis's industrial web spam startup, Mahalo, for a higher paying position at a competitor. Invariably, Calacanis went apeshit on the poor guy in a very public way, and this started a cascade of blogosphere butthurt about people in software under thirty: that we're unreliable, that we're lazy, that we're entitled.

Well I'm as unreliable, lazy, and entitled as the next guy, but that's not why I've hopped jobs in the past. People in my generation have a very low tolerance for bullshit, and software engineering, in general, is a very high bullshit career. If you couple that with the standard load of bullshit you would get from a non-technical Harvard MBA type boss — like many CEOs that you find trying to get rich in Silicon Valley by hiring some engineers to "code up this idea real quick" — it's no wonder that a good engineer will walk off the job after his one year cliff vesting.

As an engineer, you are told that you're "lucky to have a job", because there are "a hundred people lined up outside, ready to take it". (As chance would have it, there are at least a thousand lined up to take the job of rich prick who tells people what to do). This backlash is the product of diseased thinking. A CEO who makes an engineer work 80 hours a week is a driven entrepreneur, but an engineer asking for a comfy chair is a prima donna. So, when we are up to our knees in golf-course, martini-lunch bullshit, don't be surprised when we jump ship for a higher salary.

I recognize the value of business people and management. Somebody has to sell the code that I write, which in turn puts food on my table. Since I am an engineer, I like iterative optimization. Every time I have left a job, I have further refined the requirements that a person must fill before I agree to work for him. After every job, I add one or two requirements to the list, and I have found that my happiness at work improves dramatically with every step.

This is my current list:

The organization must need me at least as much as I need it.
My direct manager must have a technical background — enough to understand why programming is hard.
My direct manager must have enough experience or raw intelligence such that I can trust him/her to make decisions, even though I may not understand the reasoning.
I must have absolute faith in the business plan.
I must have absolute faith in "the business side" to execute that plan.

So, Jason, when that fellow quit Mahalo, he didn't just leave you in the lurch. He added something to his list. Maybe you should find out what that is.

Blog Upgrade

2010-04-04T00:00:00-07:00

So that's it, I'm finally done with Movable Type. I had upgraded my 4.x install to the newest 5.x, and the process was nothing but a colossal fuckup. After the fact, my site was compromised 4 times by my last count — some bot set up a handful of phishing pages here. RSA security caught it, notified Dreamhost, and they shut me down a couple of times.

Anyhow, rather than figure out the attack vector with Movable Type, I decided to scrap it and use Jekyll. It only took me a day and a couple of angry Python scripts to migrate all my shit to Jekyll from MT. Comments are still off because I don't care in the slightest what people have to say, and certainly not enough to slow my stuff down with Javascript.

One of the reasons I used MT in the first place was that it generated static HTML pages for all posts, instead of doing something silly like querying a database to generate what amounts to static content. Because of this, my pages load (first request to final render) in about 300 milliseconds on my home connection. For comparison, techcrunch.com can take upwards of 1 minute from first request to final render.

In this regard, Jekyll feels right. I can keep everything under version control, the templating is only marginally braindead, and the publishing step is rsync. After using Jekyll, I feel like every other blogging engine out there is telling me, "You'll shoot your eye out, kid!".

It's also come to my attention that the re-do with Jekyll has caused my posts to show up afresh in all of your Google Reader accounts. This was unintentional, but a nice benefit. It's true, this is the greatest web site on the internet, and everything you need to know, you can find out here.

I Can't Wait for NoSQL to Die

2010-03-04T00:00:00-08:00

They don't teach you this in college, but the fundamental theorem of the software industry is the idea that everything needs to be rewritten all the time. As a corollary, web startup engineers believe that there is no problem but scalability, and architecture is its solution. And thus, the NoSQL movement was born.

The idea is that object relational databases like MySQL and PostgreSQL have lapsed their useful lifetimes, and that document-based or schemaless databases are the wave of the future. Never mind of course that MySQL was the perfect solution to everything a few years ago when Ruby on Rails was flashing in the pan. Never mind that real businesses track all of their data in SQL databases that scale just fine. (For Silicon Valley readers, Walmart is a real business, Twitter is not.)

Invariably, all web projects start off with something like Rails or Django, most likely backed by MySQL. The data relationships are easy to model, and the application works well. If you are lucky enough that people actually use your application, eventually you will start to see some performance issues. At this point, a developer who values technological purity over gettin' shit done will advocate "rewriting the whole thing in a weekend using Cassandra". And if he's smart enough, he might just pull it off. (Of course, said developer has only migrated the app to use a different data store - all of the ancillary support code was conveniently ignored)

So you've magically changed your backend from MySQL to Cassandra. Stuff will just work now, right? Well, no. Did you know that Cassandra requires a restart when you change the column family definition? Yeah, the MySQL developers actually had to think out how ALTER TABLE works, but according to Cassandra, that's a hard problem that has very little business value. Right.

I'm not just singling out Cassandra - by replacing MySQL or Postgres with a different, new data store, you have traded a well-enumerated list of limitations and warts for a newer, poorly understood list of limitations and warts, and that is a huge business risk.

You Are Not Google

The sooner your company admits this, the sooner you can get down to some real work. Developing the app for Google-sized scale is a waste of your time, plus, there is no way you will get it right. Absolutely none. It's not that you're not smart enough, it's that you do not have the experience to know what problems you will see at scale.

Besides, did you know that Google Adwords is implemented on top of MySQL? What, that business critical code that operates at massive scale doesn't use BigTable? No, in fact there is such enormous value in sticking with what works that Google identifies problems with InnoDB at scale and submits patches, instead of saying "MySQL doesn't scale, let's dump it for something else".

NoSQL will never die, but it will eventually get marginalized, like how Rails was marginalized by NoSQL. In the meantime, DBAs should not be worried, because any company that has the resources to hire a DBA likely has decision makers who understand business reality.

Top photo credit Paul Russell

Eventlet: Asynchronous I/O for Grownups

2010-02-11T00:00:00-08:00

Event-driven asynchronous I/O is the newest chatter at the Silicon Valley High Abercrombie table. Threading, the mode of parallelism we all thought we were so smart for understanding, isn't cool anymore. Everybody who is anybody is using asynchronous I/O, and of course, there are different opinions on how it should be done. This being the software world, you can count on those opinions being vehement.

If you look at the benchmarks, all of the major async libraries for Python are basically on the same operating plane. There's Twisted, Tornado, gevent, and a handful of others, but the one that really stands out in the group is Eventlet. Why is that? Two reasons:

1. You don't need to get balls deep in theory to be productive with Eventlet.

2. You need to modify very little pre-existing code to adapt a program to be event-driven.

Eventlet's approach is that asynchronous code should look like synchronous code. Why? Because it's easy for people to understand synchronous code. Thinking about callbacks and schedulers is unnecessary, after all, we have work to do. What's more, not only does asynchronous code with Eventlet look synchronous, it can also run synchronously.

Look at this Python snippet:

def fetch_and_parse(url):
    contents = urllib2.urlopen(url).read()
    tree = lxml.html.fromstring(contents)
    # Do some parsing on the ElementTree
    return value

It looks like regular synchronous code, and ostensibly it is. The output of the URL fetch is the input to the HTML parser. However, if you have a ton of URLs to do this to, how would you parallelize it? Threads are an option, but so is Eventlet:

import eventlet
from eventlet.green import urllib2

def main():
    green_pool = eventlet.GreenPool(size = 10)
    results = []
    for result in green_pool.imap(fetch_and_parse, urls):
        results.append(result)

This is interesting because all I've done to make a seemingly synchronous piece of code run asynchronously is to patch the library it needs for I/O and give it a driver method. That driver class could have easily been a series of threads all reading from a Queue, and importing the standard library's version of urllib2.

Now hold on a second. This is a painfully contrived example, but it's such a key point: The asynchronous code looks synchronous. It can even function synchronously. All I did to make it use event-driven I/O is change the driver and patch a library. Now this is podracing!

That sort of integration has such a massive business value that I will easily disregard any pissing-contest performance gains that Twisted or Tornado may offer. I know that when you have code written in the "old" style, and the powers that be hand down the "new" style, there is an itch to re-write it, but rewriting known-working code is the worst thing you can do for your project.

The Eventlet developers have gone further than this, providing a facility to monkey-patch the existing system libraries at invocation time. For example, let's say you have a web app that does some Memcached I/O and some database I/O.

from eventlet import patcher
patcher.monkey_patch(all = True)

Oh look. Your application is now using asynchronous I/O. This call patches Python's socket module and a few others to make it all "just work" with Eventlet's internal coroutine switching mechanism. (Caveat: MySQLdb, which uses C-land sockets, needs a little bit of extra treatment, but it's only a couple of lines)

This all sounds great in theory, but I have actually made a large I/O bound program work using monkey patching and changing the driver. It is a piece of software that reads jobs from a queue and processes them, putting the result in memcached. For esoteric reasons I will not go into, the job processors could not thread the work, they had to fork. Using this setup, one production box with 8GB of RAM was consistently 7.5GB full. After a less than 5 line code change to the driver, that same production box uses only around 1GB of RAM consistently, and can handle 5 to 10x the throughput of the old system.

Now compare this to Twisted or Tornado. Twisted tries so damn hard to be Java that it really offends me personally. Those developers strike me as the alpha-programmer types who see no reason not to rewrite an existing codebase for a 20% performance gain. Tornado on the other hand is significantly less Jersey Shore douchebaggy, but they still miss the point: we are programmers who need to get stuff done. Inventing your own HTTP client class, when Python's builtin works just fine if not better is the type of hubris that gets hotshot programmers fired in their first month.

There's also gevent, which appears to be a fork of Eventlet, but is not as well documented. Partial credit.

It's hard to find a performance or scaling related open source library that values my time. Eventlet is one of those rare few.

http://eventlet.net

Break My Concentration and I Break Your Kneecaps

2010-01-24T00:00:00-08:00

I own a good set of headphones that fully enclose my ears. I am not an audiophile, I just don't like to hear other people talk at me. When I am staring at my Emacs windows with headphones on, it generally isn't a physical cue that I am looking for conversation. In fact, when I am that deep into thinking out a problem and I get interrupted, I think about the anti-workplace-violence clause in the employee handbook, and how a poorly lit parking lot probably doesn't qualify as "company property".

Interrupting a thinking programmer is a sucker punch to productivity's kidney. Of course it's still important to keep open communication channels, especially in a small team. I don't mind answering questions and helping out, so long as it's not an immediate context switch for me, i.e. I'll help you if I don't have to speak.

Instant messaging is a decent first attempt, but it's only person-to-person communication. (And no, group-IM never fucking works right) Programming teams need group chat. White-label Twitter clones like Yammer are okay, but I feel icky using a product that is hailed as a technological advance for supporting the ability to identify topics by prefixing a word with a pound sign. That, and I want to keep an eye on the conversation as I work, and my attention isn't on my IM client or browser when I'm coding. It's on Emacs.

The answer, of course is IRC.

My team recently grew, and four of us need to communicate constantly. I set up an IRC server and brought people in. One non-programmer who needed to be in the loop had never used IRC, but caught on quickly. Productivity is up, as is communication. The developer chat channel is right in front of me as I work, as a window in Emacs:

Think of developer communication like I/O. There's blocking and nonblocking. When somebody talks to me as I work, my programming train of thought needs to block. With inline chat like you see above, I can answer questions when I have spare cycles. Since the conversation is integrated into my development environment, I don't need to look around at other applications, and there's no popup notification bouncing around like a Jack Russell terrier who got into my Adderall supply. Also since it's Emacs, it's not vim. If you use vim, /quit #life.

Collaboration technology doesn't need to be re-invented every six years. The stuff we had in the eighties works just fine.

Options for Parallel Compression

2010-01-15T00:00:00-08:00

At Milo, I pretty frequently need to pull data down from production to my workstation to test some new code. That's what happens when you raise a Series A round - you can't live-edit production data anymore. I think it's in the term sheet somewhere.

Anyhow, I was pulling down a 14GB MySQL database dump today. Trying to compress it through plain Jane gzip was pretty slow, so I looked for some parallel options. The server I was pulling from has 16 cores, so I figured I could make use of them. Anyhow, here's what I found:

pbzip2 - Parallel BZIP2: Parallel implementation of BZIP2. BZIP2 is well known for being balls slow, so speed it up using multiple CPUs.
pigz - Parallel GZIP: Parallel implementation of GZIP written by Mark Adler (guy who co-authored zlib and gzip, so you can be reasonably confident he has his shit together).

On the 14GB database dump, both are faster than vanilla GZIP. Because Hacker News and Reddit both love this shit, here are the timing stats:

Plain gzip, default compression level: 11 minutes, 58 seconds. Resultant file is 2.3GB.
pbzip2, default compression level: 8 minutes, 48 seconds. Resultant file is 1.7GB.
pigz, default compression level: 1 minute, 33 seconds. Resultant file is 2.3GB.

Again this was on a 14GB database dump file, on a 16-core machine, with Intel solid state disks.

If any readers know of other parallel compression schemes I can try, e-mail me and let me know. I will post stats here.

I Love the GPL (Except When it Applies to Me)

2010-01-02T00:00:00-08:00

Boy do I love free software. It is usually pretty high quality, I don't have to pay for it, and I feel completely justified in criticizing the maintainers on public mailing lists for not supporting the exact features I need. Of course I'm not going to send patches back, because it's just way easier to bitch and moan.

Also, since my software product is a web service, I have exactly zero obligation to contribute anything back to the community, ever. Sure, I may use some GPLed software, but shit, actually following the spirit of the copyleft? Don't they know this is a business, not a charity? Fuck that noise.

I came up in the salad days of Slashdot, when the cast of villains and henchmen included Microsoft, SCO, and anyone else who wanted to turn a dime from software. We believed in the GPL, that a viral copyleft clause was good for humanity. That is, until we left academia and had to pay the rent.

Since the world appears to be moving toward software as a service (against my sage advice, mind you), it is blisteringly easy to be a champion of the ideals behind open source and free software, but still pussyfoot around when it comes to execution. What I'm talking about is the loophole in the GPL that exempts application service providers from having to release their derivative works under the same license as the libraries.

The pedantic reader who is going to talk shit will point out the difference between open source and free software. So, before you write a blog post that nobody's going to read, allow me to demonstrate.

Open Source: I want to let others use my code in whatever manner they please, and not be bound by an anti-commercial license.

Free Software: I found a loophole in my student loan documentation that lets me defer payments for decades, so long as I stay in the Ph.D. program!

If anything good comes out of Web 2.0, it's the malignant tumor on the GPL's kidney, still wrongly diagnosed as a urinary tract infection.

Back in the Slashdot days, we all thought that the fate of free software would be decided by a landmark court decision, that if the ideals of the GPL were to die, they would wind up meeting a ceremonious end like the cabinet members of a government overthrown in a military coup. But no - the free software ideal will die by the hands of a thousand poseurs, all who want the notoriety of contributing to open source, but none who are convicted enough to release any of their business's core code under a free license.

The copyleft will share the same fate as the hippie movement, now only a shell of its former self supported by college age kids who hang out in the Haight-Ashbury and smoke pot all day, and at night, drive their Lexuses over the Golden Gate, back to Marin County. But you will take off that damn Che Guevara shirt before you come back into my house, young man.

Look at all of the open source software in modern use. The vast majority of it is licensed under terms without a copyleft clause. The BSD license, Apache license, MIT license, and a handful of others are the most prevalent. In some places, the GPL still kicks around, but since we are application service providers, we are all free to ignore it.

The Affero General Public License, a version of the GPL that closes the service-provider loophole, is almost nowhere to be found. The only new-hotness software I know of that is licensed under Affero is MongoDB, and even they have a chickenshit implementation - they have structured the code such that the 99% case of a web application using Mongo is effectively bound by the Apache license.

Affero-licensing your project is a fatal defect if you want it to be used. Since the current flow of the software industry has effectively neutered the GPL, the only serious chance the copyleft has is the Affero license, and that sure-as-shit ain't gonna happen.

The toll on the Golden Gate Bridge is now six dollars.

How I Spot Valuable Engineers

2009-12-14T00:00:00-08:00

Goofy stuff happens when your company announces a funding round. We've gotten walk-in solicitors who try to sell us networking equipment, pitches for I-can't-quite-see-the-scam-but-I'm-sure-it's-there-somewhere stock exchange programs, and phone calls from slick Oracle salesmen who have their get-past-the-secretary sneak perfected so well that they could probably make better livings as industrial spies.

But most frequently, there are resumès that land in my inbox. Yes Milo is hiring, and a lot of people contact me directly instead of the "jobs" address, which I can sympathize with because I've always had this feeling that "jobs@" e-mail addresses are black holes where career dreams get sent to die.

Our general workflow for hiring engineers is to send the person our "engineering challenge" programming question and see how they do on it. If that looks good, they come in for interviews. I don't like doing interviews because I've always got enough stuff to do, but sometimes it's a good break. Necessary evil, I guess. Like Katy Perry. Have you heard a live performance? Ph33r.

Anyhow, when I interview a candidate, I'm trying to determine how valuable the candidate is, not just how smart he or she is. Because I love English semantics:

A smart candidate will do well on the engineering challenge problem.
A productive candidate will be able to explain past projects in detail.
A valuable candidate is smart and productive, but also has useful knowledge gained from experience.

To tell if a candidate is valuable, you need to piss them off. (By the way, does it make you feel icky that they can be used with a singular antecedent? This derelict language is put together with duct tape and baling wire, I swear.) A valuable candidate will likely have been personally offended by some sequence of bullshit thrown from a programming language, tool, library, or problem in past work. This is the kind of bullshit-train I'm talking about.

Need to parse XML with Python → SGMLlib feels like a kids toy → Implement it with BeautifulSoup → Fuck me, Soup is too slow → Re-implement with LXML → LXML works great for months → LXML segfaults the Python interpreter when used in a threaded environment under heavy load →

Any developer who has been around enough to accumulate valuable experience will have his personal collection of stories that have mad him rage. I have been burned by bugs in programming language implementations, bugs I call "coding slurs". I have gotten the shaft more times than I can count from pathological character set issues that make me want to run for Congress on the platform of requiring licenses before people are allowed to use computers. If you really want to find the value in a job candidate - find out what pisses him off.

The easiest way I have found of doing this is to ask a candidate "what don't you like about your favorite programming language?" You can grade their experience with the response. For example:

What don't you like about Java?
Out-of-college answer: "Java is too verbose"

Battle-hardened developer answer: "Object storage is aligned on a 64-bit boundary, at least in Sun's JVM, so if you need to allocate a lot of small storage, you really need to know JVM internals so you don't run out of memory."

What don't you like about Python?
Answer from a candidate who will write frameworks for solving problems instead of getting shit done: "Dynamic typing means you need to rely more on your tests and less on the interpreter to make sure your code is correct"

Answer from Gunnery Sergeant Hartman, your senior drill instructor: "Independent object cycles where one of the objects has a __del__ method don't get garbage collected."

People think I hate programming. Nope. What I hate is fording endless rivers of horseshit that are in the way of seemingly simple tasks. And I hate it even more when I have to explain to a non-programmer what I am doing, "building LXML against a different version of libiconv because I think it might be the source of a crash".

"But all I asked you to do was parse some documents."

Good times.

Introducing Milo

2009-11-24T00:00:00-08:00

I had mentioned a couple of months ago that I had my head down into a new project. It's been an open secret that the project is Milo.com, an online local comparison shopping engine. We index the inventory of stores nationwide and show you real-time, what is available around you. From an engineering perspective it's a cool problem because there is a lot of data to store and manage, as well as a lot of integration work to deal with the particular temperament of various retailers' inventory systems. Of course if it were easy, someone would have done it already.

From a business perspective, I like it a lot. The online comparison shopping world is very crowded, and we didn't want to be just another me-too AdWords arbitrage/affiliate marketing site. When I got into this business, I thought that online shopping was like the Stairway to Heaven of Internet business, but with the local inventory lookup, I think we really have distinguished ourselves from the others out there.

Today I'm happy to announce that we've closed a $4 million Series A investment round, led by True Ventures, with other investors such as Ron Conway, Aaron Patzer, and Jeff Clavier also participating. As a side note, I was really impressed by the True team, and am happy to be working with them. There were ups and downs to the Series A process, and I have to say that pitching the True partners was a definite up.

Oh, right. We also have a mascot. His name is Milo, of course. Here he is attacking me at my desk:

Hey Lets Bitch About SEO Again

2009-10-13T00:00:00-07:00

Hey I have an awesome idea. Let's take a field of business that many people work in to make a legitimate living, and tear it down for being immoral and accuse it of fraud. And when it comes to solving the actual problem that this business works on, apply a nice helping of sunshine-up-your-ass, and everything's just fine.

Better yet, let's do this every six to eight months, because collectively we have the attention span of a fruit fly.

Well, it's been a solid eight months, and somebody kicked the hornet's nest. Is SEO good or evil? It's good. It's great. I <3 SEO.

When you hire a legitimate white hat SEO, you are paying for domain knowledge. Is it better to use dashes or underscores to separate keywords in a URL? I know the answer, but I've spent some time researching SEO. If I were, say, an online publisher, it would be worth money to hire somebody who knows the answer to this question and a pop-quiz full of other questions that isn't in your average web developer's job description.

Every hit on SEO eventually ends with the same solution. "Just write good content or make a good web app, and the traffic will come." Oh really, it's just that simple, eh? How many unpublished novelists are there out there? How many film students whose reels go unwatched? Google is the greatest media distribution channel that there has ever been, and you expect people not to look for every advantage they can get?

Here's the failure with the "make a good app, people will come" argument. Let's say you are making an application whose target market is one person in ten. That's a respectably sized market. You tell your friends, your family, people you know through the internet. You write on your personal blog about it. Let's say you reach 1,000 people, generously. If your hit rate within that market is 50%, that's 50 people you've got who haven't immediately dumped your app. Do they care enough about it to do your marketing for you? With that small of a user base, you don't have statistically significant feedback to improve the site, you've got to gun it on intuition, which is frequently wrong.

So there you are, with your 50 users, and since you don't have to spend any time or money on distributing your app (remember these 50 people will do it for you), then you can continue to develop the app, making it "better", as you see it, in a vacuum.

And let's just count on those 50 people bringing in 10 million of their closest friends in the next month or so.

Hell no. This is the internet, son. Kill or be killed. If you can spend some money on a good SEO who will bring a steady flow of traffic to your site, then you have a way better chance than with that initial set of 50. With search engine traffic, even if you're only getting a handful of traffic every day, it's a different handful. If you have built something of value, some percentage of users will recognize this, and maybe tell a friend, maybe they'll come back to your site, and maybe they'll link to you, but you have a continuous stream of people to try it out on.

Obviously there are shysters in SEO. Going to an SEO who guarantees that you'll rank in the top 10 for mesothelioma is like taking your car to the dealership to get fixed. Of course you're going to get scammed. Buyer beware, and all that.

Anyway, keep debating on whether or not SEO is evil. The rest of us have to find ways to handle our traffic growth.

I Don't Code in my Free Time

2009-10-10T00:00:00-07:00

Why would you ever hire a programmer who doesn't program in his free time? I mean, a person who doesn't compile recreationally is probably useless on the job. You might as well hire somebody ... old. And who wants a bunch of people around the office who whine about things like healthcare benefits? Just don't get sick, duh.

I love it when twenty-something engineers take such a hard-line position on something they have so little experience with, like hiring. Saying that you wouldn't hire somebody for a programming job because they don't program in their spare time is blissfully naive. Yeah, I remember the days when my greatest responsibility to another human being was making rent on the first of the month.

If I am going to hire somebody for a programming job, I don't really give a shit what they do in their spare time, so long as that person is very good at the task at hand. I don't ask questions about what a person does in their free time in job interviews because I don't care, and because that can sometimes open the door to an illegal conversation. (What's that? There are laws about what you can ask somebody in a job interview? Who thought that up, Republicans?)

Me, I can count on one hand the number of times I've programmed outside of work or a class. There was only once when I actually enjoyed it, though. I was in college, and shared a common wall with a girl from Spain who was painfully unaware that her computer had a volume control knob. She would stay up late on AOL instant messenger, and I couldn't sleep. So, I rigged up a Python script to play AOL instant messenger sounds randomly every 5 to 10 seconds, turned up my speakers, pointed them at the wall, and went on vacation for a week. And thus, the asshole you all know and love is born.

I don't enjoy programming so much as I enjoy the satisfaction I get from cracking hard problems. In that case, computer code is a means to an end, but so is my Craftsman socket set. I like to spend free time wrenching on a car or a bike, but I don't set out on Saturday morning and say "I'm going to learn how to use a torque wrench today, because those things are the future of tools".

I would not want to work for a company that wouldn't hire me because I don't code in my spare time. Professional development? Working at a startup, I get a heaping helping of that on the job. Keeping up with new technology? Yeah, I read reddit, and again, startup. You know what's more awesome than spending my Saturday afternoon learning Haskell by hacking away at a few Project Euler problems? Fuck, ANYTHING.

Really, why should I bother spending time with my family and taking an active role in my kids' development when there's a dead-beaten math puzzle that doesn't have a good answer in Clojure? "I won't hire someone who doesn't code in their free time" is Siliconvallese for "I don't want to hire any grownups because they remind me of my parents".

Twisted vs. Tornado: You're Both Idiots

2009-09-18T00:00:00-07:00

First, a message to bloggers. If you've got the bright idea to try some new kind of benchmark that pits Twisted against Tornado, take pause. Turn off your computer, step into a public area, and reconsider your life's goals. The internet does not need another pointless network performance graph.

With that out of the way, it's become clear that the Pissing Contest of the Day, Twisted.web vs. Friendfeed's Tornado web framework, reveals that neither side of the argument is particularly right, but both sides are particularly stupid.

First, Twisted. Now, my company uses Twisted for a small piece of functionality because it was the easiest way that we found to send traffic over different network interfaces on a Linux machine. We never have any problems with it. The only reason I ever need to touch it is to see how something works.

However, Twisted is probably the douchiest programming library out there. Every time I open up that code, I feel like I've wandered into a late-night bar on the Jersey Shore where everybody's drinking Jager-bombs, and nobody is wearing a shirt. Twisted is a cool network library, but not cool enough to be named "Twisted". It's the Python programmer's version of Ed Hardy clothing and a baseball cap with the tag still hanging off the side. When I'm digging around in this code and my co-workers ask me what's up, the only appropriate response is "NOT NOW CHIEF. I'M STARTIN' THE FUCKIN' REACTOR."

Now you can see why there's so buttsore over Tornado.

Even though I advised against things like Tornado, Friendfeed still built it. From the graphs I've seen, Tornado is just marginally faster than Twisted at serving concurrent requests. Marginally. Evidently Friendfeed figured that tiny margin was enough justification to waste their time writing something that's been re-written by every developer that gets bored on the job. A Python web framework? My mercy how original. I think that's one of the ending exercises of "Learn Python in 24 Hours".

Friendfeed spent a lot of time trying to optimize the queries per second graph, but maybe they should have spent more time optimizing this graph instead:

Anyway, when it comes to Twisted vs. Tornado for a Python web framework, I use Django. Why? Because it works, and my time is valuable.

30 Helens Agree: You Can't Win Without Failing

2009-09-09T00:00:00-07:00

I read Fred Wilson's blog post on failure today, and after I was finished being impressed by his three letter domain name, it really made me think about what I learned from my last failed startup.

There's the usual Reddit material: don't write your own database, concentrate on the UI, put your users first, other such horse-beaten realities that green engineers understand after being in the field for a few years. A true failure is one that changes your life's philosophy, not one that changes your unit testing strategy.

What I really learned from the fall of Pressflip is that arrogance is more dangerous than incompetence. I believed that raw engineering prowess could make up for the complete lack of business experience, a product that really only appealed to the people who build the technology behind it, and an addressable market that could easily be mistaken for roundoff error. Couple that with the youthfully cute thought that Silicon Valley is a meritocracy, and it was only a matter of time. We had build some neat technology behind the scenes, and I was very proud of a few key parts of the system, but in the end, the users just did not come.

The trouble with this lesson is that it can only be learned the hard way. Arrogant people don't listen to criticism, they just run themselves into the wall. Incompetent people can usually be led in the right direction, even though they may execute their way into the dirt. Arrogance doesn't listen to reason, it only listens to itself.

For example, an arrogant motorcyclist will ride on the highway at twice the speed of traffic, and no matter how many times he gets pulled over, and he'll keep doing it until he crashes. An incompetent motorcyclist will drop his bike in a U-turn in front of his house, cracking a mirror.

This failure made me saltier. I now understand why old men have no patience for the modern world. However, it did not let me keep thinking that superior code is the solution to any conceivable problem. I've hunkered down a bit, concentrating on a new project that I really believe will be a winner, and started learning the business realities of a cruel Valley.

So now, if an investor asks me what I learned from past failures, I won't put him to sleep talking about schema-less versus SQL databases. Instead, I've got a good answer.

A Happy Life Without the Whining

2009-08-28T00:00:00-07:00

There's no better way to waste your time than to talk about politics. For as good as educated people are at acting intellectual, we love to bitch and moan about one side versus the other.

Politics are potato chips for the enlightened mind.

I grew up in Connecticut. New Englanders are generally pretty educated people. We keep to ourselves. We vote. We donate money to causes, and for the most part, we shut the fuck up. Personally, I'm registered to one of the two major parties in the US. (If you tried to guess, you'd probably guess wrong.) I don't get into political arguments because I've got better shit to do. I don't blog about politics because I know that nobody cares what I think. You know what? It's a good life.

Lately, I've been hearing a lot about this Glenn Beck fellow. I don't know who he is or what he said to get everyone so sore-assed, but I sure as shit don't care. I don't watch CNN or Fox News. I don't have cable TV. I get all my news from my local news channel over the air. No talking heads, no shouting matches, no six-second-attention-span scrolling tickers on the bottom of the screen. In 30 morning minutes, I get a brief summary of what the president said at such and such a meeting the other day, a look at the traffic and weather for the day, and some feel-good community segment.

The last thing I need on a 40 mile motorcycle ride to work is a head full of piss, thanks to Bill O'Reilly or Keith Olbermann.

But I can tell you that from the inside, generating butthurt is big business. Every time I've knocked an article out of the park for The Register, there's been a decent troll element to it. Not all trolls succeed, but the ones that hit a nerve really bring in the page views and comments. That's just the IT world. If I could get a job trolling politics, I'd be damn sure to demand a page view bonus. I can't knock the hustle.

The news networks aren't stupid. They know that viewership increases when people are pissed off. Walter Cronkite delivered facts, but was a crusty old book report of a man for it. I'm sure that all else equal, if national media never figured out how much fucking money there is to be made in keeping people salty, the news would still be a puff of dry air.

So I don't watch network TV. I don't blog about politics. It's a calm life. I have informed opinions on most issues, but I know that nobody cares what I think, so I keep to myself. Maybe that's why I still have trouble "getting" Twitter.

Stop Using the Word 'We'

2009-08-20T00:00:00-07:00

Yesterday, I spearheaded a new movement at the office. I stopped using the word "we", and started to say what I really meant to say. For example, instead of "We should fix that bug", I say, "You should fix that bug", and good God is it satisfying.

There are a couple of motivations for this. Firstly, one of the key things I've learned being a for-pay writer is to show some conviction. Secondly, the passive discussions about defects and delegation and responsibility really started to irritate me. Why not just tell it like it is?

When I worked at Google, I picked up on a really annoying trend in the software industry (or maybe just in Silicon Valley) that I call "fuck-you with a smile". You never want to outright blame somebody or something, rather, it's best to state the existence of an issue and then ask "the team" to fix it. We should really move that icon ten pixels to the left. We definitely need to fix that concurrency bug. We should probably have that all done before lunch.

Well then, Mr. Manager, you had better get cracking, because I've got some YouTube videos to watch.

I learned that the goal of institutional business is to keep from angrying up the blood at all costs. A productive employee is one whose personality has been bleached out to a yellow tinge. Always non-confrontational, never suggesting that any one person fucked up.

The best part about working at a startup is that I'm free to suggest that yes, you fucked this up. Yes, it's your fault, and yes, you need to fix it. Delegate! Don't waste time listing out action items, spend time telling people what to do. Everyone you work with should be a grown up, and can handle it. The other side of that is owning up to your mistakes. Instead of "There is memory leak in the code, we should prioritize it over other defects", say, "I introduced a memory leak in the code. I am going to fix it as soon as possible."

Anyway, I'm going to keep this up until somebody openly calls me an asshole. You should try it too. You don't have to be a prick about it, just be assertive. Your co-workers will be impressed at your new found confidence. It might even get you laid.

Well, probably not, but you won't be wondering when a meeting is going to end if you grab it by the balls.

Context Switches are Bad, but Stack Traces are Worse

2009-08-17T00:00:00-07:00

Every programmer works in silent fear of a manager sneaking up on him and asking him to drop everything he is doing and work on an unrelated task. Context switches like this cost us time and energy, but managers are beginning to figure out that a programmer isn't a machine that can be switched on and off, no, they're understanding that a programmer is a machine that needs a warm up phase.

I guess you could call that progress.

I'm fortunate enough to work with a company that understands engineering, but I have worked with my fair share of nontechnical managers, and I have to say that categorically that the most expensive question a manager can ask is "What are you working on?"

The danger here is when you're six or seven levels deep into yak-shaving, and your manager wants to know what you're doing and why. You need to give the manager a complete stack trace from your current frame all the way up to the original task. Each jump up the call stack is a context switch of its own, where you need to remember exactly why you made the decision that you did, and justify it as the best course of action.

"I'm compiling a new version of libxml, so I can get the Python parser working. I need to do that because LXML, the Python binding, would crash under heavy load when I used the system default version of libxml. I am using LXML because BeautifulSoup doesn't have support for XPath. I need to do XPath transforms against the input because the legacy system we interact with doesn't send well-formed XML. We tried to get the vendor to fix it but they said 6 to 8 weeks for a patch, and our project deadline is sooner than that. I need to interface with the legacy system because even though the DBAs have ported things over to Oracle, they're still sorting things out and it's not reliable enough for me to make any meaningful progress. Fortunately, I've thought this through and abstracted the data subsystem well enough that I can drop-in replace Oracle when it's ready, so long as the database sends some decent form of XML. Once I get this data subsystem done, I can finish the business logic, which attaches to the nonfunctional demo you love so much.

So yeah, I'm working on the new asset tracking system."

Stack trace, all the way up to the main method. It's not always that simple. I know I have trouble keeping that many stack frames in my head. I can't remember why I chose to go down the path that I did, other than "there was a good reason for it". After all, I'm not slogging my way though compilation bullshit for my health. When a manager demands a full stack trace like this, it sets your progress back, because you need to go over decisions that you already made, examine the circumstances, and make the same decisions again. You lose your original frame of reference, and your manager thinks you're just fiddledicking around instead of doing work.

So what's there to do? If you're awesome like me and work for a manager who understands why programming is hard, chances are you can just leave the answer to "what are you doing?" at the innermost stack frame. Everybody wins. However, if your manager is nontechnical, your goal is to get him off your ass as soon as possible, because you want to minimize the damage he does to your productivity. My recommended course of action, when asked "What are you working on?" is to slap the manager in the face and yell "YOU DON'T END A SENTENCE WITH A PREPOSITION UP IN THIS BITCH. THIS IS MY HOUSE." Thump your chest with a clenched fist and say "yeah what's now, fool" under your breath.

Failing the battery charge, the key phrase is "I explored every option". Beyond this, there's really no way out, because your manager doesn't trust you. You're basically fucked. Quit your job and come work with me.

This Is America, Take Your Unicode Somewhere Else

2009-07-04T00:00:00-07:00

There's a question that comes up on Stack Overflow every couple of months: "How do I strip diacritic marks from Unicode characters?". Popular variants include "How do I remove special characters" and "How do I convert Unicode to ASCII", but the underlying motivation is the same: characters that don't have their own key on an American keyboard have no place in modern web software.

Before you go all apeshit on me and call me a bigot and whatnot, read my story. When I was in college, Google hired me for a summer internship. One of my projects that summer was to write Google's employee directory search. Google, as I'm sure you could imagine, is a very multicultural employer. Googlers in general are very accepting of different cultures, customs, and languages. (Well, sort of. Googlers are accepting of multicultural differences like sushi, Diwali parties, and the word namaste. They're not accepting of cultural differences like Old English 800, 22 inch rims, and the word juicy. The general rule I figured out as a Googler is that you should welcome diversity so long as it doesn't make you feel guilty for making ten times as much money.)

Anyhow, as a result of pulling in a lot of foreign talent, my employee directory search had to handle UTF-8 properly. A lot of peoples' names had umlauts, tildes, and other such little nuggets that love to appear as diamonds with question marks in them. I figured, just make the database UTF-8, page encoding UTF-8, and everything should work fine, right? Well it did, in theory. But when the first super-tolerant Googler typed his colleague's name into my search engine, it didn't come up. There was an o with an umlaut in the name, but our hero of race relations simply typed "o".

And that came through to me as a bug report. "Strip funny characters." So I did, and how the searches flowed. See if you can guess how many people would input diacritic marks into the search box.

Googlers are some of the most understanding people out there, and if they can't be bothered to type Alt-148 for an o with an umlaut, then what hope does the rest of the software industry have? None. That's why I want to systematically dismantle Unicode, and have a good answer to the question "How do I strip diacritic marks?". Not because handling multibyte character sets is too hard (although that asspain is what prompted me to think about this in the first place), but rather because only a small minority of people actually care about it, and an even smaller minority will whine when their umlauts disappear.

(To satisfy the pedants, clearly if you're writing software whose job it is to handle and store UTF-8, this advice isn't for you. I'm talking about web services with user input here.)

Now, you can feel free to take an idealist approach to this problem. Yes, Americans should be more accepting of other cultures and not passively destroy intricate details of pronunciation. Well, feel free to enjoy your floating-point market share. Nobody cares but you.

End Note
I found two decent implementations of Unicode transliteration, one in Python and one in Perl. If you know of good implementations in other languages, e-mail me and I'll add them to this list, with SEO-friendly anchor text goodness.

Strip diacritic marks in Python
Strip diacritic marks in Perl
Strip diacritic marks in Java (thanks to Simon Lieschke)
Strip diacritic marks in Lua (for all 8 of you who use it. Thanks to Petite Abeille)
Strip diacritic marks in PHP (thanks to Tommy Montgomery)

Print Isn't Dying, Serious Journalism Is

2009-06-22T00:00:00-07:00

It's a tired Silicon Valley drum beat: print is dying, blogs and Twitter are the future of news. Many in the business of blogging like to think that print ad revenues are declining and subscriber bases are shrinking because online media is vastly superior to those dinosaurs. This is one area where the evidence actually seems to suggest that the bloggers are justified.

However, if you're not so full of yourself that "citizen journalism" seems like a revolution, you can understand the real reason that print is dying: newspapers' shit is all retarded.

Too many big words, articles that are way too long, and boring stuff like researched facts. Fuck all that shit, I want my news as it happens, and I don't care how true it is. Bloggers call this process journalism...whatever. That's just writers trying to convince themselves that they're serious when they know deep down that their readership is only interested in sensational titles and text no longer than 300 words. Any more than that, well, shit's all retarded.

The only satisfying part of journalism turning into shinythings.com is watching intellectuals whine about it. See, I probably should be an intellectual. I've got a degree in mathematics, I'm a computer programmer by trade, but every time I've knocked an article out of the park for The Register, it's been a great troll. That's the only way to get by in online media, and even the New York Times knows this.

Take for example, NYT columnist Paul Krugman. He won a Nobel Prize in economics, and has been writing the same op-ed column for NYT for the past 8 years: "Republicans are the cause of all the world's ills." Someone who's shit is arguably all retarded has been reduced to trolling to get page views. And it really works.

If, as a blogger, you're above trolling, then the only other way to be popular is by printing blatant falsehoods. In 2008, people actually started to pay attention to CNN's iReport because somebody wrote that Steve Jobs had a heart attack. Apple lost 10% of its market capitalization in 10 minutes. Now that's fucking power. TechCrunch's Michael Arrington, showing an obvious tell of a manic depressive, keeps going off on Last.FM with lies about them giving data away to the recording industry. None of it is true, but it brings readers.

It certainly doesn't hurt that TechCrunch shies away from words longer than eight letters.

Print media isn't hurting because it's an outdated business model, print media is hurting because it's boring. Blogs and Twitter are succeeding because their shit is clearly not retarded. And you know what? I love it. Intellectualism is dying, and the news is now anything we want it to be.

I just can't wait until 4chan figures that out.

Startups: Keep It In Your Pants

2009-06-09T00:00:00-07:00

I've worked with a lot of engineers around the Valley, some who are genuinely competent and some who can fake it pretty well. One trend that I've noticed with alot of really good engineers is that they like to swing their dicks around when it comes to implementation.

You start a project with one of these guys, and the first thing to come up is how MySQL isn't going to scale, and how you're going to have to write your own data store. With that settled, you'll also need your own object-relational mapper, and you might as well make your own web templating language because well, it will fit in better with the architecture.

This, gentlemen, is dick-swinging, and it is the most colossal waste of time for a startup.

Now, it's a well known fact around Northern California that I'm the greatest programmer who ever lived, and I even fell victim to this. At my last startup, we were absolutely convinced that we were building ourselves into a corner by using MySQL, so we wrote our own data store. It started off as an RPC wrapper around some magical key/value store in Erlang (parallelism, fuck yeah), and ended up as a different RPC wrapper around BerkeleyDB. All in all, it went through three major rewrites, and the end product was something that took months to develop and would crash under moderate load.

But hey, it was a cool architecture.

As another small example, again at the last startup I spent a few hours one day writing a feedforward neural network implementation in Java, just to try my hand at implementing an algorithm. Again, a small waste of time, but it was my attitude toward it that signaled a larger problem: I wanted to see how awesome I really was (answer: pretty fuckin' awesome).

It's not just apartment-bound startups that fall victim to this, either. Kosmix, which is a well funded science project that's fooled itself into thinking it can be a major player in search, wrote its own data store in C++. It's basically a clone of Google's GFS because hey, if Google's doing it, then we should too, right? Who knows how much time, energy, and money was wasted on this thing, but that's all time, money, and energy that could go into making their final product not such a joke.

Kosmix falls to a different sword: they are well funded and assume they have all the time in the world. Maybe a serious venture round buys you time, but when you spend it all writing a file system that's not core to your product, you start talking Series C, Series D, and so on.

Fortunately, trench-level engineers aren't concerned with dilution. Oops.

At my current startup, we've got business-focused leadership. We have a good engineering team, and we don't let our hubris get the best of us. There are so few instances where a startup will need to write something like a file system, and we're not one of them.

As an entrepreneur, you should be prideful of your idea, now how big you think your compiler-cock is.

My Twitters: Let Me Show You Them

2009-06-02T00:00:00-07:00

I signed up for Twitter. Do you people have any idea how fucking important I am? It's a good thing I'm benevolent enough to clue you people into the glory of my day to day operations.

You should consider it a fucking honor to read my Twitters.

http://twitter.com/dozba

Hacking Domains by Proxy

2009-06-02T00:00:00-07:00

Remember how Uncov.com lapsed registration, and somebody bought it with Domains by Proxy? I'm sure other people have faced this problem: how do you find out who owns a proxy domain? Well, I successfully hacked the system.

Here's how it works. When someone registers a domain with Domains by Proxy, the e-mail provided to the DNS system for administrative and technical contacts proxy through to the person who actually registered it. If that person directly replies to an e-mail, you can see who actually owns the domain.

As usual with anything technical, the weakest link is the human. The KGB used to say "it's easier to break fingers than it is to break codes". And it's easier to exploit greed than it is to subpoena Domains by Proxy or hack their computers.

Check this shit out:

Names hidden to protect the douchey, but if you've got ten thousand extra dollars hanging around, you can have uncov.com all for yourself.

Don't be a Menace to South Central

2009-05-25T00:00:00-07:00

The internet has this nasty habit of violently breaking down market inefficiencies. For example, the music market was inefficient because driving to a store and exchanging money for a record was too much effort. Downloading an album for free is a virtually frictionless process, so Napster, KaZaA and others thrived.

Now, I think the word "disruption" is mostly used by half-wits in the media to pimp something. When you read that something is disruptive, that's an obvious tell that the journalist doesn't have a fucking clue about the technology, but is pretending that he does. Journalists love that shit, and so do editors. However, as an entrepreneur, you haven't created a disruption until a group of powerful old men convene in a board room to figure out how to shut you down. You haven't created a disruption until the government is trying to regulate you. You haven't created a disruption until there's a media campaign against you.

That being said, I love the idea of paid posting and sponsored conversations: companies paying bloggers to talk about their shit. Why? Because it's really pissing off people who make a living out of public relations.

When I was writing Uncov, I would get several e-mails daily from PR agencies, pitching me a story on such and such a shitty startup. This is how it works: your company pays a PR agency for the size of their Rolodex. The PR agency spams the publications with your press release in hopes that the story gets picked up. In the tech media, the hit rate for PR isn't terribly high, so you end up spending upwards of $10,000 per month on PR that only gets your company a few writeups. It's a scam.

A company like PayPerPost, now Izea, has removed that market inefficiency. You can simply pay bloggers directly to write about you. Whether or not they disclose that they're being paid, well, who gives a shit? You get the Google link juice, you get the attention, and if it comes out that you paid for it, the internet has the attention span of a fruit fly, so everyone will forget about in 24 hours.

Any blogger that takes a stand against paid posting is delusionally self-important. There is no morality to "citizen journalism" by definition. The idea is that the traditional media will die in favor of hundreds of thousands of individual reporters, all working for themselves. The good ones will rise to the top, but everybody will keep talking. In this type of scheme, there is no force whatsoever that can stop paid placement. With a few large media outlets, like The New York Times, The Washington Post, and other newspapers, paid placement isn't an issue because so much credibility is at stake.

Not so on the internet. Complain about it all you want, but paid placement is a necessary side effect of user-generated media. Regulations against it are like speed limits: there's bound to be some marginal enforcement, but by and large, it does nothing to prevent it.

It's what we wanted, now it's what we've got.

Startup Dad

2009-05-21T00:00:00-07:00

The first time you hold somebody else's screaming baby, you understand immediately why prostitution is the world's oldest profession. The first time you hold your own screaming baby, you understand immediately why a federal prisoner who gunned down three police officers needs no moral justification for sticking the sharpened end of a toothbrush into a freshly jailed child molester's kidney.

It's that first revelation that scares many first time fathers into escaping responsibility like a jackrabbit from a coyote. The difference between the ones who run and the ones who stay, really, is how the news was broken to them: a great comedian will tell you that they key to any good joke is the delivery.

A runner was out on a Tuesday night with some of his friends at a bar, somewhere between his third drink and fourth cigarette, when the girl he'd been fucking calls him up and says that she's pregnant. A dad who sticks around is concentrating on some manly order of housework, like changing the oil in a car, when his wife or girlfriend calls him in to make sure that the little blue plus sign actually does mean “pregnant”, and that she's not just misreading it.

With a baby in the house, the bedroom is going to lose its understated but victorious smell of Astroglide and unwashed sheets in favor of a strong presentation of rancid breast milk. When there's a child to take care of, getting falling down drunk to the point where you're willing to argue with a street vendor over the price of a 2AM hot dog isn't really an option in the list of things to do this weekend. With a baby, all of the money you used to spend on video games and car accessories is going to be repurposed for child care.

When a man runs from fatherhood, he's not really running from responsibility, he's running from the guilt of a mediocre life.

Without the responsibility of a baby, there's still time to salvage it. A month after disappearing, though, a runner realizes the vicious truth: that no amount of time or things-not-to-be-responsible-for will turn an unaccomplished life into one your eventual children will look up to. Fleeing your responsibility and making that new year's resolution to get your life on track is as effective as telling yourself that you have the courage to ask out a girl as you masturbate. No number of promises will ever amount to motivation.

For those of us that stay in the picture, ambition has a new meaning. I'm twenty five years old, a software engineer on the startup circuit in Silicon Valley. I'm not in the business so that I get invited to speak at conferences. I'm not an entrepreneur because I want to feel important. I'm in this game now to provide for my family. At first, I thought that a startup was the only part of my youth left breathing, but now, I know that having a picture of my daughter stuck to my monitor is the best motivation there is. If you're the type to man up to what's demanded of you, a baby won't throw your entrepreneurship game off.

Just make sure you're funded.

Disable The Annoying Thing In Ubuntu Jaunty

2009-04-25T00:00:00-07:00

I upgraded to Ubuntu Jaunty Jackalope today. It was a positive experience until I opened Pidgin and Evolution and started using my computer. Jaunty has this new feature called NotifyOSD that application makers can use to bug the shit out of you at every possible moment. Someone signed online? Bug the shit out of the user. Received an e-mail, bug the shit out of the user. Joined a wireless network? You guessed it. Let's bug the shit out of the user.

The old notifier used to stay out of your way. Get a little message or whatnot when you got a new e-mail. It was unobtrusive and didn't distract you while you're trying to figure out why some little bit of SQLAlchemy code is making too many calls to a database. But now, Canonical has found it necessary to make sure you're abundantly aware of every excruciating detail of your computer's operation. Productivity be damned.

I don't know whose bright idea this feature was, but whoever it is is trying to spread their terminal case of attention deficit disorder to the rest of the world. Fuck you. Grind up your Adderall pills and snort them until your heart shits out like a Chevy.

Anyway, if you don't know what I'm talking about, this is the offending window:

How To Turn This Thing Off

Open up a command line. Type this:

sudo mv /usr/share/dbus-1/services/org.freedesktop.Notifications.service /usr/share/dbus-1/services/org.freedesktop.Notifications.service.disabled

And restart your computer. You could probably restart the dbus daemon, but that makes a lot of things go ill on your machine.

This disables the notifier for good. Now you can get back to work.

DiggBar is a Howl of Desperation

2009-04-10T00:00:00-07:00

Since the most recent death of Uncov, I've tried to lay off the Web 2.0 shit. However, as a conoisseur of fail, I thought the DiggBar is worth an examination.

DiggBar is a URL shortening service from Digg, the internet's largest community of whiners, armchair political activists, inconsolable Book-of-Steve-Jobs bible beaters, and automatic voting bots. The long and short of it is this: you can put any address into it, and it will give you a way to view that URL through Digg.com. For example, http://digg.com/u1hrO brings you back here, except with a Digg toolbar at the top.

There's been a small wave of butthurt over this little scheme, because every link on the front page of Digg.com now leads you to one of these toolbars instead of to the actual content. So, when a Digg user clicks through, he never actually leaves Digg.com. They've done some of the stuff necessary so that publishers don't get shafted on the traffic or the PageRank (and still managed to fuck that up), but still, that little bar adds little to no value to the user.

Then why did they do it?

When an entrepreneur raises any useful amount of money from an investor, he needs to answer to that investor. The CEO of a company reports to the board of directors, which usually includes the investors. Every quarter, the CEO must lay out goals and objectives, and for internet companies, these goals and objectives always, always include traffic growth.

Digg has raised 40 million dollars to date. With that kind of money, investors demand explosive growth. Since the economy has gone to shit, there's a very slim chance that Digg will see a sale before it needs to raise more money. As a small company, Digg could have been a very profitable business, but instead they took too much money and made too many expectations for themselves. I can guarantee you that Jay Adelson (CEO) and Kevin Rose have some demanding goals to meet, and lately, they haven't been meeting them.

Hence the introduction of this DiggBar business. When a link makes its way to the top of Digg, it gets republished quite a bit. Now that all these links will land a user at Digg.com, Digg that collects the unique users from this collateral linkage. And it's working, too. In a recent interview, VP John Quinn of Digg said that the DiggBar has given them a 20% boost in unique visitors.

This move shows that not only is Digg willing to pull some sleazy shit to increase their unique visitors, but that they also need to pull this sleazy shit, because they need more unique visitors.

Damn. And I would have gotten away with it too, if it weren't for you meddling kids.

Footnotes.

Uncov.com died again. I never owned the domain name, one of my business partners did. The registration on it recently lapsed, and somebody picked it up. It now redirects to a Twitter search for "kevinrose". If you're the one who bought it, good show. Thanks for not being a spammer. I'm not willing to buy it back from you, but if you want to give it to me, I will take you out for a beer to congratulate your achievement.
No, I did not get fired from The Register. My wife and I had a daughter last week, and I am taking some time off. Not sure when I'll return yet, but I will. I just need to get my sleeping back on schedule.

Be a Better Blogger. Stop Reading Blogs.

2009-03-08T00:00:00-08:00

The greatest hope of internet generation is that you can share your thoughts with everybody in the world. The greatest letdown of the same generation is that nobody cares. Still, that doesn't keep us from trying.

Bloggers are good people, generally. We're self confident in a passive aggressive sort of way, we're opinionated, and best of all, we can type quickly. But what separates bloggers from each other? Those who can break exclusive news usually have a good following, but what about the rest of us? How do you actually get better at blogging?

Not that you asked for it, but this is my advice: read more, write less. By "read more", I mean books. Newspapers are useless, because the job of every newspaper editor is to remove any semblance of personality from all of the text. It's just facts, and facts are fuckin' boring. If you want to cruise programming.reddit a few times a day and write reactionary articles, fine, live with that crowd. It's not an interesting place to be, though. Telling the world why you think DHH is wrong about some programming methodology isn't going to get you a column at Rolling Stone.

Getting on the front page of Digg is not an accomplishment.

Blogarrhea begets blogarrhea. There's a continuous global discussion on the internet, and if you're not the one who started it, you're just background noise. People like Paul Graham and Dave Winer never really say anything original, they just enjoy the act of typing. Graham has been re-writing the same three essays for almost a decade, and Winer, well, Winer doesn't have much to do during the day, and at least blogging keeps him away from drugs and rap music. I guess it's a positive influence.

If you want to keep that company, do so, but like programming, writing is so much better when you value elegance as well as functionality.

Which brings me to my second point. Write less.

For the last two months, I have been working my way through a pile of books: everything ever published by Chuck Palahniuk (tl;dr: the guy who wrote Fight Club). I'm almost done, a book and a half to go. Chuck likes to do these writers' workshops, and somebody once asked him what he does when he's stuck. He knows where the story needs to go, but just doesn't know how to get it there.

Chuck's response: "Did you ever go into the bathroom and try and take a shit when you didn't have to go?"

Whenever I sit down to write a post here, it's because I really have to take a dump. Incidentally, sometimes when I write for The Register, it feels like I'm really trying to squeeze one out. If I end up dead from an aneurysm, that's what happened. Setting a post-per-week quota for yourself is like setting a lines-of-code quota at work.

Don't write just because you want to spend some time on the pot. Do it because you really have to go.

Effective Vices for the IT Professional

2009-02-08T00:00:00-08:00

There's a blog post that snakes through the programming community every three months: the one about only hiring programmers who program in their spare time. It's always the same person who writes it, too.

He's a sequentially numbered employee at a company with a well-tracked ticker symbol, and his only outlet of authority is sweating down some poor sod in a windowless interview room, asking questions about sorting integers in linear time.

The truth of it is, after a day of writing JUnit tests to achieve the corporate-policy-mandated code coverage metric, you don't need to go home to a Haskell compiler. You need to go home to a tall drink and a depraved presentation of human sexuality. Corporate coding sucks, and if there's no vice to counteract it, you'll be dead of an aneurysm by age forty. They'll find you on the toilet, pants down, your copy of Design Patterns unceremoniously splayed open on the floor.

Programming isn't a glamorous job, and pretending that it is won't make you any better at it.

I've been studying some techniques for decompressing the tension built up by JBoss and WebSphere in my personal lab for quite some time now. I'm not a corporate coder anymore, but when I was, I studied ways to make it easier on the head. I'm now ready to share my results with the scientific community.

Drinking

Alcohol is the most obvious medication because it's cheap and readily available. Parked on the couch, racing your way to the bottom of a highball glass of Chivas Regal is a fantastic way to forget that the hour you spent in a meeting watching two type-A personalities fiercely debate Scrum versus XP is one hour less of the life you wanted. The downside is that one drink usually leads to three or four, and you waste the drunkenness on an early sleep because you need to get up early the next day and do it all over again.

Alcohol interrupts your sleep, and if you're going to stay sharp at work, you need a good rest. If you're one of the damned souls like me, you get vicious hangovers, to the point where swimming in drink for a night isn't even worth it, if you're going to spend the next day wishing that you'd died of alcohol poisoning.

That being said, at some time in your programming career you need to go to work with a severe hangover, out of sticking it to the man by way of martyrdom.

Drugs

Here's where the process gets dicey. When you wear a button-down shirt to work, you're not the usual type of person who ends up in county lock-up on a possession charge. Marijuana is certainly a better option than alcohol in every way, shape, and form, but like it or not, it is illegal. There are some exceptions here in California, but you're still rolling the dice Getting pinched could mean getting fired.

The scary shit comes as rocks or powders. Again, being the khakis-and-necktie crowd, nobody really expects you to be shooting black-horse heroin in the shower.

There is a convenient edge case when it comes to drugs, though. Prescription painkillers, when used appropriately, really take the edge off of reality. Again, you run the risk of upsetting John Q. Law, so make sure it's legit. While they make for good entertainment in the evening, there's a real possibility that you can get addicted, and once a vice starts interfering with your work, then you're fucked.

Strippers

If you're the type that easily takes to strippers, you had better come ready to peel of the cabbage: this vice doesn't come cheap. There's also a simple but unrelenting set of rules you need to learn to keep from getting your ass kicked by a bouncer. It's the kind of thing you'll pick up as you go.

For the programmer or IT professional, strippers are an excellent choice. You usually show up to the gentleman's establishment with a bit more money than any of the other clients, so you'll be Mr. Popular. Just be respectful of what's going on: it's not so much a smut show as it is a first hand demonstration in a loosely regulated free market. The dancers are there to make a buck, and don't you forget it.

Tobacco

Tobacco is a great vice for the programmer because it's a performance enhancing drug as well as an escape. I recommend either cigars or smokeless tobacco to avoid the growing anti-cigarette movement. A cigar is gangster, and chewing tobacco is stealth.

After putting down a nice stogie wrapped in Connecticut shade, you'll feel like it's time for action. Nicotine is a fantastic stimulant - better than caffeine. If you're the work-from-home type, smoking a cigar twenty minutes before you start will send you on your way in a hurry. Avoid the dregs, though: don't buy a cigar in any place that sells gasoline.

Chewing tobacco is often overlooked. Yeah, you say it's more of a staple with the Nascar crowd, but that's really just a stereotype invented by the Nascar crowd, designed to keep you damned hoity-toity folks from driving up the cost of a can of chaw.

The key part about dip is that you can do it at your desk. Spit into an empty Coke bottle. Nobody will come by to bother you. Plus, think of how authoritative you're going to be at a meeting when you start it off by lipping a fat digger out of a tin of Skoal.

Just Keep It Within Reason

You can judge any vice on two dimensions: how good is it, and how likely is it to interfere with your work. Once a vice becomes more than a vice, you're going to wish you were that guy who goes home to code Haskell.

However, there is a convenient side-effect to the addictiveness. If you are aware enough to see your vice getting out of hand, it's probably time to quit your job.

Just don't do anything illegal.

Corporate Blogs: It's The PageRank, Stupid

2009-01-19T00:00:00-08:00

If you're running an online business and have hired a consultant who tells you that you should have a corporate blog to "better connect with the community", fire that consultant.

If you have a corporate blog that is only marginally more interesting than a press release wire, you're wasting your time.

A corporate blog should serve only one primary purpose: distribution. And I'm not talking about building brand recognition by getting people to read your blog. Nine times out of ten, the text on your corporate blog is a chore to read. Even Google fails this - their pathological cuteness and lame humor comes off as contrived. It's not funny. It's irritating.

Anyway, how does a blog get you distribution if you're not concentrating on branding? PageRank. You can and should use your blog for link-building and search engine optimization.

A great example of this is Mint.com's blog. Mint is a personal finance web product that competes with desktop apps like Quicken. Mint publishes longer articles about personal finance to their blog, and have several thousand readers. That alone is interesting, but not mind-blowing. The trick is that their content is useful. It's basically a magazine about personal finance without the advertisements. Social media picks up on Mint's content, and it gets a lot of inbound links.

Mint takes gross advantage of those inbound links. That's the whole point. At the bottom of every blog post is this little nugget:

A-ha, I see what you're doing there. Mint is juicing their PageRank with the popularity of the blog. If you're a personal finance website, chances are you want to optimize for some of these keywords. And it's really working for them.

If you use Google's Keyword Tool to estimate the traffic for these keywords, find Mint's rank in the result page for each of them, and then multiply keyword traffic by the distribution of clicks for the top results in Google, you'll see that Mint is raking in at least 100,000 uniques per month from Google for these keywords.

If you hire a writer to post on your corporate blog, you could be seeing this kind of traffic, too. By "writer", I don't mean "Peggy in accounts receivable who majored in English thirty years ago". No, I mean someone whose words are worth reading. A decent freelancer will run you 50 cents per word. A good length blog post is 1,000 words, and you should publish at least once per week. 5 posts like this per month will cost $2,500.

Now let's compare that to buying traffic from Google by bidding on these keywords. A really, really conservative estimate of a bid price for keywords like this is 10 cents (but good luck ranking with that bid, cheapskate). To buy 100,000 uniques would therefore cost you $10,000 per month, and you don't get the PageRank.

Of course, the success of this strategy isn't as quantifiable as buying ads, but eventually you'll see traffic throughput. Any writer worth his salt will be able to game social media sites like Digg and Reddit, which will bring in the backlinks. All you need to do is figure out what keywords to optimize for, and put them in the blog template.

Every day I'm hustlin'

Advice to Old Men from a Young Man

2009-01-17T00:00:00-08:00

1. Unless you were shooting Kennedy, nobody cares where you were when Kennedy was shot.

2. The left lane on the freeway is a young man's game.

3. Things will always get more expensive. Bitching about the cost of gasoline isn't going to make it any cheaper. Corollary: nobody cares that a gallon of gasoline used to cost a nickel.

4. War stories: keep them coming.

5. If you have a prosthetic hook-arm, it's your duty to use it to scare children.   Corollary to #4, your prosthetic hook-arm makes a war story way better. If you didn't lose your arm in a war, make up a good war story to explain it. Nobody will know the difference.

6. The world doesn't owe you anything.

7. Respect your youngers. We're the ones who will pay your Social Security and take care of you when you're enfeebled.

8. Advice you offer to young men should fall into one of these three categories:
   A. The finer points of tolerable behavior when it comes to strippers
   B. Recommendations on quality whiskeys
   C. Sticking it to the man

9. If you're past the point where people depend on you, eat, smoke, drink, and gamble. We young men must control our vices, but you've earned the right to indulge with reckless abandon. Show us what we have to look forward to.

10. You keep getting older, but they stay the same age. From a young man's perspective, a 65 year old man with a 23 year old woman isn't a shame, it's a victory.

Buying Sea Salt? You Might Be a Sucker.

2009-01-11T00:00:00-08:00

If there's one thing I have a lot of contempt for, it's neo-hippie bullshit. However, my appreciation for fresh produce just barely overthrows this contempt, so sometimes I go shopping at The Berkeley Bowl for all kinds of fruits and vegetables that I've never heard of. Really, they have some wonky shit there. Ever see a Buddha's Hand?

Anyhoo, they sell sea salt there. Salt, like 'out the ocean. And people buy it. And those people are morons.

If you buy sea salt, you're paying a premium for the luxury of being a douchebag. It's salt. It has no discernible flavor other than salty, it has no metric of quality other than not mixed with dirt and glass shards, and it should have no variation in price other than cheap.

You can buy 4 pounds of standard issue table salt for $5.37 on the internet. Alternatively, I've seen 4 ounces of sea salt for sale for $2.39. That's a markup of roughly 712%. It's a pretty good business if you're selling salt.

In fact, sea salt might even be bad for you. Regular salt has been used for years as a vehicle for iodine, a chemical your body needs to keep you from becoming a retard. No bullshit, iodine deficiency can cause mental retardation. It only costs a dollar or so to iodize a ton of salt, so it really is ideal. Most sea salt isn't iodized, because it's sold as "natural". Boy, a lesser product for way more money? Where do I sign up?

Some people claim to be able to distinguish the "superior flavor" of sea salt. These are the same kinds of people who keep a fridge stocked with gallons of bottled water and don't use the tap for anything but watering a house cactus. If you are one of these people, you should kill yourself as a public service. The only real difference between sea salt and table salt you'll feel when you eat it is the coarseness of sea salt. That's it. And coarse salt isn't worth fucking ten dollars a pound.

tl;dr if you're buying sea salt, consider yourself successfully marketed to. It's like Fiji water. You got hustled.

There Will Be No Web 3.0

2008-12-21T00:00:00-08:00

The recession reached its hand into Silicon Valley's now lukewarm tub and yanked the plug. It's still draining out, and I wish it would go faster, because there are just too many fucking people in the San Francisco Bay Area. I'm talking about you, guy in your Prius taking the left hand turn on to Middlefield Road too slowly. Leave, now. And don't come back. Bonus points for wrapping your expression of environmental consciousness around a tree. Be one with nature.

The guy who drives the Prius likely works at a Web 2.0 company that's burning its way through the $4 million it raised from Me2 Ventures, one of the many sheep-funds in the Valley who follow the trends of top-tier investors like Sequoia or DFJ but don't have the connections to pull liquidity out of hype.

In two years, this guy's company will finally run out of money, having failed to raise another round because investors are too busy conjuring up the next bubble. The failure of Web 2.0 was a live demonstration in I-Told-You-So, as was the first bubble. Both times, the world looked on and thought "what the fuck are you doing?", and Silicon Valley replied "shut up and bring me my Vaseline". We went from bad business plans to no business plans, and saw much less liquidity this time. The big bang was YouTube, and it was all down hill from there.

The Only Easier Money is Marijuana

So what will the next bubble be? Green technology. Green energy. Green computers. Green pants. Green vomit after an Absinthe adventure.

Al Gore did a wonderful job creating awareness of global warming. Awareness isn't the right word, but neither is hysteria. Both are close enough.

San Franciscans were more motivated than usual by this cause, and have begun to care about their carbon footprints or other such nonsense. Making a San Franciscan feel like he alone can make a difference is the best way to control his actions. See also: spending habits. Al Gore, with his nonthreatening voice and relentless assault of data has the power to cultivate the same feeling in stay-at-home-moms and college students.

Unfortunately, the average American mind can only be concerned with one crisis at a time. Purveyors of fine doom-and-gloom are continuously vying for this spot. Presently, it's the economy. Foreclosures. You're going to lose your house. Oh fuck, you'll lose your house, your family, your car, and did we mention that you'll be living on the street? Fear not. Here's some shit you can buy to make it all better. Here's a politician you can vote for who will fix everything.

Fear cycles last a few years. Remember when we were afraid of terrorism? What about peak oil? Global cooling anyone? When money comes back to the Valley, it's going to be aligned perfectly with the beginning of the next fear cycle, and the next fear cycle is going to be global warming. Or climate change. Or polar bear rescue. You can call it whatever you like, as long as you spend money to fix it. Do your part. It's your obligation as a citizen of the earth.

Still Waiting For That Twitter Business Plan

Green tech hasn't taken off yet because liberal guilt can't support a very big market. What you need is government collusion. You need somebody with a gun to step in and say that if you emit more than 100 tons of carbon per year, you need to pay. You need that same person with a gun to say that these carbon emission credits have value, and can be traded. It helps if your typical Silicon Valley entrepreneur or investor believes the call to action.

That last part is easy. Web 2.0 was all about San Francisco values. Sharing. Caring. Understanding. What would Web 3.0 be about? Many say it's some semantic bullshit. Those are the same people who have figured out what Twine does (any hints?). Whatever we can dream up to do over the internet won't draw any money; investors will be bored with web companies after this debacle. The money will go to green tech, because there will be an obvious business plan, popular support, and a government mandate. How can you lose?

The entrepreneurs will follow suit. Silicon Valley types love to feel like they're making a difference, and green tech will practically let them fellate themselves. (In Web 2.0 the Silicon Valley types fellated one another, so this is the natural extension) It will be different people, as an extensive knowledge of Python doesn't give you much insight into solar panel construction, but the same kind of people.

I believe this because it's satisfying. No more "get users, do something, get bought out". This time, it's "invent something, build it, sell it". Sure, we'll be turning a profit by taking sick advantage of alarmism, but it's a business.

Shut Your Face, Commons Httpclient

2008-12-18T00:00:00-08:00

If you're like me and every other user on the planet, you don't give a shit when an SSL certificate doesn't validate. Unfortunately, commons-httpclient was written by some pedantic fucknozzles who have never tried to fetch real-world webpages.

If you want to turn off SSL certificate validation in httpclient, do this:

1. Put not-yet-commons-ssl.jar on your classpath.
2. Execute the following method before you start any SSL connections:


public static void trustAllCerts() throws GeneralSecurityException, IOException {
	ProtocolSocketFactory sf = new EasySSLProtocolSocketFactory();
	Protocol p = new Protocol("https", sf, 443);
	Protocol.registerProtocol("https", p);
}

This essentially makes commons-httpclient accept every SSL certificate it gets. Yeah, that's what I thought. Who's bitching now?

Python Makes Me Nervous

2008-12-06T00:00:00-08:00

The amount of time saved by using Python as opposed to something like Java is inversely proportional to the number of people working on the project.

As a programmer in a team, you need rules. You need structure. You need order. Freewheeling your way around a software project is going to create more problems than it solves.

What I'm butthurting about here is Python's duck typing. It's cute when you're a lone wolf working on a simple Django application, but add a few more people to the project and it quickly becomes unmanageable. Why? Because with duck typing, you need to keep a lot more state in your head to interact with other peoples' code.

Pydev for Eclipse Sucks Too

Method signatures are virtually useless in Python. In Java, static typing makes the method signature into a recipe: it's all the shit you need to make this method work. Not so in Python. Here, a method signature will only tell you one thing: how many arguments you need to make it work. Sometimes, it won't even do that, if you start fucking around with **kwargs.

Calling a colleague's method isn't as easy as looking at the signature. You need to look at the method definition itself to see what it does with its input.

Let's look at an example from Thrift, Facebook's open source RPC server. Here's the signature to a TServer constructor in Java:


protected TServer(TProcessorFactory processorFactory, TServerTransport serverTransport)

And there are a few other constructors that take different args. Pretty straight forward, if you look at this, you know what you need to instantiate to get your TServer up and running. Now let's look at the Python version:


def __init__(self, *args):

So, how do you use it? Big fuckin' mystery! You can't overload constructors in Python, so they had to mash the several different constructors into one. To figure out how to instantiate a TServer, you need to look at the constructor implementation. As a user of the library, the implementation is none of my concern, unless I'm programming in Python.

What a waste of time.

Whatever You Do, Don't Do It Wrong

What about errors? Python exceptions are what really make me nervous. Your code can run fine for the longest time then shit out with a runtime exception. How do you know what exceptions a method can throw? Well, you don't, unless you look at the method definition. Fantastic.

Java has a well thought out hierarchy of checked and runtime exceptions. Sure, handling checked exceptions means you need to write a bit more code, but it's better to spend the time in development than in debugging at 4am.

Another example is in order. In Java, the constructor to FileInputStream throws a FileNotFoundException if something goes wrong. Since it's a checked exception, you need to deal with it somehow. The fact that this exception is thrown is made obvious in the documentation, and your code won't compile if you ignore it.

Python, on the other hand, prefers to leave things up to chance. This is the documentation for the open() builtin, that opens a file:

Help on built-in function open in module __builtin__:

open(...)
    open(name[, mode[, buffering]]) -> file object
    
    Open a file using the file() type, returns a file object.
(END)

How does this function handle a failure? Does it raise an Exception? Does it return a special value? Nobody seems to know! Ah, fuck it, that's a runtime problem, right?

Sure, runtime exceptions happen in Java, but they are usually things that are indicative of a big fuckup like a NullPointerException, not something stupid like a file not being found.

Programming a large project in Python makes me uneasy. Perhaps I'm just doing it wrong? Do other Pythonistas drop a Valium before they begin the day?

Analog Debugging is Hard

2008-11-24T00:00:00-08:00

I took a new job at a software company in Palo Alto, which means an 80 mile commute every day through Bay Area combat traffic. The first two weeks wore hard on my motorcycle - a 14 year old Ninja 500. Last week on my ride home, the left turn signal stopped working. Fuck.

If you thought debugging a software problem was hard, try debugging a hardware problem. There are some salient facts about hardware problems that make them a real bitch:

You need to buy a lot of tools.
There's a real possibility that you will fuck something up beyond repair.
There's a real possibility that you will injure yourself.
If it's your primary vehicle, you need to have it up and running on Monday.

A blinker light stops working, which means electricity isn't flowing. Sounds easy, but to get access to the wires, I needed to take the whole damn thing apart:

You know how when you're writing software for a client, and they completely underestimate the amount of time and effort required to build something? Yeah, the same goes for auto mechanics. Don't bitch about a shop's $75/hr labor rate or their diagnosis fee.

Avoiding NLP At All Costs

2008-11-13T00:00:00-08:00

I'm working with a startup now on a text summarization project. The requirements are fairly loose: "take all this text and make it smaller", solving the tl;dr problem (too long; didn't read). There are a couple of critical details, namely identifying the sentiment of the text, and a few others that are excruciatingly domain-specific.

At first glance, this seems approachable with some natural language processing libraries. Oh no. There be dragons. At Pressflip, I had myself into a few NLP libraries, and the only takeaway I got from all that experience was "Don't use NLP. Ever."

Why? NLP is yucky. It's complicated, the field is rife with academic shitheaddery, there are some major-asspain licensing issues with a couple of software packages and best of all, it's balls slow. Plus, if you venture down the road of natural language processing, the law of diminishing returns will pull you into a dark alley, pummel you with a tire iron, take you wallet, and then just to be a prick, steal your shoes so you need to walk home barefoot.

My point is, for 99 practical projects out of 100, you can cheat your way out of NLP. Cook up some fancy shit with word frequencies and logarithms. Reach back into your information retrieval notes for inspiration. TF*IDF can take you a long way if you know how to use it.

When I was brainstorming the project I'm working on, my first thought was some hand-waving business about a part-of-speech tagger and a Markov Chain to figure out probabilities of part-of-speech transitions and all that fancy shit. Factor in a little bit of sentiment detection from God-knows-where and that was my sketch. Then practicality set in: how much time do you want to spend on this? If you are considering NLP as the answer to a real problem, it's virtually certain that you're overthinking it.

That being said, NLP does have its place: making the best fucking Wikipedia search engine there ever was with technology licensed from Xerox and then selling yourself to Microsoft.

I Have Never Seen Ubuntu Get Upgrades Right

2008-11-01T00:00:00-07:00

Ubuntu upgrades always, always, always fuck up the same things: network connections and graphics drivers. Without fail, if you upgrade, your wireless connection won't work and any closed-source video card drivers you need will get ill. 8.10 Intrepid Ibex is no exception.

I figured out how to get my wireless connection back, but the NVidia drivers are still a mystery. I don't care about free software idealism, I care that my shit works. I'm willing to jump through minor hoops to make it work, like Ubuntu's "restricted drivers" lecture, but now that doesn't even work:

Why is this such a pain in the balls? I press the activate button and nothing happens. AWESOME. Just because I'm used to wasting hours fixing things like this doesn't mean I enjoy it.

NVidia has 70-ish percent market share of all GPUs. If your shit doesn't work out of the box on 70% of the graphics cards out there, who has failed?

[Update, the next day]: I found the fix. Navigate to System > Administration > Synaptic Package Manager. From there, go to Settings > Repositories. In the "Ubuntu Software" tab, check the "Proprietary device drivers" box. Or edit /etc/apt/sources.list if you want to show your chest hair.

I'm glad to see that passive-aggressive Debian superiority is alive and well.

Auto Mechanics: A Good Hobby for Programmers

2008-10-19T00:00:00-07:00

When I was a kid, I used to help my Dad work on the family cars. We changed the oil, brake pads, repaired a broken hydraulic line, and fixed faulty air conditioning.

It wasn't too long before I was able to do most basic repair and maintenance myself. In college, I spent hours fixing an electrical problem that caused my rear turn signals to go out.

Recently, my car started to make a weird "coughing" noise from the muffler, and I fixed that, too.

This Is Going Somewhere I Swear

Why is this relevant to programming? Well, one of my favorite parts about programming production services is forensic debugging. Some process crashes in the middle of the night and all you're left with is a beeper going apeshit and a stack trace. What went wrong? How do you debug that?

Debugging a car is the same thing, but a lot harder. With a car, the symptoms of the bug aren't usually very concrete: a funny noise, a bad smell, a jittery feeling. Compared to a computer, a car is a very simple machine, but because it's so simple it's much harder to debug. Newer cars have an electronic interface to tell you what sensors are indicating faults, but that doesn't always solve the problem. Plus, the sensor readers are like $300.

The more you work on a car, the more you develop an intuition about it. In code, you can narrow your bug down and fix it. With a car, you narrow the fault down to a couple of suspect parts and start by replacing the cheapest one. For me, fixing a car problem is much more gratifying than fixing a code problem because of the tangibility of it.

So, if you're enjoy debugging and problem solving, you'd probably like auto mechanics. There are a couple of collateral upshots to it: you save some money, you can give your friends car advice, and you get to buy a bunch of really awesome tools.

No one can pull your man card when you have a specialized wrench for an O2 sensor.

Most Programming Interviews are a Waste of Time

2008-10-07T00:00:00-07:00

Interviewing a candidate is so much fun because you get to passively assert your superiority and be professorial enough that you can justify those nine years you spent in graduate school studying compiler optimizations only to get a job maintaining a failure-prone database driven web app.

Interviewers spend almost as much time Googling for interview questions as candidates do.

I've been on both sides of the interview, and I'm here to dump a big load of truth on you about what interviewers really think.

The Technical Question

What The Interviewer Says: How would you find a cycle in a singly linked list?

What The Interviewer Thinks: This job has nothing to do with linked lists. In fact, I don't think anyone has used a singly liked list since the seventies. I wonder if you're good at PHP and MySQL, because that's what all the work is here, but I'm not going to ask you anything about actual job requirements, because that doesn't afford me the opportunity to be pathologically pedantic.

The Follow-Up

What The Interviewer Says: And how would you refine your solution to use O(n) time and O(1) space?

What The Interviewer Thinks: I haven't actually solved this problem myself, nor have I formally proven the "right" answer to be correct. I have done no preparation for this interview beyond looking on the internet for programming interview questions. I'm basically dead wood in this organization, and I pray every day that nobody figures this out. As such, if you come up with the answer quickly, I'll either think you cheated and looked up programming interview questions on the internet, or you're genuinely smart enough to expose my own uselessness should you get hired. In either case, your best course of action here is to pretend like you don't know and let me explain the correct answer with a shit-eating grin on my face.

The Bullshit

What The Interviewer Says: Where do you see yourself in five years?

What The Interviewer Thinks: I have no idea what I'm doing. I really need to keep this room from going dead-air, so I'll give you something to talk about. Just start talking. I really don't care, say anything. I'm not listening. I'm using this moment to think about the woman working in HR that I want to bone, but wants nothing to do with me because I'm an introverted nerd who will never work up the sack to ask her out. Fuck my life.

Finally It's Over

What The Interviewer Says: Do you have any questions for me?

What The Interviewer Thinks: We're 35 minutes through a 45 minute interview. If this doesn't take up ten minutes, I can blame ending the interview early on my clock being fast.

So What Is A Good Interview?

For once, I'd like to have a 45 minute candid conversation with an interviewee. Talk about interests, shoot the breeze, and get a general idea of the guy's technical aptitude and fit with the company. Talk about projects, see if the guy gets animated. Combine that with some pre-submitted code samples, and you can get a genuine idea of how suited the candidate is.

I don't know about you, but I would not want to work for or with somebody who is this passive-aggressive.

Asking pedantic and useless questions like this is just a waste of everyone's time.

Java subList Gotcha

2008-09-13T00:00:00-07:00

Don't ever try to do anything sneaky when you're programming. It will always bite you in the ass. If you still want to be sneaky, read the documentation.

Last week, we had a problem with one of our processes hanging and burning 100% CPU. The first time it happened we chalked it up to mysteries of the universe and restarted the process (a time-honored startup tradition), but the second time, I actually got off my ass and investigated.

Through the miracle of jstack, I could look at the stack trace of a currently running Java process. This is what I found:

"main" prio=10 tid=0x0a030800 nid=0x10a6 runnable [0xb7d7d000..0xb7d82218]
   java.lang.Thread.State: RUNNABLE
     at java.util.SubList$1.nextIndex(AbstractList.java:713)
     at java.util.SubList$1.nextIndex(AbstractList.java:713)

...snip... about 100 lines

     at java.util.SubList$1.hasNext(AbstractList.java:691)
     at java.util.SubList$1.next(AbstractList.java:695)
     at java.util.SubList$1.next(AbstractList.java:696)

...snip... about 100 lines

     at com.pressflip.pipeline.standard.deduper.ShingleDupeDetector.
        dedupBatch(ShingleDupeDetector.java:139)
     at com.pressflip.pipeline.standard.deduper.DeduperPipelineStep.
        innerProcess(DeduperPipelineStep.java:115)

... and right down to the main() from here.

The suspect line in all this is ShingleDupeDetector.java:139, which is one of those how-the-hell-are-you-hanging-on-this lines:

for (Integer x : someCollectionOfIntegers) {

So what the shit, right?

I was using this collection as a cache of sorts, where on every run, I chopped some data off the front of it and added some data to the back, keeping the collection size constant. To accomplish this, I used the subList method on java.util.List, something like this:

someCollectionOfIntegers = someCollectionOfIntegers.subList(fromIndex,
                                                    someCollectionOfIntegers.size());
someCollectionOfIntegers.addAll(incoming);

Well it turns out that subList didn't do what I thought it did. I assumed that I just got a new List that contained the elements in the given range of the original. Oh no, subList returns a view of the original list where only elements in the given range are addressable. A look at AbstractList.java's source reveals this:

 
public List<E> subList(int fromIndex, int toIndex) {
        return new SubList<E>(this, fromIndex, toIndex);
}

And the SubList object keeps a reference to this, as well as an offset to know where iteration starts, so as I updated the "cache", iterating over it became recursive. Oh, balls. That's why it's running slow.

A Web OS? Are You Dense?

2008-09-06T00:00:00-07:00

People are calling Google Chrome a "Web Operating System" and a "Cloud Operating System". Some are even calling it a Windows killer.

I think it's time to nip this horseshit in the bud, before it gets out of hand.

How Does Arringtons Know What Operating Systems Is?

He doesn't. It is TechCrunch's official position that Google Chrome will compete full on with Microsoft Windows, and computers will be sold with Chrome only, having the Windows layer "stripped out". I am not shitting you, he actually said that. Yeah, I get where the argument is going about web apps being more dominant than desktop apps. That prediction is a crock of shit. A 2007 survey found that 73% of Americans have never even heard of Google Docs, and 94% have never tried an online office suite. Yeah, desktop apps aren't going anywhere.

But I'm not here to talk shit on Web 2.0 today. I'm going to present a glimpse of the hole that the incompetent programmers are digging for us.

When Times Were Simple

Let's have a look at the application stack that we all know and love: programs compiled to run in an environment with a C library.

Fuck me, life is good.

Making It Easier On Programmers

I first learned to program in C++ and then later on I learned Java in college. I thought the whole Java Runtime Environment thing was kind of weak, but if it means I don't have to manage memory, that's cool. Same goes for Python, Ruby, and whatever else has its own VM or interpreter.

This situation is pretty agreeable, and lets us prototype applications rapidly. Sure, there's a small trade-off with execution speed, but they have multi-gigahertz processors nowadays. No big deal.

Making It Easier On Idiots

After a while, everybody wanted to be a programmer. Since programming is actually kind of hard, many of these folk landed in PHP and HTML, hence the explosion of webapps. As such, the browser became a feeble example of a "runtime".

Now, with Google Chrome being lauded as a Web Operating System, the stack gets way bigger. This is what it looks like on my computer, considering I run Linux and Google hasn't released their Operating System for the Linux Operating System (that makes sense, doesn't it?)

Users have pretty basic needs when it comes to computers. They want word processing, spreadsheets, communications, and games. These needs have not changed much since the advent of the personal computer. So, when your Aunt asks why her 1.2GHz computer isn't fast enough to run an online word processor that has the same fucking features as the 1987 version of Corel WordPerfect, you don't have an answer for her. There is no justification.

The "Web Operating System" just highlights how much journalists don't know about computers.

Porter Stemming Makes Me Rage

2008-07-23T00:00:00-07:00

I have no formal training in natural language processing. As such, I figure out a lot of this shit on my own.

One of the simplest concepts in NLP/text mining is stemming. If you're not in the know, to stem a word is to remove all the unnecessary shit after its root.

For example, "computer", "computing" and "compute" all stem to "comput". Same root, virtually the same meaning.

Something like this is clearly useful in a search engine like Pressflip, because if somebody searches for "iphone" (and a lot of you people are), the engine should pull up documents that contain the plural (iphones) of the word.

The canonical algorithm for doing this sort of thing is called the Porter Stemming Algorithm, which considers each word on its own. Porter works great 99% of the time, but when it fails, it fucks you hard.

Why You Keep Tryin To Say That Word?

A good example of this comes from the pressflip query logs. A user searched for "marketing". Perfectly reasonable. Porter stemmed that to "market", which returned a bunch of search results about the Dow Jones and Nasdaq. Ouch. Right in the butt.

What went wrong? In smart-talk, the bare infinitive that corresponds to the gerund has a different meaning than the gerund. Again, I know dick-shit about NLP, so maybe you guys have a serious-business name for this sort of thing.

So yeah, gerunds make Porter suck sometimes.

There are some other failure cases I've discovered. Proper nouns will give it to you Clydesdale-style, too. More specifically, proper nouns that don't stem to themselves. Example: "Mariners" and "Marin" both share the same stem. So potentially, someone searching for the baseball team from Seattle will come up with news about the hoity-toity town across the Golden Gate Bridge from San Francisco.

What's the answer to this? If you're a company with millions in VC lottery winnings, you can pay Basistech $100,000 for a 3-year license of their context sensitive stemmer. If you're me, though, you make exclusion lists. Big ones.

That being said, after a large re-processing this weekend, Pressflip search quality is going to improve.

Build Google Protocol Buffers Without Maven

2008-07-07T00:00:00-07:00

Google released protocol buffers as open source, which, with a proper transport, will give both XML-RPC and Thrift a run for their money.

Anyway, it's kind of a pain in the balls to build the Java version. If you import the Java source into Eclipse, it's got all sorts of build errors, all stemming from a missing file: DescriptorProtos.java.

If you've got Maven installed, it will make DescriptorProtos.java for you (this file is generated via protoc). But Maven is stupid, because it didn't work immediately after apt-get install and I couldn't figure out how to fix it within 30 seconds. I have no patience for this kind of bullshit.

So, to build DescriptorProtos.java without Maven, you make it by hand:

protoc --java_out=/home/ted/some_directory \ 
/path/to/protobufsrc/src/google/protobuf/descriptor.proto

(You already compiled protoc, didn't you?)

Drop the output file into Eclipse and protocol buffers will build. There are still a bunch of compilation warnings, but only chumps listen to those.

Corporate Competence

2008-07-04T00:00:00-07:00

I really love it when people just do their jobs. I feel gifted whenever I call a company and get a customer support representative who know what they are doing and actually cares about me.

It's rare, but it happens.

Worst ISP Ever

For a while, I had Comcast's cable internet service. It was clear after two years of putting up with their horseshit that they don't care about customers at all.

Oh, wait, they set up a Twitter account.

Fantastic, but my BitTorrent shit still didn't work on their network. Their installation staff is rude and has questionable hygeine, and their customer support representatives are downright lazy.

Switch to AT&T Now

When I moved, my first order of business was to call Comcast and tell them it's over. They said my service wouldn't end until I brought back my cable modem, and of course, the place I need to bring it back to is only open during working hours.

I took off work early to get this little brick of dissatisfaction back to its rightful owner, because fuck them.

At the same time, I was waiting for AT&T to show up and install U-Verse internet service. They did, and shit was impressive.

They told me the tech would be at my house any time from noon to 2pm on a Sunday. The tech showed up at noon on the dot.
It took him about an hour to set up the service. When he left, he gave me a card with his direct cell phone number. If I had any problem in the next ten days, I called him directly and he would come fix it.
An hour after he left, the service went out. I called him, and he was back at my house within 30 minutes. It turns out there was something wrong with the line from the street to my house, and he had to get another tech out to fix it. That guy showed up, fixed the problem, and was on his way. The two of them were at my place until 8pm on a Sunday until the job was done right.

I've been using the service for almost a week now and it's great.

No BitTorrent fuckery. All my torrents work great, and I can seed.
10 megabits downstream, 1.5 megabits upstream.

Great job, AT&T, you actually care about the people paying your salaries.

Practical Unique Identifiers

2008-07-01T00:00:00-07:00

There have been a handful of places within the Persai pipeline where I have needed unique identifiers of varying length. 64 bits here, 32 bits there. I'm not the only one to ever have to solve this problem, but I could never find a concise toolbox of information on it.

Automatic Increment or Not

MySQL has the AUTO_INCREMENT modifier for integral record keys. That's great, if you're using MySQL. In general, prefer a non-automatically increasing record identifier, unless you have a specific reason. Here's why:

You may actually have to think about thread synchronization at some point when creating records.
If these identifiers become publicly visible, they can leak information about how many records are in your database.
If you make identifiers out of other pieces of data (say URLs), then you can't get the identifier value of a given datum without a table lookup. And even then, you'll need another index on that field.

There are a few cases where automatic increment identifiers are good, though:

You are using a MySQL database and are setting up a simple structure of tables. (i.e. MySQL handles synchronization for you and it's actually harder to not use automatic increment)
The creation order of records is really important to you, but not important enough to store a timestamp field.

Making an Identifier Out Of Arbitrary Data

Easy, right? Just hash whatever data you've got. It's not reversible and spread uniformly over the identifier space. However, many times the output of a standard hashing algorithm is too big. SHA-1, for example, is 160 bits wide. Way too long for most purposes.

In this case, I truncate the output. Yes, this is mathematically valid, because any good hashing algorithm's output will be uniform over the range of the function. And by uniform, I mean really uniform. For example, if you take the first 64 bits of a 160-bit SHA-1 hash and call that your unique identifier, the probability of a collision is going to be uniform over the space of all 64-bit numbers. If it wasn't (i.e. the first 64-bits of a SHA-1 hash were distributed, say normally), then the hash function would be cryptographically insecure.

Don't try to swing your dick around and come up with your own hash function. You'll screw it up. I know I have.

GUIDs

I got an e-mail from a reader about using GUIDs for unique identifiers. This fits with the hashing scheme, but for the most part, I think GUIDs are far too large, especially if you are storing a lot of records. GUIDs are 128 bits wide, so if you have a hundred million records, that's about 1.5GB worth of identifiers. Use a 64-bit identifier, and your space is halved, without a significant increase in collision probability.

Making An Identifier Easier On The Eyes

If you need to put a unique identifier in a URL, it can't look too nerdy. For example, this URL looks like shit:

http://www.website.com/document?id=1b25a53bf21d0206

Too many numbers. So, to make it look better, Base-64 encode it. It will lengthen the code a little, but it's much easier to look at:

http://www.website.com/document?id=ZnJvc3RlZCBidXR0cw==

Eh, well it looks better to me. Personal taste, I guess.

You'll need to make sure that your Base-64 alphabet doesn't include the + and / characters: they aren't URL safe.

Sort Orderings

Don't worry about sort ordering unless you have to worry about sort ordering. Duh. The vast majority of Persai's data is stored simply as files, and for most purposes we don't have to care about the processing order. We're fortunate in that regard (well maybe not fortunate, I mean that's like saying you're fortunate that you're not fat because you exercise and eat sensibly).

Anyway, there are a couple of places in Persai where sort order matters. The ordering of recommendations, for example. There, though, we're just ordering by time, and we need to display the exact time, not just the relative times of the recommendations, so we store a date field and order data by it in the store.

This drives one of my earlier points home: if you need ordering by time, don't count on an automatic increment unique identifier to do it. It's much more robust to store a timestamp.

In fact this point goes deeper. Very rarely do you actually need records sorted by record identifier. What you need is the records sorted by some other value that happens to be reflected in the record identifier by virtue of automatic increment and the insertion order. It's always more robust to store the actual value you need to sort by.

I'm Not Going To Tell You How To Write Code

Because I don't really care. This is how I do it, though.

An Engineer's Guide To Weight Loss

2008-05-20T00:00:00-07:00

After a year of the Google cafeteria and another year of eating low cost Startup FounderChow, I put on a few pounds. Now, I'm starting to shed them, and I'll tell you how.

Before I get into it, I want to lay down a few prerequisites. There are a lot of diet guides out there that will bullshit you into thinking that the process is easy. This is a lie. Dieting and exercising suck. This is possibly the most miserable thing you can do to yourself. You are not going to have fun.

To that end, if you are more than 50 pounds overweight, are unmarried, have no children, and your only reason to get up in the morning is your shitty software job, the healthy lifestyle is not for you. You are better off eating yourself to the grave: you will get much more satisfaction out of life by eating cheeseburgers than you will by torturing the pounds of fat off your gut.

Simple I/O Operation

The science of weight loss is simple: eat fewer calories than your burn. You have heard this before, I trust. To follow this principle, you need to start quantifying. I use a web service called FitDay to track the calories I eat versus the calories I burn.

Start by running a 1,000 calorie per day deficit. To lose a pound of fat, you need to burn around 3,500 calories, so you'll lose two pounds in a week. Just to be clear, doing this sucks ass. However, there are a few easy ways to trim calories here and there.

Go Easy On The Drinking
When I go out on a Friday night to have a few beers, it's not hard for me to consume 800 calories worth of booze. Yes, liquor helps to numb the pain of writing XML parsers all day, but it comes at an expense. To compensate, take up smoking. I smoke more cigars now: it's a good zero-calorie alternative.

Eat One Serving
There are eight servings in a box of Barilla pasta. I used to eat half a box of pasta in a single sitting, 4 servings worth. Since you're counting your calories anyhow, you'll already be monitoring servings. You will also spend less money this way: since I started counting my calories, I've been spending 50% less per week on food. More money for cigars.

Drink More Coffee
Caffeine is an appetite suppressant. In large enough quantities, it can be used as an amphetamine. Drink up.

Ice Cream Keeps You Sane
Low-fat ice cream has around 120 calories per half cup. Fat calories keep you feeling satiated for longer. The Dreyer's brand (sold as Edy's on the east coast) doesn't suck that much.

After you have been limiting your calorie intake for two weeks, your stomach will shrink enough that it takes significantly less food to satisfy you. So that's step one: stop eating so damn much. Step two is exercise. And yes, it's awful.

See What Condition My Condition Was In

The only real benefit to exercise is being able to hold your nose over people who don't exercise. That's pretty cool if you're looking to take your misery out on your co-workers. Protip: it's better for your general well-being to be a prick to your colleagues than to your family.

You will lose more weight by dieting than by exercise if you are eating 1,000 fewer calories per day than you burn by doing nothing, so use exercise only as a supplement to your calorie loss.

If you're going to exercise, use an elliptical machine. Treadmills are terrible: they make you run. If you're like me, you have horrific flashbacks of being 10 year old, sucking wind, being the one that got nailed by the cops because your friends were all physically fit and managed to get away. Failure.

If you're going to pussyfoot around and work out for 30 minutes in your "fatburn" zone three times a week, don't even bother. You're just wasting your time. One hour per day, hard. You should be close to vomiting by the end of that hour.

Well, That's It

Easy, huh. Stop eating so damn much and get off your fat lazy ass.

Machine Learning Is Not As Cool As It Sounds

2008-05-14T00:00:00-07:00

I don't like to talk about my job. Don't get me wrong, I like what I do, I just don't like having to explain things to people who are feigning interest. It's a waste of everyone's time.

When I do have to go into more detail than "I write software", I sex it up by saying "I write artificial intelligence software for recommendation systems". Sounds pretty awesome when you say it like that, huh?

Truthfully, that's like describing a summer job at Burger King as "caloric energy distribution engineer".

Yes, one of the things I do is implement machine learning methods for a news recommendation system. The prerequisite amount of pain-in-the-ass, why-did-I-go-to-college-for-this work, though, dominates the cool stuff.

Vector Space Model AI... Sounds Hot

The idea here is that you turn your data into N-dimensional vectors and let loose a bunch of linear algebra on that shit. In return, you get stuff like classification and clustering. If you want to sound like you know what you're talking about here, you can mention stuff like separating hyperplane, sigmoid kernel function, or k-means++.

I deal mostly in the vector space model. As awesome as all of this sounds, most of the work is a real pain in the balls. Writing a sequential minimum optimization routine for a support vector machine is a good exercise, but it's not useful in practice. Somebody else has already written it for me, and besides, that's not the problem I need to be concentrating on.

Most of the methods that deal with VSM machine learning are well defined and fairly easy to implement. What remains a mystery, though, is the generation of the input. How you translate your data into vectors is the most important problem to solve. It's also the most boring. After that, you can worry about shaving 3 nanoseconds off of your dotproduct routine.

Then You Need To Deal With The Academics

Publish or perish. Yeah, that's cute, but in the real world it's profit or perish, and that means getting useful results. Academics love machine learning because it affords them the opportunity to make slight variations to undergraduate level mathematical procedures, quantify the result, and write it up in LaTeX with graphs and shit.

It's easy to write a paper showing the effects on precision and recall for a perceptron classifier on the Reuters corpus using normalized vs. non-normalized vectors. It's not easy to generate data as clean as the Reuters corpus from a web crawl. Not only is this task hard, it's about as much fun as chemotherapy. As such, there are no useful papers coming out of academia about how to parse HTML. Unfortunately, problems like these are the ones that need solving.

When I talked earlier about all of the prerequisite bullshit, this is what I meant. You get the most testicular pain when dealing in text content, and the real deep-rooted ball ache comes from web content. We put a ton of effort into our HTML parsing routines, and it has paid off.

For reference, altering a method that helped with removing boilerplate content from a web page (boring) had a greater benefit to the accuracy of our classifier than did dimensionality reduction and normalization combined (sexy). If you're not picking up what I'm putting down here, I'm saying that the really hard and less science-y improvements made the machine learning better than any of the shit you would read about in an ACM journal.

This Is Going Somewhere I Promise

This really is just a buttsore blog post, but I'm on a roll now.

When I am working with the machine learning part of my job, I am rarely working in my development environment. Most of the real stuff is done in Excel. Well, at least it used to be, until I figured out that GNU R is so awesome it makes me want to fuck myself up with a chainsaw.

When I make a change to the inputs of a machine learning method (support vector machine in this case), I need to verify that the change I just made was actually positive. And since that can't be done with a JUnit test, I have to get all scientific-method on that shit. Remember in college when you snoozed through advanced statistics because it sucked? Yeah, me too. Good thing I kept the book.

Weekend Science Project

2008-05-10T00:00:00-07:00

I built an HD antenna today. Using these instructions and less than $15 worth of materials, I can get a few local channels in HD over the air. Check this sucker out:

Ugly as hell, but I can watch Lost in HD without paying Comcast an extra dime.

JavaFX Native Look and Feel

2008-05-10T00:00:00-07:00

I have been toying around with JavaFX, Sun's answer to Adobe AIR and Microsoft Silverlight. Since JavaFX is pretty much an easy way to do Swing, you can get Swing's pluggable look and feel in Java FX programs. Thank Christ, because the Swing UI components look like shit:

Versus the GTK look and feel:

This was done by adding the following snippet to my JavaFX code:


import javax.swing.UIManager;
 
 UIManager.setLookAndFeel(
            UIManager.getSystemLookAndFeelClassName());

Oh neat, native UI components.

Eclipse Crashes in Ubuntu Hardy Heron

2008-04-26T00:00:00-07:00

I just upgraded my workstation to Hardy and got a SIGSEGV when starting Eclipse. It appears to be a bug in the Sun JVM that ships with Hardy, and it only happens on AMD64.

If you're set on using this runtime, the fix is to disable the JIT compiler by launching Eclipse with -Xint, but that's comes with a severe performance penalty.

The fix I used was to simply downgrade the Hardy JRE (6-06-0ubuntu1) to the Gutsy version (6-03-0ubuntu2). You'll have to edit /etc/apt/sources.list to add a Gutsy repository.

I'm pretty sure this is the Sun bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6614100

And this is the Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/eclipse/+bug/174759

I'm Going To Scale My Foot Up Your Ass

2008-04-24T00:00:00-07:00

Engineers love to talk about scalability. It makes us feel like the bad ass, dick-swingin' motherfuckers that we wish we could be.

After we talk about scalability with our co-workers (Yeah, Rails doesn't scale!), we flex our true engineering prowess by writing a post about it on our blog. Once that post hits Reddit, son, everyone will know how hardcore you really are. Respect.

People Who Talk Big About Scalability Don't Need To Worry About It

Fact: every chest-thumping blog post I have seen written about scalability is either about architecture, Memcached, or both. Some asshole who writes shitty code starts pontificating about "scalable architecture" with data storage, web frontends, whatever-the-fuck. Dude, your app isn't having scalability problems because of the architecture. It's having scalability problems because you coded a ton of N^2 loops into it and you're too self-important to get peer reviews on your commits.

And let's not forget the tools who discover Memcached for the first time, install it on a web server, and notice how fast their app runs now. Yeah, welcome to the modern age. Hope you know what a cache expiry policy is.

If You Haven't Discussed Capacity Planning, You Can't Discuss Scalability

You don't need to worry about scalability on your Rails-over-Mysql application because nobody is going to use it. Really. Believe me. You're going to get, at most, 1,000 people on your app, and maybe 1% of them will be 7-day active. Scalability is not your problem, getting people to give a shit is.

Unless you know what you need to scale to, you can't even begin to talk about scalability. How many users do you want your system to handle? A thousand? Hundred thousand? Ten million? Here's a hint: the system you design to handle a quarter million users is going to be different from the system you design to handle ten million users.

Of course you'll point to the engineer's wet dream: linear scalability. Lulz but when we get more users we just add more machines you are so stupid ted. uncov sucks.

Yeah, great, well it doesn't exist. Oh no, go ahead and try out Amazon SimpleDB and think to yourself that it will scale linearly. Then, when you get enough users that the latency becomes a problem, blame it on "those shitty Amazon datacenters".

Choosing Technology Don't Mean Shit If You Don't Know How To Use It

The most common butthurt about scalability is this: choose a technology. If you like the technology, claim "technology X scales better!" If you don't like it, claim "technology X doesn't scale!"

Saying "Rails doesn't scale" is like saying "my car doesn't go infinitely fast". Alternatively, saying "We'll have no problems scaling because we're using Django" is like saying "I will win every race because my car is the most powerful". Maybe so, but you suck at driving, and you're up against professionals.

If you're having scalability problems and blaming it on a single technology, chances are, you're doing it wrong.

tl;dr

Shut up about scalability, no one is using your app anyway.

Don't Serialize Java Objects In Hadoop SequenceFiles

2008-04-08T00:00:00-07:00

Not if you can avoid it, at least.

Hadoop provides you with the Writable interface if you want to write your object to a SequenceFile. It's up to you to implement the write() and readFields() methods for your object. It's easy if your object is simple: just write each of your instance variables to a DataOutput and read them back in the same order from a DataInput.

Don't Write Your Object As A Serialized Byte Array

I got lazy when I was implementing the Writable interface with one of our classes because it had a ton of instance variables. I figured I'd just serialize it to a byte array, then write the array length and the whole array to the DataOutput. And on the read, well, just unserialize the object from the byte array. This was my write():


@Override
public void write(DataOutput out) throws IOException {
	ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
	ObjectOutputStream objectOut = new ObjectOutputStream(byteOutStream);
		
	objectOut.writeObject(getContainedObject());
	objectOut.close();
		
	byte[] serializedObject= byteOutStream.toByteArray();
		
	out.writeInt(serializedObject.length);
	out.write(serializedModel);

}

Naw, dude. Bad idea.

I knew that I'd be paying some overhead in both space and time for this little scheme, but I didn't know how much. It was just a little bit per object, but when we started seeing MapReductions take way too much time in I/O, it was time to revisit this.

What This Cost In Space And Time

First, the Java serialization space overhead. On a toy example of this object, serialization to a byte array used 953 bytes. Properly writing out the instance variables consumed 296 bytes. In production, doing it the right way shrunk a 1,600-record SequenceFile from 1.4GB to 825MB.

Time savings were great, too. In the same toy example, it took my JVM 7.2 milliseconds to serialize the object and 1.7 milliseconds to unserialize. Doing with with stream I/O only took 76,000 nanoseconds to serialize, 58,000 nanoseconds to unserialize.

I love order-of-magnitude improvements.

Lesson learned: get off your lazy ass and do it right.

Plugged in the New York Times

2008-03-20T00:00:00-07:00

Me, Persai, and Uncov got a plug in the New York Times today. We've been in VentureBeat, Slate, and now NYT, but not TechCrunch. Something tells me that's not an accident.

A Magic Elixir

2008-02-27T00:00:00-08:00

In software, there are no silver bullets. In internal combustion engine mechanics, however, there are plenty. And I just discovered one.

It's called Sea Foam, and it will cure what ails 'ya.

My wife's first motorcycle was a Honda Rebel 250. She upgraded too late in the season and couldn't sell the starter before winter showed up. Winter time is a dead zone for the used motorcycle market in the San Francisco Bay Area, so the Rebel sat in the parking garage for 5 months. Being lazy, I didn't properly store it.

We went to fire it up yesterday to prepare it for its 15 minutes of Craigslist fame, and it wouldn't turn over. Gasoline, if left long enough, will degrade into a mucky varnish that cakes the inside of your carburetors.

I poured half a can of Sea Foam into the tank and let it sit for a few minutes. I cranked it again and it made a few pathetic putts. A few more cranks, a few more putts, but after about 5 tries, the Rebel roared to life.

A six dollar bottle of some petroleum distillate has the same end effect as a three hundred dollar carburetor job.

I am detecting much win in this sector.

Core Dumps Disabled By Default In Ubuntu

2008-02-19T00:00:00-08:00

Enable them with this command:

ulimit -c unlimited

The Road To Hell Is 64 Bits Wide

2008-02-14T00:00:00-08:00

Java is an awesome language because you get to ignore hard stuff like memory allocation. Write once, run anywhere. Sweet, where do I sign up?

The privilege of not having to manage memory comes at a cost: you aren't allowed to question how the JVM works. Move along, coder. Keep making those objects. Don't ask how much memory things take up. In fact, to keep you from getting curious, we're not even going to have a sizeof function.

How My Complacency Made Me Fail

When you're coding in Java, it's easy to buy into this mentality. You never really have to worry about how much space anything takes up, and if you get an OutOfMemoryError, just give the JVM more memory. Problem solved.

But there are times when you need to be conscious of how the JVM actually works. For example, when you're trying to squeeze every bit of performance out of the crappiest of machines, such the small Amazon EC2 instances (where Persai is hosted).

Here's a real world example of how I got burned:

Persai does a lot of work with high-dimensionality, sparse vectors. To save space, we compact the vectors. Since most of the values in the vectors are zero, we simply do not store them. What we store amounts to a list of the nonzero element indices and the corresponding values. This is our basic data structure:

class sparseNode {
    public int index;
    public double value;
}

So a vector is an array of sparseNode objects. Sounds easy enough, and for a while, it was. That is until I was tasked with storing as many of these in memory as I could.

Where's the fail here? How big is an int primitive in Java? 32 bits, right? Sort of. The Java specification says that an implementation must provide 32 bits of workable space for the programmer using an int, but makes no mention of how how the virtual machine must store this variable.

In Sun's HotSpot JVM, object storage is aligned to the nearest 64-bit boundary. On top of this, every object has a 2-word header in memory. The JVM's word size is usually the platform's native pointer size. Alright, two words for the object header, one word for the int, two words for the double. That's 5 words: 160 bits. Because of the alignment, this object will occupy 192 bits of memory. Effectively, the int value is taking 64 bits! In an array of these things, I've wasted N times 32 bits. Figure, a typical vector is about 200 elements long, so that's 800 bytes out the window for each one.

This fun fact would have been good to know when I was doing my initial back of the envelope calculation of how many vectors I can fit in a gigabyte of memory.

Yes, I know I should be complaining about the same thing when using C structs. But you know what? When you learn C, you are introduced to many harsh realities. When you learn Java, you are introduced to XML. They protect you from the hard things. Live and learn, I guess.

The Fix

After knowing this, it was easy to rescue myself. Java primitive arrays fall to the same alignment issue, but in our case, they can help solve the problem. Instead of representing a vector as an array of objects, we'll represent a vector as an object of arrays.

class sparseVector {
  public int[] indicies;
  public double[] values;
}

This way, we're going to lose at most 4 bytes per vector with the alignment of the int[] array. This sure beats the ~800 byte loss with the other solution.

Why Machine Learning Isn't Getting Any Easier

2008-02-02T00:00:00-08:00

I learned the basics of machine learning in college. Classifiers, clustering, all that jazz. Every undergrad computer science major loves this crap because they can understand it with a passing knowledge of linear algebra. Machine learning. Sounds pretty sexy.

In Practice, It's A Lot Harder Than What You Did In College

Up front, let me say that I implemented the vast majority of the machine learning technology behind Persai. In the beginning, I thought it was going to be a breeze. Take some documents, turn them into features, train a classifier, and off you go. The harsh reality is that less than one percent of the time I spent on this system went into the "sexy part" of machine learning, and most of that was done by the guy who wrote the SVM library we use!

The lion's share of time, and the source of most of the hair-pulling, was spent dealing with the data. Data coming in off of the open internet is dirty. Conflicting character set declarations, boilerplate removal, duplicate detection: these things will drive you to insanity and back.

I was fooled by the simplicity when I first learned this stuff. This is what they teach you about vector space model based classifiers:

Turn your data into vectors.
Specify the positive and negative samples.
Train your classifier.
Tune your vectorization scheme and classifier parameters until the classifier is good.

What they don't teach you is this: Step 1 is a bitch.

Publish Or Perish

If turning data into vectors is such a hard problem, why aren't the academics churning out papers about it? Because it's not sexy. There are no numerical nuances to deciding how to handle a document whose declared character set is ISO-8859-1 but is actually encoded in UTF-8. There's no Turing award coming your way for finding a way to make reasonable text out of horrifically malformed HTML that makes you curse Firefox and Internet Explorer for accepting as renderable.

When I started Persai, I admitted that somebody else has already done the mathematical programming better than I ever could. I didn't spend years of my life studying numerical analysis, so chances are, if I attempted to write my own SVM library, I would fail. So, in the interest of success and avoiding Not-Invented-Here syndrome, I used somebody else's library.

People have busted my chops for this, too, as if I am somehow less of an engineer if I use a third-party library. However, one thing has become painfully obvious: the quality of a classifier depends much, much more on your ability to sanitize data than on the algorithm you use.

Hermetic RPC Unit Testing With Thrift and jMock

2008-02-01T00:00:00-08:00

Unit testing is a pain in the ass. I will admit it, I hate doing it. More often than not, you just write a few obvious JUnit tests that you know will pass and say you're finished.

Testing code that makes RPC calls is especially discouraging. You'll say "I can't unit test it, it needs to set up an RPC server and that's too complicated for JUnit", or, if you're like me, you won't even make up an excuse.

Of course, this laziness comes back to bite you when the code goes into production, the RPC server throws a one-in-a-million exception, and your entire service bites the dust because you never tested that execution path.

So, given that you don't like to be woken up at 3AM by sysops when you have been out drinking all night, let's unit test our RPC clients. Let's do it without having to start up an RPC server when the test runs, and it would be nice to be able to have fine-grained control over the RPC methods.

She's Thrifty - She's Just My Type

This is the Thrift RPC definition we will be using for this example:

service MyRPCService {
  i64 getDocidForUrl(1: string url),
}

Simple. We'll be looking up a 64-bit integral document identifier for a given URL. Our client code will make a decision about the state of the document given that identifier.

This is the class we will be testing:

public class ProgramToTest {

	// class constants
	private static final int RPC_SERVER_PORT = 3141;
	private static final String RPC_SERVER_HOST = "rpcserver.widgetsandshit.com/teddziuba";
	private static final long DOCID_IS_OLD_IF_LESS_THAN = 1000;
	public static enum DocumentStatus { OLD, NEW, UNKNOWN };
	
	// instance variables
	private MyRPCService.Iface myRpc;
	private TSocket socket;
	
	public ProgramToTest() {}
	
	private void init() throws TTransportException {
		socket = new TSocket(RPC_SERVER_HOST, RPC_SERVER_PORT);
		TProtocol protocol = new TBinaryProtocol(socket, true, true);
		myRpc = new MyRPCService.Client(protocol);
		socket.open();
	}
	
	public Enum getDocumentStatus(String documentUrl) {
		try {
			long docId = myRpc.getDocidForUrl(documentUrl);
			if (docId < DOCID_IS_OLD_IF_LESS_THAN) {
				return DocumentStatus.OLD;
			}
			return DocumentStatus.NEW;
		} catch (TException e) {
			return DocumentStatus.UNKNOWN;
		}
	}
	
	public void finished() {
		socket.close();
	}
	
}

If I were still in CS class in college, I would get dinged for having multiple return statements, but the best part about being a grown up is that when I want a cookie, I can have a cookie.

The getDocumentStatus is really the only thing we need to test, as clients of this class will be responsible for dealing with a TTransportException if the socket initialization fails. The unfortunate part about testing that method is that it makes an RPC call. Sockets. Exceptions. Icky. Even though it's easier to say screw it and go have a beer, remember: you gotta do what you gotta do.

Making a Mockery

JMock is a clever unit testing library that makes mock objects really easy. If you're new to mock objects, read more about them here. The basic idea is that we will make an object that "mocks" the behavior of the RPC server, but without doing any I/O. That way, we have complete control over the operations of the server, and can actually test how your client code interacts with that one-in-a-million exception.

We'll be mocking out the MyRPCService.Iface interface that is autogenerated by Thrift, and defining our own behavior for it. If you've got some experience with JMock, this should be pretty straight forward, and if not, then you'll catch on quick. JMock's syntax focuses on making the testing conditions human readable.

Prepare The Class For Testing

Since we will be providing the ProgramToTest class with a mocked version of this interface, we need to add a constructor to the class for testing only:

public ProgramToTest(MyRPCService.Iface testOnlyIface) {
	this.myRpc = testOnlyIface;
}

JUnit. We In It.

We'll test the low-hanging fruit first. Using our mock to control the return value of the RPC call, we can make sure the logic works:

@Test
public void testHandlesOldDocid() throws TException {
	final MyRPCService.Iface mockedRpc = context.mock(MyRPCService.Iface.class);
	ProgramToTest testObject = new ProgramToTest(mockedRpc);
		
	final long rpcCallReturnValue = 100L;
	final String testUrl = "http://www.widgetsandshit.com/teddziuba/";
		
	context.checking(new Expectations() {
		{
			one(mockedRpc).getDocidForUrl(with(equal(testUrl)));
			  will(returnValue(rpcCallReturnValue));
		}
	});
				
	assertEquals(ProgramToTest.DocumentStatus.OLD,
                               testObject.getDocumentStatus(testUrl));
}

That is pretty cool. Without a whole lot of effort, we've managed to make a unit test for a method that depends on an RPC server. This test does not require any network I/O and runs very quickly. It can be run in a self-contained environment, like an automated test server. I call this kind of test hermetic, because nothing outside of the test code can affect its outcome.

We can also use JMock to test what happens when an exception is thrown. If a Thrift RPC server throws an exception somewhere in its handler method and that exception is not caught server-side, it will be thrown up to the client as a TException. To simulate this, we simply change one line of the test expectations:

@Test
public void testHandlesException() throws TException {
	final MyRPCService.Iface mockedRpc = context.mock(MyRPCService.Iface.class);
	ProgramToTest testObject = new ProgramToTest(mockedRpc);
		
	final TException rpcException = new TException("something awful has happened.");
	final String testUrl = "http://www.widgetsandshit.com/teddziuba/";
	
	context.checking(new Expectations() {
		{
			one(mockedRpc).getDocidForUrl(with(equal(testUrl)));
			  will(throwException(rpcException));
		}
	});
				
	assertEquals(ProgramToTest.DocumentStatus.UNKNOWN,
                                testObject.getDocumentStatus(testUrl));
}

Go And Do Likewise

JMock is an incredibly useful library. If you're a lazy tester like me, it beats the pants off of subclassing. Now, you have no excuse for leaving RPC calls untested.