Scott builds Software

[Ed: This post is a whitewashed transcription of my invited talk at the Amazon Web Services meetups in New York and Boston in 2008.]

In 2007 I led a team building an online video advertising service for 5m consumers.  My team was excited about the new Amazon web services. Having worked with our own machines for years, I too was curious, but had no real experience with it. It seemed like the natural evolution of hosting, something we had dreamed about at IBM in the late 90’s.

The chart you see here tells the whole story. The Y axis shows the amount of bandwidth being generated. The X axis is time. The red color represents bandwidth from our managed hosting account, feeding external pipes. The yellow color represents bandwidth from our S3 buckets into the web.

First thing we did was fix our process for building software. The company was replete with producers, the kinds of people who love to see an entire script, to read something end-to-end, to ensure that everything has been covered. Those complete, detailed, hundred page “specs” work great for television and movies. Its horrible for software.

We switched to an agile process, akin to what some new friends at Pivotal Labs are espousing. That allowed us to be more flexible with what we build, when, tuning and adjusting every week, committing to deliverables in 4-week sprints.

Next, we called our providers. We told them we’d adjust our monthly commitments as we were “going into the cloud.” I think they were snickering at us on the other end of the line. The cloud was a lot of hype when we started.

Then we got started. I didn’t have a lot of extra budget. In fact, my budget was cut, and had to cut further. We had to squeeze what we’re doing into the current spend. We looked at our architecture, poured over traffic logs, and realized that a lot of traffic was being served from our origin servers that should be served from the edge.

After a bit of wrangling and reswizzling our backend to be more REST-like, we chopped up our search results, browse results, channel results, and more into bite-sized pieces with finely tuned caching rules. We deployed this technology in late February, which resulted in the first drop (1) in cost.

We used the savings from this architectural work to fund our Amazon experiments. Note how the chart starts to show a sliver of yellow at (2). That’s the early experiment with S3, where we were writing Bash scripts, PHP classes, and test clients in Flash and JavaScript to pull videos from S3. After a month or so of testing, we ran a pilot. Here we took our videos that were created in 2006 and served them directly from S3 instead of a hosted SAN.

The pilot worked. We built up confidence, held a meeting, and looked each other in the eye.

“Are we ready guys?”

“Sure, why not.”

We threw down the hammer (3). We fired up our cron scripts and started migrating our videos en masse to S3. The result was dramatic. Within days the bandwidth dropped by over 90% from Rackspace, where transfers were costing us $1.00 per Gigabyte, and storage was eating us alive at $8.00 per Gigabyte per month on a high-end SAN. Our new cost structure was $0.45 per Gig to store the videos, and a measly $0.17 to deliver a Gig into the Internet. The $0.45 was “high” as we kept three different copies of our videos on three availability ones.

Our CFO was delighted (well, as delighted as a bean counter can be, which is actually a half-smirk, almost a smile). The CEO couldn’t believe the savings. We actually ripped 90% of our cost out of the monthly bill!

I’ll admit it. We were getting a little ahead of ourselves, feeling a tad overconfident. We decided our next target should be Akamai. Let’s attack that bill! Let’s knock a six-figure
bill down to the low thousands!

We fired up more cron scripts and turned our “suck-o-meter” all the way up. The “suck-o-meter” was a device that took a percentage of our traffic from high-end CDN deliver at Akamai to storage “that sucks” in a cloud S3. We assumed hardware was terrible, probably SATA drives in aging 1U PCs.

Initially this worked, which gave us the big spike (4). My Amex bill from Amazon was over $8,000 that month! Within days, however, we were getting calls from advertisers. Our head of product came to see me, as did our CEO. All were complaining about choppy videos.

Sure enough, we ran tests and discovered that S3 delivery was choppy. The chart here shows the bandwidth throughput of a connection to S3. Our videos were encoded at 500 kilobits per second. That’s about midway through the choppy part of the graph. We had to be in solid green for the videos to deliver smoothly. In fact, we saw that some S3 connections were slower than a 1993 modem at 56 kilobits per second! Other times we’d get Akamai quality at 1000 kilobits and more.

We turned the suck-o-meter back to zero, leveling out our costs. We’d live to tackle CDN costs another day. That’s a future post. Amazon just released the CDN private beta, and we were invited to join!

Filed under:
Cloud computing by admin