[Ed: This post is a paraphrased transcription of my talks at the Amazon Web Services meetups in New York and Boston last week.]
Last summer I met David Carson, the colorful and creative cofounder of Heavy. We shared a burger at a nondescript hotel in Westchester. While I was munching on french fries, David began to describe his model for making money with video. Instead of selling ads directly in the video stream, his team was selling an enormous ad that surrounded the video. He called this a “video skin.”
I asked David how many videos he had. How many videos use this skin?
“Oh, about 20,000.”
That was nothing. We’d upload that before breakfast at Photobucket. Yet I was curious. “How much money are you making with these skins?”
David paused, looked at me. “I’d say somewhere between $10 and $20 million this year.”
I nearly choked on my cheeseburger.
That’s incredible! Here was this guy making millions out of a fraction of the videos we were producing at Photobucket. If this was true, and indeed it was, he had a real shot at making big bucks. Polaris wanted to turn David’s ad model into a network, enabling other video producers to profit, helping advertisers reach tens of millions of viewers… not just the kids watching skateboard stunts and girls on Heavy. A few weeks later I signed on as CTO.
Within a month I saw some real challenges under the hood. Yes, they were indeed making tons of cash off these videos. The revenue was very lumpy. Dave would make a killing during the Holiday season, when retailers would reach out to customers seeking the latest gizmo for their kids. During the dark winter, he’d suck wind. Revenue would dry up.
This wouldn’t be so bad, except as CTO I was faced with six figure monthly bills from Rackspace and Akamai. That was fine when revenue flowed. It was a disaster for the books when revenue dried up. Heavy’s newly hired CFO looked across his desk at me, frowning, saying I had to “do something” about these costs.
My team was excited about the new Amazon web services. Having worked with our own machines for years, I too was curious, but had no real experience with it. It seemed like the natural evolution of hosting, something we had dreamed about at IBM in the late 90’s.
The chart you see here tells the whole story. The Y axis shows the amount of bandwidth being generated. The X axis is time. The red color represents bandwidth from our Rackspace account, feeding external pipes. The yellow color represents bandwidth from our S3 buckets into the web.
First thing we did was fix our process for building software. Heavy was replete with producers, the kinds of people who love to see an entire script, to read something end-to-end, to ensure that everything has been covered. Those complete, detailed, hundred page “specs” work great for television and movies. Its horrible for software.
We switched to an agile process, akin to what some new friends at Pivotal Labs are espousing. That allowed us to be more flexible with what we build, when, tuning and adjusting every week, committing to deliverables in 4-week sprints.
Next, we called our providers. We told Rackspace that we’d likely adjust our monthly commitments as we were “going into the cloud.” I think they were snickering at us in Texas. The cloud was a lot of hype when we started.
Then we got started. I didn’t have a lot of extra budget. In fact, my budget was cut, and had to cut further. We had to squeeze what we’re doing into the current spend. We looked at our architecture, poured over traffic logs, and realized that a lot of traffic was being served from our origin servers that should be served from the edge.
After a bit of wrangling and reswizzling our backend to be more REST-like, we chopped up our search results, browse results, channel results, and more into bite-sized pieces with finely tuned caching rules. We deployed this technology in late February, which resulted in the first drop (1) in cost.
We used the savings from this architectural work to fund our Amazon experiments. Note how the chart starts to show a sliver of yellow at (2). That’s the early experiment with S3, where we were writing Bash scripts, PHP classes, and test clients in Flash and JavaScript to pull videos from S3. After a month or so of testing, we ran a pilot. Here we took our videos that were created in 2006 and served them directly from S3 instead of Rackspace.
The pilot worked. We built up confidence, held a meeting, and looked each other in the eye.
“Are we ready guys?”
“Sure, why not.”
We threw down the hammer (3). We fired up our cron scripts and started migrating our videos en masse to S3. The result was dramatic. Within days the bandwidth dropped by over 90% from Rackspace, where transfers were costing us $1.00 per Gigabyte, and storage was eating us alive at $8.00 per Gigabyte per month on a high-end SAN. Our new cost structure was $0.45 per Gig to store the videos, and a measly $0.17 to deliver a Gig into the Internet. The $0.45 was “high” as we kept three different copies of our videos on three availability ones.
This is when I received my first unsolicited call from Rackspace. Their IPO was imminent. An executive called me and wanted to hear more about what we’re doing. He said they’d be releasing cloud services soon. The account team later called me and offered to drop the storage price by more than 75%, down to $1.50 per Gig, “if I’d just keep what storage we had at Rackspace.”
Heavy’s CFO was delighted (well, as delighted as a bean counter can be, which is actually a half-smirk, almost a smile). Heavy’s CEO couldn’t believe the savings. We actually ripped 90% of our cost out of the monthly bill!
I’ll admit it. We were getting a little ahead of ourselves, feeling a tad overconfident. We decided our next target should be Akamai. Let’s attack that bill! Let’s knock a six-figure
bill down to the low thousands!
We fired up more cron scripts and turned our “suck-o-meter” all the way up. The “suck-o-meter” was a device that took a percentage of our traffic from high-end CDN deliver at Akamai to storage “that sucks” in a cloud S3. We assumed hardware was terrible, probably SATA drives in aging 1U PCs.
Initially this worked, which gave us the big spike (4). My Amex bill from Amazon was over $8,000 that month! Within days, however, we were getting calls from advertisers. Our head of product came to see me, as did our CEO. All were complaining about choppy videos.
Sure enough, we ran tests and discovered that S3 delivery was choppy. The chart here shows the bandwidth throughput of a connection to S3. Our videos were encoded at 500 kilobits per second. That’s about midway through the choppy part of the graph. We had to be in solid green for the videos to deliver smoothly. In fact, we saw that some S3 connections were slower than a 1993 modem at 56 kilobits per second! Other times we’d get Akamai quality at 1000 kilobits and more.
We turned the suck-o-meter back to zero, leveling out our costs. We’d live to tackle CDN costs another day. That’s a future post. Amazon just released the CDN private beta, and we were invited to join!
