Tinkering with Amazon's S3 and EC2

I was really excited about Amazon's EC2 (Elastic Compute Cloud) when it first became available sometime last year. Unfortunately it was (and still is) a "limited beta" service (yeah I know) so all I could do was sign up for it and hope that my day would come.

It did :) (big surprise!).

Anyway I've been playing around with my SliceHost VPS (which they of course call a "slice") for a week and was hoping to have access to a similar service at low cost to compare, so the timing was just about perfect. Note that I also had to wait for my SliceHost slice, but it was a lot quicker (about 8 weeks). I have to get a hold of the specifics for both services to compare price and performance, but my main concerns right now are ease of deployment and flexibility, so that'll have to wait.

My initial guess is that both services will behave pretty much the same after they're setup. You start with a basic OS to play with anyway. I probably won't trust anyone else's image and just roll out my own customizations.

Back to the Amazon services, you can't just use EC2 out of the box. First you need to sign up for S3 (Simple Storage Service), which is pretty awesome on itself but I hadn't used yet (mostly for lack of real need). So I did that (no wait for that one) and decided to explore it a bit before moving on with my EC2 experiments.

At first a nice command-line client to S3 would be really nice, to get a general feel for the service without having to dive into the API. I did a quick search and started using s3cmd. It's written in Python, which I like a lot, and has a very clean feel to it. The only problem was that it didn't use my outgoing HTTP proxy (Squid) like other Python apps usually do.

If you have a http_proxy environment variable, the Python libs will generally detect that and pass your requests onto the proxy. After inspecting s3cmd's code, I realized that due to a multi-step process in generating the webservice requests, that functionality was being sidestepped. This required some minor tweaking to add proxy support to s3cmd's config file and then use that to properly craft the requests for proxy and non-proxy situations. I created a patch and sent it to the maintainer. Of course you can just download the patch and apply it yourself--it's against version 0.9.3 of s3cmd.

Other interesting projects related to S3 are s3sync.rb (a Ruby backup script similar to rsync) and Backup Manager (a general backup tool that I've been told supports S3 now), but I haven't tested them yet.

Having played with S3 I decided to complete my first EC2 setup. After jumping through all the hoops in the Getting Started guide, I chose Paul Dowman's image for my first boot. It worked like a charm, and in less than 1 minute (!!) I was able to login as root to my newly created instance, complete with Rails, MySQL and an up-to-date Ubuntu Feisty install.

BTW, the EC2 tools (in Java) also don't honor the http_proxy environment variable (sigh). Here's the environment variable you have to set to make it all work, assuming your proxy is not password protected:

export EC2_JVM_ARGS="-DproxySet=true -DproxyHost=<host> 
  -DproxyPort=<port> -Dhttps.proxySet=true 
  -Dhttps.proxyHost=<host> -Dhttps.proxyPort=<port>"

Having access to EC2 opens up a lot of possibilities in deployment and scaling. From what I've been reading, the main complaint about EC2 is the lack for persistent storage. I don't see that as too much of a problem because you do have storage as long as the instance is running, and even if it crashes (doubtful) from my understanding the data is intact. I rebooted my instance and it came back unscratched. Only if you forcibly shutdown ("shutdown now -h") do you destroy the image and lose your data. Plus, you can always backup to S3, even the whole image block by block if you'd like. Another common complaint is about the lack of static IPs, but that can also be worked around with dynamic DNS services.

After successfully booting my first EC2 instance I now plan to work on some deprec recipes to deploy rails apps on either SliceHost or EC2, transparently. My target system will be an Ubuntu Feisty base, with PostgreSQL 8.2, Apache 2.2 and Mongrel.

Using the Compaq PA-1 with Linux

I purchased one of these little guys in 2001 and used it very little at the time. It took forever to transfer songs, only worked under Winblows, and the capacity was very limited (they come with two 32MB MMC cards).

Fast-forward 4 years and podcasting is born. It's really nice to always have new interesting content to listen to while you're comuting or working outside. So I downloaded a bunch of podcasts (in MP3 format for now) and proceeded to install the Compaq-bundled RioPort software inside a Winblows session in VMWare. File transfers worked one third of the time, and drained the batteries pretty badly. The real problem, however, was the time wasted in other parts of the process. Waiting one full minute for RioPort to read the list of files transfered was too much. As with anything else in Winblows, it's not the apps that suck that much, but the OS just makes the user experience a real nightmare.

Being a happy resident of a non-DMCA encumbered country, I decided to reverse engineer the filesystem used to store the files to the flash cards. This way I can transfer files without using the PA-1 itself, which saves on the USB hassles and uses zero battery power. The first step was to dump a working flash image and examine it using a binary editor (bvi in this case).

It turns out that the filesystem was created by a company called Eiger M&C, which doesn't seem to be doing business anymore. I even tried emailing their contact listed on their website (last updated in 2002), but of course got no reply. To make a long story short, I ended up successfully reverse engineering most of the filesystem format, and used a bare bones version of it as the basis for a small Python script.

And so was born jdeigerfs v0.1 (3.4KB) :) It allows you to generate a mm.img file that contains a filesystem image that you can copy to any flash card. I use 32MB and 64MB MMC cards on my device, but your device may use other cards/sizes. All should work, up to 128MB per card. From what I can tell the format used for the FAT reserves 128KB for a 1 to 1024 mapping of the flash card, so anything over 128MB would actually cause the FAT to overwrite the first file in the card.

The script is barely useable. Actually it's a bit better than that, and in a works-for-me state. I decided to release it early so that if anyone else has any use for it I can get feedback at an early stage, although I don't plan on making any major improvements to it. Now I can finally test if the claim to support AAC is true (hard to believe for 2000 hardware). Later.

If you're a Dilbert fan...

...then surely you must be a geek, just like me. I can't say that I'm insanely into comic strips or anything like that, but a visit to comics.com made me want to have all Dilbert strips really bad.

While I'm not prepared to pony up for their paid service, I did subscribe to the free "Basic" service to see what it looks like. Not being able to wait to start my collection, I decided to fetch the 30 or so strips that are available in the Archive for free.

Obviously I'm not the first person to have such an urge. The guys at comics.com don't use a very common filename scheme for their content, presumably for the exact reason of making mass-fetching harder. A quick search on Google showed that every geek and his grandma has already written a script to fetch these strips. Instead of making my life easier, this simply proved that every minimally proud Dilbert fan must make his/her own script.

And so I proceeded to code my hack, jdilbert. Don't expect much--it's just a hack--but it does work. So much so that my collection now contains exactly 31 strips :)

Update: I rewrote this in ruby and it now handles more comics. Fetch jdstrips now :) I put it in a weekly cron job so that I never miss any.