Alexa cleaning up interfaces

Now that Alexa's thumbnails are no longer supported via AWS (breaking their own site for a while :P) it seems that they decided to also revamp their remaining services and do so cleaning up. This broke a script of mine used to fetch per-country stats. It's a simple change, the UrlInfo call with RankByCountry now returns a Code property in the XML instead of the old code property (note the case change). So if you're using HPricot (like I am), you'll use a line like:

(h/"aws:country[@Code='#{countryCode}']"/"aws:rank").innerHTML.sub(/\s*$/, '')

to filter out the resulting rank.

jdresolve on GitHub

I'm trying to force myself to switch to Git from Subversion for my current projects. The safest way to do it seems to be using an older project as a guinea pig :) So I just pushed jdresolve over to GitHub. I'll scavenge for the uncommitted patches and the unanswered emails regarding jdresolve and will commit to the Git repository soon (enough).

Google geocoding charset encoding currently broken

Update (2008-09-09): Google seems to have finally fixed this.

About 2 weeks ago I started seeing weird characters when geocoding addresses via Google using the YM4R gem. The addresses are outside the US and so contain plenty of accented characters that used to be properly encoded in UTF8. Although Google's XML claims to return UTF8, it currently doesn't, sending what looks like ISO-8859-1 encoded characters in some fields instead. This is more than likely a problem with their outsourced partners not having properly setup UTF8 environments and updating GIS information using local encodings.

I search for the issue and found if mentioned in ep's blog. He ended up seeing the same error message:

#<REXML::ParseException: Missing end tag for 'DependentLocalityName' (got "DependentLocality")

That'll usually result in a nice 500 error in your Rails app (if that's what you use) because it raises an exception.

His solution (forcing further charset translations to occur) worked quite well, except that I had to mention the origin charset in my case. So instead of calling to_utf8 plainly, I pass it the charset with to_utf8('iso-8859-1'). This is a very ugly hack in all so I hope that Google fixes the issue soon. I personally didn't report the bug 'cause I never got any feedback from any information I ever sent their way or any requests that I've made in the past.

Fixing mint tables after a 32bit to 64bit migration

I just migrated a web site from a regular 32bit processor (Intel(R) Xeon(TM) CPU 2.40GHz) to a 64bit slice (Dual Core AMD Opteron(tm) Processor 265). This made mint v2.14 (or actually the underlying MySQL database) start showing the visitor IPs as mostly 127.255.255.255. A little searching got me to here and here and here. I don't run PhpMyAdmin so here are the raw SQL command for you to run on your console:

alter table mint_visit modify ip_long int(10) unsigned;
alter table mint_visit modify referer_checksum int(10) unsigned;
alter table mint_visit modify domain_checksum int(10) unsigned;
alter table mint_visit modify resource_checksum int(10) unsigned;
alter table mint_visit modify session_checksum int(10) unsigned;

After running these a show columns from mint_visit should return the following:

+--------------------+---------------------+------+-----+---------+----------------+
| Field              | Type                | Null | Key | Default | Extra          |
+--------------------+---------------------+------+-----+---------+----------------+
| id                 | int(11) unsigned    | NO   | PRI | NULL    | auto_increment |
| dt                 | int(10) unsigned    | NO   | MUL | 0       |                |
| referer            | varchar(255)        | NO   |     | NULL    |                |
| referer_checksum   | int(10) unsigned    | YES  | MUL | NULL    |                |
| domain_checksum    | int(10) unsigned    | YES  | MUL | NULL    |                |
| referer_is_local   | tinyint(1)          | NO   | MUL | -1      |                |
| resource           | varchar(255)        | NO   |     | NULL    |                |
| resource_checksum  | int(10) unsigned    | YES  | MUL | NULL    |                |
| resource_title     | varchar(255)        | NO   |     | NULL    |                |
| search_terms       | varchar(255)        | NO   |     | NULL    |                |
| img_search_found   | tinyint(1) unsigned | NO   | MUL | 0       |                |
| browser_family     | varchar(255)        | NO   |     | NULL    |                |
| browser_version    | varchar(15)         | NO   |     | NULL    |                |
| platform           | varchar(255)        | NO   |     | NULL    |                |
| resolution         | varchar(13)         | NO   |     | NULL    |                |
| flash_version      | tinyint(2)          | NO   | MUL | NULL    |                |
| local_search_terms | varchar(255)        | NO   |     | NULL    |                |
| local_search_found | tinyint(1) unsigned | NO   | MUL | 0       |                |
| window_width       | smallint(5)         | NO   | MUL | -1      |                |
| window_height      | smallint(5)         | NO   | MUL | -1      |                |
| ip_long            | int(10) unsigned    | YES  | MUL | NULL    |                |
| session_checksum   | int(10) unsigned    | YES  | MUL | NULL    |                |
| visitor_name       | varchar(255)        | NO   |     | NULL    |                |
+--------------------+---------------------+------+-----+---------+----------------+
23 rows in set (0.00 sec)

The sooner you fix your database the less info you'll lose. Took me a couple of hours to realize this was happening...

Keeping VNC clipboards in sync

My typical "desktop" consists of two environments: the local machine (usually my notebook) and a VNC session to some fancier hardware. The logic behind this is that I can play around with my local apps and even reboot if I need to without losing any real work, which is usually done on the remote machine.

This works out quite well except for sharing information between environments. Quite often I'm browsing a web site inside the VNC session and want to open it up in my local environment, to have access to multimedia features or simply not to bloat my pristine remote session. Trouble is that X11 has a concept called the "cutbuffer" and another for "selections", which depend on which apps you're running. And, you've guessed it, VNC syncronizes the wrong one (namely the cutbuffer). So to simply copy a URL between sessions I had to select the URL using the mouse, paste it (middle-click) into an xterm window, select it again (!), switch to the other environment, paste it (middle-click again) into another xterm, then re-select it, and finally paste it into my other browser. This took care of syncing all of the related buffers manually.

Today I finally got fed up with it after doing this 3 times within 10 minutes. I found a project called autocutsel that does this automatically for me. Now all I have to do is keep autocutsel running in the background, and every time I copy something inside the local browser (using Ctrl-C) it is instantly available for use inside the VNC session (and vice-versa if I run autocutsel both in my local and my remote VNC environments).

Setting WEP keys in Kamikaze

I finally had a pressing need to OpenWRT-ize a Linksys WRT54GS v4 I bought over a year ago. The idea is to use the unit for field testing, so I want to be able to constantly change settings without rebooting or running complex scripts. Basically I want to SSH into the WRT and change settings at will as easily as possible.

The version of OpenWRT that I flashed was Kamikaze 7.09 and it has been working beautifully. Very fast boot times, organized filesystem and configuration structure, and plenty of RAM and Flash to spare. The radio in this box is a Broadcom, which I'm not very used to. I have a BCM4318 in my budget Acer Aspire 3002 laptop, and it works quite well, but I basically use it for connecting to a single access point using WPA encryption.

On the WRT box (which I fondly hostnamed 'wart') I wanted to try out WEP (yes, the old, insecure, useless WEP), mostly because a lot of access points in my area use it. Typically one could just use something like iwconfig wl0 enc <WEP key> to accomplish that. Sadly this doesn't work, although no error message is returned. Simply nothing happens, and the status of the interface ends up like:

wl0       IEEE 802.11-DS  ESSID:"MUSIK"
          Mode:Managed  Frequency:2.462 GHz  Access Point: 00:02:2D:0D:1B:39   
          Tx-Power:19 dBm   
          RTS thr:2347 B   Fragment thr:2346 B   
          Encryption key:<too big>
          Link Signal level:-42 dBm  Noise level:-96 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:516  Invalid misc:0   Missed beacon:0

Notice the Encryption key:<too big> line. It seems that the wireless tools don't play too well with the Broadcom's "advanced" settings. Even after I got this to work I still got the same output from iwconfig.

After some searching I found out that the utility that controls most of the internals of the Broadcom hardware is called wlc. Sure enough it was a "wepkey" option that should do the trick. The syntax seemed to be wlc wepkey <WEP key>, only that always returned:

Command 'set wepkey' failed: -1

I couldn't find a docs for wlc by searching online, so I fetched the full Kamikaze source tree to check its source code (hail to Open Source!). It turns out that the syntax expects you to inform the key slot to store to (1 through 4), and to use "=" to assign the "PRIMARY KEY", which is the slot that will be actually used. Thus, the command that worked for me was wlc wepkey =1,<WEP key>. Notice that I'm using an hex WEP key. If you use an ASCII WEP key, prefix it with "s:".

And that did it for me. Now with a simple call to wlc I can change WEP keys on the fly and get instant results.

Tinkering with Amazon's S3 and EC2

I was really excited about Amazon's EC2 (Elastic Compute Cloud) when it first became available sometime last year. Unfortunately it was (and still is) a "limited beta" service (yeah I know) so all I could do was sign up for it and hope that my day would come.

It did :) (big surprise!).

Anyway I've been playing around with my SliceHost VPS (which they of course call a "slice") for a week and was hoping to have access to a similar service at low cost to compare, so the timing was just about perfect. Note that I also had to wait for my SliceHost slice, but it was a lot quicker (about 8 weeks). I have to get a hold of the specifics for both services to compare price and performance, but my main concerns right now are ease of deployment and flexibility, so that'll have to wait.

My initial guess is that both services will behave pretty much the same after they're setup. You start with a basic OS to play with anyway. I probably won't trust anyone else's image and just roll out my own customizations.

Back to the Amazon services, you can't just use EC2 out of the box. First you need to sign up for S3 (Simple Storage Service), which is pretty awesome on itself but I hadn't used yet (mostly for lack of real need). So I did that (no wait for that one) and decided to explore it a bit before moving on with my EC2 experiments.

At first a nice command-line client to S3 would be really nice, to get a general feel for the service without having to dive into the API. I did a quick search and started using s3cmd. It's written in Python, which I like a lot, and has a very clean feel to it. The only problem was that it didn't use my outgoing HTTP proxy (Squid) like other Python apps usually do.

If you have a http_proxy environment variable, the Python libs will generally detect that and pass your requests onto the proxy. After inspecting s3cmd's code, I realized that due to a multi-step process in generating the webservice requests, that functionality was being sidestepped. This required some minor tweaking to add proxy support to s3cmd's config file and then use that to properly craft the requests for proxy and non-proxy situations. I created a patch and sent it to the maintainer. Of course you can just download the patch and apply it yourself--it's against version 0.9.3 of s3cmd.

Other interesting projects related to S3 are s3sync.rb (a Ruby backup script similar to rsync) and Backup Manager (a general backup tool that I've been told supports S3 now), but I haven't tested them yet.

Having played with S3 I decided to complete my first EC2 setup. After jumping through all the hoops in the Getting Started guide, I chose Paul Dowman's image for my first boot. It worked like a charm, and in less than 1 minute (!!) I was able to login as root to my newly created instance, complete with Rails, MySQL and an up-to-date Ubuntu Feisty install.

BTW, the EC2 tools (in Java) also don't honor the http_proxy environment variable (sigh). Here's the environment variable you have to set to make it all work, assuming your proxy is not password protected:

export EC2_JVM_ARGS="-DproxySet=true -DproxyHost=<host> 
  -DproxyPort=<port> -Dhttps.proxySet=true 
  -Dhttps.proxyHost=<host> -Dhttps.proxyPort=<port>"

Having access to EC2 opens up a lot of possibilities in deployment and scaling. From what I've been reading, the main complaint about EC2 is the lack for persistent storage. I don't see that as too much of a problem because you do have storage as long as the instance is running, and even if it crashes (doubtful) from my understanding the data is intact. I rebooted my instance and it came back unscratched. Only if you forcibly shutdown ("shutdown now -h") do you destroy the image and lose your data. Plus, you can always backup to S3, even the whole image block by block if you'd like. Another common complaint is about the lack of static IPs, but that can also be worked around with dynamic DNS services.

After successfully booting my first EC2 instance I now plan to work on some deprec recipes to deploy rails apps on either SliceHost or EC2, transparently. My target system will be an Ubuntu Feisty base, with PostgreSQL 8.2, Apache 2.2 and Mongrel.