Second p0st

There's not much I can say that everyone else hasn't already. GAE is out, and it appears to be good. I've downloaded the SDK and created a silly little test app (now just a landing page) to see what the sandbox allows, and it looks very promising.

This is very very different to something like EC2, which acts more or less as a very very fast-paced (instant deployment, pay by the hour) dedicated (unmanaged) server hosting provider. It's a shared hosting environment, like what you might get somewhere like GoDaddy, only seriously locked down -- you can only use Python and Google's proprietary datastore, there's next to no filesystem access, no threading, no long-running processes and no sockets. This makes it far less flexible than typical shared hosting environments, but locking you into this particular model makes it feasible for Google to 'auto-scale' your application, distributing it across many servers as required. I expect that the usual upper limits of scaling will still apply -- things that would take a really long time for something like MySQL to do will simply be disallowed within GAE -- but this may make things like database sharding less important.

The awesome thing about it, though, is that it appears to scale *down* very well, just like shared hosting. It's trivial to create and deploy an application here, and each additional app won't consume 40-70M of memory like a single Mongrel running a tiny Rails app would. Google handles monitoring, networking, load balancing and hopefully backups for you. So this will be *perfect* for hobby projects, while at the same time being quite adequate for many commercial projects. Projects wanting to do things like run Jabber gateways, background tasks and asynchronous stuff can do that on servers outside GAE and push/pull data as required over HTTP.

Oddly, I can't think of a pure website-type project that I'd like to build right now. But my next 'hack' type project will most likely be targetted to GAE.

(On second thoughts, taking a look back at my list of old hacks/projects, I see a couple, like the Radio Comment Monitor and Feed Normalizer, that I'm currently running as Python CGI scripts, that would be perfect for GAE. So maybe one day when I feel like taking some load off the server they're on, I'll port them over and let Google run them for me. If anyone's still using them, that is!)

I have a bunch of servers scattered around the place: a physical server in one colo, some boxes at home and a couple of Xen VMs. I've been meaning to tie all this together to give my personal stuff some measure of reliability for some time, but keep putting it off. I just slotted one of the pieces into place, though: set up OpenVPN point-to-point links on three servers. So now I have a secure and pretty persistent tunnel between each pair of servers that MySQL can use for replication, Nagios can use for monitoring, and so on.

Getting there ...

Sad to see the Christchurch office of Humanware, where I had my first job after college, (effectively) closing. I had a lot of fun working there, especially in the early days when there were only 5-6 of us in the development division, and later on when MyReader development really got going.

Ian Bicking does a stunning job of explaining what I've been pondering about for a while; especially relevant right now after the conversation started by DreamHost about Rails deployment on shared hosting.

His main point is that PHP has a CGI-like model: everything gets loaded on every request. Amazingly, this turns out to be fairly practical, because all the core stuff is written in C, so you can get pretty decent performance out of a system like this. mod_php keeps the core of the language and all C extensions in memory, so you don't have to keep reloading the interpreter like you do for 'real' CGI, but (at least without APC/eAccelerator installed) it doesn't keep any compiled PHP code or data around between requests.

To make web apps in any other language work, you usually end up needing to put something together that preloads the app and serves lots of requests -- mod_perl for Perl, some SCGI server for Python (Django, TurboGears), Mongrel for Ruby, Tomcat/Jetty/Resin for Java. This works well in the end but is limiting for the shared host, as the process that preloads stuff generally can't be shared between customers, or they'll step on each other's toes. Perl/Python/Ruby people love VPSes for this reason: you get a fairly isolated environment in which you can install and run what you like. Whereas in PHP's case, you can throw a bunch of apps together on one Apache install and generally it'll all work out OK.

This isn't to say that PHP is necessarily *better* than any other language, but it's certainly easier to deploy. I used to do a lot of web stuff in Python, and I'm having a lot of fun working with Rails these days, but for pretty much anything experimental, I use PHP, because I can just throw a few scripts up on my web server and forget about them, as opposed to having to set something up to keep a separate process alive on the server.

Another interesting though: ASP.NET seems to have the deployment/process model sorted out quite nicely as well. I'm not that interested in actually developing with it, but Microsoft has obviously put some thought into making it usable for shared hosts, by enabling a decent level of isolation between customers.

How could one do something like this for a system like Rails or Django? Maybe nginx with some magic to autostart Mongrels as required? If behind Apache, maybe something like mod_fastcgi but with a separate pool handler, so you could keep a number of per-customer pools of Ruby processes around? It feels pretty feasible but I imagine that the technology isn't there yet. The best thing I can think of off the top of my head is something like this:

- Perlbal in front, with a manager process that reconfigures it as appropriate when new Mongrels come online. This is feasible as Perlbal is reconfigurable without a restart.

- A UNIX user for each customer, with a very lightweight (read: doesn't use much memory) manager process that starts and stops Mongrels as required by load (i.e. if Perlbal starts to build up a backlog of requests for one customer's app, it starts more, and if the pool isn't seeing much activity, it kills them off) and sends messages to the Perlbal manager as appropriate.

This doesn't feel like something that would be *too* painful to prototype, if you do all the manager processes in Python or Ruby to start with.

This is probably old news to anyone who's run anything spammable on the web for long enough, but there are bots out there that don't do HTTP particularly well. I had this problem with the Topic Exchange years ago and it just recently hit a couple of BBM's sites. It's now fixed (for some definition of "fixed") in both cases with a Python proxy I wrote to buffer uploads and responses.

Curious as to whether other "standard solutions" out there handle this situation, and also so I could test out my proxy, I wrote a script that makes many HTTP connections to a site, writes POST headers with a large Content-Length, then feeds a byte into each of them every now and then so they don't time out, but never actually finishes the POST.

It took down my development box within seconds, driving it so badly out of memory that the Linux OOM killer kicked in and tore the thing to shreds. Reducing MaxClients in the Apache config improved the situation to the point where the machine would stay up, but the script would still make the site inaccessible (and fill up the accept queue so that nobody would be able to get in even if a request ever did get fully posted). Apache stops accepting requests when it runs out of children, so the script just started getting rejected connects after a while. Killing the script (and closing all the sockets) resulted in the site being instantly accessible again.

Trying it out with my proxy in front, I ran it for a few minutes but killed it after it established lots of HTTP connections without affecting the site. My proxy's not particularly clever so you could cause the machine to run out of swap by feeding it gigabytes of data, but it seems fine with lots of connections, at least.

Testing it on a Rails site running on a Mongrel cluster behind nginx gave similar results: no memory exhaustion (as no child processes are spawned with Mongrel/nginx) but an inaccessible site. Interestingly, the site didn't come back for a few minutes after killing the script; nginx or Mongrel took a while to process all the disconnects, or something. Correction: nginx handled the attack flawlessly. I didn't realise, but in the first test there was actually a different proxy in front of nginx, which wasn't so well behaved. So Rails hackers running nginx+mongrel_cluster can rest easy; your sites are probably not susceptible to this!

One bit of software I thought would be able to take a beating without batting an eyelash is Perlbal. And it did... to a point. It seems to use quite a lot of memory per connection, and after 500 connections (at which point it was using something like 3G of virtual memory) it printed "Out of memory!" and died. This is kind of scary, as Perlbal doesn't auto-respawn by itself (at least with the provided debian/perlbal.init script). So if you're running Perlbal you might want to run it under something like supervise or monit, or just in a simple shell script like this:

I've pinged the Perlbal dev list about this (I'm sure it would be possible to get it to stop accept()ing in low memory situations, or perhaps have it limit the number of simultaneous connections per IP -- or if this is a bug, to fix it.)

If anyone has a service running behind a different proxy/balancer that they wouldn't mind me running the script against, please drop me a line. Or for that matter if anyone has a web service that they're concerned is easily taken down, let me know... I'd be interested to see how resilient other proxies and web servers are.

Update: Thanks to Bruce Fitzsimons for letting me take down his 3gtelcotools.com Erlang server! New result: a fairly small web server (256M RAM, single CPU) running YAWS, accepted about a thousand sockets then died completely, as with Perlbal. The error was EMFILE: too many files open. The experiment will continue after Bruce does some tweaking :)

Update 2: Bruce upped his ulimit, and I have since been unable to bring his server down. So chalk this up as a success for YAWS, with appropriate configuration. (Details: it handles 3000 simultaneous connections quite happily. Appears to be forcibly closing old connections after getting > 1000 from one IP address - or this may be a bug in my code. More results later!)

Update 3, 2008-01-4: Summary of results so far:

Apache - blocked up immediately

Squid + Apache - blocked up immediately

nginx + Mongrel - happily accepts lots of connections without service interruption

Perlbal - happily accepts lots of connections, but dies eventually due to a bug?

YAWS (Erlang) - happily accepts lots of connections without service interruption

So the safest thing to do if you're running Apache is to throw nginx in front of it. Personally I'm waiting for a Perlbal patch, then I'm going to use it, as it has some cool features and looks very debuggable.

Happy New Year! I came across some alarming results last night when testing out a script that simulates badly behaved spambots and am curious as to whether popular load balancers take care of the situation. (I ended up solving the issue in this case by using a reverse proxy I wrote myself a few years ago, but I'm sure someone else has done the same...)

First on the list to try is Perlbal.

Latest Debian install instructions:

sudo apt-get install subversion libbsd-resource-perl libcompress-zlib-perl libnet-netmask-perl libio-stringy-perl libwww-perl libdanga-socket-perl libdbi-perl

svn co http://code.sixapart.com/svn/perlbal/tags/Perlbal-1.60/

cd Perlbal-1.60

perl Makefile.PL

sudo make install

Debian has a libio-aio-perl package, but it doesn't seem to work for me at the moment (it may only be in unstable) so I installed it from CPAN:

sudo perl -MCPAN -e shell

install IO::AIO

Now you can run Perlbal like this:

perlbal --config=config filename

Andrew 'Bunnie' Huang's series on making stuff in China is interesting. He mentions a very cool trick in his quality control entry about automated testing: logging all test data then sending it back to the home base for analysis.

I designed and built the test hardware for these motorized skylights and the autofocus board for these CCTV magnifiers - and this could have made life much easier. Neither of my testers even had network cards, and the second didn't involve a PC at all - it was all done with an Atmel AVR - as involving a PC the first time had made life difficult when transporting the whole unit around. Nowadays you can save so much time and gain huge amounts of debuggability (and loggability, as Bunnie demonstrates) by starting with a cheap laptop and a bunch of USB ADCs...

"How dare you treat us like second-hand citizens!"

A security vulnerability was recently discovered for PeopleAggregator and given the NIST ID CVE-2007-5631.

It's quite serious, allowing code injection, however is only a problem if you are running PA on a server with PHP's register_globals directive turned on. This directive is turned OFF on all Broadband Mechanics servers, so if you are hosting with us, you aren't in any danger. It's also off by default on most modern Linux distributions, so generally if you're running PHP5 you're probably OK.

I've seen shared hosts with it turned on, though, so it's quite possible that there are some exploitable PA installs out there. PeopleAggregator throws up a big red-lettered warning if you attempt to install with register_globals on, but will continue to run if you ignore the warning, and the exploit will still work if you upload but don't configure it, so if you hit the warning then go away but don't delete it from your server, you're still vulnerable.

So, if you're running PA on a host which has register_globals turned on (or you don't know that it's definitely turned off), please upgrade to v1.2pre6+1, the security fix for v1.2pre6. v1.2pre7, which also includes the fixes, plus some extra hardening, will be coming out soon, but please don't wait :)

As always, the code is available at update.peopleaggregator.org.

Vulnerable versions are v1.2pre6-release-53 and anything earlier, and the fixed version is v1.2pre6-release-55.

Postscript: No thanks to the discoverer of this vulnerability, who went ahead and posted it publicly without informing us.

Interesting to see that Joost is embedding Meebo. Can others do this? We've got AIM's chat widget embedded in PeopleAggregator (or at least we used to - not sure if went into the released code or not), and it would be nice to support other IM networks as well.