It seems this week that I should reconsider my vocation, as work with pipes (no, not just the usual tubes) has been featuring prominently. Firstly, I had to put in a drainage pipe under the garden bed which I’ve (slowly) been building for Rach. Hopefully it will save the paving against the side of our house from flooding every time it rains. That would be super cool…
The second incident, and one which some of you might have noticed, is that SwitchPipe has been acting like a bitch. You may have noticed my sites being down at random intervals lately? Yup, that’s SwitchPipe dying off for no good reason. And after I’d said so many nice things about it!!!
I was away when it first started, receiving a phone call from truelocal (of all folks!) letting me know that the Achernar Solutions site was down. Great. I then got an email from Peter – you know, the dude who wrote SwitchPipe – saying something along the lines of “Hey! Notice your site is down, hope it’s not because of SwitchPipe”. Gak!
Several other people on the SwitchPipe group had been having the same problem – SwitchPipe would sit around for ages, working perfectly, and then it would just die with no explanation.
I’ll spare you the more boring bits about finding and hunting down the problem. Suffice to say that it involved lots of extra logging calls, and hacking logging calls into the EventMachine library.
Just now, I managed to reproduce the problem, fix it, create a patch and fire it off. Hopefully, that’ll be the end of this freaking irritating bug.
What seemed to be causing it (and how I finally reproduced it) was making a “GET / HTTP/1.0” request to SwitchPipe. The key bit there is the request for ”/”. A fairly safe request, one would think. Well, not quite. I’m fairly certain that mod_proxy sends requests just like this off to back-end servers periodically, just to see if they’re alive. This is likely what was happening in my case.
See, SwitchPipe first tries to find which site to serve based upon the directory name in the request. ”/” has no directory. Oops! As such, that code returns false (as in, the FalseClass object), where it would normally return a string of the directory name. Not that_bad so far. However, later on, this string was turned into a symbol using to_sym(). Ok, BUT FalseClass doesn’t have a to_sym() method! So what happened? Exceptions were raised, Threads died, and eventually, SwitchPipe would come to a standstill.
This sort of stems from the fact that EventMachine creates 20 Thread objects when it’s running, but never manages or restarts them. It just assumes that they’ll always be alive until it says otherwise. So, when a Thread died thanks to the Exception being thrown, the pool just continued getting smaller and smaller until, finally, there are no more Threads to handle the requests.
Hopefully with this patch things will be back to normal with my sites, and they should stay up. Aside from this one small (but nasty) bug, SwitchPipe has been working like a charm. Ironically, the problem only surfaced after I made the post about it. Typical!
You may have seen or heard about SwitchPipe – a tool written by Peter Cooper, the dude who writes for RubyInside. Effectively, SwitchPipe is a proxy-esque Ruby app which listens on a port for requests and palms them off to Mongrel, thin, WEBrick, and other applications according to which site the request belongs to. The magic comes in that it’s able to dynamically start, stop and manage the number of processes you have running for your sites, helping you to use your memory and cycles better.
For example, for most production Rails sites you’ll want to have at least 2 mongrels running to ensure that it’s responsive even when someone is doing something that takes a while (eg. slow upload, long running query, etc.) However, when the server is idle, or the site is not busy, there’s a mongrel kicking about doing nothing but chewing up a big batch of memory. SwitchPipe allows you to say things like “for this site, keep 1 mongrel running at all times, and if it can’t handle it because it’s too busy, you’re allowed to start up to 3 other instances”. Then, after a period of inactivity (which you can define) your mongrel instance dies off, freeing up memory again. You also have the benefit of being able to “burst” a bunch of instances for a short period of time. For example, you might get a rush of users making slow requests – SwitchPipe can start up more instances of your application to keep your site responsive, and you can still set limits easily to ensure that one site can’t overload your server.
It’s a totally sexy tool – I’m using it exclusively on my new SliceHost slice. As the slice currently only has 256Mb of RAM, having more than a few mongrels running at any time (especially with some of my larger applications) causes the server to slip into a swapping nightmare. So, I’m using SwitchPipe to dynamically start and stop the instances as they’re required. This means that my quieter sites (this blog, for example) isn’t always chewing up memory, and that memory can be used by the busier applications.
There is a slight hit when starting a new instance, but it’s not terrible – usually a few seconds, depending on how fast the server can start the Mongrel instance. For example, on my server I say that I don’t want any mongrels always kicking around for this blog, but once they’re started they should live for a few minutes before dying off. This means that when a visitor navigates to my site, they wait a couple of seconds while Mongrel is started (unless it’s already started), and then so long as they make another request before the instance times out, it’ll all be super fast and sexy – reusing the already started instance. Of course, I also use Apache to serve my static content so as that images, etc. will always be super fast to be delivered.
Not convinced? Not just this blog is running through SwitchPipe – so is the Achernar Solutions site, and 5 or so other sites on this server. And they’ve all been running in this way for well over a month with absolutely no problems. I haven’t even bothered to install pal on the new slice (pal is the tool I wrote to make managing Rails sites easier for administrators) because SwitchPipe is simpler, and has more features. SwitchPipe manages all the ports for me automatically and manages starting and stopping all the processes for me. Why would I need pal when I just need to drop a YAML configuration file into SwitchPipe’s configuration directory and it will pick it up on it’s own and start handling the site?
So, if you’re hosting Rails apps, Merb apps, Django apps, and a whole host of others, SwitchPipe is going to make your life simpler. Check it out!
Just a super short post in case anyone hits the same problem I did with setting up extensions for RadiantCMS.
If running
db:migrate:extensions gives you an error, such as :
# rake production db:migrate:extensions
...
rake aborted!
undefined method `page' for #<Radiant::AdminUI:0x45b4a38>
(See full trace by running task with --trace)
The issue may be because the other extensions are loading before shards, as extensions are loaded in alphabetical order.
In my case, reorder was loading before shards, and hence the error.
Once I fixed up the load order in config/environment.rb, everything went great :
config.extensions = [ :shards, :all ]
Well, today’s the day. As of the 1st of February 2008, I am no longer part of Griffin Multimedia as it has been purchased by Paul and his friendly cohorts at in silico.
We originally set up GMM in March 2003 – just out of high school, just starting in University and really just learning. It’s been fun, but all good things must come to an end.
Over those 5 years, I’ve had the opportunity to work with a whole host of people and businesses – some that I’m proud to say that I’ve worked with, and others that I’m glad to no longer be associated with. It’s definitely been a learning experience – one which I’d be lying if I said that I loved every moment of, but one that has been fun, and one that I don’t regret (too much :P).
In those 5 years, I abandoned Uni, we’ve changed servers 3 times, Craig has finished his degree, Danny’s been to Japan twice, I’ve moved more times than I can count, finally ending up in New South Wales, and we helped a variety of businesses and community groups to develop a presence on the Internet.
So, I thought I’d put a shout out to all the folks that have been an inspiration to me over the 5 years, and those folks who have helped us along our way.
Firstly, definitely has to go to Paul O’Neill. I originally met Paul doing work experience at the small Perth ISP – v-App – which Paul was one of the principals of. Paul has a lot to be responsible for – including my first introduction to The Pragmatic Programmers, teaching me OO better, faster and more coherently than University, and for giving me support in developing from some teenager who loved writing code to a young dude being able to write code for a living. Through those 5 years we’ve both had ups and downs – both professionally, and personally, and I’m privileged to have him as a friend through all of that. Not to mention his awesome skills as a rubber duck and someone to throw zany code ideas around with.
I’m glad that Paul, Naomi and all the guys at in silico are taking over our clients – I can’t think of any other company that I would trust to treat them as well.
Next, Danny and Craig, my ex-partners. These guys deserve a huge thanks just for putting up with me, for being awesome dudes, and for reminding me that just because we’re in business doesn’t mean that everything has to be totally serious all the time. Again, we’ve had our ups and downs, but they’ve never let that get in the way of business, and I appreciate both of them for that. GMM did well to make it as far as we did, and it wouldn’t have been anywhere near as fun without them. The only regret? All of the cool apps that we wanted to build that we never got around to. They’re both super clever and talented dudes, and I’m sure they’ll do great in the future – whether that’s in IT, design, photography or music.
To all of you folks that have been good enough to throw work at us over the past 5 years – to Zanchey, who’s sent me enough referrals over the years to leave me eternally indebted to him; to Dirk, who (unbeknownst to him) has often reminded me that I love writing code, and if it’s not fun you’re not doing it right; to George, who seems to take a sadistic delight in getting a perfectly good design sorted, and then sending me a “revision” when I’ve just finished the first one; and to Andy, who will probably never read this, but hopefully is doing great with all of the amazing ideas that he has. To all of you, thank you for the friendship and trust that you’ve shown to us in the past. There’s nothing harder than referring a friend to someone, and without you we’d have closed up shop long ago.
Finally, to my family. Despite the fact that they’ve probably never quite understood just what I’m rabbiting on about (since I was 8, probably), and have looked on with fear and concern as I’ve worked myself into the ground over one project or another (or from caffeine abuse), they’ve always just been there – which is much more important than it sounds. Whenever I’ve had problems with business, need some advice, or just need someone to bitch and moan to, they’ve been there.
So, what’s happening now?
As I mentioned in a previous post, I’ve set up a new business – Achernar Solutions, where I’m looking at doing far more development and consulting work. Coding, thinking, building – the stuff that I love. I’m available to build web applications, for Ruby on Rails consulting, and remote Linux support.
I’m also staying on with in silico as their Systems Administrator – making sure that my baby (the server) stays running for long enough until they can easily maintain it themselves.
Aside from that, who knows? It’s the start of a brand new year, and I’m looking forward to seeing everything that it has to offer.
May the next 5 years be as successful as the past 5!
