Griffin::blog()

Maybe I Should Be a Plumber?

Fri, 04 Apr 2008 04:48:10 GMT

It seems this week that I should reconsider my vocation, as work with pipes (no, not just the usual tubes) has been featuring prominently. Firstly, I had to put in a drainage pipe under the garden bed which I’ve (slowly) been building for Rach. Hopefully it will save the paving against the side of our house from flooding every time it rains. That would be super cool…

The second incident, and one which some of you might have noticed, is that SwitchPipe has been acting like a bitch. You may have noticed my sites being down at random intervals lately? Yup, that’s SwitchPipe dying off for no good reason. And after I’d said so many nice things about it!!!

I was away when it first started, receiving a phone call from truelocal (of all folks!) letting me know that the Achernar Solutions site was down. Great. I then got an email from Peter – you know, the dude who wrote SwitchPipe – saying something along the lines of “Hey! Notice your site is down, hope it’s not because of SwitchPipe”. Gak!

Several other people on the SwitchPipe group had been having the same problem – SwitchPipe would sit around for ages, working perfectly, and then it would just die with no explanation.

I’ll spare you the more boring bits about finding and hunting down the problem. Suffice to say that it involved lots of extra logging calls, and hacking logging calls into the EventMachine library.

Just now, I managed to reproduce the problem, fix it, create a patch and fire it off. Hopefully, that’ll be the end of this freaking irritating bug.

What seemed to be causing it (and how I finally reproduced it) was making a “GET / HTTP/1.0” request to SwitchPipe. The key bit there is the request for ”/”. A fairly safe request, one would think. Well, not quite. I’m fairly certain that mod_proxy sends requests just like this off to back-end servers periodically, just to see if they’re alive. This is likely what was happening in my case.

See, SwitchPipe first tries to find which site to serve based upon the directory name in the request. ”/” has no directory. Oops! As such, that code returns false (as in, the FalseClass object), where it would normally return a string of the directory name. Not that_bad so far. However, later on, this string was turned into a symbol using to_sym(). Ok, BUT FalseClass doesn’t have a to_sym() method! So what happened? Exceptions were raised, Threads died, and eventually, SwitchPipe would come to a standstill.

This sort of stems from the fact that EventMachine creates 20 Thread objects when it’s running, but never manages or restarts them. It just assumes that they’ll always be alive until it says otherwise. So, when a Thread died thanks to the Exception being thrown, the pool just continued getting smaller and smaller until, finally, there are no more Threads to handle the requests.

Hopefully with this patch things will be back to normal with my sites, and they should stay up. Aside from this one small (but nasty) bug, SwitchPipe has been working like a charm. Ironically, the problem only surfaced after I made the post about it. Typical!

SwitchPipe for Fun and Profit

Thu, 06 Mar 2008 00:59:44 GMT

You may have seen or heard about SwitchPipe – a tool written by Peter Cooper, the dude who writes for RubyInside. Effectively, SwitchPipe is a proxy-esque Ruby app which listens on a port for requests and palms them off to Mongrel, thin, WEBrick, and other applications according to which site the request belongs to. The magic comes in that it’s able to dynamically start, stop and manage the number of processes you have running for your sites, helping you to use your memory and cycles better.

For example, for most production Rails sites you’ll want to have at least 2 mongrels running to ensure that it’s responsive even when someone is doing something that takes a while (eg. slow upload, long running query, etc.) However, when the server is idle, or the site is not busy, there’s a mongrel kicking about doing nothing but chewing up a big batch of memory. SwitchPipe allows you to say things like “for this site, keep 1 mongrel running at all times, and if it can’t handle it because it’s too busy, you’re allowed to start up to 3 other instances”. Then, after a period of inactivity (which you can define) your mongrel instance dies off, freeing up memory again. You also have the benefit of being able to “burst” a bunch of instances for a short period of time. For example, you might get a rush of users making slow requests – SwitchPipe can start up more instances of your application to keep your site responsive, and you can still set limits easily to ensure that one site can’t overload your server.

It’s a totally sexy tool – I’m using it exclusively on my new SliceHost slice. As the slice currently only has 256Mb of RAM, having more than a few mongrels running at any time (especially with some of my larger applications) causes the server to slip into a swapping nightmare. So, I’m using SwitchPipe to dynamically start and stop the instances as they’re required. This means that my quieter sites (this blog, for example) isn’t always chewing up memory, and that memory can be used by the busier applications.

There is a slight hit when starting a new instance, but it’s not terrible – usually a few seconds, depending on how fast the server can start the Mongrel instance. For example, on my server I say that I don’t want any mongrels always kicking around for this blog, but once they’re started they should live for a few minutes before dying off. This means that when a visitor navigates to my site, they wait a couple of seconds while Mongrel is started (unless it’s already started), and then so long as they make another request before the instance times out, it’ll all be super fast and sexy – reusing the already started instance. Of course, I also use Apache to serve my static content so as that images, etc. will always be super fast to be delivered.

Not convinced? Not just this blog is running through SwitchPipe – so is the Achernar Solutions site, and 5 or so other sites on this server. And they’ve all been running in this way for well over a month with absolutely no problems. I haven’t even bothered to install pal on the new slice (pal is the tool I wrote to make managing Rails sites easier for administrators) because SwitchPipe is simpler, and has more features. SwitchPipe manages all the ports for me automatically and manages starting and stopping all the processes for me. Why would I need pal when I just need to drop a YAML configuration file into SwitchPipe’s configuration directory and it will pick it up on it’s own and start handling the site?

So, if you’re hosting Rails apps, Merb apps, Django apps, and a whole host of others, SwitchPipe is going to make your life simpler. Check it out!

RadiantCMS Extensions Load Order GOTCHA

Mon, 04 Feb 2008 14:39:46 GMT

Just a super short post in case anyone hits the same problem I did with setting up extensions for RadiantCMS.

If running db:migrate:extensions gives you an error, such as :

# rake production db:migrate:extensions
...
rake aborted!
undefined method `page' for #<Radiant::AdminUI:0x45b4a38>

(See full trace by running task with --trace)

The issue may be because the other extensions are loading before shards, as extensions are loaded in alphabetical order.

In my case, reorder was loading before shards, and hence the error.

Once I fixed up the load order in config/environment.rb, everything went great :

config.extensions = [ :shards, :all ]

Every Era Has an End

Fri, 01 Feb 2008 18:16:09 GMT

Well, today’s the day. As of the 1st of February 2008, I am no longer part of Griffin Multimedia as it has been purchased by Paul and his friendly cohorts at in silico.

We originally set up GMM in March 2003 – just out of high school, just starting in University and really just learning. It’s been fun, but all good things must come to an end.

Over those 5 years, I’ve had the opportunity to work with a whole host of people and businesses – some that I’m proud to say that I’ve worked with, and others that I’m glad to no longer be associated with. It’s definitely been a learning experience – one which I’d be lying if I said that I loved every moment of, but one that has been fun, and one that I don’t regret (too much :P).

In those 5 years, I abandoned Uni, we’ve changed servers 3 times, Craig has finished his degree, Danny’s been to Japan twice, I’ve moved more times than I can count, finally ending up in New South Wales, and we helped a variety of businesses and community groups to develop a presence on the Internet.

So, I thought I’d put a shout out to all the folks that have been an inspiration to me over the 5 years, and those folks who have helped us along our way.

Firstly, definitely has to go to Paul O’Neill. I originally met Paul doing work experience at the small Perth ISP – v-App – which Paul was one of the principals of. Paul has a lot to be responsible for – including my first introduction to The Pragmatic Programmers, teaching me OO better, faster and more coherently than University, and for giving me support in developing from some teenager who loved writing code to a young dude being able to write code for a living. Through those 5 years we’ve both had ups and downs – both professionally, and personally, and I’m privileged to have him as a friend through all of that. Not to mention his awesome skills as a rubber duck and someone to throw zany code ideas around with.

I’m glad that Paul, Naomi and all the guys at in silico are taking over our clients – I can’t think of any other company that I would trust to treat them as well.

Next, Danny and Craig, my ex-partners. These guys deserve a huge thanks just for putting up with me, for being awesome dudes, and for reminding me that just because we’re in business doesn’t mean that everything has to be totally serious all the time. Again, we’ve had our ups and downs, but they’ve never let that get in the way of business, and I appreciate both of them for that. GMM did well to make it as far as we did, and it wouldn’t have been anywhere near as fun without them. The only regret? All of the cool apps that we wanted to build that we never got around to. They’re both super clever and talented dudes, and I’m sure they’ll do great in the future – whether that’s in IT, design, photography or music.

To all of you folks that have been good enough to throw work at us over the past 5 years – to Zanchey, who’s sent me enough referrals over the years to leave me eternally indebted to him; to Dirk, who (unbeknownst to him) has often reminded me that I love writing code, and if it’s not fun you’re not doing it right; to George, who seems to take a sadistic delight in getting a perfectly good design sorted, and then sending me a “revision” when I’ve just finished the first one; and to Andy, who will probably never read this, but hopefully is doing great with all of the amazing ideas that he has. To all of you, thank you for the friendship and trust that you’ve shown to us in the past. There’s nothing harder than referring a friend to someone, and without you we’d have closed up shop long ago.

Finally, to my family. Despite the fact that they’ve probably never quite understood just what I’m rabbiting on about (since I was 8, probably), and have looked on with fear and concern as I’ve worked myself into the ground over one project or another (or from caffeine abuse), they’ve always just been there – which is much more important than it sounds. Whenever I’ve had problems with business, need some advice, or just need someone to bitch and moan to, they’ve been there.

So, what’s happening now?

As I mentioned in a previous post, I’ve set up a new business – Achernar Solutions, where I’m looking at doing far more development and consulting work. Coding, thinking, building – the stuff that I love. I’m available to build web applications, for Ruby on Rails consulting, and remote Linux support.

I’m also staying on with in silico as their Systems Administrator – making sure that my baby (the server) stays running for long enough until they can easily maintain it themselves.

Aside from that, who knows? It’s the start of a brand new year, and I’m looking forward to seeing everything that it has to offer.

May the next 5 years be as successful as the past 5!

Scaremongering About Australia's Proposed Filters

Wed, 09 Jan 2008 13:00:47 GMT

In addition to my last post regarding Australia’s proposed mandatory Internet Filter, this is a copy of a comment that I have just posted in response to Phillip Malone’s post, Second Life, Skype, Even Google Won’t be blocked by The Australian Government?, which was itself a response to Duncan Riley’s recent post, Will Second Life, Skype, Even Google Be Blocked By The Australian Government?

Whilst I agree that some of what Duncan points out may be slight exaggerations, I believe it highlights the problem with the policy as it has been presented to the public to date – we don’t know what is going to be blocked under the new filter, and therefore Duncan’s suggestions could, in fact, be possible at this stage. The comment I left for Molly is as follows:

I think the main point that many folks are trying to make with these claims (which I agree are often exaggerated) is that, from what the public has been told so far, it’s not possible to say what will and won’t be filtered.

I’ve not seen anywhere refer to a black and white classification system of what will and won’t be blocked under this policy. Regardless of any of the (considerable) technical problems, we’re effectively at the whim of whoever is running the programme once it is in place. Is there a government document setting out what is and isn’t acceptable content?

From what has been publicly released thus far – a few phrases with non-specific terms such as “objectionable content” – the only thing that we can be certain is to be on the list (and quite rightly so) is child pornography. However the question is still wide open as to what else could be on that list. The fact that the classification system is not public and transparent is a huge problem.

Australia's Proposed World Class Broadband?

Wed, 02 Jan 2008 21:55:54 GMT

As Dave Hughes might say, “I’m angry tonight…”. I’m angry because Australia’s Internet is on a nasty road – the road to censorship. If you look to your left, you’ll see ISPs forced to implement a technology doomed to fail. To your right, you’ll see the Australian public getting shafted. Just up ahead, we’ll see parents taking even less responsibility for what their kids do on the Internet.

It’s not getting that much airtime outside of the IT community currently, but the ALP is pushing for it to be mandatory for all ISPs to take part in a system to filter out nasty content on the Internet. Sure, this will have it’s perks, but at what costs?

In this post, I’m going to take you on a quick tour of the pros, the cons, and probably some knee-jerk conspiracy mongering thanks to recently reading 1984. Should be fun!

Background from other folks

If you want a bit of background from other folks too, check out these links:

Some History

The ALP made a lot of noise during the recent election about giving Australia World Class Broadband – not before time! In Australia we’ve been plagued with high broadband costs, mediocre bandwidth and quotas (sorry, couldn’t resist!), and even worse infrastructure.

To give you some idea, it’s not uncommon for Australian ISPs to charge anywhere from $10/Gb to $90/Gb (That’s extreme though) for network traffic. Our particular host charges $35 down to $15 per Gb, and that’s for a co-located server, which one would expect to be cheaper thanks to the bulk of data.

Contrast this with other nations, such as the US where it’s not uncommon to see $1/Gb excess charges, or even hosting providers offering 1TB of bandwidth for $20/month.

Sure, Australia isn’t the worst, but for an advanced, affluent country, we’re paying a huge amount for Internet access. I don’t know exactly why this is the case, but I suspect a great deal of it owes to the fact that we don’t have that many options when it comes to physical connections in and out of the country – undersea cable is expensive, and satellite communications only has a certain amount of bandwidth that it can handle. As such, it’s going to take a heap of capital to get more bandwidth into Australia, and then far more infrastructure to get it delivered everywhere.

With regards to web filtering, it is not an uncommon thing to do. Home users have some solutions (albeit usually fairly unreliable), but it’s very common for schools, Internet cafes, and other businesses to do some filtering. For example, the WA Department of Education and Training uses a very coarsely grained access filter to block out common and obvious websites that are not suitable for minors. Every school I’ve ever worked at or been involved with has run their own filtering solution on their own network in order to keep tabs, and filter out what students are looking at.

A few years ago, China also announced that they would start filtering web traffic, and this received a fair amount of condemnation from those who heard about it in most Western societies. That may have been mainly because they were filtering content that was anti-governmental, and news sources external to the strictly controlled Chinese media.

Some other countries, including the UK also currently filter their Internet traffic, however I am not 100% certain to what degree, or of the specifics.

PROs

First off, the concept of filtering the Internet has a few lofty goals – we can protect children from unsuitable content such a pornography, we can protect society from websites that might incite hate crimes or violent behaviour, and it gives us another way to stop and track paedophiles and those who are involved with child pornography.

These are all important things – I don’t believe that anyone here would condone child pornography, or say that hatemongering, or exposing children to pornography or graphic violence are things that we should be defending. As a society we already do a fair deal to protect against these types of things – movies and TV shows have ratings (don’t get me started on the lack of an R-rating for games in Australia, though!); hate and violence is condemned and we have laws and legislation to discourage it; we have task forces of police and civil servants who work locally and internationally to hunt and stop child pornography and paedophilia.

However.

CONs

The concept of national level, opt-out filtering scares me. It does this for a few reasons :

it’s not going to work as well as it’s proponents might want to believe;
it’s going to be horribly inefficient, and will raise costs to push a World Class Broadband for Australia further out of reach;
managing and maintaining the solution is going to have additional costs across the board that taxpayers will have to fork out for;
it opens a whole new can of worms in terms of accountability, impartiality and corruption;
it opens a whole big can of worms in terms of monitoring, privacy and the rights of the individual;
there are far better solutions, that distribute the load and have beneficial side-effects;

Simply put, I don’t think it’s going to work as well as the proponents think it will. I primarily put that down to the fact that those espousing these sorts of solutions neither fully understand the technical details on the implementation, nor do they understand what it is like to be sitting behind one of these sorts of solutions.

Anyone who has ever been at one of the aforementioned schools which uses content filtering knows that it’s not an exact science – there are a few ways to manage these sorts of content filtering proxies, and none of them are perfect. As someone that’s both run a system of this nature, and someone who’s been trying to get around them, they’re unreliable and require a large amount of maintenance. Furthermore, updating the filter needs to be done often, and needs to be done very rapidly after unsuitable content is found.

Let’s look at a few of the technical options to run a filtering system such as this.

You can use keywords – for example, a program can look through every website as it is loaded, and if it contains certain words then it will be blocked. I’m sure you can guess most of the four-letter words that schools frequently block, and if you let the dirty side of your mind wander a little, you’ll probably work out a few more. Keyword filtering sucks. We’ve seen this in other problem spaces too – just look at SPAM. As soon as solutions emerged that looked for obvious SPAM words such as “viagra”, “penis”, etc. SPAMmers quickly took to obfuscating them. As such, we now get emails offering us “v1agr4 3n1arg3 y0ur P3N!S” and crap like that. We’ve used these kinds of solutions at schools, and I can guarantee that they’re not perfect. Is “breast” a keyword for pornography, or for Breast Cancer Awareness?

In addition to a keyword solution being unreliable, the infrastructure required to search for these keywords across all Australia HTTP traffic would be immense and expensive. Keyword filtering is not a good idea from a technical standpoint.

The other typical solution is to see what everyone is viewing and review those pages in order to find new “bad stuff” to add to your list of blocked sites. I did this for about 2 years while I was in high school, where each day I would check the traffic logs on the proxy server and have a quick look for inappropriate content. To do this reliably, every page that is viewed must be reviewed, and checked for any content deemed “inappropriate”, and therefore blocked. Not withstanding that the process of identifying these “bad pages” as they are navigated to will be a bottleneck, the manpower required to review these pages would need to be immense in order to be anywhere near useful. Certainly, much of the traffic would be duplicated (and so might not need to be checked again) however it would not be possible to effectively review content such as snippets of information returned via AJAX, pages requiring authentication or cookies, or the result of POSTed content.

Reviewing sites has the computing overhead of using keywords, plus is human-intensive, is prone to errors, and won’t even work for a bunch of the use cases. As such, that’s not a suitable option.

Finally, you could just block pages that you get complaints about. I’m not sure if I need to point out the stupidity of this idea. The likelihood of someone complaining about inappropriate content is minimal, as they actually have to visit it in order to be able to complain. Consider that goatse.cx, one of the best known shock sites, was available for over 4 years before anyone actually complained about it – I consider this to be a very prominent example, as goatse was (and still is) remarkably well known for it’s offensive nature. Yet, it still took several years before someone found it offensive enough to report it.

So let’s jump forward to an Australia where this kind of filtering has been made mandatory. Who is going to pay for the core infrastructure to maintain the lists, employ the maintainers, etcetera? It will be us, the taxpayers, who will be paying for a system that will likely never provide a suitable level of protection. Additionally, ISPs will need to expand or modify their infrastructure to handle the new technical requirements. If we assume that the Government mandates that ISPs must filter the traffic themselves (based upon filtering lists, or something similar), the cost to process that traffic will be passed on directly to customers. I can’t see iiNet, Telstra, or any ISP of any size for that matter, saying “Hey, it’s OK guys – we’ll spend a this wad of cash on new hardware, new personnel to maintain it, legal advice on whether or not we comply, etc. We won’t charge you any more!”.

Furthermore, this is an additional roadblock for new ISPs, who will not only have to compete with the big players, but have to comply with the legislation, and find out whether they comply.

However, the concept of ISPs managing the filtering themselves is a ludicrous idea – what is there to stop whoever is implementing the filters from ensuring that there is still a workaround for private or community use? Geeks are, stereotypically, against censorship and therefore I consider it inevitable that such a workaround will be made in at least one location.

Oh, and I’m also going to gloss over the fact that systems such as Tor, BitTorrent, IRC or any number of different, non-HTTP protocols could be used to disseminate this “inappropriate content”.

Ok, so let’s jump forward some more to an Australia where the filtering has been made mandatory, and it’s being implemented somewhere, whether that be at a core Governmental location, or at the ISP level.

Who the hell is going to decide what is and what isn’t appropriate? Certainly, some things are black and white – child pornography is well out, fuzzy bunnies are in! But what about the grey areas? What about tasteful (artistic) nudity? What about classic art or religious texts that suggest or promote violence and hate? What about graphic cruelty to animals such as is frequently featured on peta.org and other environmentalist sites? What about goatse, tubgirl, and all those shock sites?

You might think that the mention of goatse and tubgirl wasn’t warranted, but I believe these are strong examples of content that is very offensive, but only in certain circles. Does that make them “nationally” inappropriate? Who decides so? I’ve been exposed to young Mr. Goatse so many times that he doesn’t bother me any more (one of my 18th birthday cards included him – what a surprise!). However, I’d warrant that my grandparents (or most other people) would be less than pleased to find it in a Christmas card. It is how the legislation deals with these grey areas that best indicates the future direction of the legislation. Personally, I don’t want to see pictures of a raccoon being beaten to death with a cudgel, but people who visit peta.org are likely very passionate about such things! Can I ask for it to be blocked as inappropriate?

Will adding something to the filter need the approval of a judge or court? What about a vote of parliament? A public vote? A council of religious leaders? Before even considering building a list, we need to decide on who should maintain such a thing, and what hard and fast rules we use to place content into that list.

Why is this important? Because it’s only a small step from blocking sites that promote anti-social behaviour, to sites that promote fringe political beliefs, to sites of minor political parties or lobby groups. I sure don’t support victimless crimes such as incest between adults however I do believe that you have a right to put forward that viewpoint. I strongly agree with Duncan when he said that The problem with censorship is once it starts it rarely stops.

I’m not saying that we shouldn’t filter anything – however I do believe very strongly that the filtering has to be entirely transparent and 100% accountable.

Furthermore, the ALP is proposing an opt-out system, and it’s not a big stretch to consider consumers could possibly run the risk of being subject to additional monitoring if they were to ask to opt-out. It’s especially popular for proponents of these sorts of measures to say “If you don’t support filtering, you must be viewing child pornography, as that’s all that’s going to be filtered”. Despite that this kind of Straw Man attack is a logical fallacy, that is a fundamentally flawed perspective to take. I’m certain that there’s something about presumption of innocence in this country, and it’s illogical arguments such as these which open the doors to emotive, knee-jerk reactions and allows for people to be stripped of their rights in the name of “the greater good”.

Don’t get me wrong – I’m not under any illusion that our Internet traffic is not currently monitored and tracked for a range of purposes. Call me a raving conspiracy theorist nutjob, it’s happening. I know enough about the technology to know it’s within possibilities, and I’ve met enough people who have told me first hand of their involvement with it, or told me of their discussions with people who have been involved with it. Whether it’s by ECHELON or something else, we’re being monitored. I’m at ease with it – as someone who’s grown up on the Internet, and has more stuff on the Internet than most of my friends actually know about me, I’m under no illusions that this plan by the ALP would be the first effort to keep track of what we’re doing and reading on the Internet. I’ve got absolutely nothing to hide, but that doesn’t mean that I’m not entitled to a modicum of privacy!

That said, I don’t believe that we need any more monitoring, especially not monitoring that is being pushed using emotive knee-jerking under the misconception that it will actually make any difference in the fight against child pornography, hate crimes and the sliding of society down the proverbial toilet.

Earlier in this essay, I referred to a better solution, one that is distributed, low cost, and has beneficial side-effects.

It’s called parenting.

I’m not a parent. I don’t have kids. I am, however, a Scout Leader, and I know what it’s like to try and manage what children are up to. They’re smart, sneaky, cunning little beasts.

However, that is no excuse for you to not know what your children are up to. You’re older, you’re meant to be smarter too. If you child is on the Internet, and you don’t know what they’re up to then quite honestly, it’s your own fault when they stumble upon something you don’t want them to. I’m not saying that parents need to sit next to their children at the computer (where’s the fun in that!) but all too many parents buy Johnny and Suzie a computer, put it in a room out of the way (so as that incessant beeping can’t be annoying) and let them go wild. Put the computer in a public place, check what they’re up to. Hell, take an interest in what they’re up to!

Parents will get a bit offended at that. “But when they’re at school! When they’re at a friends place! But! But! But!”.

No. No buts!

Schools already have an interest in protecting your child from this kind of content. There isn’t a school I know of, or an education department that has any excuse for not filtering out the most run of the mill garbage from their students. It’s not easy, and it’s not perfect, but with a combination of teacher supervision and these light weight technical solutions, it does the job. If the school your children go to doesn’t have something like this, then you should be asking why.

At a friends house, you might want to expect that your child is being given the same level of supervision as you might provide at home. Do you trust the parents of your children’s friends to stop them playing with knives, matches or guns? Of course you do. The Internet is a tool, exactly the same. You should be watching your children’s friends just as you would your own.

No level of technical measures will be anywhere near as effective as having someone who knows better keeping at eye on your children. Oh, and you get the benefit of actually having a relationship with your offspring. Pfft! How would that work again!?

A Better Solution?

Ok, I’ve been pretty negative all this post, so let’s look for another alternative.

Get the Government involved in a distributed, open-source-esque initiative for blacklists. We’re talking entirely transparent, entirely open blacklists that proxies (in particular, schools) can use in order to limit children’s access to this “inappropriate” content. But make it entirely optional! Give ISPs the choice of using the solution in an opt-in arrangement. Don’t make it some sort of Big-Brother solution whereby the government has control over what is added, and what is removed. Allow folks to subscribe to the protection they want – pornography, hate and violence, gambling, etc. Produce a resource that can be used by multiple technologies, extended, and most importantly – verified.

As much as I hate the anti-spam blacklists, a model similar to that would perhaps work well – the Australian Government produces a collection of blacklists for “bad” content of different grades, and other groups can take these lists and build upon them, remove incorrectly marked content, and things like that. Give the power back to the consumer, not some power-hungry bureaucrat working towards their own end!

Still not convinced?

Suit yourself. If all of this hasn’t swayed you, then I doubt anything can. I’m just hoping that enough technical people point out the glaring stupidity of this kind of solution before more of our taxpayers dollars are wasted by yet another inept government.

Hey! Welcome to 2008! Man, this is going to be fun!

Musings on Pure Democracy After the Election

Tue, 27 Nov 2007 08:19:46 GMT

Well, with the election over, we’ve finally seen an end to those nauseating election campaign advertisements. Yaay! However, the election has got me thinking about government, power and that whole Democracy thing.

In Australia, we enjoy a Representative Democracy, one where a group of representatives, elected by the masses act for the good of the nation. Supposedly. Ask most people and they’ll swear that politicians don’t do what they would like, and they’re all bastards. This is, of course, somewhat odd, considering that the politicians are there to represent them.

A Representative Democracy has a variety of problems – politicians can misrepresent their constituents, they can be self-serving in their own interests, they can be ill-informed and make incorrect decisions. An issue that I had not realised until today was the fact that, in a Representative Democracy, an election can easily become a popularity contest – not a content of the actual merits of each side of Government.

Regardless of political standpoint, I believe that the recent Australian election had no alternative outcome. Why? Because I don’t believe that it it was an election between two differing ideas of Government, or policies, but rather it was an election about the popularity of Rudd vs Howard (or Rudd vs Costello, whichever way you want to look). I posit that any politician who has been in power for 11 years has had considerable time to attract a “critical mass” of public opponents (it doesn’t take much to piss people off), regardless of policies or performance. Whether Howard’s policies were good or bad, I believe that this critical mass was going to take the Liberal party out of power regardless of anything that may have been done by either party.

This isn’t a discussion of the merits of each side of government’s policies (after all, it’s been said that the election promises were very similar anyway) but rather on whether the way the election was carried out sits OK with us all. (As an aside, no, I’m not accusing either party of being as bad as Hitler, regardless of the Wikipedia content :P) In my eyes, it turned into an election to decide which politician’s personality is better, and which we’d rather deal with for a while. It seemed to me that key parts of the Labor campaign hinged not on policy, but on personalizing a vote for the Liberal Party as a vote for the unpopular Costello. Other parties did the same (although perhaps not quite as blatantly as the ALP did, in my opinion) by using the Union movement as a scare tactic against the ALP. This is

We effectively had an election as farcical as watching a group of 8 year olds pick their team for sport, or one of those crazy blind-date shows where the woman asks the guys a few questions to decide which one she will have and hold.

I’m not particularly phased by the change of government, however I do believe very strongly that choosing a government based upon the personality of the figureheads is no way to run a country.

However, in the title I alluded to Pure Democracy.

A differing system of Democracy, one that is rarely used, is that of Direct Democracy (Pure Democracy). In this system, the people themselves are given the power to vote on laws. It tends to be championed somewhat as a cure for self-interested politicians, and giving power back to the people. Switzerland is one such country – could we learn something from them?

Whilst I certainly believe that pure democracy goes a fair way to easing these sorts of problems that we see in government, I don’t believe that pure democracy would fix things entirely. For one, pure democracy offers no way to protect the nation from an election being personality driven rather than policy driven. Just as parties advertise heavily before an election with messages filled with FUD and trying to stir up emotion, so too can this happen with important laws in a pure democracy. Any form of government breaks down when people start making decisions without thinking about them adequately.

So a solution? Not a solution per se, but perhaps a refinement, or a melding of the features of both.

Firstly, I believe that that politicians and political parties are indispensable – it is unreasonable for us to expect that the public have enough time (or attention, or intelligence) to take part in every law that is discussed. Similarly, an uneducated decision is worse than no decision, and so any solution must protect the nation from the inherent stupidity (or apathy) of the masses.

However, we must make these politicians and political parties accountable for their actions and their promises. If we elect a party on a promise, they are expected to hold up to those promises they were elected on. This played a big role in this recent election, and for good reason. I don’t really know how best to ensure this is done, but I have a few ideas.

A rather extreme solution to this may be to have politicians salaries dependant on how completely they fulfil their core promises in each year. I do not mind if you under promise and over deliver (or even under promise and barely meet that promise), however it becomes a problem when you under deliver. We would not accept a contractor under delivering on a contract, and yet we allow politicians (who are, in essence, working for us) to under deliver what they have quoted to be able to provide. Incidentally, I’m interested to see GetUp.org.au’s Promise Watch which is doing something about holding them to their promises.

Secondly, I believe that we need to remove the personalities and emotions from politics, in order to prevent this kind of (for want of a better term) emotional gerrymandering. Whilst I don’t believe this is ever entirely possible, I do believe that we can go to certain lengths to do ease the situation as it stands.

Just as things like The Political Compass can give you your position on the political scales based upon a series of questions, why could we not hold elections in a similar, questionnaire based manner? Given that each party and politician has a series of different policy decisions on similar topics, surely it would be possible to build a fairly simple questionnaire in order to gauge which issues are important and which outcomes are actually desired by the public. For example:

Education Funding:
( ) This is not important to me
(-) More money should be spent on Primary and Secondary education
( ) More money should be provided to teaching staff
( ) More money should be provided to Tertiary Institutions

Education Technology:
( ) This is not important to me
(-) Every student should recieve a laptop computer for school use
( ) We should increase computers in schools
( ) Technology in education needs no reform

Where each answer has been provided by one or more politicians or party (condensed into one answer where appropriate).

In this way, there is no knowledge of which decision is related to which politician or party – it becomes an election purely based upon the outcomes that people want, rather than who the people choose. Also, the default answer of “This is not important to me” for each question would allow people to express their apathy without the catch of making a Donkey Vote.

The questionnaires could be done via post, electronically, via phone, on paper, or a combination of all these options (and others). Those that have the facility could vote electronically from home, whilst those without that facility can vote in public places similar to how we do now. With the correct security measures (and an open and transparent technology), I see no reason why this could be any less reliable than the rather lacklustre identity checks done at local polling places.

The election winner would be calculated by matching the votes for each outcome to the politician(s). In such a case, one questionnaire may award partial votes to a variety of politicians. Eventually, one politician would be found to be the most preferred, and would be elected. If you were to adopt the “performance based” salary as I suggested above, this process would also the calculation of which issues are of most importance, and so the effect they will have upon their salary.

There are other benefits to this approach too:

election advertising becomes about the issues not about politicians with fake smiles kissing babies;
people actually get a say in the issues and outcomes that are important to them, rather than having to pick from a group of politicians where (chances are good) they don’t agree entirely with any of them;
there is an active reason for parties to educate the public on issues, rather than using FUD to scare them, or make them vote emotively. This fact can then be verified and undergo scrutiny by educators, action groups, etc.;
the possibility of lower costs, as much of the polling could be done electronically via the Internet or phone;
less likely to be heckled by those insanely irritating people that yell at you brandishing flyers on election day.

Do I believe this is a perfect solution? Hell no. However, I do believe it’s another idea to throw into the mix, and that our current electoral system needs fine-tuning, if not an overhaul. We have the technology, we just need to use it cleverly.

I look forward to seeing what the next few years brings for this country. Let’s hope it sees us well.

My Late Night Productivity Myth

Mon, 26 Nov 2007 09:56:54 GMT

I find that, for me personally, the best time to work is late at night – usually from about 10pm onwards I find myself most productive. Often, I’ll keep working until I’m physically exhausted – usually around 5am or so. Mentally I’m still sharp at that stage, but my body starts to get tired, and I start to wind down. I’ve always thought that if I had to pick a contiguous block of work time, 2pm to 10pm or 4pm to midnight would be a good choice for me.

This might be the time that I feel that I’m most productive, but obviously this can really upset the rest of my life – I’m awake when everyone else is asleep (and vice versa), I’m not alive during business hours the day after (and cantankerous if I need to be), and it puts huge a strain on the rest of the house as everyone tries to be quiet whilst going about their day (lest they wake the most angsty bear they’ve ever met). Not to mention the fact that having one evening like this completely blows out my sleeping pattern completely for the next few days.

It’s hyper productive for me at the time, sure, but I doubt it’s the optimal solution.

Interestingly enough, I’ve also found myself to be very productive the past few days that I’ve been getting up early. Let’s get things straight – I don’t do early mornings. Anyone that’s met me knows that mornings just aren’t something that I subscribe to. Ever. As such, getting up before 6am sounded like a big task. However, it’s not been as bad as one might expect.

However, this post isn’t meant to be about me getting up early per se, but rather a revelation that I’ve had with regards to my “super productive night-time mode”.

I’ve recently noticed things that suggest that my evening productivity “rule” is a great big myth, like the tooth fairy or honest politicians – I don’t think that the time of day has absolutely anything to do with how productive I am. However, I do think the rest of the day has a very large impact upon how productive I am.

I posit that I am productive at night not because of the time of day, but rather because there’s nothing left of the day. During the day, my mind is filled with things that are coming up. Right from the first thing in the morning, there are things coming up in the day – I need to eat, I need to make some calls, I need to go out, there are things that I need to think about, emails I need to write. You get the idea. I know I have all of these things to do (and a task list to remind me) and that’s exactly the problem I think – I know that I’m going to have distractions, so my mind is already trying to get ready for them. I consider this a sort of premature optimization type problem – I know that I have to make an email later, so in the back of my mind I’m already keying up for it. I know I have at least 2 more meals in the day, so what the hell am I going to cook?

I guess you could say that were my brain a computer, it’s in “swap city” at that time – thrashing between what I’m working on, and all of the other processes polling every now and again going “oh man! is it time for me yet? am I ready?”.

At 10pm though, there is none of that – I’m effectively clear for the rest of the evening. I have no more meals to worry about, I can’t make any more calls, I have nowhere to go. My brain is clear from all of that clutter. Again with the computer metaphor, I have one process running and it can take as much of the memory as it likes without things needing to swap all the time.

So, is there a way to get this “single process” mentality during the day, when there are other things coming up? I’m not really sure yet. I’m mainly happy to realise that I don’t have to keep making late runs just to be super productive, however I do need to find a solution that actually works for me. I’m going to experiment with a few different options in limiting the “background processes” in my mind, and I’ll be sure to post how it goes. Hopefully, there’ll be something interesting in it.

The Human Mind and Other Inane Ramblings

Mon, 19 Nov 2007 23:57:21 GMT

This post is going to be a break from the usual “I don’t do anything but write code” kind of posts. Partially because that’s not why I started blogging, and partially because, well, it’s my damn blog so I’ll write what I want! Danny can get away with posting obscure things on his blog – how about me? :D

I just managed to catch the 2nd episode of Professor Robert Winston’s BBC Series, The Human Mind (on ABC), and some of it got me thinking.

The show was very interesting, especially if, like me, you have a fascination with how folks tick. This particular episode examined some of the fundamental parts of personality, including how the brain and personality are related, and how physiological changes in the brain through life lead to personality changes – things like why children can often have little control over their urges, and why all those angsty teenagers really are so angsty!

The first example to get me thinking was a demonstration wherein two identical twins were put in environments in order to stimulate certain moods in order to see their effect upon personality – one listened to happy music, watched a happy movie, and read uplifting comments; the other listened to sad music, watched a sad movie, and read depressing comments. The example went on to show the temporary change that this had upon their personality (in this case shown by shopping habits, which apparently is a useful way to gauge personality). Obviously the twin in the happy environment was happy, spent more, and was more outgoing, whilst the other twin did very little, and regretted her purchases.

What intrigues me is that providing that this information is true (and I’d be compelled to believe Professor Robert Winston if I believe anyone), this lends scientific credibility to the idea of positive affirmations having measurable effect upon the mind. For a long time these have been perceived to be “bad-science”, in the realm of New Age crazies, or as a self-help trick for underachieving students and salesmen.

However – this suggests that the science is there, and that things such as affirmations may have a long term affect on the brain and personality. The idea is that the words invoke thoughts and emotions, which in turn invoke chemical responses that eventually lead to physiological changes in the brain – either by rewiring new pathways (learning) or by making parts of the brain more or less receptive to certain chemicals and hormones.

This is interesting to me – most of my family subscribes to beliefs and ideas that are very “New Agey”, and so (perhaps as a consequence?) I tend to be more open to those sorts of ideas than most. However, I’m also a geek, and fiercely logical – I make a point of making sure that I can justify most of my personal beliefs with fact and science as much as I can seem to manage. That said, I’m certainly in the “New Agey” side of the chart – however I make a point to be able to justify what I believe in. In many respects, this information about affirmations represents a meshing of two parts of me – the purely scientific and the part of me that is prepared to entertain things that seem logical, but have no hard scientific basis.

Anyhu, the second thing that got me thinking was their examination of a dude with anger problems. They explained that what was happening (in his specific case) was that one part of the brain was overproducing the chemical associated with “anger” and his brain wasn’t able to control his reaction to it. Effectively, when something triggered his anger, he really did lose control. Apparently, the part of the brain handling emotional control (which this man was lacking “strength” in) is the frontal lobe, which spurred the question from me – isn’t that the bit that deals with things like language?

Well, I was partially on the right track. It seems that whilst much of language processing the brain does is in the Temporal Lobe there certainly are some elements of language processing done in the frontal lobe (such as Broca’s Area).

Now I’m not at all medically trained (so I’d welcome anyone who knows more on the topic than me) but if that’s the case, could that possibly support an idea that a lack of emotional control, and unregulated anger (if not a symptom of another, more complex condition) could be somehow related to the frontal lobe not receiving suitable stimulation during development? Could this mean that folks with problems with emotional control, or other “regulation” problems are a result of not being adequately stimulated as a child or teenager? Could this suggest that encouraging problem solving, teaching complex tasks and encouraging language development (as in expanding the vocabulary) might help to reduce the susceptibility to rage and anger later in life? Does this mean that education plays an even more vital role in building safe and happy communities than we might have thought?

I don’t have any answers, but it’s interesting to think about anyway. I don’t even know if what I suggest about might “strengthen” the frontal lobe, or whether something like this might have an effect, but I find it something interesting to muse on regardless.

Regardless of my inane ramblings, I strongly suggest you check out the show if you get the chance – The Human Mind and How to Make the Most of It, Monday nights, 11:30pm-12:30am on ABC.

Ruby Net::HTTP and Content Encoding : http_encoding_helper

Sun, 04 Nov 2007 20:58:33 GMT

Working on LifeBox has given me a great opportunity to get my hands dirty with the Ruby Standard Library.

I’ve blogged previously about Net::HTTP , which is a very nice library for working with HTTP and HTTPS requests in Ruby when you need a bit more control than open-uri allows. You know, things like sending custom headers, handling 302 Redirects, Last-Modified-Since, etc.

Even though we’re still in development, we are currently checking just under 400 feeds – some of which we check every 15-20 minutes. That is a serious amount of bandwidth unless we can take advantage of some of the bandwidth saving features of the HTTP spec. With a combination of Last-Modified-Since (which not everyone supports for their feeds) and compression (which not every server supports) we’ve made a huge reduction in bandwidth.

However, there’s a small catch with Net::HTTP requesting that servers send compressed content using the Content-Encoding header – Net::HTTP can easily add the header to request the content, but it currently provides no mechanism to actually handle the compressed data. (Although it looks like some other folks are looking into it . There’s an update at the bottom of this post) It’s fairly simple to do, however.

If you’re reading this, now it’s even simpler, as you can get my 21 line library (excluding code) that will do it for you – http_encoding_helper .

All that the library does is add a single method – plain_body which will work out if the content is compressed, and decompress it (using gzip or deflate, as the case might be). All that you have to do is request it using the Accept-Encoding header.

Check out the project page for a code sample, and more information.

Updated: 10Nov2007 I’ve been swapping a few emails with Hugh Sasse, who is responsible for the effort to get the Net::HTTP library to use compression by default (It’s his email I linked to above). Since that post, he’s also made a revision

One comment that Hugh made to me was that his patch and mine are fundamentally different in philosophy – his patch seeks to make compression the default, whereas mine is more of a “as you need” kind of patch.

The reason that my patch is a separate library, rather than a patch to the Net::HTTP library itself is purely for simplicity, and management. As I deploy code to a variety of servers, mangling the net/http code itself is not convenient. Similarly, my patch doesn’t enable compression by default purely to keep it simple without having to mangle too much of the Net::HTTP code.

I’m eager to see Hugh’s code to make it into the Ruby-1.9 Net::HTTP library so as that everyone’s code can begin to use compression by default – being worked into the library itself makes it much more powerful, and much easier to keep working between changes in core.

Until that happens, my code will be available for use.

On a side note, it’s interesting to see that lighttpd supports bzip2 compression – I don’t know what UserAgents can handle that, but interesting none the less.