Damian Brady’s Blog

So when I wrote that post yesterday, to my surprise I found three other draft posts of varying ages.  I know myself well enough to know that I probably won’t go back and revisit them, so here’s what they were about:

Salling Clicker

That’s all that was in this one – just the title.  I know what I was going to say though.  I was going to heap praise on this tiny but personally very useful app.

I have a beautiful (expensive) Sony LCD TV in my living room with a very handy VGA port.  My computer is right nearby so it’s permanently connected.  What Salling Clicker does is provide a way to control my computer from my mobile phone via Bluetooth or Wifi.  This is amazingly useful when watching movies from the couch.

The interface is solid and deployment is very easy.  Best of all, it’s cheap.

Understanding threading is more important than ever

This was a post I was going to write about a now old Ars Technica article.  The article was about the future of processors according to Intel where they suggested there’d be many many cores, not just two or four.  I didn’t get too far on my post:

We were taught threading in the very first programming subject I took at uni.  Initially, it can be difficult to get your head around.  Eventually I got there.  I was convinced though that plenty of people passed that course without any firm idea of how threading worked.

With the announcement from Intel that they’re heading down the track of many, many CPU cores, understanding threading is more important than ever for a budding software developer.

What I was going to get to was that developers will have to start thinking about parallel processing.  If we’re stuck on 3GHz per core and you want to do some crazy-complex stuff, it better be able to be spread across the cores or it will be S.L.O.W.

Will the compiler/OS/processor be able to handle this for you?  Well, yeah, possibly to an extent, but writing your software to take advantage of new processor capabilities will be a necessity for performance.

At the very least, you’ll have to have a detailed knowledge of how threading works.  It’s the future, man.

Religious Coding

The oldest draft of the lot was also the closest to being finished.  Here’s what I wrote, minus the half-written final sentence (seriously, I stopped halfway through a sentence):

A mate of mine working in the US sent me a link to a Mozilla Labs tool called Ubiquity.

Essentially, the idea behind it is to let users describe what they want to do with the Internet rather than where they want to go.  The web at the moment is really based around sites that you have to actually visit to be able to use the information.  Their idea is to skip this visiting stuff and let the tool (or commands written for it) do all the work for you, giving you only the information you were after.  They have a really good description on their blog post (linked above), and I can’t really think of a better way to describe it than them (which is probably good), so have a read if you’re interested.

It’s really lightweight and very easy to install, but at the moment it only works for Firefox.  It’s also surprisingly easy to write new commands.  I’ve been a .Net kid for a while now but it only took me a couple of hours to hack up a new command.  Admittedly the Ubiquity language is Javascript so it’s not terribly unfamiliar.  Still, that’s a pretty short learning curve.

Commands have the ability to give you a realtime ajax preview before you actually execute the command.  In most cases, I don’t even use the execute functionality and I suspect that for the small things, most people won’t.

Anyway, when I was trying it, the built-in “define” command wasn’t working for me.  It would look up a word when you pressed enter to execute the command, but it didn’t give me a preview.  So, in order to get a nicer dictionary lookup and at the same time try out this funky little thing, I decided to create my own.  Of course once I’d got mine working, the define command started working properly.  I like mine better – it gives prettier results.  It’s not just because I made it myself… but that’s the main reason.

You can go here to have a look at it.  Subscribe to it if you want to use it or hey, just steal the code or whatever, I don’t care, it’s public license – do what you want.

So in summary, it’s easy to use and I think it has the potential to be something really big.  If you have a look at the examples of what they’re ultimately trying to be able to do, it’s pretty impressive.  Imagine typing “find restaurants near me with reviews over 4 stars” and being given a google map with markers and summarised reviews of each.  Very useful, no?

Hmm, combine that with voice recognition and you’ve got a genuine futuristic computer from the movies!  Rad.

So I haven’t written a post in a while, but no, that’s not the reason for the title.  I’ve just been concentrating on other things and haven’t felt sufficiently motivated to write about anything.

But then I read this post from Steve Yegge on Stevey’s Blog Rants.  His title was “Business Requirements are Bullshit” which, while clearly designed to catch the eye, doesn’t entirely represent what it was about.

Steve’s post was aimed at people developing a new (or better) product to take to market.  He wasn’t talking to consultants or employees who are producing something to meet a specific company’s business needs, but someone who was creating something new to fill a perceived hole in the market.

His point (adapted from Warren Buffet) was that you should build something you already know about; something that actually meets your needs.  If you’re doing that, then you already know what you want.  You know what compromises you can make and what the unspoken and tacit deal-breakers are.  If you’re trying to gather business requirements from a group of people who may or may not want the product while trying to understand what it is you’re actually making, then you’re probably going to fail.

It sounds like great advice, but right now, I’m not in the category of people building something new to take to market.  I’m in the other group.  The stuff I build and maintain (and now I’m going to slip back into software) is supposed to meet an immediate business need for a specific client.  It more than likely won’t be used by me, but I have to build it anyway.

So can Steve’s advice be applied to my situation?  Sure it can, to an extent.

Steve’s main point was that if you’re not investing in something you understand, then you’re walking into very dangerous territory.  As an end-to-end software developer, I’m aware that it’s difficult to know exactly what the customer wants.  Sure, you can grill every potential user for days, you can write a comprehensive list of requirements, you’ll check it and recheck it over and over again to make sure you know single possible piece of functionality that they want and need.  But when it comes to the crunch, no matter how much work you’ve done, you’ll start getting negative feedback about some specific things that you hadn’t thought of and the customer hadn’t mentioned.

That complex report you were asked to include, the one with all the tables and forecasts and things?  It turns out that 10 different people print that 20 times a day.  So even though it’s perfect, it takes 15 minutes to run each time, and they can’t wait that long each time.

So how do you avoid this? In my experience, if you want to write truly useful software, you need to understand why each piece of functionality is being written.  Spend a lot of time with the clients and find out what it’s like to be in their shoes.  If you see how they operate day-to-day you’ll start to get an idea of what they actually want, not what’s written in the requirements doc.

Steve’s advice was that you shouldn’t invest in what you don’t understand.  So if you have to produce something you don’t understand, make a genuine effort to understand it first.  You might not be as enthusiastic about or deeply involved in the business your client is in.  But by honestly trying to see what they’re trying to achieve, you’ll learn what they really want your software to do for them; over and above the list of required functions.

I stumbled across Part 6 of a Programming Job Interview Challenge on the Dev102 blog and thought, “hey, I know this!” I couldn’t help responding.

So the question was, given this bit of code, what will the output be?

   1: ArrayList a = new ArrayList();

   2: ArrayList b = new ArrayList();

   3:

   4: a.Add(1);

   5: b.Add(1);

   6: a.Add(2);

   7: b.Add(2.0);

   8:

   9: Console.WriteLine((a[0] == b[0]));

  10: Console.WriteLine((a[1] == b[1]));

Ok, so I’ll break the post here in case you want to work it out for yourself, but you should really go to the original post – not least because there are five other challenges there.

Read more »

It’s great when software is designed and implemented with careful planning and plenty of documentation so someone can pick it up when it falls down.  Unfortunately, it’s often the case that a) You’re working on an existing project that has no documentation, or b) You’re working on an existing project that has documentation that is now out of date.

I’m doing the former right now.  The software is old.  Like 12 years old.  It has no documentation.  It has duplicated code, code where it shouldn’t be, hard-coded values, code that does the wrong thing before being corrected by other code.  Think of an anti-pattern and it’s in there.

The application is being phased out, but it must be supported until it’s completely gone.  It’s also in an environment where changes are inevitable and legally mandated so there’s no chance of a complete code-freeze.  The original programmers are long gone.  It’s not fun to work with.

Now don’t get me wrong.  I don’t go in and make changes that just add to the spaghetti – I refactor when it’s practical.  If I’m changing some code that’s repeated in three places, I’ll pull it out into its own method.  If there are hard-coded values that could change, I’ll set them up so they’re configurable.  In short, I’ll do some cleanup, but I’m not going to do it all.  Refactoring everything would mean nearly rewriting the thing from scratch, and that’s been done (hence the phase out).

So what about documentation?  Writing detailed UML for this software is just not practical.  For one thing, it’d be obscenely hard to do, but more important is the fact that this program is unlikely to be in use in 12 months time.  Priority doesn’t go to documenting legacy software.  The fact of the matter is that it’s just not worth the time and effort to pull the thing apart to work out how every little piece ticks.

So I’ve come up with my own alternative.  I call it Post-traumatic Documentation.

I’ve set up a document that I’ve titled the Application Body of Knowledge.  Every time I dive in to make a change or fix a bug, I’ll absorb a whole lot of information about how the application fits together.  There are basic things like which files do what, but most of the important stuff is in the details.  I might spend half an hour working out that some assignments in method A get overridden in certain cases when method B is called, or that a class called ABC actually provides functionality for DEF and perhaps doesn’t get called at all.  I learn these things the hard way and I make notes as I go.

Whenever I finish one of these traumatic quests to change something small, I go through my notes and add anything of interest to my Application Body of Knowledge.  There’s a section dealing with the overall architecture of the application, one for each of the main modules, one for deployment, and one for general notes.  It’s deliberately heavy on keywords and jargon to facilitate searching.

This document is obviously far from complete, but it is immensely helpful to me when I next have to make a change, and it’ll be even more helpful to the guy who takes the reigns from me when I get hit by that programmer-killing bus.

Damo

Two of my favourite blogs have announced in the last few hours the upcoming launch of www.stackoverflow.com and I, for one, am excited.

The site will be a combined effort by two of the legends in the software design world, Jeff Atwood and Joel Spolsky.  Jeff runs the well-known Coding Horror blog, and Joel is chief guy for the Joel on Software blog as well as CEO of Fog Creek.  They’ve decided to combine forces to start a community site that’s essentially a free programming Q&A site.

Every programmer knows the recurrent problems encountered when searching for a programming dilemma in Google with the hopes of finding an answer.  There are a couple of problems that haunt me constantly.  Joel mentions in his blog post the situation where you find your exact problem and potential answers on sites that require registration and payment.  Experts Exchange is a classic for that; so much so that I generally use some google-fu to remove any Experts Exchange pages from my search results when searching for solutions.  Don’t get me wrong, I’m sure Experts Exchange has some fantastic content, but I can’t justify spending that much money to save me a few hours every few months.

The other problem I encounter time and time again are the forum posts with no solutions.  Often, when I search for a specific problem in Google, I’ll get a ton of results that appear to exactly match my particular issue.  Excitedly, I’ll open them all up in new tabs and go through them looking for the bit of code that will be my saviour.  Far too often, the threads don’t provide answers at all – just a community of people lamenting the same fault and looking for cures.  Even more frustrating are the instances when the initial poster finishes up the thread with a post along the lines of, “Thanks everybody, but I found out how to do it” without providing the solution.

Hopefully, this site will hasten a timely death for these frustrations.  If there’s anyone I’d want on the case, it’s these two guys.

Damo

Slashdot today carries a link to a story claiming that the CAPTCHA algorithm for Hotmail (or Windows Live Hotmail or whatever it’s called now) has been defeated by a spambot and the exploits have started.  So that’s Gmail, Yahoo Mail, and now Hotmail.

CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a great idea, but if it doesn’t work, then it doesn’t work.

CAPTCHAs were developed to tell humans apart from software.  They’re essentially a Turing Test across a very limited domain, and because of the limited domain, they’re much easier to attack.  In the case of a standard warped-text CAPTCHA, the attacker knows that the challenge will be an image with a certain number of letters and/or numbers, and that it will be warped in one or more ways.  The software can be written with this in mind.  Additionally, even if there is only a miniscule success rate, it’s often worthwhile for a spammer, particularly if attempts can be automated and run several times a second.

So what’s the solution?

Slashdot made a tongue-in-cheek reference to Kitten Auth, suggested in 2006.  It may have been a playful suggestion, but I think they’re on the right track.  Kitten Auth basically presents the user with a number of pictures of cute fluffy animals, and tells the user to select all the kittens.  The premise is the same as the text-based CAPTCHAs – easy for humans, hard for computers – but it doesn’t use text, making OCR useless.

Something like Kitten Auth could work as long as there’s no predictability.  If the same images are repeatedly used, a brute force attack would work.  If you needed to select three kittens out of nine pictures, all you need is one random success and bam, you have copies of three images that are kittens.  Given enough time, the software could learn enough images to be viable as a solution.

Alternatively, if OCR can be trained to learn letters and numbers that are very warped and modified, then why not pictures of kittens?  It’s harder, sure, but if we mere mortals can tell a kitten apart from a possum, then why not a computer? These spammers and malware authors are pretty determined you know.

So what else?

Maybe the problem with CAPTCHAs is the “CA” part.  Completely Automated.  What about PAPTCHA? Partially Automated. Sure, it ruins the contrived acronym, but it might be more effective.

Arguably, Kitten Auth is already an PAPTCHA.  The pictures of kittens can’t really be completely automated unless there are 3D models of kittens rendered from different angles with different lighting each time… hmm… that’s an idea… but I digress.

If Microsoft and Google and Yahoo were to put some effort into changing their “PTCHA” regularly, by real people, maybe there’s a solution.

Here’s how it could work:

• Twenty people, armed with cameras, walk the streets for a few hours taking photos of random objects or scenery.
• They get back to the office and upload the photos to today’s collection.
• They link each photo to some standard questions (e.g. “what is the main object in this photo?”) and provide acceptable responses.
• They provide additional specific questions for each photo (e.g. “How many white horses are there in the field?”) and provide acceptable responses.
• One or more other staff members look at the photo and each question for quality control.  They can add more acceptable answers, remove them, or reject photos or questions outright.
• Photos are retired after a time to prevent them being learned.

As a very rough estimate, I’d expect that a person would be able to add at least fifty photos with ten questions each every day.  With 20 people, that equals 10,000 new PTCHAs every day – 50,000 per working week. Surely that’d be enough.  Is 20 people too many?  Even with five people you’d have 12,500 new challenges every week.  If you expire the questions after a month, you’d still have an incredibly large number to choose from.

Current CAPTCHAs effectively have an infinite number of possibilities, however they’re still in a narrow domain.  By expanding the domain to include any question about any photo, there’s no pattern to learn – no possible algorithm to solve the problem.

Is it foolproof?  Definitely not.  However, I’d suggest that implemented properly (and that means a lot of QA), it would be a lot harder to break than current CAPTCHA methods.

There could be a business in this you know… I’d be interested to know what you think!

Damo

Edit: I’ve been having a discussion with a friend of mine who has outlined exactly why 50,000 new challenges per week is not enough.  In short, if x people are creating these challenges, then some fraction of x can be employed to decipher them (answering is quicker than asking).  The answers get added to a massive database along with copies of the images, and there’ll be enough solutions saved to give some malicious code a decent success rate.  If the image and question match one in the database, then the answer will be there.

Repetition of challenges is therefore a significant problem.  A challenge that presents an “image and question” that is repeated every 200,000 requests (4 weeks of 50,000 per week) is far too repetitive.  If the malicious code runs one request every fifteen minutes on 1,000 nodes, you’d have seen every challenge in just over 2 days.

So to overcome this, here are some ideas:

• Use existing CAPTCHA technology such as warping the question text and putting it directly on the photo in a semi-random place.  You’d get no exact repeats.  The obvious problem is that this may still allow a malicious program to recognise sections of the photo that haven’t been altered.  With every photo and answer saved, there’s still a one in ten chance (given 10 questions per photo) of getting the question right.  Very unacceptable.
• Warp not only the text, but the image as well.  Obviously it’d still need to be recognisable, so overlaying a random, semitransparent pattern or something might be all you could do.  It might be enough to slow down matching of the image though.
• Include a bevy of questions that bear no relation to the image.  These could be added to any of the images.  For example, you could have a picture of a field of horses which renders with the question, “How many legs are most people born with?”

So now I have a system where a modified image is rendered with an overlayed warped-text question which may or may not have anything to do with the image.

Of course all I’m really doing is adding complexity, but as long as it’s complex enough to withstand attacks for the length of time it’s used (one month in my example), it should work.

My other suggestion, the CG kittens, got more interest.  In this case, there would be essentially no repeated images.  You’d probably only need a handful of animal models with a few variables set at random to make it feasible.  Perhaps fur colour, lighting, camera position, and some posture or face variables.

So I finally bit the bullet and upgraded the version of WordPress.  This blog is now running the brand spanking shiny new version 2.5!

I’ve gotta say, the upgrade went really, really smoothly! Literally about 10 minutes from thinking about doing it to now (including download time)

While it’s far from a one-click upgrade solution, it is very straightforward.

Having said all that though, I’m dreading the upgrade of the QUT Volleyball Website.  The most dangerous thing about these upgrades is apparently the plugins – and the QUT Volleyball Website is filled with them.  Videos, Polls, etc.

I’ll let you know how that one goes when I finally get the courage!

Damo

I was doing some blog-hopping the other day and came across an old post called “The Scott Adams Meltdown: Anatomy of a Disaster” on the Ask Tog site.

Basically, Scott Adams (creator of the Dilbert comics) had an incident in early 2006 where he accidentally (permanently) deleted a post as well as 500 comments that were attached to it.

Tog identifies a “misleading metaphor” as one of the issues and highlights the importance of using appropriate metaphors when designing software and educating users on how to use it.

Several articles in his site talk about other misleading and confusing metaphors and how they contributed to bugs or problems, and it got me thinking about the use of metaphors a bit more.

Now, I’m a big fan of metaphors – I use them constantly, particularly when I’m talking about anything IT to a “layperson”. Communication between “nerds and normal people” is something that is typically not handled well. Frequently the nerd doing the explaining gets frustrated at the user’s lack of understanding and the user gets frustrated at the jargon and poor explanations. Metaphors can help, but only if they’re used properly. Similarly, when writing software, user-interface metaphors are frequently used. Think of the recycle bin in Windows or the Home button in your browser. These are (usually) effective metaphors.

My sister is fairly heavily involved in AFL. She was talking the other day about how coaches teach young kids the correct techniques for handballing and marking. They tell the kids to imagine the ball as a spaceship and the little valve in the middle of the laces as the spaceman. When you’re kicking, the spaceship should be pointing up, and the spaceman should be pointing to where you want to kick it.

The problem with this is that the intended behaviour doesn’t match the metaphor terribly well. A spaceship should point where it needs to go, right? In fact, the spaceman should probably point in that direction as well. Essentially, you’re telling kids to imagine the ball as a spaceship, but a spaceship that doesn’t really mimic the behaviour of a spaceship. It’s misleading.

It gets worse though. When they teach the kids to handball, they tell them to hold the spaceship in one hand with the spaceman at the top up and the spaceship pointing in the direction it needs to go. Ok, not bad so far. Then, they tell the kids to imagine there’s an icecream in their other hand. To handball correctly, they should smash the icecream into the back of the spaceship. What?

I don’t think I need to point out the problems with that one.

This, to me, is a series of very poorly thought-out metaphors. A metaphor should be something that someone can relate to to help them understand the concept in question. The properties and behaviour of the metaphor should closely resemble the model you’re trying to present. This is why metaphors like “bookmarks” in browsers and “address books” in email programs work reasonably well, and others like “Clippy” (Word’s abandoned instructional paperclip) confused many users.

Damo

Joel on Software has an absolutely magnificent article explaining the difficult situation Microsoft is in with Internet Explorer 8.

Essentially, he explains how and why the web browser is in a no-win situation. Microsoft can enforce “standards” and allow existing web pages to break if they don’t meet the standards (something no browser has done before) or they can continue to be backwards compatible and support all those workarounds people have been putting in for years to make-it-work-for-browser-X.

If they choose the first option, which is their current plan, they’ll have a hard time convincing the developers of all those existing websites (if they’re around any more) to update their sites to conform to the HTML4 and CSS1 standards amongst others. If not, all those webpages will work, but what’s the point of standards if they’re not enforced?

It’s a longish article, but it’s very well written and it provides great insight if you’re a web developer.

Damo