Damian Brady’s Blog

So I haven’t written a post in a while, but no, that’s not the reason for the title.  I’ve just been concentrating on other things and haven’t felt sufficiently motivated to write about anything.

But then I read this post from Steve Yegge on Stevey’s Blog Rants.  His title was “Business Requirements are Bullshit” which, while clearly designed to catch the eye, doesn’t entirely represent what it was about.

Steve’s post was aimed at people developing a new (or better) product to take to market.  He wasn’t talking to consultants or employees who are producing something to meet a specific company’s business needs, but someone who was creating something new to fill a perceived hole in the market.

His point (adapted from Warren Buffet) was that you should build something you already know about; something that actually meets your needs.  If you’re doing that, then you already know what you want.  You know what compromises you can make and what the unspoken and tacit deal-breakers are.  If you’re trying to gather business requirements from a group of people who may or may not want the product while trying to understand what it is you’re actually making, then you’re probably going to fail.

It sounds like great advice, but right now, I’m not in the category of people building something new to take to market.  I’m in the other group.  The stuff I build and maintain (and now I’m going to slip back into software) is supposed to meet an immediate business need for a specific client.  It more than likely won’t be used by me, but I have to build it anyway.

So can Steve’s advice be applied to my situation?  Sure it can, to an extent.

Steve’s main point was that if you’re not investing in something you understand, then you’re walking into very dangerous territory.  As an end-to-end software developer, I’m aware that it’s difficult to know exactly what the customer wants.  Sure, you can grill every potential user for days, you can write a comprehensive list of requirements, you’ll check it and recheck it over and over again to make sure you know single possible piece of functionality that they want and need.  But when it comes to the crunch, no matter how much work you’ve done, you’ll start getting negative feedback about some specific things that you hadn’t thought of and the customer hadn’t mentioned.

That complex report you were asked to include, the one with all the tables and forecasts and things?  It turns out that 10 different people print that 20 times a day.  So even though it’s perfect, it takes 15 minutes to run each time, and they can’t wait that long each time.

So how do you avoid this? In my experience, if you want to write truly useful software, you need to understand why each piece of functionality is being written.  Spend a lot of time with the clients and find out what it’s like to be in their shoes.  If you see how they operate day-to-day you’ll start to get an idea of what they actually want, not what’s written in the requirements doc.

Steve’s advice was that you shouldn’t invest in what you don’t understand.  So if you have to produce something you don’t understand, make a genuine effort to understand it first.  You might not be as enthusiastic about or deeply involved in the business your client is in.  But by honestly trying to see what they’re trying to achieve, you’ll learn what they really want your software to do for them; over and above the list of required functions.

On several occasions recently I’ve had to work with data in legacy applications.  It’s never fun, and it got me thinking about the origins of the problem.  It’s fairly obvious that difficulties encountered when working with legacy data are a result of decisions made when the system was originally written.

New systems are not immune, however.  The same difficulties are bound to crop up for someone trying to work with your data at a later date unless you think about it now.  There’s obviously no definite way of knowing how your data will be used in the future, and the tendency is often to focus only on current requirements.  With a little care though, you can make it easier for a future person to work with.

Thus, I present my guiding principles for data capturing.

By way of disclaimer, I should mention that these principles are based on business applications where users, be they staff or clients, enter information manually.  That’s not to say that they can’t apply in other scenarios, but keep in mind that I was looking at it from a business application angle.

So here they are:

1. Am I capturing everything I need?
2. Am I capturing information in a way that allows it to actually be used later?
3. Am I restricting myself - can I capture more (or different) information later if it’s needed?

1. Am I capturing everything I need?

If you’re not sure, capture it anyway.  For example, if you’re not sure whether the sex of a client is required, put it in anyway.  Obviously there are legal restrictions on collecting some information, but remember that these restrictions can change.  Allow for that.

More to the point, capture information with as much detail as you can.  Don’t hide behind a usability excuse - if you think hard enough, you can get around that.  Just because you store information in a particular way doesn’t mean the user has to enter it that way or even enter it at all - some information can be assumed.  Of course there’s always some trade-off with usability, but it’s worth thinking about how you can get around the usability problem rather than leaving a potentially important piece of information out.

For example, timestamps on file comments probably don’t need to be accurate right down to the tick, but what’s the harm if they are?  Let’s face it, storage space is cheap.  In terms of hiding behind usability, your user shouldn’t have to enter a time to that detail (or probably at all), but it doesn’t mean an automatic process you write at a later date isn’t able to.

2. Am I capturing information in a way that allows it to actually be used later?

This seems obvious but is often overlooked.  Don’t store dates (or any other numeric information) as strings, don’t store information in binary fields, and use a database not a file to store information.  If your system takes a piece of information from somewhere, be it a person or another system, it should store that information in a way that can be used easily later.  There’s no point retaining information if it can’t be used.

Let’s say the business wants you to store a whole lot of information about a client’s car.  The designer might give the user fields to fill in the licence plate, the colour, make, model, VIN, age, etc.  The easiest way to store most of these pieces of information is in string fields.  Sure, this will allow users to look at the data, probably interpret it, and use it to glean information, but what if you need to analyse it in bulk later?  You’ll have a lot of work to do to group all the “White”, “white”, “wite”, and even “wh9ite” cars.  It’s hard to work out that cars that are “12mths”, “12 months”, “1Y”, and “one yr” old are all the same age.  Don’t do it to yourself.

Typically, the designer might ask of the business, “Are you ever going to search on this data?” That’s a mistake.  Don’t even ask.  They will say no.  However, some years down the track, they might actually want to know whether there’s any correlation between having a new car, and being a good client.  It may seem far-fetched now, but by allowing for it, you don’t take it off the table.  The business should never hear, “I can’t give you that information” when they know for a fact their staff have been filling it out every time.

It all comes down to working out what the information actually is and what it means, then storing it the same way.  If it’s a length of time, work out how to store it as a timespan.  If it’s a colour, try working out how to actually store it as a colour, not someone’s misspelt description of a colour. Do the research and find out how a VIN or a licence plate is structured so you can validate it.  If it’s really not possible to validate, at least process it as much as you can.  Remove spaces or capitalize all letters if it doesn’t matter to make it as easy as possible to work with at a later date.

3.  Am I restricting myself - can I capture more (or different) information later if it’s needed?

Don’t save space in fields when it could come back to bite you. Don’t use a 4-bit integer to save space because you know you’ll never have more than 16 different payment methods or vehicle makes.  Even though you’re only operating in Australia, use string encoding that allows Chinese characters.  What are you saving by skimping, really?  Again, storage is cheap.  Don’t restrict yourself from the start.

If you have to use your judgement to determine the size of a field, then overestimate significantly.  Like, really a lot.  Can’t imagine someone having a surname longer than 30 characters?  Me neither, but I’ll allow 100 or more.  I’m not having some client with a four-way-hyphenated-surname break my application.

Summary

Don’t take the easy way out.  I know if you’re stressed for time and under the pump, adding a new Bank Name field as an non-validated string is going to be a lot easier than trying to match it to a list of known banks (with the option to add a new one).  Fight the urge to make it easy now and potentially very hard later.

Remember that your design decision is usually a one-way street.  It’s not often that you can do it the easy way now and fix it up later.  Even if you can, will you?  Really?  You’re too busy to deal with it now, what makes you think you’ll have plenty of spare time to do it later?  It’s worth the extra time to do it properly.  You’ll thank yourself later.

In a previous post, I talked about the limitations with CAPTCHA systems and proposed a partially-automated turing test to keep non-humans out.

I had a few discussions about this with a friend and he was more interested in the CG Kittens idea I very briefly alluded to in the main text. To summarise, the idea was that if the kittens in KittenAuth were 3D models rendered on the fly, you’d get an essentially unlimited number of images.

Well someone has gone and done something very similar. On a small site called SpamFizzle, there’s a description of a 3D CAPTCHA design that renders simple objects and asks the user questions about them. It looks like a great idea to me. It could require some fine-tuning, but I think the premise is sound.

I’d love to credit the author some more, but there are no details on that page. I presume it’s the same author as the only other page on the site, Michael G. Kaplan.

Regardless, it’s a great idea and worth a read if you’re interested.

I found this article by Steve Yegge claiming that the Singleton pattern is stupid and evil.  I know it’s old, but I found it via reddit (which I’ve been getting into a lot more than Digg lately - probably more on that later) and despite having his blog on my feed, I hadn’t gone back that far into his archives.

I read about half of the article before I decided I’d write my own post pointing out that he’s actually attacking not the singleton pattern itself, but its use in inappropriate circumstances.  I use the pattern occasionally, and I’d like to think I use it when it should be used or at least when it’s useful.  The examples he gives throughout represent inappropriate uses for the singleton pattern, not a failure of the concept.

Then I read the rest of it, including the final note which says:

Note: I think there are some good uses for the Singleton pattern, but it’s so incredibly overused, and serves as such a convenient crutch for people who don’t understand OOP, that I figured I’d just take the stance in this article that it’s basically Evil.)

Huh.  So here I am writing a post about nothing.

If you’ve visited here before, you’ll have noticed that the template I’m using has changed.

While the old one looked good to me for a while, the more I looked at it, the more I thought it just looked too busy.  There were borders all over the place and gradients and background textures and all sorts of things.  It was too much.

So I’ve gone the minimalist approach and have chosen a template that’s very simple.  I hope you like it.  I think it makes the posts easier to read.

Damo

It’s great when software is designed and implemented with careful planning and plenty of documentation so someone can pick it up when it falls down.  Unfortunately, it’s often the case that a) You’re working on an existing project that has no documentation, or b) You’re working on an existing project that has documentation that is now out of date.

I’m doing the former right now.  The software is old.  Like 12 years old.  It has no documentation.  It has duplicated code, code where it shouldn’t be, hard-coded values, code that does the wrong thing before being corrected by other code.  Think of an anti-pattern and it’s in there.

The application is being phased out, but it must be supported until it’s completely gone.  It’s also in an environment where changes are inevitable and legally mandated so there’s no chance of a complete code-freeze.  The original programmers are long gone.  It’s not fun to work with.

Now don’t get me wrong.  I don’t go in and make changes that just add to the spaghetti - I refactor when it’s practical.  If I’m changing some code that’s repeated in three places, I’ll pull it out into its own method.  If there are hard-coded values that could change, I’ll set them up so they’re configurable.  In short, I’ll do some cleanup, but I’m not going to do it all.  Refactoring everything would mean nearly rewriting the thing from scratch, and that’s been done (hence the phase out).

So what about documentation?  Writing detailed UML for this software is just not practical.  For one thing, it’d be obscenely hard to do, but more important is the fact that this program is unlikely to be in use in 12 months time.  Priority doesn’t go to documenting legacy software.  The fact of the matter is that it’s just not worth the time and effort to pull the thing apart to work out how every little piece ticks.

So I’ve come up with my own alternative.  I call it Post-traumatic Documentation.

I’ve set up a document that I’ve titled the Application Body of Knowledge.  Every time I dive in to make a change or fix a bug, I’ll absorb a whole lot of information about how the application fits together.  There are basic things like which files do what, but most of the important stuff is in the details.  I might spend half an hour working out that some assignments in method A get overridden in certain cases when method B is called, or that a class called ABC actually provides functionality for DEF and perhaps doesn’t get called at all.  I learn these things the hard way and I make notes as I go.

Whenever I finish one of these traumatic quests to change something small, I go through my notes and add anything of interest to my Application Body of Knowledge.  There’s a section dealing with the overall architecture of the application, one for each of the main modules, one for deployment, and one for general notes.  It’s deliberately heavy on keywords and jargon to facilitate searching.

This document is obviously far from complete, but it is immensely helpful to me when I next have to make a change, and it’ll be even more helpful to the guy who takes the reigns from me when I get hit by that programmer-killing bus.

Damo

Every few months, my bank card decides it’s not going to work.  Usually generic ATMs or those of other banks refuse to accept it, but this time it was an ATM belonging to the bank I’m with.  Once again, I walked inside the branch and arranged for another card to be sent to me.

In addition to ordering me another card, the customer service rep offered to link my Visa card to my savings account so I could use it to withdraw money from my main account.  I’ve never had problems with the Visa, so this may be a more permanent solution.  Regardless, I needed to choose a new PIN for this to work.

Here’s where it gets interesting.  I was asked to type in a new four-digit PIN with various obvious restrictions such as no birth dates or other important dates, but also with no consecutive double digits.  My initial reaction was that it was an unusual requirement (given that I’ve had PINs with double digits assigned to me by the same bank in the past), but I didn’t think much of it.  That was my initial reaction.  As I considered it more, I thought, wouldn’t this would have the opposite of the desired effect?

Now my probability mathematics is a little rusty, but here goes.  There are 10,000 possible four digit PINs.  If a truly random PIN is chosen, the probability of guessing it on the first try is one in ten thousand.  Given you usually only get three tries before the machine swallows your card, the odds aren’t good.  If you let people choose their own PIN with no restrictions, it could be a little easier - you’d probably try their birth date, or an anniversary or something like that.  Let’s face it, people need to be able to remember this number so they’ll have to choose something that relates to them.  The odds of guessing a PIN like this are a lot better.

I was asked to choose a PIN that didn’t correspond to any important dates, so given that restriction, we’ll make an assumption that the PIN isn’t guessable.  Brute force (albeit three attempts) is really all that’s left, so we’re back to one in ten thousand.  But if you add a rule that says you can’t have double digits, then the number of options available decreases to 7,290.  A person who finds my card has a better chance of guessing my PIN than if this rule didn’t exist.

This seems like a case of implementing decisions that sound good without any real analysis.  Have I got this totally wrong?  Does this seem silly to anyone else?

Damo

From a few places comes news that CNN is selling t-shirts with news headlines on them.  An interesting/unusual idea, sure, but they’ve done one thing very wrong.

The text on the shirt is part of the URL.

All of which means I can presumably order (but at least create an image of) a shirt that says, “CNN slept with your mum” or “I like short shorts”.  The shirts come with a line underneath the headline that says, “I just saw it on CNN.com” and the date of the news story.  A person could have fun with that…

Now I’m all for RESTful architecture when it’s appropriate, but there should really be some checking to make sure the headline exists…

Damo

Slashdot today carries a link to a story claiming that the CAPTCHA algorithm for Hotmail (or Windows Live Hotmail or whatever it’s called now) has been defeated by a spambot and the exploits have started.  So that’s Gmail, Yahoo Mail, and now Hotmail.

CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a great idea, but if it doesn’t work, then it doesn’t work.

CAPTCHAs were developed to tell humans apart from software.  They’re essentially a Turing Test across a very limited domain, and because of the limited domain, they’re much easier to attack.  In the case of a standard warped-text CAPTCHA, the attacker knows that the challenge will be an image with a certain number of letters and/or numbers, and that it will be warped in one or more ways.  The software can be written with this in mind.  Additionally, even if there is only a miniscule success rate, it’s often worthwhile for a spammer, particularly if attempts can be automated and run several times a second.

So what’s the solution?

Slashdot made a tongue-in-cheek reference to Kitten Auth, suggested in 2006.  It may have been a playful suggestion, but I think they’re on the right track.  Kitten Auth basically presents the user with a number of pictures of cute fluffy animals, and tells the user to select all the kittens.  The premise is the same as the text-based CAPTCHAs - easy for humans, hard for computers - but it doesn’t use text, making OCR useless.

Something like Kitten Auth could work as long as there’s no predictability.  If the same images are repeatedly used, a brute force attack would work.  If you needed to select three kittens out of nine pictures, all you need is one random success and bam, you have copies of three images that are kittens.  Given enough time, the software could learn enough images to be viable as a solution.

Alternatively, if OCR can be trained to learn letters and numbers that are very warped and modified, then why not pictures of kittens?  It’s harder, sure, but if we mere mortals can tell a kitten apart from a possum, then why not a computer? These spammers and malware authors are pretty determined you know.

So what else?

Maybe the problem with CAPTCHAs is the “CA” part.  Completely Automated.  What about PAPTCHA? Partially Automated. Sure, it ruins the contrived acronym, but it might be more effective.

Arguably, Kitten Auth is already an PAPTCHA.  The pictures of kittens can’t really be completely automated unless there are 3D models of kittens rendered from different angles with different lighting each time… hmm… that’s an idea… but I digress.

If Microsoft and Google and Yahoo were to put some effort into changing their “PTCHA” regularly, by real people, maybe there’s a solution.

Here’s how it could work:

• Twenty people, armed with cameras, walk the streets for a few hours taking photos of random objects or scenery.
• They get back to the office and upload the photos to today’s collection.
• They link each photo to some standard questions (e.g. “what is the main object in this photo?”) and provide acceptable responses.
• They provide additional specific questions for each photo (e.g. “How many white horses are there in the field?”) and provide acceptable responses.
• One or more other staff members look at the photo and each question for quality control.  They can add more acceptable answers, remove them, or reject photos or questions outright.
• Photos are retired after a time to prevent them being learned.

As a very rough estimate, I’d expect that a person would be able to add at least fifty photos with ten questions each every day.  With 20 people, that equals 10,000 new PTCHAs every day - 50,000 per working week. Surely that’d be enough.  Is 20 people too many?  Even with five people you’d have 12,500 new challenges every week.  If you expire the questions after a month, you’d still have an incredibly large number to choose from.

Current CAPTCHAs effectively have an infinite number of possibilities, however they’re still in a narrow domain.  By expanding the domain to include any question about any photo, there’s no pattern to learn - no possible algorithm to solve the problem.

Is it foolproof?  Definitely not.  However, I’d suggest that implemented properly (and that means a lot of QA), it would be a lot harder to break than current CAPTCHA methods.

There could be a business in this you know… I’d be interested to know what you think!

Damo

Edit: I’ve been having a discussion with a friend of mine who has outlined exactly why 50,000 new challenges per week is not enough.  In short, if x people are creating these challenges, then some fraction of x can be employed to decipher them (answering is quicker than asking).  The answers get added to a massive database along with copies of the images, and there’ll be enough solutions saved to give some malicious code a decent success rate.  If the image and question match one in the database, then the answer will be there.

Repetition of challenges is therefore a significant problem.  A challenge that presents an “image and question” that is repeated every 200,000 requests (4 weeks of 50,000 per week) is far too repetitive.  If the malicious code runs one request every fifteen minutes on 1,000 nodes, you’d have seen every challenge in just over 2 days.

So to overcome this, here are some ideas:

• Use existing CAPTCHA technology such as warping the question text and putting it directly on the photo in a semi-random place.  You’d get no exact repeats.  The obvious problem is that this may still allow a malicious program to recognise sections of the photo that haven’t been altered.  With every photo and answer saved, there’s still a one in ten chance (given 10 questions per photo) of getting the question right.  Very unacceptable.
• Warp not only the text, but the image as well.  Obviously it’d still need to be recognisable, so overlaying a random, semitransparent pattern or something might be all you could do.  It might be enough to slow down matching of the image though.
• Include a bevy of questions that bear no relation to the image.  These could be added to any of the images.  For example, you could have a picture of a field of horses which renders with the question, “How many legs are most people born with?”

So now I have a system where a modified image is rendered with an overlayed warped-text question which may or may not have anything to do with the image.

Of course all I’m really doing is adding complexity, but as long as it’s complex enough to withstand attacks for the length of time it’s used (one month in my example), it should work.

My other suggestion, the CG kittens, got more interest.  In this case, there would be essentially no repeated images.  You’d probably only need a handful of animal models with a few variables set at random to make it feasible.  Perhaps fur colour, lighting, camera position, and some posture or face variables.

Every now and then I’ll get an email from someone that’s been addressed to a number of people. I can tell how many and who they are because all of our email addresses are listed in the “To:” header. My email address has been sent to a list of people I don’t know.

There are a couple of reasons I don’t want my email address broadcast to the world.

Spam et. al.

I often don’t know many of the other people on the list, and more to the point, I don’t know anything about their computers. It’s possible, probably likely, that in an email addressed to 20 people, at least one of them has a fairly insecure computer and probably has at least one virus or a trojan. When the insecure computer receives the email, my email address going to be visible to this malware.

In short, I can have the best and most robust security on my computer, and I can ensure my email address is never published on the web, but all of this is useless if a single person sends my email address to a compromised machine.

Email addresses can be personal

I may be in the minority here, but I maintain a large number of email addresses. Having my own domains means that I can set up separate email addresses for job applications, friends and family, work, web enquiries, and so on. Right now, off the top of my head, I can think of about twenty email addresses that I use regularly. Most are simply forwarders, but they allow me to categorise incoming emails efficiently.

The other thing this lets me do is control communication. If I am wary about giving a company or a person my email address, I’ll create a new one. If I start getting emails I don’t want, or if for some reason I don’t want them to contact me any more, I can delete the email address.

Now you can see the dilemma. If I’ve given a particular email address to one company and they broadcast it to other people in a group email, I lose control.

Easily Fixed

If you’re reading this blog, I’d be surprised if you didn’t know about BCC, but I’ll summarise just in case.

When sending an email, you can put recipients’ email addresses into the “To:” field, the “CC:” field, or the “BCC:” field. “To:” and “CC:” behave the same, but “CC:” indicates that the person is being given a “carbon copy” - a legacy name from the paper days.

“BCC:” stands for “blind carbon copy”. These people will still receive the email, but the email addresses in this section will not be included in the header. They will be kept private.

The problem of sending everyone’s email addresses out with the email is obviously easily fixed. Just put all the email addresses in the “BCC:” section. For emails amongst groups of friends and family, it’s often not a big deal, but in business it’s frankly unprofessional.

Design flaw

People still don’t know about BCC. I sometimes feel compelled to educate the sender of a group email about BCC and the usual response is surprise. They usually aren’t even aware of this function.

I think the problem is deeper than just a lack of education. There are fundamental design flaws here. Now, email is old - Wikipedia claims that it’s been around since about 1965. So I’m not going to suggest any fundamental technical changes. Such changes would be infeasible in a system that a) works, and b) is older than the Internet.

A significant part of the problem is that it’s called “BCC”. What the hell does that mean to the average person? Even expanding it to “blind carbon copy” doesn’t really help - it doesn’t describe its behaviour.

“Carbon copy” is relatively easy to understand. It’s at least reasonably clear that the people in this section will be getting a “copy” of the email - it’s not directly addressed to them, but they’ll see it anyway. But what does “blind” mean? That it will be invisible? It’ll be transmitted in Braille? People won’t know what BCC does until they’re told.

The other problem I can see is that “BCC:” isn’t presented as a default field in many email clients. The main email clients I use are Outlook and Gmail. In both cases, “BCC:” must be explicitly turned on.

Solutions

1. Change the name.
   “BCC” doesn’t mean anything - even when it’s expanded to “blind carbon copy” it doesn’t mean anything.
   Obviously, this change can’t be fundamental, but it can be cosmetic. If an email client changed “BCC:” to “Discreetly To:” or something similar, it might help with people’s understanding.
2. Change the behaviour.
   In an office environment, group emails to external domains should, by default, include everyone in the “BCC:” field rather than the “To:” field. If that’s too extreme, it should prompt the user, suggesting that perhaps they don’t want to share the list of email addresses with all the recipients. At the very least, hiding email addresses from the other recipients should be a very visible option.

In the meantime? Teach your staff about BCC. Make sure they use it when it’s appropriate.

Damo