Damian Brady’s Blog

On several occasions recently I’ve had to work with data in legacy applications.  It’s never fun, and it got me thinking about the origins of the problem.  It’s fairly obvious that difficulties encountered when working with legacy data are a result of decisions made when the system was originally written.

New systems are not immune, however.  The same difficulties are bound to crop up for someone trying to work with your data at a later date unless you think about it now.  There’s obviously no definite way of knowing how your data will be used in the future, and the tendency is often to focus only on current requirements.  With a little care though, you can make it easier for a future person to work with.

Thus, I present my guiding principles for data capturing.

By way of disclaimer, I should mention that these principles are based on business applications where users, be they staff or clients, enter information manually.  That’s not to say that they can’t apply in other scenarios, but keep in mind that I was looking at it from a business application angle.

So here they are:

1. Am I capturing everything I need?
2. Am I capturing information in a way that allows it to actually be used later?
3. Am I restricting myself – can I capture more (or different) information later if it’s needed?

1. Am I capturing everything I need?

If you’re not sure, capture it anyway.  For example, if you’re not sure whether the sex of a client is required, put it in anyway.  Obviously there are legal restrictions on collecting some information, but remember that these restrictions can change.  Allow for that.

More to the point, capture information with as much detail as you can.  Don’t hide behind a usability excuse – if you think hard enough, you can get around that.  Just because you store information in a particular way doesn’t mean the user has to enter it that way or even enter it at all – some information can be assumed.  Of course there’s always some trade-off with usability, but it’s worth thinking about how you can get around the usability problem rather than leaving a potentially important piece of information out.

For example, timestamps on file comments probably don’t need to be accurate right down to the tick, but what’s the harm if they are?  Let’s face it, storage space is cheap.  In terms of hiding behind usability, your user shouldn’t have to enter a time to that detail (or probably at all), but it doesn’t mean an automatic process you write at a later date isn’t able to.

2. Am I capturing information in a way that allows it to actually be used later?

This seems obvious but is often overlooked.  Don’t store dates (or any other numeric information) as strings, don’t store information in binary fields, and use a database not a file to store information.  If your system takes a piece of information from somewhere, be it a person or another system, it should store that information in a way that can be used easily later.  There’s no point retaining information if it can’t be used.

Let’s say the business wants you to store a whole lot of information about a client’s car.  The designer might give the user fields to fill in the licence plate, the colour, make, model, VIN, age, etc.  The easiest way to store most of these pieces of information is in string fields.  Sure, this will allow users to look at the data, probably interpret it, and use it to glean information, but what if you need to analyse it in bulk later?  You’ll have a lot of work to do to group all the “White”, “white”, “wite”, and even “wh9ite” cars.  It’s hard to work out that cars that are “12mths”, “12 months”, “1Y”, and “one yr” old are all the same age.  Don’t do it to yourself.

Typically, the designer might ask of the business, “Are you ever going to search on this data?” That’s a mistake.  Don’t even ask.  They will say no.  However, some years down the track, they might actually want to know whether there’s any correlation between having a new car, and being a good client.  It may seem far-fetched now, but by allowing for it, you don’t take it off the table.  The business should never hear, “I can’t give you that information” when they know for a fact their staff have been filling it out every time.

It all comes down to working out what the information actually is and what it means, then storing it the same way.  If it’s a length of time, work out how to store it as a timespan.  If it’s a colour, try working out how to actually store it as a colour, not someone’s misspelt description of a colour. Do the research and find out how a VIN or a licence plate is structured so you can validate it.  If it’s really not possible to validate, at least process it as much as you can.  Remove spaces or capitalize all letters if it doesn’t matter to make it as easy as possible to work with at a later date.

3.  Am I restricting myself – can I capture more (or different) information later if it’s needed?

Don’t save space in fields when it could come back to bite you. Don’t use a 4-bit integer to save space because you know you’ll never have more than 16 different payment methods or vehicle makes.  Even though you’re only operating in Australia, use string encoding that allows Chinese characters.  What are you saving by skimping, really?  Again, storage is cheap.  Don’t restrict yourself from the start.

If you have to use your judgement to determine the size of a field, then overestimate significantly.  Like, really a lot.  Can’t imagine someone having a surname longer than 30 characters?  Me neither, but I’ll allow 100 or more.  I’m not having some client with a four-way-hyphenated-surname break my application.

Summary

Don’t take the easy way out.  I know if you’re stressed for time and under the pump, adding a new Bank Name field as an non-validated string is going to be a lot easier than trying to match it to a list of known banks (with the option to add a new one).  Fight the urge to make it easy now and potentially very hard later.

Remember that your design decision is usually a one-way street.  It’s not often that you can do it the easy way now and fix it up later.  Even if you can, will you?  Really?  You’re too busy to deal with it now, what makes you think you’ll have plenty of spare time to do it later?  It’s worth the extra time to do it properly.  You’ll thank yourself later.

In a previous post, I talked about the limitations with CAPTCHA systems and proposed a partially-automated turing test to keep non-humans out.

I had a few discussions about this with a friend and he was more interested in the CG Kittens idea I very briefly alluded to in the main text. To summarise, the idea was that if the kittens in KittenAuth were 3D models rendered on the fly, you’d get an essentially unlimited number of images.

Well someone has gone and done something very similar. On a small site called SpamFizzle, there’s a description of a 3D CAPTCHA design that renders simple objects and asks the user questions about them. It looks like a great idea to me. It could require some fine-tuning, but I think the premise is sound.

I’d love to credit the author some more, but there are no details on that page. I presume it’s the same author as the only other page on the site, Michael G. Kaplan.

Regardless, it’s a great idea and worth a read if you’re interested.

I just read a fantastic article by John Miano called, “The Myth of the Interchangeable Programmer: Can’t We Just Offshore Him?“.

Basically, the premise is that there’s a very common misconception that all programmers are alike and have the same productivity.  It’s definitely worth a read.

While his article focuses on the practice of offshoring programming to cheap overseas countries, the idea is also relevant when talking about hiring new (local) programmers either to replace others or in response to increased workload.

The misconception that all programmers are the same can lead to an almost apathetic approach to hiring by managers.  If someone quits, that’s fine, we’ll just hire another programmer to take over the work.  If there are new requirements that mean we need a new developer, then fine, we’ll just hire another one.  One problem is that there’s usually a significant lead time filled with learning about the project before even the best programmer can be truly productive.  The other problem is that which is outlined in Miano’s article; one programmer can be many times more effective than another.

I’m not sure what the solution to this is.  I’ve preached this line many a time but because it’s in such stark contrast to the person x time = fixed productivity formula that is relatively true for most other professions, it falls on deaf ears.

David from 37signals posted a blog entry about people who have succeeded with ideas you may have thought of a while ago.

He’s right about it all of course – an idea by itself it worth nothing.  If you do nothing with it, then you have nobody to blame but yourself if someone else makes a whole lot of money from a similar idea.

Of course this doesn’t necessarily stop the nagging feeling that it could have been you with all that income…

My big idea that I did nothing with?  I think it was around 1998 when I thought it would be useful to have a real estate website to which rental agencies could submit available properties complete with photos and searchable descriptions.  Seriously.  Oh well.

The Queensland Government recently introduced the Go card to provide a single intelligent ticketing mechanism for (almost) all public transport in South East Queensland.

The technology was developed by Cubic Transportation Systems and similar cards are in use all over the world.  The idea is when you get onto a bus or a train or anything, you touch your Go card to the sensor.  When you get off, you touch it again and the appropriate amount of money gets debited from your card balance.  Presuming it works, it’s a sensible system in my opinion.

Despite catching a bus to (not from) work nearly every working day, I had originally avoided the new system for a few reasons.  The main one was that it provided no financial benefit to me.  There’s a refundable deposit that’s payable when you buy a card, and the cost of an individual one-way ticket was the same whether you used the card or paid cash on the bus.  Discounts only came when you used it more than 6 times in a week.  I very, very rarely travel by bus more than half a dozen times a week.  In early August however, the fares will come down for the Go card only.  This makes it more attractive, so I went to purchase one.

The TransLink website provides an online web ticketing service that lets you purchase a card online.  Presumably they send it out to you but I didn’t get that far because frankly, I was too scared.  Let me show you.

After a couple of short screens asking you about the type of card you want, you come across this screen (click to enlarge):

Notice the “Billing Account Question” at the bottom.  There’s no more information on what this is for, but I presume it’s some kind of verification question you have to answer in order to make payments or maybe changes to your billing details.  The default question is, “What is my name?”.  That’s probably the worst security question I’ve ever heard! Ok, I’m generous, so I’ll give them the benefit of the doubt here and assume that this isn’t used for anything important.  You can change it anyway, and if you’re sensible, you probably will.

Let’s look at the next screen:

The first thing I noticed was that there was another “Cardholder Question”.  Is this different from the other one?  Again, there’s no help available to tell you what it’s for.  At least the question is slightly more difficult to guess this time.  I wasn’t terribly concerned at this point, so I continued.

Here’s the next screen:

Now I’m quite concerned.  Firstly, it appears that despite this being a Queensland Government website, I’m suddenly being charged in pounds.  On one of the first screens, I was told that the charge was $5 so I could probably assume that they just got the currency symbol wrong, but this is a big deal.  What if I am going to end up paying the equivalent of just over $10? I had a look at the address bar to make sure I was still in the right place, and yes, it’s an Australian domain.  I’m growing more and more reluctant to sign up to this thing.  Of course by this stage, I’ve already given them my credit card details, and who knows whether they’ve been stored.

So next, I clicked on the terms and conditions link at the bottom of the page.  Here’s what the pop-up window said:

So that’s it.  I’m done.  No way I’m going to buy online using a credit card from a site with that many problems. The other thing that the terms and conditions error showed me was that they appear to be using Lotus-Domino version 4.6.7a.  The current stable version is version 8.  And does that “a” indicate an alpha version?  The Wikipedia page on Lotus Domino doesn’t even recognise the software before version 5, and the page on Lotus Notes suggests that version 4.6.7 was released sometime prior to 1999.  I’d hate to think what kind of exploits could be carried out on that server.  Colour me scared.

Now, I’m sure I could have continued on my merry way, bought the card, and everything would have worked out fine, but I wasn’t convinced that the transaction would work or even that my information was safe.  SSL or no, the currency problems and the information gathered from that error page just scare me too much.

To be honest, I’m not sure I’m comfortable buying the card at all any more.  The cards have to be registered, so I assume I have to give them some kind of personal information.  With web software that old, I simply can’t trust that it’s safe.

I certainly hope they sort all this out soon if they plan to decommission their other ticketing options.

Damo

I want to preface this post with a couple of disclaimers.  When I talk about passwords in this post, I’m talking about the ones that matter.  I’m talking about your Windows Login password or your Internet Banking password, not the password you use for Facebook or Digg. Let’s be honest, nobody really cares enough about those accounts enough to break into them.  Also, I’m not going to address how the password is stored or the physical security of the resources.  Ideally, there’d be more than just a password.  However, for the purposes of this post, I’m assuming that the password is the only way in.

Now to my blunt opinion.  Most password policies are bad.  They fail spectacularly to solve the problem they’re meant to solve.

Usually, password policies (particularly in workplaces or universities) enforce a few rules.  Some of these rules are useful and some aren’t.  So called “standards” or “industry norms” are followed, but there seems to be little thought put into the password policy.

A large number of password policies I’ve come across follow these two rules:

• The password must be technically complex, usually judged by a basic algorithm; and
• The password must be changed frequently.

While they both seem sensible on the surface, I ask you to actually look at them in terms of the problem they’re supposed to solve.  It’s probably not much of a spoiler, but I’ll let you know that I have a problem with both of these rules.  They don’t effectively solve the problem they’re meant to solve.

First, let’s look at the problem a password is meant to solve.  Obviously, a password is meant to prevent unauthorised access to a particular resource; usually a computer, a network, or some software.  The assumption is that unauthorised access is bad and must be prevented, and don’t worry, I’m not going to argue with that.

So, looking at the first password policy rule listed above, does having a technically complex password solve the problem of unauthorised access?

The problem is that true complexity is very difficult to judge with software.  If a judgement is made on a character-by-character basis, it is essentially useless.  A password like Pa$$w0rD for example, will almost always be judged as technically complex, but would be relatively easy for a sensible password cracking algorithm to break.  It’s not enough to assume a password is good because it meets your easily-testable rules.  I submit that the only good way to test whether a password is secure is to try your hardest to break it.

Enforcing a rule that makes sure a person has a certain number of characters and that they belong to several different character groups (uppercase, lowercase, numbers, symbols) means that a brute-force attack would probably be infeasible.  But as a software developer, if I really wanted to find someone’s password, there’s no way I’d be doing a blind brute-force attack.  I’d start with a dictionary of common words and I’d add things like the person’s date of birth, their partner’s name, their children’s names, their pets’ names – essentially any personal information that I could find.  I’d write some code to try things in a sensible way using these words prefixed or suffixed with numbers or other characters.  I’d try changing the letter O to the digit 0, i to 1, a to @.  Of course in the end, I’d probably just find a tool that did all this for me, but I wouldn’t just go trying random characters.

So does the first rule meet our goals?  Does enforcing 8 characters including three different groups guarantee a good password?  No it doesn’t.  You can set a terrible password and still follow the rules.

Onto the second rule – regularly changing the password.  Let me first say that this is a rule I hate.  Not only does it not meet the goal of protecting an account, it actually makes things worse when combined with the first rule.  Let me explain.

The reaction most users have when you tell them their password has to be (for example) at least 8 characters long and contain uppercase, lowercase, and numbers is not entirely positive.  It’s not always easy to think of a password that meets all of these requirements as well as the unspoken one; you have to remember it.  For obvious reasons, unless they’re instructed otherwise most people will choose something easy to remember like their surname followed by their birthday.  It meets the requirements and they can remember it.  To combat this, sensible administrators will explain how important it is that the password can’t be guessed and encourage another method of choosing a password.  They might suggest turning a sentence into a string of characters.  For example, the sentence “I’m going to try to remember this password” could become “ImG2t2RthP@ss”.  That’s a pretty good password – it meets all the rules, it looks very random, it will survive a dictionary attack, and most importantly it can be remembered.  Basically, it’s going to take a very long brute-force attack to guess.

Now, what if they know they’ll have to choose a new one to remember every month? Are they going to pick a hard password then?  I’d suggest that it’s far less likely that they’ll go through this process of turning a sentence into a password every month.  Even though it’s easier to remember than a random string of characters, it’s won’t stick instantly.  It might take them a few days before they can type it without thinking.  And if they have to do this every 30 days, it becomes that much harder to properly cement it in.

The user is thinking, “why do I have to change it?  My last password was good enough!”, and you know what?  They’re right.

The common argument is that a password should be changed frequently in case it gets compromised.  If someone discovers the password, they’ll only be able to access the system until it’s changed.  While this may be technically true, what does this really accomplish? How long does it really take for someone to do whatever they wanted to do by gaining access to your computer?  The fact is, as soon as someone else is able to access something they shouldn’t access using your password then that’s it – mission failed.  Are you really going to hang the strength of your security on the hope that the attacker won’t have enough time to do what they want to do before the password changes again?  It’s made even worse by the fact that the password is less likely to be a good one.  If I discovered that your current password is “January101182″ or “Smith111, it probably won’t take me long to work out your next one.

To summarise, a password policy like the one in the bullets above that enforces some level of technical complexity and makes the user choose another one every month fails in a couple of ways.

Firstly, enforcing technical complexity is the ultimate false sense of security.  It doesn’t force a user into choosing a good password.  In fact, combined with a rule enforcing frequent password changes, it encourages bad passwords.  Also, the warm fuzzy feeling the admin gets from knowing a password is at most 30 days old is fools gold.  The damage is done as soon as the password is discovered, and it’s more likely to be discovered with a 30 day turnaround because the password will be worse.

I know this is an essay already, but it would be unfair of me to end after attacking common practice without offering an alternative.

So here’s what I suggest:

• Passwords should be technically complex, but they must not be easy to guess; and
• Passwords should be changed at most once a year unless there’s a suspicion they’ve been compromised.

How do you achieve this?

Instruct each user to use a method like the the one suggested earlier; compressing a sentence to choose a password.  Make it hard – 10 or 12 characters in all four categories.  They won’t have to change it regularly so who cares if they take a while to remember it.  If they absolutely need to, let them write it down and keep it in their wallet until they remember it.  If they lose their wallet, then change the password.

Of course as I mentioned earlier, the only real way to test the strength of a password is to try to crack it.  So do that.  Set up a simple system that stores a hash of the person’s password, and try to break in using whatever means necessary.  In particular, look for personal information to work with.  If it’s too easy, you should be able to break in quickly, and if you do, make them change the password to something harder.

So there you have it.  My opinion on why most password policies are bad.  Sometimes you actually have to revisit the problem you’re trying to solve rather than just follow so called tried-and-true policies.

-Damo