Damian Brady’s Blog

I solved a problem the other day.  It was a terrible solution. It works, but it’s difficult to be proud of how I solved it.

The problem was with MSMQ, but let me describe the problem with a needlessly overwrought metaphor.

Every week for the last 5 years, you’ve been sending a bill to one of your clients.  You have an infinite supply of envelopes and when Friday comes around, you print out an invoice, seal it in an envelope, send it by courier to your client, and a couple of days later, you get paid.  With me so far?

One day, you decide to start using a different invoice management software package – it’s much prettier and more stable than the old one.  You know, however, that the client doesn’t want anything in that invoice to change.  You’re not sure how they process it, but one time your printer smudged an invoice slightly and you didn’t get paid.  So everything in that invoice has to be exact.  Luckly, the new software is capable of printing a pixel-perfect version of the old invoice layout.  So far so good.

You also think you should start using a different courier.  Your existing courier company (COM+) is fine, but frankly they’re a bit behind the times.  The delivery drivers are about 80 and they’re driving some vans that are generations old.  So you set up a deal with a new company called C-Sharp 3.5 to do the delivery for you.

Still with me?  I told you it was overwrought.

So anyway, you give your new system a try.  You print out the invoice, compare it to the old one (spot-on, 100% the same), put it in an envelope, and send it off using the new courier company.

You don’t get paid.

You contact the customer but they don’t say much.  Just that they didn’t receive a valid invoice.  The new courier company swears they delivered it to the correct address and they provide proof.

You decide to try again next week and do some investigating.

The next week, you print out the invoice, compare it to the old ones (still the same), put it in an envelope, and organise for the new courier to pick it up.  This time though, you follow him.

He takes your letter, gets in his van, and drives to the client’s address.  He gets out, puts your letter in the mailbox and drives off.  Nothing wrong so far, as far as you can tell, so you wait to see what happens.

Soon, a guy comes out of the house (which by the way looks exactly like a big black box), opens the mailbox, picks the letter up, and takes it inside.

You go home, satisfied that the letter made its journey this time.  It must have been a once off.  But you still don’t get paid.

Again, you contact the client and all they’ll tell you is that they didn’t get a valid invoice.  You protest, telling them that you saw them pick it up but to no avail.

Over the next few weeks, you try everything you can think of to get this new system to work.  You try looking at the message again after it’s put in the letterbox and you try sending an invoice created by your old program.  Every time, the seemingly perfect letter gets picked up my the man in the black box, and nothing happens.

No matter how hard you try to work out what’s going on, nothing helps.  So, like any sensible person,  you give up and call the old courier company.  They turn up, pick up your invoice, and two days later, you get paid.  Despite the fact that they’re apparently doing exactly the same as the new guys, their deliveries get you paid, and the other deliveries don’t.  You resign yourself to using the old couriers until they or their vans all die.  It’s just a matter of time…

Fun story, huh?

So in case you’re a bit slow and didn’t work out what it was all about, I replaced an app that put a message in a Windows message queue for another (black box) program to pick up.  The old app was VB6 using a COM MSMQ library, and the new one was C# using the native .Net MSMQ library in the 3.5 version of the .Net Framework.

No matter how hard I looked, I couldn’t find ANYTHING different between the messages and where they got put.  I even dug up a copy of the black-box code and stepped it through a debugger.  When it got to the line saying Queue.Receive(), the message disappeared from the queue, and nothing came back.  There was no exception thrown!  I watched the message get picked up!  I did a binary comparison of the message contents from the old program and the new program!  No difference at all!  Inexplicable!

The fact that I was putting the message into the Windows Message Queue using a .Net library and picking it up using COM+ seemed to be all it took to break the thing.

This is the first time I can remember being absolutely, 100% stumped by what was going on.  I’d analysed what was going on as deeply as was practical and had come up with nothing.

So I did the logical thing and referenced the COM+ dll from my new app.  When I used this library to send the message, it all worked perfectly.

I really don’t like this fix. Despite the fact that everything works perfectly now, it’s still a failure in my eyes.  It’s like a brand new Merc with duct tape holding the wheels on.

However, given that I’d spent way too long already on something that really shouldn’t have been a problem, it was probably the right thing to do.  Cut my losses and take the easy way out.  I’m not proud.

So I finally joined the dark side and I got myself an iPhone.

Now, if you’ve been following this blog for a while, you’ll know that often it’s the little things that interest me.  Small bits of functionality and the tiniest ideas sometimes make the biggest impression.

With that in mind, here are the top things that have impressed me about the iPhone.

1. UI during phone calls

When I’m on a call and I’m holding the phone up to my ear, there’s no need for the screen to be on… so it’s not.  If I take the phone away from my ear, the accelerometers in the phone notice this and I get a useable display again.

A tiny thing, but it shows that Apple is thinking.

2. Music in my car

I connect the iPhone to my car stereo using a standard 3.5mm headphone plug.  Great sound. It works perfectly. Not that impressive.

However when I get out of my car, I pull the cable out.  The music stops.  It could quite easily continue – there are speakers on the phone after all, but it’s smart enough to assume that having unplugged th speakers/headphones/whatever, I probably don’t want to listen to the music any more.

Again, it’s simple but intelligent.

3. Silent switch

My old phone was a Nokia E71 and I loved it.  It did everything I wanted it to, but I’m a sucker for stuff that’s nice to use (usually) so of course I wanted an iPhone.

One very slight annoyance I’ve had with every phone thus far (including the E71) is that to turn the phone to silent, it takes a few steps.  On the E71, you have to unlock it, press the power button at the top, scroll down to Silent, press the OK button, and lock the phone again.

On the iPhone, you flip the switch on the side.

4. Network connection switching

I’ve set up my iPhone with the details of my home wireless network and the network at my parents’ house.  I did each setup a grand total of once.

Now, whenever I’m at home or at Mum and Dad’s, any browsing I do uses those networks.  The phone never asks me, never checks to see whether I might actually want to pay for access, no.  It assumes I’ll want the fastest (and cheapest) connection available.

5. Application Installation

Hands down the cleanest installation for any software I’ve experienced.

Go to the app store and find something you want, touch the price button and then install, and you’re done.  There could be a couple of extra touches here and there and you might have to put a password in so your account gets charged too, but it’s incredibly easy.

It downloads in the background, there are no restarts, and you can immediately see it in your list of apps (”springboard” apparently).  Brilliant.

So that’s it – probably not the same list as anyone else, but like I said, it’s the small attention to detail that impresses me.

So my keyboard at work just broke.  Doesn’t happen often, but one of the keys just stopped y’know… putting it’s letter on the screen.

So I got given a replacement from somewhere.

Here’s the thing – the layout is different.  Any programmer or techie or otherwise-nerd will know what I mean when I say that it’s FREAKING ANNOYING!!

Now, there aren’t any big changes really.  I mean, all the letters are in the same place, and the F-keys are still above the keyboard and the number pad is the same, but there are enough differences to frustrate someone like me.

The Home-End-PgUp-PgDown-etc. chunk of keys is different.

This is what I used to have:

Here’s what I have now:

What that means is that I keep hitting Delete instead of End.  Because it’s right below Home now.

I only have one Windows key

And it’s on the left – the one I don’t go for when I’m locking my computer with Win-L as I stand up to go somewhere.

The F-keys are grouped in threes

Seriously, in threes?  F5 is by far the most common F-key I hit, and now it’s in the middle of a cluster instead of on the left.  I have to look at my keyboard now to hit it.

My right Ctrl key is slightly to the left of where it was

Not a big deal you’d think?  Well truthfully it doesn’t affect me much – I still usually hit it first go, but it’s been moved left to make way for the arrow keys.  So when I don’t hit Ctrl, I hit the left arrow key.  That’s confusing.

So you’re probably thinking, “Oh, you poor little baby! Would you like some cries with your waa-burger?”  And yes, yes I would.

Seriously, it’s frustrating.  I know it’s only temporary – I’ll get a new one soon with a normal key layout.  But my keyboard is by far the most frequently-used tool in my arsenal.  Small things like this slow me down.

Ok, I’m finished complaining now.  You can proceed with your day.

My last post talked about how I wasn’t getting anywhere with my project.  Well now I am.

I might have mentioned that I’m using php and while it’s pretty easy to do stuff, I sometimes feel uncomfortable with the looseness of it all.  If you’ve used php in addition to a “real” OO language, you’ll know what I mean.  Lots and lots of globally-accessible functions with no namespaces, duck-typing, all that sort of stuff.

So I’m following some better practices (as suggested by Jon in his comment on the last post).  Nothing is very groundbreaking and I’m sure most newer php projects use the same kind of things.  Anyway, here’s what I’m doing:

I’m using an ORM library to get from my database to my code and vice versa.  Specifically Doctrine.  I’m reasonably familiar how these things work having used NHibernate for .Net (a port of Java’s Hibernate), but I’ve been impressed with how easy it’s been this time.  I got Doctrine working beautifully for my spider-web of a MySql database inside an evening with no experience.  I’ll let you know how it works out in the long run.  And yes, MySql is the database right now but Doctrine allows it to change very easily as expected.

I’m using an MVC pattern so my code doesn’t get too mixed in with my UI.  Now I’m sure there are plenty of libraries and frameworks out there that would help me with this, but I’m rolling my own… if you can call it that.  I’m used to .Net’s code-behind method so I’m really just copying that to an extent.  Each php page will have an include at the top that references the code-behind.  The “code-behind” php file will set up all the data and will provide functions that can be called by the front one.  Simple, but it should be effective.

Finally, I’m using JQuery for UI prettiness.  I had a serious look at a lot of javascript libraries and JQuery came out on top for what I needed.

So that’s it.  That’s what I’m doing.  So far it’s all fitting together quite nicely, but I’ll be sure to post updates if I want to rave about something or if something sends me into a fit of frustration.

As always, comments are welcome.

Damian

So when I wrote that post yesterday, to my surprise I found three other draft posts of varying ages.  I know myself well enough to know that I probably won’t go back and revisit them, so here’s what they were about:

Salling Clicker

That’s all that was in this one – just the title.  I know what I was going to say though.  I was going to heap praise on this tiny but personally very useful app.

I have a beautiful (expensive) Sony LCD TV in my living room with a very handy VGA port.  My computer is right nearby so it’s permanently connected.  What Salling Clicker does is provide a way to control my computer from my mobile phone via Bluetooth or Wifi.  This is amazingly useful when watching movies from the couch.

The interface is solid and deployment is very easy.  Best of all, it’s cheap.

Understanding threading is more important than ever

This was a post I was going to write about a now old Ars Technica article.  The article was about the future of processors according to Intel where they suggested there’d be many many cores, not just two or four.  I didn’t get too far on my post:

We were taught threading in the very first programming subject I took at uni.  Initially, it can be difficult to get your head around.  Eventually I got there.  I was convinced though that plenty of people passed that course without any firm idea of how threading worked.

With the announcement from Intel that they’re heading down the track of many, many CPU cores, understanding threading is more important than ever for a budding software developer.

What I was going to get to was that developers will have to start thinking about parallel processing.  If we’re stuck on 3GHz per core and you want to do some crazy-complex stuff, it better be able to be spread across the cores or it will be S.L.O.W.

Will the compiler/OS/processor be able to handle this for you?  Well, yeah, possibly to an extent, but writing your software to take advantage of new processor capabilities will be a necessity for performance.

At the very least, you’ll have to have a detailed knowledge of how threading works.  It’s the future, man.

Religious Coding

The oldest draft of the lot was also the closest to being finished.  Here’s what I wrote, minus the half-written final sentence (seriously, I stopped halfway through a sentence):

I stumbled across this Slashdot post the other day which talks about the potential for mobile phones to generate enough power to run by using piezoelectric materials.  In short, the vibrations you and those around you make when you talk could generate enough electricity to keep your phone going.

Now here’s the point in this post where I really wish I could remember the word I want to use.  You know when you hear about something for the first time in your life, then amazingly you hear about it several times in the next short while?  You wonder how you’re suddenly being bombarded with this new concept when you only just found out about it?  Yeah, there’s a word for that and I’ve forgotten what it is.  That word goes here.

I picked up a Popular Science magazine a few days before I found this Slashdot post and amazingly, there was a whole section in it about “free” energy.  Awesome concept.

There’s a whole host of low-powered devices in common use today, and there’s superfluous energy being expended everywhere.  Surely this energy can be used to power these low-powered devices?

The simplest example in the Popsci magazine was the humble light switch.  When you wire up a house, you have wire for power going to the actual light, as well as wire for control going from the light to the light switch.  When you flick the switch to turn the light on, you’re expending energy.  Not much, granted, put apparently enough to send a wireless signal to the light (or a controller next to it) to tell it to turn on.  With a setup like this, there’s no need for wire to run down the walls to the switches.  Yeah, it’s not much of a power saving and it’s probably not worth the extra initial cost right now, but in the future it could, and combining a few dozen simple ideas like this in every house could surely save some energy.

The fact is, we expend a lot of energy when we do anything.  There’s also ambient energy all over the place – the obvious ones like wind and light, but also vibrations and body heat.  If we could capture a fraction of that energy and put it to use, we’d have much less need for power plants.

There are thousands of options out there.  If you go for a walk, chuck your mobile phone in the little holster on your belt, and as you walk, the up-down motion of your steps can charge it.  Or maybe the energy absorbed when rain lands on your car roof could be enough to power your windscreen wipers.

Perhaps after a time, when it becomes cheap enough, device manufacturers may get into the habit of building these “free” power sources into their devices.  A mobile you didn’t have to plug into a wall or a sound-powered LED light you could just stick onto a ceiling would certainly have a market.

Clearly this stuff interests me.

So I haven’t written a post in a while, but no, that’s not the reason for the title.  I’ve just been concentrating on other things and haven’t felt sufficiently motivated to write about anything.

But then I read this post from Steve Yegge on Stevey’s Blog Rants.  His title was “Business Requirements are Bullshit” which, while clearly designed to catch the eye, doesn’t entirely represent what it was about.

Steve’s post was aimed at people developing a new (or better) product to take to market.  He wasn’t talking to consultants or employees who are producing something to meet a specific company’s business needs, but someone who was creating something new to fill a perceived hole in the market.

His point (adapted from Warren Buffet) was that you should build something you already know about; something that actually meets your needs.  If you’re doing that, then you already know what you want.  You know what compromises you can make and what the unspoken and tacit deal-breakers are.  If you’re trying to gather business requirements from a group of people who may or may not want the product while trying to understand what it is you’re actually making, then you’re probably going to fail.

It sounds like great advice, but right now, I’m not in the category of people building something new to take to market.  I’m in the other group.  The stuff I build and maintain (and now I’m going to slip back into software) is supposed to meet an immediate business need for a specific client.  It more than likely won’t be used by me, but I have to build it anyway.

So can Steve’s advice be applied to my situation?  Sure it can, to an extent.

Steve’s main point was that if you’re not investing in something you understand, then you’re walking into very dangerous territory.  As an end-to-end software developer, I’m aware that it’s difficult to know exactly what the customer wants.  Sure, you can grill every potential user for days, you can write a comprehensive list of requirements, you’ll check it and recheck it over and over again to make sure you know single possible piece of functionality that they want and need.  But when it comes to the crunch, no matter how much work you’ve done, you’ll start getting negative feedback about some specific things that you hadn’t thought of and the customer hadn’t mentioned.

That complex report you were asked to include, the one with all the tables and forecasts and things?  It turns out that 10 different people print that 20 times a day.  So even though it’s perfect, it takes 15 minutes to run each time, and they can’t wait that long each time.

So how do you avoid this? In my experience, if you want to write truly useful software, you need to understand why each piece of functionality is being written.  Spend a lot of time with the clients and find out what it’s like to be in their shoes.  If you see how they operate day-to-day you’ll start to get an idea of what they actually want, not what’s written in the requirements doc.

Steve’s advice was that you shouldn’t invest in what you don’t understand.  So if you have to produce something you don’t understand, make a genuine effort to understand it first.  You might not be as enthusiastic about or deeply involved in the business your client is in.  But by honestly trying to see what they’re trying to achieve, you’ll learn what they really want your software to do for them; over and above the list of required functions.

On several occasions recently I’ve had to work with data in legacy applications.  It’s never fun, and it got me thinking about the origins of the problem.  It’s fairly obvious that difficulties encountered when working with legacy data are a result of decisions made when the system was originally written.

New systems are not immune, however.  The same difficulties are bound to crop up for someone trying to work with your data at a later date unless you think about it now.  There’s obviously no definite way of knowing how your data will be used in the future, and the tendency is often to focus only on current requirements.  With a little care though, you can make it easier for a future person to work with.

Thus, I present my guiding principles for data capturing.

By way of disclaimer, I should mention that these principles are based on business applications where users, be they staff or clients, enter information manually.  That’s not to say that they can’t apply in other scenarios, but keep in mind that I was looking at it from a business application angle.

So here they are:

1. Am I capturing everything I need?
2. Am I capturing information in a way that allows it to actually be used later?
3. Am I restricting myself – can I capture more (or different) information later if it’s needed?

1. Am I capturing everything I need?

If you’re not sure, capture it anyway.  For example, if you’re not sure whether the sex of a client is required, put it in anyway.  Obviously there are legal restrictions on collecting some information, but remember that these restrictions can change.  Allow for that.

More to the point, capture information with as much detail as you can.  Don’t hide behind a usability excuse – if you think hard enough, you can get around that.  Just because you store information in a particular way doesn’t mean the user has to enter it that way or even enter it at all – some information can be assumed.  Of course there’s always some trade-off with usability, but it’s worth thinking about how you can get around the usability problem rather than leaving a potentially important piece of information out.

For example, timestamps on file comments probably don’t need to be accurate right down to the tick, but what’s the harm if they are?  Let’s face it, storage space is cheap.  In terms of hiding behind usability, your user shouldn’t have to enter a time to that detail (or probably at all), but it doesn’t mean an automatic process you write at a later date isn’t able to.

2. Am I capturing information in a way that allows it to actually be used later?

This seems obvious but is often overlooked.  Don’t store dates (or any other numeric information) as strings, don’t store information in binary fields, and use a database not a file to store information.  If your system takes a piece of information from somewhere, be it a person or another system, it should store that information in a way that can be used easily later.  There’s no point retaining information if it can’t be used.

Let’s say the business wants you to store a whole lot of information about a client’s car.  The designer might give the user fields to fill in the licence plate, the colour, make, model, VIN, age, etc.  The easiest way to store most of these pieces of information is in string fields.  Sure, this will allow users to look at the data, probably interpret it, and use it to glean information, but what if you need to analyse it in bulk later?  You’ll have a lot of work to do to group all the “White”, “white”, “wite”, and even “wh9ite” cars.  It’s hard to work out that cars that are “12mths”, “12 months”, “1Y”, and “one yr” old are all the same age.  Don’t do it to yourself.

Typically, the designer might ask of the business, “Are you ever going to search on this data?” That’s a mistake.  Don’t even ask.  They will say no.  However, some years down the track, they might actually want to know whether there’s any correlation between having a new car, and being a good client.  It may seem far-fetched now, but by allowing for it, you don’t take it off the table.  The business should never hear, “I can’t give you that information” when they know for a fact their staff have been filling it out every time.

It all comes down to working out what the information actually is and what it means, then storing it the same way.  If it’s a length of time, work out how to store it as a timespan.  If it’s a colour, try working out how to actually store it as a colour, not someone’s misspelt description of a colour. Do the research and find out how a VIN or a licence plate is structured so you can validate it.  If it’s really not possible to validate, at least process it as much as you can.  Remove spaces or capitalize all letters if it doesn’t matter to make it as easy as possible to work with at a later date.

3.  Am I restricting myself – can I capture more (or different) information later if it’s needed?

Don’t save space in fields when it could come back to bite you. Don’t use a 4-bit integer to save space because you know you’ll never have more than 16 different payment methods or vehicle makes.  Even though you’re only operating in Australia, use string encoding that allows Chinese characters.  What are you saving by skimping, really?  Again, storage is cheap.  Don’t restrict yourself from the start.

If you have to use your judgement to determine the size of a field, then overestimate significantly.  Like, really a lot.  Can’t imagine someone having a surname longer than 30 characters?  Me neither, but I’ll allow 100 or more.  I’m not having some client with a four-way-hyphenated-surname break my application.

Summary

Don’t take the easy way out.  I know if you’re stressed for time and under the pump, adding a new Bank Name field as an non-validated string is going to be a lot easier than trying to match it to a list of known banks (with the option to add a new one).  Fight the urge to make it easy now and potentially very hard later.

Remember that your design decision is usually a one-way street.  It’s not often that you can do it the easy way now and fix it up later.  Even if you can, will you?  Really?  You’re too busy to deal with it now, what makes you think you’ll have plenty of spare time to do it later?  It’s worth the extra time to do it properly.  You’ll thank yourself later.

In a previous post, I talked about the limitations with CAPTCHA systems and proposed a partially-automated turing test to keep non-humans out.

I had a few discussions about this with a friend and he was more interested in the CG Kittens idea I very briefly alluded to in the main text. To summarise, the idea was that if the kittens in KittenAuth were 3D models rendered on the fly, you’d get an essentially unlimited number of images.

Well someone has gone and done something very similar. On a small site called SpamFizzle, there’s a description of a 3D CAPTCHA design that renders simple objects and asks the user questions about them. It looks like a great idea to me. It could require some fine-tuning, but I think the premise is sound.

I’d love to credit the author some more, but there are no details on that page. I presume it’s the same author as the only other page on the site, Michael G. Kaplan.

Regardless, it’s a great idea and worth a read if you’re interested.

I found this article by Steve Yegge claiming that the Singleton pattern is stupid and evil.  I know it’s old, but I found it via reddit (which I’ve been getting into a lot more than Digg lately – probably more on that later) and despite having his blog on my feed, I hadn’t gone back that far into his archives.

I read about half of the article before I decided I’d write my own post pointing out that he’s actually attacking not the singleton pattern itself, but its use in inappropriate circumstances.  I use the pattern occasionally, and I’d like to think I use it when it should be used or at least when it’s useful.  The examples he gives throughout represent inappropriate uses for the singleton pattern, not a failure of the concept.

Then I read the rest of it, including the final note which says:

Note: I think there are some good uses for the Singleton pattern, but it’s so incredibly overused, and serves as such a convenient crutch for people who don’t understand OOP, that I figured I’d just take the stance in this article that it’s basically Evil.)

Huh.  So here I am writing a post about nothing.