Damian Brady’s Blog

I solved a problem the other day.  It was a terrible solution. It works, but it’s difficult to be proud of how I solved it.

The problem was with MSMQ, but let me describe the problem with a needlessly overwrought metaphor.

Every week for the last 5 years, you’ve been sending a bill to one of your clients.  You have an infinite supply of envelopes and when Friday comes around, you print out an invoice, seal it in an envelope, send it by courier to your client, and a couple of days later, you get paid.  With me so far?

One day, you decide to start using a different invoice management software package – it’s much prettier and more stable than the old one.  You know, however, that the client doesn’t want anything in that invoice to change.  You’re not sure how they process it, but one time your printer smudged an invoice slightly and you didn’t get paid.  So everything in that invoice has to be exact.  Luckly, the new software is capable of printing a pixel-perfect version of the old invoice layout.  So far so good.

You also think you should start using a different courier.  Your existing courier company (COM+) is fine, but frankly they’re a bit behind the times.  The delivery drivers are about 80 and they’re driving some vans that are generations old.  So you set up a deal with a new company called C-Sharp 3.5 to do the delivery for you.

Still with me?  I told you it was overwrought.

So anyway, you give your new system a try.  You print out the invoice, compare it to the old one (spot-on, 100% the same), put it in an envelope, and send it off using the new courier company.

You don’t get paid.

You contact the customer but they don’t say much.  Just that they didn’t receive a valid invoice.  The new courier company swears they delivered it to the correct address and they provide proof.

You decide to try again next week and do some investigating.

The next week, you print out the invoice, compare it to the old ones (still the same), put it in an envelope, and organise for the new courier to pick it up.  This time though, you follow him.

He takes your letter, gets in his van, and drives to the client’s address.  He gets out, puts your letter in the mailbox and drives off.  Nothing wrong so far, as far as you can tell, so you wait to see what happens.

Soon, a guy comes out of the house (which by the way looks exactly like a big black box), opens the mailbox, picks the letter up, and takes it inside.

You go home, satisfied that the letter made its journey this time.  It must have been a once off.  But you still don’t get paid.

Again, you contact the client and all they’ll tell you is that they didn’t get a valid invoice.  You protest, telling them that you saw them pick it up but to no avail.

Over the next few weeks, you try everything you can think of to get this new system to work.  You try looking at the message again after it’s put in the letterbox and you try sending an invoice created by your old program.  Every time, the seemingly perfect letter gets picked up my the man in the black box, and nothing happens.

No matter how hard you try to work out what’s going on, nothing helps.  So, like any sensible person,  you give up and call the old courier company.  They turn up, pick up your invoice, and two days later, you get paid.  Despite the fact that they’re apparently doing exactly the same as the new guys, their deliveries get you paid, and the other deliveries don’t.  You resign yourself to using the old couriers until they or their vans all die.  It’s just a matter of time…

Fun story, huh?

So in case you’re a bit slow and didn’t work out what it was all about, I replaced an app that put a message in a Windows message queue for another (black box) program to pick up.  The old app was VB6 using a COM MSMQ library, and the new one was C# using the native .Net MSMQ library in the 3.5 version of the .Net Framework.

No matter how hard I looked, I couldn’t find ANYTHING different between the messages and where they got put.  I even dug up a copy of the black-box code and stepped it through a debugger.  When it got to the line saying Queue.Receive(), the message disappeared from the queue, and nothing came back.  There was no exception thrown!  I watched the message get picked up!  I did a binary comparison of the message contents from the old program and the new program!  No difference at all!  Inexplicable!

The fact that I was putting the message into the Windows Message Queue using a .Net library and picking it up using COM+ seemed to be all it took to break the thing.

This is the first time I can remember being absolutely, 100% stumped by what was going on.  I’d analysed what was going on as deeply as was practical and had come up with nothing.

So I did the logical thing and referenced the COM+ dll from my new app.  When I used this library to send the message, it all worked perfectly.

I really don’t like this fix. Despite the fact that everything works perfectly now, it’s still a failure in my eyes.  It’s like a brand new Merc with duct tape holding the wheels on.

However, given that I’d spent way too long already on something that really shouldn’t have been a problem, it was probably the right thing to do.  Cut my losses and take the easy way out.  I’m not proud.

I’m positive I’ve ranted about maintaining legacy code before, but I going to do it again.  At least this time I’m just reading the code so I can understand it and rewrite it rather than having to apply more bandaids.

Oh yes, and I’ve jumped on the “Top X List!” bandwagon.  It’s cosy up here.  So in no particular order, here are my top 6 things I hate seeing in legacy code:

1. Comments and variable names that didn’t update with the code

Yes, the code changes as you write, I understand that.  But seriously, can you please update the comments to reflect what you’ve done?

It’s very confusing seeing TODO comments for things that have been done, or comments about things that should be considered before the code goes into production.  Or maybe there’s a variable called currentIndex which is now actually a string containing a key because the original programmer changed from an array to a dictionary.

These things sometimes take ages to work out.  For example, I recently spent several hours trying to figure out what was happening in a chunk of code.  Looking at the following line:

It turns out:

1. It was in fact implemented (and was very, very necessary)
2. The failedMessageList variable contained all messages, not just failed ones

So seriously, if you change your code – take a glance at the comments to see if they need changing too.

2. Poorly named files

I had to add this one in. Not because it’s a big problem very often, but because when it occurs, it can be really painful. It’s obviously frustrating when classes and variables are named badly, but if files are named badly it can be even worse.

Let’s say you’re reading through the code and find a line that instantiates a Person class. Unfortunately, there’s no Person.cs (or whatever language) file. After some searching, you find that the Person class definition is in Teacher.cs – a file that wasn’t renamed when the code changed.

Or perhaps you’re looking for the UI for the window titled, “Database Maintenance” and have to work out that it’s actually a class called Restoring which is found in a file called FrmBackRest.vb. Sadly, a true story.

3. Inconsistent ways of doing things

I don’t have a problem if you do something a different way than I do; that’s fine – we all have our preferences. I might prefer to write to the database with parameterized stored procedures that I’ve written myself, while you use an ORM and just call a Save() method on an object you’ve just updated. That’s fine – both work.

What I really don’t like, though, is when someone does both. Like, in one place they’re calling stored procedures, in another they’re building SQL queries on the fly, and in another they’re retrieving DataSets via queries, updating the values, and calling a Save() method.

Nothing screams I-just-hacked-this-together-without-a-plan as much as large inconsistencies like this.

4. Even legacy-er code in legacy code

Sometimes I’ll come across legacy code that makes it clear that the original developer came from an even older background and didn’t quite understand this new technology. As a .Net developer, the most common stuff I see is good old VB6 code in a .Net app.

I came across a Goto statement in a bit of .Net code the other day. Another true story. It was a few lines after “On Error Resume Next”.

5. Error handling that hides errors

This is pretty common, and unfortunately it’s really only something that gets noticed when you have a problem. While things are working perfectly (and you don’t care how it works), there’s no problem. As soon as there’s an error though, you can’t find any details about it because the error handling is crap.

For example:

The end result of this? Here are some examples:

1. The transmitter.SubmitMessage method throws an exception. It gets caught, written to an error log, and thrown again. Ok, that’s not too bad…
2. The result from the SubmitMessage method comes back invalid. We increment our error count, get the details of the problem, and throw an exception. Then, oh no! That exception gets caught immediately, and another incredibly generic “CONNECTION_ERROR” exception gets thrown – completely abandoning the information gathered. But wait, there’s more! That exception then gets caught and logged and the error count gets incremented again! Yay! A “CONNECTION_ERROR” occurred… no more information… and an incorrect error count…

See? Don’t do it. If you can’t test for proper failures (sometimes it’s very hard), at least think about what’s going to happen.

6. Unused tracts of code

This one comes back to updating everything relevant when you have to revisit your code. A lot of the legacy code I’ve seen has codepaths that can’t possibly be used, methods that are never called, and in some cases classes, modules, and even entire libraries that aren’t referenced.

A recent project even had two code files, Connection.cs and Connection2.cs – they had different class names (Connection and Connection2 predictably), but they had the same methods with different implementations. Connection.cs was never used.

If you have to traverse your way through hundreds of lines of poorly-written code, it can get very frustrating when you find that half of it isn’t even used. It’s even more frustrating when you’re trying to match the behaviour you’re seeing with the code you’re reading, only to find out that it’s actually a completely different near-duplicate that’s running instead.

And that’s it – my top 6 annoying legacy code things. Feel free to comment.

My last post talked about how I wasn’t getting anywhere with my project.  Well now I am.

I might have mentioned that I’m using php and while it’s pretty easy to do stuff, I sometimes feel uncomfortable with the looseness of it all.  If you’ve used php in addition to a “real” OO language, you’ll know what I mean.  Lots and lots of globally-accessible functions with no namespaces, duck-typing, all that sort of stuff.

So I’m following some better practices (as suggested by Jon in his comment on the last post).  Nothing is very groundbreaking and I’m sure most newer php projects use the same kind of things.  Anyway, here’s what I’m doing:

I’m using an ORM library to get from my database to my code and vice versa.  Specifically Doctrine.  I’m reasonably familiar how these things work having used NHibernate for .Net (a port of Java’s Hibernate), but I’ve been impressed with how easy it’s been this time.  I got Doctrine working beautifully for my spider-web of a MySql database inside an evening with no experience.  I’ll let you know how it works out in the long run.  And yes, MySql is the database right now but Doctrine allows it to change very easily as expected.

I’m using an MVC pattern so my code doesn’t get too mixed in with my UI.  Now I’m sure there are plenty of libraries and frameworks out there that would help me with this, but I’m rolling my own… if you can call it that.  I’m used to .Net’s code-behind method so I’m really just copying that to an extent.  Each php page will have an include at the top that references the code-behind.  The “code-behind” php file will set up all the data and will provide functions that can be called by the front one.  Simple, but it should be effective.

Finally, I’m using JQuery for UI prettiness.  I had a serious look at a lot of javascript libraries and JQuery came out on top for what I needed.

So that’s it.  That’s what I’m doing.  So far it’s all fitting together quite nicely, but I’ll be sure to post updates if I want to rave about something or if something sends me into a fit of frustration.

As always, comments are welcome.

Damian

So I have this project that I’m doing for my own interest and I constantly find myself getting nowhere with it.  I’ve done plenty of design work and I have my schemas and stories sketched all over the pages of a notebook, but there’s nothing really tangible electronically yet.

The most common excuse I give myself is lack of time, but I don’t think that’s really true.  After all, I have tons of design work done and you’d think that’d be the boring part right?  I think the problem is lining up spare time, motivation, and resources.

Spare time exists in snippets everywhere.  When I’m on the bus, in a particularly boring lecture, at home after work, etc.  Motivation is more fleeting.  Sometimes I’m really keen to get into this project, other times it’s the furthest thing from my mind.  Finally, resources are either available or they’re not.  I haven’t had a laptop for a while, so I intended to develop mainly from home.  I’ve borrowed a laptop, but I don’t want to fill it with all my development tools.  Having said that, I’m doing it in php and I don’t need much of an IDE to cut code.  I could probably get some meaningful stuff done in notepad if need be.  Testing? That’s harder, particularly as I’d need a database running and a php web server.

So you see that even though I’m motivated more often than not, those three things don’t always line up well.  I don’t often have much motivation when I get home from work, even though I have time and resources.  When I’m sitting in a boring lecture, I’ve got plenty of motivation and time, but I don’t have the resources.  You get the picture.

So, how do I change this?  Well you’ll be pleased to know I have a plan:

1. I bought a laptop.  It’ll arrive in about a week.  It’s powerful and I can put whatever the hell I want on that thing so you can rest assured it’ll be running all the dev tools I want.
2. Portable Apps.  I rediscovered these when I was thinking about how I could improve the situation.  If a computer is available, I can spin up a full WAMP stack as well as GIMP, Notepad++, and FileZilla.
3. Target deadlines.  I have them.  And work breakdowns, heirarchical todo lists and all that stuff.  I’m organised.

So when I wrote that post yesterday, to my surprise I found three other draft posts of varying ages.  I know myself well enough to know that I probably won’t go back and revisit them, so here’s what they were about:

Salling Clicker

That’s all that was in this one – just the title.  I know what I was going to say though.  I was going to heap praise on this tiny but personally very useful app.

I have a beautiful (expensive) Sony LCD TV in my living room with a very handy VGA port.  My computer is right nearby so it’s permanently connected.  What Salling Clicker does is provide a way to control my computer from my mobile phone via Bluetooth or Wifi.  This is amazingly useful when watching movies from the couch.

The interface is solid and deployment is very easy.  Best of all, it’s cheap.

Understanding threading is more important than ever

This was a post I was going to write about a now old Ars Technica article.  The article was about the future of processors according to Intel where they suggested there’d be many many cores, not just two or four.  I didn’t get too far on my post:

We were taught threading in the very first programming subject I took at uni.  Initially, it can be difficult to get your head around.  Eventually I got there.  I was convinced though that plenty of people passed that course without any firm idea of how threading worked.

With the announcement from Intel that they’re heading down the track of many, many CPU cores, understanding threading is more important than ever for a budding software developer.

What I was going to get to was that developers will have to start thinking about parallel processing.  If we’re stuck on 3GHz per core and you want to do some crazy-complex stuff, it better be able to be spread across the cores or it will be S.L.O.W.

Will the compiler/OS/processor be able to handle this for you?  Well, yeah, possibly to an extent, but writing your software to take advantage of new processor capabilities will be a necessity for performance.

At the very least, you’ll have to have a detailed knowledge of how threading works.  It’s the future, man.

Religious Coding

The oldest draft of the lot was also the closest to being finished.  Here’s what I wrote, minus the half-written final sentence (seriously, I stopped halfway through a sentence):

A mate of mine working in the US sent me a link to a Mozilla Labs tool called Ubiquity.

Essentially, the idea behind it is to let users describe what they want to do with the Internet rather than where they want to go.  The web at the moment is really based around sites that you have to actually visit to be able to use the information.  Their idea is to skip this visiting stuff and let the tool (or commands written for it) do all the work for you, giving you only the information you were after.  They have a really good description on their blog post (linked above), and I can’t really think of a better way to describe it than them (which is probably good), so have a read if you’re interested.

It’s really lightweight and very easy to install, but at the moment it only works for Firefox.  It’s also surprisingly easy to write new commands.  I’ve been a .Net kid for a while now but it only took me a couple of hours to hack up a new command.  Admittedly the Ubiquity language is Javascript so it’s not terribly unfamiliar.  Still, that’s a pretty short learning curve.

Commands have the ability to give you a realtime ajax preview before you actually execute the command.  In most cases, I don’t even use the execute functionality and I suspect that for the small things, most people won’t.

Anyway, when I was trying it, the built-in “define” command wasn’t working for me.  It would look up a word when you pressed enter to execute the command, but it didn’t give me a preview.  So, in order to get a nicer dictionary lookup and at the same time try out this funky little thing, I decided to create my own.  Of course once I’d got mine working, the define command started working properly.  I like mine better – it gives prettier results.  It’s not just because I made it myself… but that’s the main reason.

You can go here to have a look at it.  Subscribe to it if you want to use it or hey, just steal the code or whatever, I don’t care, it’s public license – do what you want.

So in summary, it’s easy to use and I think it has the potential to be something really big.  If you have a look at the examples of what they’re ultimately trying to be able to do, it’s pretty impressive.  Imagine typing “find restaurants near me with reviews over 4 stars” and being given a google map with markers and summarised reviews of each.  Very useful, no?

Hmm, combine that with voice recognition and you’ve got a genuine futuristic computer from the movies!  Rad.

So I haven’t written a post in a while, but no, that’s not the reason for the title.  I’ve just been concentrating on other things and haven’t felt sufficiently motivated to write about anything.

But then I read this post from Steve Yegge on Stevey’s Blog Rants.  His title was “Business Requirements are Bullshit” which, while clearly designed to catch the eye, doesn’t entirely represent what it was about.

Steve’s post was aimed at people developing a new (or better) product to take to market.  He wasn’t talking to consultants or employees who are producing something to meet a specific company’s business needs, but someone who was creating something new to fill a perceived hole in the market.

His point (adapted from Warren Buffet) was that you should build something you already know about; something that actually meets your needs.  If you’re doing that, then you already know what you want.  You know what compromises you can make and what the unspoken and tacit deal-breakers are.  If you’re trying to gather business requirements from a group of people who may or may not want the product while trying to understand what it is you’re actually making, then you’re probably going to fail.

It sounds like great advice, but right now, I’m not in the category of people building something new to take to market.  I’m in the other group.  The stuff I build and maintain (and now I’m going to slip back into software) is supposed to meet an immediate business need for a specific client.  It more than likely won’t be used by me, but I have to build it anyway.

So can Steve’s advice be applied to my situation?  Sure it can, to an extent.

Steve’s main point was that if you’re not investing in something you understand, then you’re walking into very dangerous territory.  As an end-to-end software developer, I’m aware that it’s difficult to know exactly what the customer wants.  Sure, you can grill every potential user for days, you can write a comprehensive list of requirements, you’ll check it and recheck it over and over again to make sure you know single possible piece of functionality that they want and need.  But when it comes to the crunch, no matter how much work you’ve done, you’ll start getting negative feedback about some specific things that you hadn’t thought of and the customer hadn’t mentioned.

That complex report you were asked to include, the one with all the tables and forecasts and things?  It turns out that 10 different people print that 20 times a day.  So even though it’s perfect, it takes 15 minutes to run each time, and they can’t wait that long each time.

So how do you avoid this? In my experience, if you want to write truly useful software, you need to understand why each piece of functionality is being written.  Spend a lot of time with the clients and find out what it’s like to be in their shoes.  If you see how they operate day-to-day you’ll start to get an idea of what they actually want, not what’s written in the requirements doc.

Steve’s advice was that you shouldn’t invest in what you don’t understand.  So if you have to produce something you don’t understand, make a genuine effort to understand it first.  You might not be as enthusiastic about or deeply involved in the business your client is in.  But by honestly trying to see what they’re trying to achieve, you’ll learn what they really want your software to do for them; over and above the list of required functions.

I stumbled across Part 6 of a Programming Job Interview Challenge on the Dev102 blog and thought, “hey, I know this!” I couldn’t help responding.

So the question was, given this bit of code, what will the output be?

   1: ArrayList a = new ArrayList();

   2: ArrayList b = new ArrayList();

   3:

   4: a.Add(1);

   5: b.Add(1);

   6: a.Add(2);

   7: b.Add(2.0);

   8:

   9: Console.WriteLine((a[0] == b[0]));

  10: Console.WriteLine((a[1] == b[1]));

Ok, so I’ll break the post here in case you want to work it out for yourself, but you should really go to the original post – not least because there are five other challenges there.

Read more »

It’s great when software is designed and implemented with careful planning and plenty of documentation so someone can pick it up when it falls down.  Unfortunately, it’s often the case that a) You’re working on an existing project that has no documentation, or b) You’re working on an existing project that has documentation that is now out of date.

I’m doing the former right now.  The software is old.  Like 12 years old.  It has no documentation.  It has duplicated code, code where it shouldn’t be, hard-coded values, code that does the wrong thing before being corrected by other code.  Think of an anti-pattern and it’s in there.

The application is being phased out, but it must be supported until it’s completely gone.  It’s also in an environment where changes are inevitable and legally mandated so there’s no chance of a complete code-freeze.  The original programmers are long gone.  It’s not fun to work with.

Now don’t get me wrong.  I don’t go in and make changes that just add to the spaghetti – I refactor when it’s practical.  If I’m changing some code that’s repeated in three places, I’ll pull it out into its own method.  If there are hard-coded values that could change, I’ll set them up so they’re configurable.  In short, I’ll do some cleanup, but I’m not going to do it all.  Refactoring everything would mean nearly rewriting the thing from scratch, and that’s been done (hence the phase out).

So what about documentation?  Writing detailed UML for this software is just not practical.  For one thing, it’d be obscenely hard to do, but more important is the fact that this program is unlikely to be in use in 12 months time.  Priority doesn’t go to documenting legacy software.  The fact of the matter is that it’s just not worth the time and effort to pull the thing apart to work out how every little piece ticks.

So I’ve come up with my own alternative.  I call it Post-traumatic Documentation.

I’ve set up a document that I’ve titled the Application Body of Knowledge.  Every time I dive in to make a change or fix a bug, I’ll absorb a whole lot of information about how the application fits together.  There are basic things like which files do what, but most of the important stuff is in the details.  I might spend half an hour working out that some assignments in method A get overridden in certain cases when method B is called, or that a class called ABC actually provides functionality for DEF and perhaps doesn’t get called at all.  I learn these things the hard way and I make notes as I go.

Whenever I finish one of these traumatic quests to change something small, I go through my notes and add anything of interest to my Application Body of Knowledge.  There’s a section dealing with the overall architecture of the application, one for each of the main modules, one for deployment, and one for general notes.  It’s deliberately heavy on keywords and jargon to facilitate searching.

This document is obviously far from complete, but it is immensely helpful to me when I next have to make a change, and it’ll be even more helpful to the guy who takes the reigns from me when I get hit by that programmer-killing bus.

Damo

Two of my favourite blogs have announced in the last few hours the upcoming launch of www.stackoverflow.com and I, for one, am excited.

The site will be a combined effort by two of the legends in the software design world, Jeff Atwood and Joel Spolsky.  Jeff runs the well-known Coding Horror blog, and Joel is chief guy for the Joel on Software blog as well as CEO of Fog Creek.  They’ve decided to combine forces to start a community site that’s essentially a free programming Q&A site.

Every programmer knows the recurrent problems encountered when searching for a programming dilemma in Google with the hopes of finding an answer.  There are a couple of problems that haunt me constantly.  Joel mentions in his blog post the situation where you find your exact problem and potential answers on sites that require registration and payment.  Experts Exchange is a classic for that; so much so that I generally use some google-fu to remove any Experts Exchange pages from my search results when searching for solutions.  Don’t get me wrong, I’m sure Experts Exchange has some fantastic content, but I can’t justify spending that much money to save me a few hours every few months.

The other problem I encounter time and time again are the forum posts with no solutions.  Often, when I search for a specific problem in Google, I’ll get a ton of results that appear to exactly match my particular issue.  Excitedly, I’ll open them all up in new tabs and go through them looking for the bit of code that will be my saviour.  Far too often, the threads don’t provide answers at all – just a community of people lamenting the same fault and looking for cures.  Even more frustrating are the instances when the initial poster finishes up the thread with a post along the lines of, “Thanks everybody, but I found out how to do it” without providing the solution.

Hopefully, this site will hasten a timely death for these frustrations.  If there’s anyone I’d want on the case, it’s these two guys.

Damo