Your system will fail

Two jobs ago (a lifetime it seems!) I worked with this great programmer who came from a job with the military. The Air Force to be exact. His job was to maintain a system that managed inventory for parts for aircraft. Seems like a simple system, but when he got into the depth of the inventory it blew my mind. They tracked everything down to each nut, each bolt, exactly where each part was on the aircraft, it was amazing the detail they had to keep track of.

The main reason for all this detail was the massive amount of redundancy in the construction. Basically if you’re flying into enemy territory and start taking fire you don’t want a bullet or shell to hit the wire that links the joystick to the wings. Or any wire for that matter (turns out they’re all pretty important) . So you put the wire in a place that has the lowest percentage of being hit.

But there’s still a chance (even small) it could get disabled. And when millions of dollars of aircraft and at least one human life are at stake, you want to get as close to 0% as possible. So you add in another wire that does the same job in a different place. Just in case.

But there’s still a chance right? So you add another wire in another spot.

The end of the story is, there are several redundant wires, nuts, bolts, hardware, software, everything. All in different spots. All to back up the other. All to make sure that if one fails the plane doesn’t fall out of the sky.

And why not? If one little wire is the difference between life and death, and there’s room and time and money to put it in, why not?

Luckily Twitter does not work for the military

A less dire example (and the point of this rant) is Twitter. For me the twitter service is sketchy at best. I set it up to get the tweets from the people I want to hear the most directly on my phone. The rest I monitor through the web or rss.

It rarely works 100%.

Case in point, this morning I got an email from twitter about a direct message from someone I follow. It was about some potential work. Even though I have my account set up to send his tweets to my phone I never got the message that way. I went to my feed online (and rss) but since it was a direct message I didn’t see the notification there. Luckily I’m a big fan of redundancy so I have direct messages sent to my email. Crisis averted. And luckily so because the work that came from it looks to be a good gig, and has tons of upside for a future working relationship.

The question is, does your web site or application have that redundancy? If someone is trying to contact you, do you have fail safes in case one way doesn’t work? Will it lose you money if it fails? Do you give your users the opportunity to be paranoid, sometimes justly?

If not, maybe now is a good time to check out your processes and add in that extra wire.