Yesterday I wrote about the difficulty of replacing existing systems, the challenges of meshing waterfall and agile (with reference to a currently running project) and proposed some options that could help move work forward. There is, though, one big caveat.

Some legacy systems are purely “of the moment” – they process a transaction and then, apart from for reporting or audit reasons, forget about it and move on to the next transaction.

But some, perhaps the majority, need to keep hold of that transaction and carry out actions far into the future. For instance:

– A student loan survives over 30 years (unless paid back early). The system needs to know the policy conditions under which that loan was made (interest rate, repayment terms, amount paid back to date, balance outstanding etc)

– Payments made to a farmer under Environmental Stewardship rules can extend up to a decade – the system retains what work has been agreed, how much will be paid (and when) and what the inspection regime looks like

In the latter case, the system that handles these payments (originally for Defra, then for Natural England and now, I believe, for the RPA) is called Genesis. It had a troubled existence but as of 2008 was working very well. The rules for the schemes that the system supports are set every 7 years by the EU; they are complicated and whilst there is early sight of the kind of changes that will be made, the final rules, and the precise implementation of them, only become clear close to the launch date.

Some years ago, in the run up to the next 7 year review, GDS took on the task, working with the RPA, of replacing Genesis by bundling it with the other (far larger in aggregate, but simpler in rules and shorter in duration) payments made by the RPA. As a result, Defra took the costs of running Genesis out of its budget from the new launch date (again, set by the EU and planned years in advance). Those with a long memory will remember how the launch of the RPA schemes, in the mid-2000s, went horribly wrong with many delays and a large fine levied by the EU on the UK.

The trouble was, the plan was to provide for the new rules. Not the old ones. An agreement could be made with a farmer a week before the new rules were in place, and that agreement would survive for 10 years – and so the new system would have to inherit the old agreements and keep paying. Well, new agreements could have been stopped ahead of a transition to the new system you might say. And, sure, that’s right – but an agreement made a year before would still have 9 years to go; one made 2 years before would have 8 years to go. On being told about this, GDS stripped out the Genesis functionality from the scope of the new system, and so Genesis continues to run, processing new agreements, also with 10 year lives … and one day it will have to be replaced, by which time it will be knocking on 20 years old.

Those with good memories will also know that the new system also had its troubles, with many of the vaunted improvements not working, payments delayed, and manual processes put in place to compensate. And, of course, Defra is carrying the running costs of the old system as well as the new one, and not getting quite the anticipated benefits.

IT is hard. Always has been. It’s just that the stakes are often higher now.

When replacing legacy systems where the transactions have a long life, sometimes there is a pure data migration (as there might be, say, for people arriving in the UK where what’s important is the data describing the route that they took, their personal details and any observations – all of which could be moved from an old system to a new system and read by that new system, even if it collected additional data or carried out differnt processing from the old system). But sometimes, as described above, there’s a need for the new system to inherit historic transactions – not just the data, but the rules and the process(es) by which those transactions are administered.

My sense is that this is the one of the two main reasons why legacy systems survive (the other, by the by, is the tangled, even Gordian, knot of data exchanges, interfaces and connections to other systems).

There are still options, but none are easy:

– Can the new system be made flexible enough to handle old rules and new rules, without compromising the benefits that would accure from having a completely new system and new processes?

– Can the transactions be migrated and adjusted to reflect the new rules, without breaching legal (or other) obligations?

– Can the old system be maintained, and held static, with the portfolio of transactions it contains run down, perhaps with an acceleration from making new agreements with individuals or businesses under the new rules? This might involve “buying” people out of the old contracts, a little like those who choose to swap their defined benefit pension for a defined contribution deal, in return for a lump sum.

– Can a new version of the old system be created, in a modern way, that will allow it to run much more cheaply, perhaps on modern infrastructure, but also with modern code? This could help shave costs from the original system and keep it alive long enough for a safe transition to happen.

Some of these will work; some won’t. The important thing is to be eyes open about what you are trying to replace and recognise that when you reach from the front end into the back end, things get much harder and you forget that at your peril. Put another way, Discovery is not just about how it should be, but about how it was and how it is … so you can be sure you’re not missing anything.

Over the last year I’ve worked in various different places, public and private, reviewing projects.

The most common line I’ve distilled from dozens of conversations is

“We are X years into our Y years transformation”

Which is greater, X or Y?

Obviously, X should be smaller than Y. Eg we are 1 year into our 3 year programme to digitise (whatever we are digitising).

Sadly, the opposite has been true in every case. We are 2 years into our 1 year plan to make change happen. Or even 3 years into our 1 year plan to turn the world on its head.

Things take longer than expected. Invalid assumptions are made. The specific outweighs the general. Every time.

In early 2001 I presented to an audience of civil servants from the Inland Revenue, 300 or so grade 5s up to the Permanent Secretary (grade 5 is old money for what is now SCS 1).

The idea was to walk them through the changes that the Internet was bringing – and, particularly, what it would mean for the way they engaged with taxpayers (oh the debates that were had about customer versus taxpayer versus citizen versus many other words).

At the time Self Assessment was the only online transaction available from the IR’s website – we’d put that online in April 2000, though the application itself came on a floppy disc (we moved it to a truly online app in time for the 2001 tax year). Those were, indeed, the days.

I’ve pulled a couple of the slides out

Cell-phones as the next generation access tool? Who would have thought.

The conversation I wanted to have, mainly, was around the change in the customer experience that would come about from moving online – a shorter, more interactive, more reponsive process that had the power to change everything about the way tax was collected (timing, amounts, compliance etc)

Today, many would call this “digital transformation.” But, it’s not. It’s streamlining the process – getting rid of redundant steps, using technology to replace paper, speeding the turnaround time, providing feedback etc. The customer experience is certainly transformed; now a taxpayer / citizen / user / customer can reliably send their tax return in 1 minute before the annual deadline and know that it has been received safely. The numbers of people who send their tax returns in on Christmas Day was surpisingly high, from day one of putting this service online. Much of what we call “digital transformation” is like this – getting a paper process online and making it easier to use isn’t transformational at the macro level, although it can be in terms of customer experience. Do it when you want to, not when we want you to.

There are constraints of course – the Internet was new for many, but expectations were quickly moving much higher:

Looking at that all these years later, I wonder if I got hierarchy and capability the wrong way round, or whether I was making a different point. I think the former.

But we had a plan

True transformation needs a fundamental re-engineering. It changes not the process but the outcomes, removes constraints and affects people, the economy and the government fundamentally.

There are precious few examples of Real Transformation (TM), except where people throw the T word around to gloss over a lack of innovation, or lack of new thinking and just want to pretend that the planned changes are a big deal.