Building New Legacy

How do we know the systems we are building today aren’t tomorrow’s legacy? Are we consciously working to ensure that the code we write isn’t spaghetti-like? that interfaces can be easily disassembled? that modules of capability can be unplugged and replaced by other, newer and richer ones?

I’ve seen some examples recently that show this isn’t always the case. One organisation, barely five years old, has already found that its architecture is wholly unsuitable for its current business looks, let alone what it will need to look like as its industry goes through some big changes.

Sometimes this is the result of moving too quickly – the opportunity that the business plan said needs to be exploited is there right now and so first mover advantage theory says that you have to be there now. Any problems can be fixed later goes the thinking. Except they can’t, because once the strings and tin cans are in place, there are new opportunities to exploit. There’s just no time to fix the underlying flaws, so they’re built on, with sedimentary layer after layer of new, often equally flawed, technology.

Is the choice, then, to move more slowly? To not get there first? Sometimes that doesn’t help either – move too slowly and costs go up whilst revenues don’t begin soon enough to offset those losses. Taking too long means competitors exploit the opportunity you were after – sure they may be stacking up issues for themselves later, but maybe they have engineered their capability better, or maybe they’re going so fast they don’t know what issues they’re setting up.

There’s no easy answer. Just as there never is. The challenge is how you maintain a clear vision of capability that will support today’s known business need as well as tomorrow’s.

How you disaggregate capability and tie systems together is important too. The bigger the system and the more capability you wrap into it, the harder it will be to disentangle.

Alongside this, the fewer controls you put around the data that enters the system (including formats, error checking, recency tests etc), the harder it will be to administer the system – and to transfer the data to any new capability.

Sometimes you have to look at what’s in front of you and realIse that “you can’t get there from here”, and slow down the burn and figure out how you start again, whilst keeping everything going in the old world.

Balancing Capex and Opex

Government has a cash problem. It simply doesn’t have enough cash allocated to running costs for IT. Projects that were traditionally funded out of capital are, in the cloud world, funded out of operating budgets. This is going to hurt.

For many years, IT projects have been funded by capex (capital expenditure). Whatever came out of the project – servers, software licences, code, automation tools etc – sat on the balance sheet and was depreciated over an agreed period. Usually, for software, it was thought to be too long a period, but given that many of our systems are still working 20, 30 or even 40 years after launch, and so long since depreciated to zero, we clearly under-estimated the longevity of code. Similarly, we probably over-estimated the life of laptops and mobile phones where 5-7 years depreciation is common, but they have quickly become replaceable after 2 or maybe 3 years.

With the move to cloud, the entire infrastructure base switch from capex to opex – that is, it’s funded out of day to day expenses and nothing is held on the balance sheet. Millions of pounds of servers (and all the switches, routers and other kit associated with them, as well as some software, where SaaS products are used) left the balance sheet.

Governments tend to be capital rich – there are few departments who complain about not having enough capex. Capex buys actual things – in IT terms, servers with flashing lights and spinning disks that can be looked at, making the spend tangible (hence the use of tangible and intangible assets for different kinds of IT assets).

This has created a challenge for some departments who want to spend their capital, but also want to move to the cloud. There was a similar challenge early in the cloud era when VAT was not recoverable, putting further pressure on strained opex budgets.

I’m seeing a change though, now, where even software development is run as an opex project – on the basis that the code is expected to turn over rapidly and be replaced through an iterative agile approach. If a project goes wrong – at a micro or macro level – there’s no write-off (which can be important to some). At the same time, treating everything as opex means that, in some cases, there’s a building soon to be legacy code base (becuase it’s a fallacy to think that this code is iterated and replaced regularly) that is going unmaintained, meaning that there’s ever more spaghetti code that isn’t being looked at or tweaked. Knowledge of that code base is held by a smaller and smaller set of people … and changes to it become more difficult as a result.

It’s a strange move – one that perhaps implies that there is less scrutiny over opex spend, or that the systems being built will not be in use for the long term and so don’t quite count as assets. But IT systems have a habit of surprising us and sticking around for far longer than expected – ask the developers, if you can find them, of the big systems that pay benefits, collect tax, monitor imports, check passports at border etc what the expected life of their system was when they built it and the answer will never (ever) be “oh, decades.”

That’s not to say that there isn’t a case for classifying some IT spend as opex. If you are a fast moving startup building products for a new market and striving to reach product/market fit, you might be crazy to think that it was worth having IT on the balance sheet. If you know that you are building a prototype and will throw it away in a few weeks or months, it would, again, be crazy to capitalise it. If you’re doing R&D work and you’re not sure what will come out of it, you might well classify it as opex initially and revisit later to see if assets were created and then re-classify it.

I suspect that the tensions between capex and opex in government still have more room to play out

Royal Parks Half Marathon

Easily the most well organised race in the London area (I’ve run most of them from 10k to full marathon), the Royal Parks Half keeps, ahem, knocking it out of the park. The start area today could have substituted for a Tough Mudder, but that didn’t dampen anyone’s enthusiasm.

I ran the inaugural race in 2008 and have only missed a couple since, collecting my 10th race t-shirt today. I don’t run it quite as fast as I used to but it’s still the most enjoyable race on my calendar.

I hope to return next year in better shape, for my 11th shirt. Kudos to the team who put this awesome event together each year. You rock!

Sub 2 hour Marathon

Eliud Kipchoge’s 1:59:40 marathon, run in the beautiful city of Vienna (my home for a while long ago) is an astonishing run. Before the race Kipchoge said “it’s like the first man to go to the moon … this is about history … it’s about leaving a legacy.”

He went on to say “I want to show the world that when you trust in something and have faith in what you are doing, you will achieve it, whether you’re a runner, a teacher or a lawyer.”

Inspirational stuff. This is a man who rarely loses – he’s won 11 out of 12 marathons and has the two fastest official times on record (the current best in a race is 2:01:39, though he ran 2:00:26 in Monza with a similar setup to the Vienna attempt).

I’ve crossed a few marathon finishing lines around the world myself. Just finishing is a huge buzz. Being the first to finish in under 2 hours is just extraordinary.

Those who have run the London Marathon in a “typical” time (let’s call that a little under 4 hours, which is what roughly 20% of people achieve) will likely know that moment when they turn right having crossed Tower Bridge, at roughly 10 miles, and see “the elites” coming the other way, having long since left Docklands behind, as they hit 22 miles or so. At that point there are usually still half a dozen in the lead group and a few scattered behind, all running with metronomic precision. It’s simultaneously uplifting – up close and personal with incredible athletes – and soul destroying, as you realise they’re near the end and you still have 16 miles to go. Were it not for the amazing crowds there, some might be tempted to throw in the towel.

At the London 2012 Olympics and, at many other London marathons, I’ve sat in a stand at the finish line watching everyone come home. It’s a thrill to watch the great athletes, but it’s just as big a rush to see everyone else complete the race, some struggling as they come round the bend by Buckingham Palace, and most picking up speed as they see the finish line. It is, after all, a race.

Kipchoge is first. Others will doubtless follow. Perhaps soon, but perhaps not. Sub 2 in race conditions is the next extraordinary marathon achievement, but it feels like a stretch from here.

Me, I’m happy finishing a half marathon in less than 2 hours these days.

Dyson and EVs

Dyson announced yesterday that it’s planned foray into Electric Vehicles was no longer commercially viable. The original investment was expected to be £2-2.5bn.

EVs are no more for Dyson, it seems, but it’s still going to press ahead with its investment in batteries, particularly solid state varieties. If I recall correctly, about half of the original investment was for the car itself and half for batteries – so this is still an investmnet of over £1bn in new capabilities (those batteries could still supply cars for other manufacturers of course, or could be solely for Dyson’s current and any future products).

Why decide this now?

  • Subsidies for EVs have been reduced across the world (recently in the UK but, particularly in China, the largest current market). Do Dyson see fewer buyers, despite the near universal commitments governments have made to take petrol and diesel cars off the road over the next 2-3 decades? That seems unlikely.
  • Is a £2.5bn investment not enough? Tesla has consumed more than £14bn to date, and not made a profit. VW is expecting to spend $33bn over the netx few years. Joint ventures, mergers and deals between car manufacturers, “taxi” firms (such as Uber and Lyft) are proceeding at a pace. Perhaps Dyson couldn’t keep up with the expected spend rate?
  • Cars are hard. Are they too hard for a new entrant? It’s a volume game, for most companies at least. 87m cars were sold globally in 2018. Tesla is shipping only (I say “only” in a relative sense) 100,000/quarter (or thereabouts, with ambitions for more). Perhaps Dyson think it’s not commercially viable in the short term – that the hockey stick of demand is too far away for investment now?
  • Did they pick the wrong place? Singapore is not known for its car manufacturing. It’s an expensive, though stable, location to pick. Maybe it was the wrong place to setup a cost effective and efficient factory?

It could, and probably is, all of these. The ROI profile must look daunting, especially in an uncertain market where realistic large scale take up is a decade away (at the mass market, global level – the point of pickup in volumes is perhaps 3-5 years away).

Perhaps the most important point is Dyson went all in, quickly. And then all out, quickly. He saw an opportunity and funded it, but then saw that it wasn’t going to pan out as planned, and so pulled the funding. J

This strikes me as the mind of an instute businessman, used to carrying out experiments, working at full capacity

– He saw an opportunity and allocated investment capital to it. We don’t know how much of the £2.5bn has been spent, or is wasted effort, or can’t be unwound of course.

– Early in the experiment it was clear that there were significant hurdles, e.g.

  • Car safety, with people travelling at 70mph+, comes with different regulations than his existing portfolio, for instance)
  • Huge competition (other well capitalised companies investing much greater amounts)
  • Long payback (governments cutting subsidies, adoption still slow, proposals to halt sales of petrol and diesel still two decades away)

And so he reviewed his investment/experiment and decided it would take more time and money than he wanted to commit. Could he have figured that out by commissioning studies to assess feasibility and so on? Sure he could, and maybe he did and that’s why it’s been cancelled, but the way of the engineer is to start building it and see whether it will work, and it sounds to me that that’s how he approached it.

Just as I’ve written here before, projects fail. They fail all the time. The trick is to see the failure coming, be sure it’s really failing and can’t be picked up, and that it’s not failing becuase of an elementary error (that is, you have learned the lessons that others learned before you), and then address that.

It’s a bold decision. And his investments live to make a return another day. Does this mean anything for the overall move to EVs though? Seems unlikely – it’s evidence that new entrants will struggle and reinforces why partnerships are increasingly the thing both within the existing motor industry and for those trying to break in.

The Legacy Replacement Caveat

Yesterday I wrote about the difficulty of replacing existing systems, the challenges of meshing waterfall and agile (with reference to a currently running project) and proposed some options that could help move work forward. There is, though, one big caveat.

Some legacy systems are purely “of the moment” – they process a transaction and then, apart from for reporting or audit reasons, forget about it and move on to the next transaction.

But some, perhaps the majority, need to keep hold of that transaction and carry out actions far into the future. For instance:

– A student loan survives over 30 years (unless paid back early). The system needs to know the policy conditions under which that loan was made (interest rate, repayment terms, amount paid back to date, balance outstanding etc)

– Payments made to a farmer under Environmental Stewardship rules can extend up to a decade – the system retains what work has been agreed, how much will be paid (and when) and what the inspection regime looks like

In the latter case, the system that handles these payments (originally for Defra, then for Natural England and now, I believe, for the RPA) is called Genesis. It had a troubled existence but as of 2008 was working very well. The rules for the schemes that the system supports are set every 7 years by the EU; they are complicated and whilst there is early sight of the kind of changes that will be made, the final rules, and the precise implementation of them, only become clear close to the launch date.

Some years ago, in the run up to the next 7 year review, GDS took on the task, working with the RPA, of replacing Genesis by bundling it with the other (far larger in aggregate, but simpler in rules and shorter in duration) payments made by the RPA. As a result, Defra took the costs of running Genesis out of its budget from the new launch date (again, set by the EU and planned years in advance). Those with a long memory will remember how the launch of the RPA schemes, in the mid-2000s, went horribly wrong with many delays and a large fine levied by the EU on the UK.

The trouble was, the plan was to provide for the new rules. Not the old ones. An agreement could be made with a farmer a week before the new rules were in place, and that agreement would survive for 10 years – and so the new system would have to inherit the old agreements and keep paying. Well, new agreements could have been stopped ahead of a transition to the new system you might say. And, sure, that’s right – but an agreement made a year before would still have 9 years to go; one made 2 years before would have 8 years to go. On being told about this, GDS stripped out the Genesis functionality from the scope of the new system, and so Genesis continues to run, processing new agreements, also with 10 year lives … and one day it will have to be replaced, by which time it will be knocking on 20 years old.

Those with good memories will also know that the new system also had its troubles, with many of the vaunted improvements not working, payments delayed, and manual processes put in place to compensate. And, of course, Defra is carrying the running costs of the old system as well as the new one, and not getting quite the anticipated benefits.

IT is hard. Always has been. It’s just that the stakes are often higher now.

When replacing legacy systems where the transactions have a long life, sometimes there is a pure data migration (as there might be, say, for people arriving in the UK where what’s important is the data describing the route that they took, their personal details and any observations – all of which could be moved from an old system to a new system and read by that new system, even if it collected additional data or carried out differnt processing from the old system). But sometimes, as described above, there’s a need for the new system to inherit historic transactions – not just the data, but the rules and the process(es) by which those transactions are administered.

My sense is that this is the one of the two main reasons why legacy systems survive (the other, by the by, is the tangled, even Gordian, knot of data exchanges, interfaces and connections to other systems).

There are still options, but none are easy:

– Can the new system be made flexible enough to handle old rules and new rules, without compromising the benefits that would accure from having a completely new system and new processes?

– Can the transactions be migrated and adjusted to reflect the new rules, without breaching legal (or other) obligations?

– Can the old system be maintained, and held static, with the portfolio of transactions it contains run down, perhaps with an acceleration from making new agreements with individuals or businesses under the new rules? This might involve “buying” people out of the old contracts, a little like those who choose to swap their defined benefit pension for a defined contribution deal, in return for a lump sum.

– Can a new version of the old system be created, in a modern way, that will allow it to run much more cheaply, perhaps on modern infrastructure, but also with modern code? This could help shave costs from the original system and keep it alive long enough for a safe transition to happen.

Some of these will work; some won’t. The important thing is to be eyes open about what you are trying to replace and recognise that when you reach from the front end into the back end, things get much harder and you forget that at your peril. Put another way, Discovery is not just about how it should be, but about how it was and how it is … so you can be sure you’re not missing anything.

Agile Meets Waterfall

This morning I’ve been looking at the release schedule for a major project launching next year. Chances are that you will be affected by it somehow. No names disclosed though. It’s a replacement for an existing capability and will use an off the shelf, quasi-cloud product (that is, software that is already in use by others and so can, in theory, be configured but that, in practice, will need some development to make it do what is really needed).

The top level project plan looks like this

Spring 2019Discovery
Summer 2019Design and Build
Spring 2020Testing
Summer 2020Go-live
Spring 2021Transition completed
ThereafterContinuous improvement

Plainly the team want to be agile. They want to iterate, adding features and capabilities and seeing how they land before improving them – going round the discovery / alpha / beta / live process repeatedly. But, at the same time, they have a major existing system to replace and they can’t replace just part of that.

They also, I suspect, know that it’s going to be harder than everyone expects, which is why they’re sticking to those nebulous seasonal timeframes rather than talking about particular months, let alone actual dates. The rhythmic cadence of Spring/Summer milestones perhaps suggests an entirely made up plan.

Ever since I first joined the Inland Revenue and heard a team saying that they “hoped to be up to speed by the Autumn” I’ve laughed whenever seasons are mentioned in the context of delivery. It’s as if there are special watches issued with 4 zones marked on them – one for each season.

What do you do when meshing the need to replace an existing widely used system that is no longer doing what it needs to do and that needs radical upgrades, with a new capability that can’t be introduced a bit at a time. This is not a start up launching a new product into the market. They’re not Monzo who were able to start with a pre-paid debit card before moving into current accounts, credit cards etc.

These kinds of projects present the real challenge in government, and in any large organisation:

How do you replace your ageing systems with new ones whilst not losing (too much) capability, keeping everyone happy and not making an enormous mistake?

At the same time, how do you build a big system with lots of capability without it going massively over-budget and falling months or years behind plan?

There are some options to consider:

  • Is a NewCo viable? Can a new capability be set up in parallel with the old, and customers, users or businesses migrated to that new capability, recognising that it doesn’t do everything. This is ideal MVP territory – how much do we need to have to satisfy a given group of customers?Ideally have less complicated needs than the full set of customers, and we can shore up the NewCo with some manual or semi-automated processes, or even by reaching into the old system
  • Can we connect old and new together and gradually take components away from the old system, building them in the new world? This might particularly work where a new front end can be built that presents data far more clearly and adds in data from other sources. It might require some re-engineering of the old system which perhaps isn’t used to presenting data via APIs and has never been described as loosely coupled.
  • Is there standalone capability that will make for a better experience, perhaps using data from the old system (which can be extracted in lots of different ways from cumbersome through to smooth), with that new capability gradually expanded?

None are easy, of course, but they are all likely better than a big bang or the risk of a lengthy project that just gets longer and more expensive – especially as scope increases with time (people who have time to think about what they want will think of more things, and industry will evolve around them giving them more things to think about).

There are, importantly, two fundamental questions underpinning all of these options (as well as any others you might come up with):

1. What do we do today that we don’t need to do tomorrow. Government systems are full of approaches to edge cases. They are sedimentary in nature – built over the years (and decades) with layer after layer of policy change. Today’s system, and all of the capability it has, is not the same as what is needed. This allows a jackhammer to be taken to the sedimentary layers so that the new capability isn’t trying to replicate everything that went before.

2. What do we need to do tomorrow that we don’t do today. This gets the policy arm the chance to shape new capabilities, including simpler policies that could have fewer edge cases, but still cater for everyone. It will also allow new thinking about what the right way to do something is, considering everything that has gone before but also what everyone else is doing (in other industries, other government departments and other countries).

Asking those questions, and working the answers really hard, will help ensure that the solution reflects the true, current need, and will also help get everyone off the page of “we need to replace what we have” when, in reality, that’s the last thing that anyone should be trying to do.

There is, though, one giant caveat to all of this which I will take a look at tomorrow.

The Race To 5G

Jonathan Margolis, the FT’s erudite technology editor, claimed in the How to Spend It magazine this weekend that “one of the surprises about the iPhone 11 … was that it does not have 5G” and then went on to laud the recently launched Samsung Galaxy S10 5G.

I found that an odd view. Indeed I’m surprised that he thought that it was a surprise. Apple has rarely added network features until they were widely available. That’s both because there needs to be a viable market for a service (if I turn on my expensive 5G phone and don’t see, immediately, a 5G symbol in the top left or right, I will surely feel let down and perhaps worse) and also because Apple will want to be sure components that provide the service are robust, reliable and ready for delivery to as many as 200m phone buyers a year.

Every few years we go through the xG hype cycle. I was closely involved in the 4G rollout some years ago and, despite best efforts, reality was far behind PR for many of the network operators (arguably only EE got delivery done in line with their PR and that was because of some clever reuse of spectrum).

There are plenty of 5G challenges ahead starting with a lack of agreed standards. This, in turn, means risk in buying components which may not be entirely compatible with the final versions. It also means that components are not yet optimised – they are much larger than their 4G equivalents, run much hotter, and take up valuable space that is needed for batteries (which, given those same components will be shorter). And, besides, we know that heat and batteries don’t mix.

There are other challenges ahead for 5G, particularly for operators. Network equipment manufacturing is in the hands of only a few suppliers, installation is a lengthy and cumbersome visit involving many 3rd parties and a dozen visits to each mast, the spectrum available is at a far greater mix of frequencies than previous rollouts (which means modelling and planning are more complicated and that there will be a need for infill masts, especially in highly populated areas) and so on.

The truth is that 5G is still some years away, at a general population level. There will be exceptions – specific use cases – of course. But for most of us, it is 2-3 years away.

So, no, it’s no surprise that Apple hasn’t shipped 5G compatible iPhones. It’s also no surprise that other companies have – some like to be first, some like to claim “me as well”, some want to trial new things to differentiate otherwise lacklustre offerings, and some want to get a sense of how the technology works in the field. This is what makes a market. The early adopters adopt. Some wait a while. Some wait much longer.

Cornish Lithium

A while ago I looked at the ingredients the UK has to support a successful EV design and build industry. I noted that whilst we were not Australia or Chile, the largest sources of lithium for batteries, we did have some in Cornwall.

I read recently that Cornish Lithium, the aptly named company with mining rights over much of the county, has recently raised £1.4m so that it can begin drilling at test sites. Doesn’t sound like a lot of money but doubtless if they find enough, the next funding round will be bigger and will let them explore more. This comes on top of previous funding rounds and also some government grants.

Lithium production may be profitable, depending on production at the time. Battery making, so far, isn’t, especially since China (the largest current market for EVs) cut subsidies. Like many industries, there will be consolidation and it will become a scale game. Local production, done ethically, could be a valuable addition for the right brand though. It will take more than Lithium of course – there’s still cobalt and various other rare earth elements needed.

I wish Cornish Lithium good luck at finding (first) enough and then getting sufficient funding to extract it safely and ethically. We don’t want this to turn into Sirius Minerals part 2.