FOI – Army of One

iStock_000004817156XSmall.jpgLast December I suggested that government could adopt a 3 stranded approach, making use of existing tools like Facebook, Wikepedia and others to replace things that have traditionally been built and managed in-house. One of the ideas was a service that would put all FOI requests and the associated responses for all departments on one site. I said:

[W]hat if we took every FoI request – and its response – and published it online with a simple search application, driven by google or windows live or any other engine- so that before you asked your question you could see what else had been asked that was similar; you’d then either just use that information and not bother to ask your own question or you’d refine yours to get a better take. Smart journalists would use the search tool to bring together previously unrelated questions and draw even more conspiratorial conclusions. Smarter ones would phrase their next question to take advantage of the freely obtained knowledge that they already have to find something new. Government would respond, one would hope, by getting smarter about its operations and processes and would use this leverage to drive greater change and efficiency.

And then

The deluge of information would be enormous. The fragments of data would require an entire army to stitch it together into meaningful conclusions. But, let’s be honest, government itself is never going to have a big enough internal army to do this stitching but, the outside world, those who want to be part of an open-source government, now maybe they’d have the willing, the time, the intellect and the energy to sort, distill and publish the very best pieces – and government, of course, would pay for such pieces once and once only. Sadly, the name YouGov is already taken by a very clever chap called Nadhim Zahawi, but maybe he’d be open to offers. Failing that, we could always go back to, the vision of access to government coined in 2000 following the [necessary] demise of

I was thinking that someone in government might take this on. What I should have realised is that somewhere in the UK, Francis Irving, one of MySociety’s people was already working on exactly such a service.

It’s not completely done but it works and I think it’s brilliant. William Heath, who must think I disagree with him at every opportunity, will be surprised by that remark I’m sure. In this context, we disagree about only a couple of things –’s overall usefulness and the search approach within the same site.

What the [to be named] service does is let you ask a question of any department. It badges the request, ties it to a unique email address and then handles the correspondence with the department. The question is posted on the site for all to see along with its status and any responses. There are only a few questions on there so far and even fewer full responses, but it has potential.

I can see that a lot of people on the inside will want to stop this – there will be complaints that somehow FoI was never meant to be published on the Internet, that it was meant to be a bi-directional transaction between government and one citizen. Plainly that doesn’t make any sense. If a journalist asks for information and then writes an article about (as has been done a 1001 times already) then what’s the difference between that and this site? This site has, of course, the potential to be a much better archive of such requests and, over time, it could save a lot of money by reducing duplication and, particularly, by allowing government to say “we’ve already answered that – and here’s the link” to mysociety’s site. I think that will take some time and there will be plenty of resistance. So for now, there’s voyeur mode – you and I get to see what kind of questions others are asking and engage mode where you get to ask the question yourself.

Funnily enough, William suggests the site be named – which I’d completely missed in my thinking in the paragraph I highlighted from the previous post – although I’d even referred to that site. He’s right, though, would be a great name. But I suspect they might have to settle for something else. is for sale, a snip at $7,200. is apparently registered. seems to have something to do with Missouri government. Simple combinations of those words (and adding words like “free”) all appear to be taken.

One question that is asked on the site, by William (on a mission from God), is, effectively, what was the story behind using as the domain name for the UK government’s first aggregating site and did money change hands with the ISP, I wasn’t around then and I’d be astonished if there is a story still; and more astonished if money actually changed hands. I think all they got was the footer on, which I think we removed somewhere in version 2 or possibly 3 (mid 2002?). But there are few, if any, people around from that time, e-mail systems have changed 3 or 4 times and I wonder how good the archives are.

Anyway, this new site is great. It does much of what I said, better than I thought, quicker than I could possibly have imagined, before I’d even put finger to keyboard and all on a shoestring. Give it the traffic it deserves and visit it.

e-government 3.0

Wait! What? 3.0? How did that happen? I, along with many others, have long had a theory that government should – and often does – sit behind the private sector in terms of technology. A government department taking a leap into the unknown, what used to be (and probably still is) called the bleeding edge is a rare and brave one. Often, at least more than the laws of probability would allow, such efforts end in failure. Web 2.0 is probably more like e-Government 4.0 or maybe even 5.0. Thinking about this has been languishing in my blog drafts folder for many months – the article in the Economist last week prompted me to dig this out and refresh it a little. It’s all about what do governments do next? They’ve had the “get everything online” phase – and many have done pretty well; most moved on to driving up usage of transactions (and, again, many have done well). But what next? What’s in the third phase?

Electronic Government 21.06.2001.jpg DWP Conference Slides - 30.01.2003 - stages.jpg

The slide on the left was something I first put up in early 2001 to explain the stages I thought government would go through on the journey to a truly transformed operation. If I remember correctly, the words on the bottom (Publish, Transact, Interact, Transform) were from a report writtten the previous year called “e-Government @ IT’s best.” I no longer have that report, but I’m pretty sure that when I created the slide I reversed the second and third stages – I felt strongly that interaction would come after transactions, i.e. that transactions were going to be one way at first (taxpayer sends tax return) and would then become two way (government says, “thank you for your tax return. we need this bit of information. did you know that you can also do this. and here’s your refund transaction” etc). The yellow octagon (bottom left) is where I thought we were then (i.e. heavily mail dependent, quite some way into the publish stage but with a big bump just ahead of us as we tried to orchestrate transactions).

Later, I put some dates on the slide suggesting that each stage might take at least a year to break through – but that online private sector companies were already moving ahead of government and making the move to some truly dramatic changes. I also used a much simpler background – I have no idea what I was doing with the whole sunset at sea thing. The opening slide from the conference where I showed this slide noted that the FT was saying that we’d already spent £3 billion on e-government in the UK alone, by January 2003. By mid-2003 I think I’d abandoned the slide – I can’t find it in any later conference pitches. Evidently I’d realised it didn’t say anything by then (and that I was, of course, hopelessly optimistic about the dates by which any of this could be achieved).

But in 2008, I think it has relevance again. Governments, generally, seem to have stalled at the “transact” stage. All of the big plans to move to real interaction, to transformation (dramatic or otherwise) appear – to me viewing from inside government but outside the e-government race (and it was a race) – to have stalled.

If e-government 1.0 was publishing – several thousand websites per country – and e-government 2.0 was about transactions (moderate take-up of a small number of transactions, high take-up of a couple and low take-up of the rest), then what is e-government 3.0?

For me e-government 3.0 is the set of actions that need to be completed to truly have a high level of online interaction between citizen, business and government. The creators of that interaction will be both private and public sector-based. Interaction means lots of things to lots of people. I mean: high levels of usage, bi-directional flows of information (that is, citizens send to government and government engages in dialogue, including offering access to services not yet taken up by that citizen), third parties engage in service provision wrapping their existing offerings alongside government services (so, for instance, third party pension providers will show you your state benefits alongside what your employer benefits could be and potential for additional benefits with separate schemes).

In e-Government 1.0 and 2.0, one of the big surprises for me – and this is a global, not local point – was the lack of engagement from non-government businesses in providing services – that is, few third parties took up the challenge of providing services alongside or in competition with government. From 2000-2003 I figured that we were only a step or two away from seeing that engagement grow strongly. Early on I was involved, loosely I think it’s fair to say, with schema development for various financial transactions with government and lots of software vendors and perhaps that gave me a false sense of how the market might develop. In hindsight, there were a few clear reasons for the absence of third party involvement:

1) It’s hard to compete with something that’s free. If government does something reasonably well, why would you try and charge for something that appears to only offer modest advantages. In the UK an example here is HMRC’s Self Assessment process. Tools that tried to compete were provided by Which (TaxCalc) and Microsoft (TaxSaver I think, although there was another one called DigiTax perhaps) – they were good, but they weren’t good enough and they died out. There is also a famous story of an online fishing licence process that with its attendant publicity, in one go, pretty much killed off aspirations for many 3rd parties.

2) Governments are not necessarily able to be open about the complex methods of interaction. The UK still, I believe, leads the field with its govtalk certification (see but even that still contains many draft schema dating as far back as 2002 and lacks important schema in, say, the health records domain. The last major update to the documentation was in March 2005 and that only really to note that the OeE was no the eGU. Interacting with government is complicated and takes a lot of documentation to figure out. When Sun partnered with Software AG to design and build a local box (known as a DIS box) that would connect a department to the Government Gateway, reams of documentation were necessary to handle the various messages that could travel back and forth.

3) Allowing third parties in makes things complicated for government. At present, if the main provider of a service is government and they realise that there’s a problem with their online service, a quick change and everything can be fixed. What if you’ve actually certified 5 external providers to provide that same service and furnished them with protocols, schema and instructions? And then find there’s a problem. The quick change is no longer possible and overall service can suffer quite dramatically.

4) Finally, at the beginning government figures it needs to seed the market. It doesn’t know how many people are going to use a service, and it can’t make promises to one third party over another about the business opportunity. So figuring out how much market share is there for the taking by 3rd parties is a challenge. Government doesn’t know. To stay with a UK example, HMRC’s Self Assessment saw 3.7 million returns last tax year (plus a few thousand more in the couple of days after the filing deadline) – that’s a pretty good number, probably close to 50% of the total returns but it’s taken 7 years to build to that number and it’s probably a good 2 million short of the total that could be delivered today with minimal effort from those sending in tax returns. I don’t think anyone – certainly not me – starting out on the e-government trail in 2000, would have thought that it would take this long to get 50% of the returns in electronically.

I think e-government 3.0 has to take these, and other, issues head-on in an effort to really drive up usage by the population – and use the space that creates amongst the internal workforce to drive real change to processes, dramatically simplifying them, joining them up where it makes sense to do so (and recognising that “sense” is sometimes a hard word to define at the get go) – bringing it all together under a new overall mission, allied to the somewhat waning transformational government agenda. I believe people need a new flag to rally under – the momentum has gone from the original drive to get things online, progress is only incremental and the buzz has gone. E-government should, by now, be business as usual of course; there is, perhaps, no need even to continue to use the word but, at the same time, the original aims of governments that embarked on the course have not yet been entirely realised and so we need something to rally the troops.

Here are the 6 things I would propose a government do to channel their efforts towards busting the next barrier to progress in e-government:

1. A clear statement on the direction of travel for the government adopting this plan. For instance, they will no longer develop new e-government services where there is potential for commercial involvement and competition, where there are existing services offered by such third parties or in a specific list of defined transactions that are as yet undeveloped, difficult to develop, or have the potential to be widely used in specific markets. Essentially, a government would publish a list of 50 or 100 transactions that it wanted the private sector to adopt, embrace, drive usage on and join up. There I’ve said it: join up. Some transactions that already exist do so only as stove-pipe transactions. Government would allow third parties to join those up and not compete with them.

2. An OpenAPI will be developed, documented and published detailing how to get to and from government. I noted above that govtalk had gone a long way down this path. but there is plainly more to do. I was intrigued to see an announcement today regarding health clinics in the USA developing connections to Google for storing health information – with the idea being that it can be retrieved as the patients move from state to state or doctor to doctor. This strikes as a perfect example of an API as I imagine it. Plainly there are, in this case, privacy issues to be concerned about – but Google reads your email and may know as much about your health if you mail updates to friends and family as anyone. This would be put on a fast path, using the hit list of transactions in (0) above as the base. Government would do what the software industry does and always support at least n-1 versions, perhaps more.

3. Government would develop rules engine capability “in the cloud” – i.e. processing would no longer need to be behind the full set of government firewalls, but could be handled a step away with only the final results of transactions entering government. Government would commit to make these engines highly available, scaleable and up to date. It would likely partner with suppliers who have proven that they can do this – inevitably, that is a small list but it plainly includes the big Internet companies such as Amazon, Google, eBay and Microsoft; there would also need to be local (national) companies in this list to ensure that the local economy developed along the same path as the titans, where possible.

4. A way to encourage small businesses to get into the provision of such services would be set up. I imagine a Dragon’s Den style competitions with businesses receiving funding to deliver services, based on the hit list with bigger incentives for the harder ones or where government already offers reasonable services. To date, I think governments around the world have fought shy of such an action, worried that it somehow breaks procurement rules or gives some players an unfair advantage. As I see it, anyone who wants to play can get a meeting, but they have to show up with a business case and plan that the angels (or is it dragons?) say is viable and that government is prepared to put its money into. To ensure the government lives up to its end of the deal, it will commit to invest in X deals for £Y a year – and whatever X and Y are, they’d be public and reported on monthly on the national portal – that would be in the UK, etc

5. To facilitate longer term development and operation, a model for paying transaction fees for businesses meeting government certification and providing accurate, timely transactions would be developed. The fees would start low and grow as volumes grew, incenting providers to seek out bigger markets, giving players that were growing fastest the opportunity to achieve real economies of scale. At some point, fees would level off and perhaps even reduce year on year, incenting the providers to find ways to deliver more efficiently. Government would pay these transaction fees by abandoning development of services where competition was being introduced, not entering new markets, no longer needing to upgrade servers and so on for its own websites. Government would have to provide some degree of technical and business support, and this would be factored into the costs.

6. Third parties would be granted the right to white label government content, with acknowledgement, using simple tools that made necessary policies, rules and instructions available to them. Ideally, content would come from the national portal which is the most likely place for the investment to be made to allow this but also may become, if it isn’t already, the only source of up to date, joined up content.

Making these 6 points happen would be no easy task. It would require a step up in co-ordination inside most governments, a new reaching out to the private sector (and a large number of cynicism bypasses there), a rapid set of decisions on where the priority areas were and some fast delivery to prove that it was viable. It’s very likely that there are some folks out there waiting in the wings, who have services that could be offered as pilots for this or have tools that could be used to develop services quickly, but they don’t know who to talk to, how to get to them, how to fund it or how they will construct a business model.

I’d envisage some small businesses getting off the ground this time round – there were attempts at this during the early dotcom boom days; some survived, most did not – and growing into strong national transaction players who would look to do interesting stuff like take paper processing off the hands of government as a vehicle to make it electronically more quickly. There could be a mixed economy of big and small players emerging over time. Just as likely though is that existing industries – banks, accountants and so on – would step into providing their clients additional services and value.

In the background, government would work on its bigger picture for transformation – once assured that increasing numbers of transactions were coming in electronically, the smart folks on the inside of government would look for ways to redeploy staff to more customer focused areas, to develop new business models that could be based on the assumption that all transactions for, say, a start up business would be electronic and so on.

This would take time – 5 years or more. That might mean 10 for some governments. But it seems to me that governments have a better sense of time than most. At the moment, though, few have set out what it is that they want from their online services over the next 2-3 years let alone 5-10. I think a model like this could just see some real breakthroughs that would significantly benefit national economies – both on consumer and producer sides.

Dan Jellinek Transform slides - 13.05.2004 - inandout.jpg DWP Conference Slides - 30.01.2003-opening up.jpg Mark Forth Slides - Year 3 - 17.11.04 - storm.jpg

Economical E-Government

I love The Economist. Like The New Yorker and Mother Jones it has a style of journalism that is highly readable, exceptionally good at distilling a complex subject into interesting chunks, enormously educational and impossible to read in the time period between issues. I wish I could write like that. A guy I was at school with is one of their editors – I met him at a conference in the USA, quite by chance (in fact, I met him twice there – and there were about a million people there); I’m very impressed, he was a talented guy then and is plainly even more so now. The problem with reading those 3 magazines is that I always end up with a backlog – there’s only so much reading you can get done on a crowded tube.

The February 16th issue has a feature on e-government headlined “The electronic bureaucrat.” I’ve just got through it. First, kudos to Jeremy Gould for being marked out as a guy with an excellent blog – and, whilst it doesn’t say it directly, one of the few on the payroll civil servants with any kind of a blog let alone an excellent one.

I thought there were some great threads to pull out and, cheekily, I’ve followed some of the bullets with slides from my collection of moderate hits, 2000-2004:

1. Inevitably e-government doesn’t exist by itself. It must merge and conform to the physical word. The example is for those looking to get a visa for America: you apply online, pay money online, get an email confirmation, print it and then take it to the embassy in London (or doubtless elsewhere) where they check the barcode, allow you access, and let you collect your visa. A brilliant melding of a process that needs you to be there but doesn’t need you to be there the whole time.

EDS workshop slides - 12.02.2003.jpg
IDeA slides - 28.09.2004.jpg

2. Governments don’t face competition. In fact, the sentence is “Governments rarely face competition”. Alistair Darling besieged by non-domiciled bankers who are all off to Geneva in the light of recent taxation demands may beg to differ. Competition exists at multiple levels. Local authorities compete for businesses to site themselves within their boundaries; regions compete for investment by big companies; cities compete for banks to put their HQ in town; countries compete for the big capital projects, major manufacturing plants and so on. This hasn’t yet visibly manifested at a transaction level – but years ago I used to tell the Inland Revenue that if they made it easy for companies to set themselves up (PAYE, VAT, Corporation Tax, overall regulation etc) then they would attract new businesses because the cost of set-up would fall and that would be attractive. That hasn’t changed. Sure, the average person isn’t going to move to a new country just because the online services are better there (unlike if the tax treatment is felt to be better), but they are going to change their behaviour if the online service is very much better than the offline (and I say “very much better” because slightly better doesn’t overcome the inertia that is present with the status quo).

IR Scotland - Future Business Challenges-comp.jpg
IR Scotland - Future Business Challenges-notsep.jpg

3. Personalisation is important. This is rule 1 of 3 by a chap from PA Consulting. Frankly I think he’s off-base and even if he’d said it in 2003 rather than 2008 he’d still have been off-base. He says government should be like online shopping and record your preferences, that it should be available round the clock and that services have to be easy to use. He wants government sites to be “beautifully designed.” Personalisation was a massive sink-hole from 2001-2004, chased by every department looking for a website as a “must have” feature, even if no one actually knew what it was. It didn’t mean anything then and it doesn’t mean anything now. You interact with government on so many levels and with so many personalities that most of the time, you’re someone new each time you come back. Reminding me of what my tax return looked like last year doesn’t help me if I’ve changed job, saved money, spent money, bought shares, sold shares or whatever. As to being around all the time and being compelling, yes, I agree. But then everyone agreed with that in 2000. Beautiful design – I’m already thinking of parrots hanging from trees – won’t attract the customer. Many travel sites that I use every day are still nearly impossible to use – a rant for another time – but you use them because they’re the best way to get the job done. Government needs to show it can get the job done and better than the offline equivalent. The online VAT form in the UK is still not attractive because the offline system is so fantastically well done. Fill in 9 boxes – and probably 6 if you don’t trade across EU boundaries – and you’re done. The only way to make that easier is to do the sums for me. Aha! Maybe we’re on to something there.

Saudia Arabia slides - 23.04.2002.jpg

4. i-government is the most prevalent form of e-government. That is, putting everything you possibly can online is still very common. In the early days of the web this was called shovelware or brochureware. And it is still depressingly common. It’s better than not having it online, but it doesn’t deal with the massive and inevitable duplication, the confusion that entails when different pages (let alone different sites) appear to contradict each other and, as The Economist says, it doesn’t mean that when you do eventually interact, your service will be any better.

Electronic Government 21.06.2001.jpg

5. Websites are closing down. This sounds like good news – and Lord knows I’m a huge fan of reducing websites (see many, many posts here – but I haven’t yet seen the evidence. The statement in the article is that 551 of the 951 central government websites “have closed” – i.e. past tense. Not will or are planned to, but have. if it’s true, that’s a brilliant step forward and I’m delighted. When the guys in OeE put together, the expectation was that other sites would be starved of visitors and so would eventually shut down through lack of traffic. Closing them down before that is even better. It narrows the information sources and forces the main player,, to be more authoritative. If it isn’t, people switch off and don’t come back. It’s like the French Encyclopaedia, Quid, no longer being printed – if there’s a better source (Wikipedia in this case), people go there. But as to 400 sites closing, I’m sceptical – show me the money as Jerry Maguire might say. There’s a sidebar here on the search engine in which is hardly a new topic for this blog; I’ll deal with that another time. It does appear, though, that the DG search team have read the article because the search terms used to illustrate the point now claim the third spot (as opposed to not being in the top 100).

Dan Jellinek Transform slides - 13.05.2004 - website curve.jpg

6. The digital have-nots number in the billions. My late and very dear friend, Angela Vivian, campaigned tirelessly (ok, so she tired everyone else out, but she was, herself, tireless) on this very topic for years. Whilst internet access has increased in the UK it flattened out in the high 50s percent (nudging 60% now, but only just), as it has in other developed countries. In less developed countries, it hasn’t even got to the high 5s or 10s. The Economist refer to a Southern Indian State, Andhra Pradesh, that appears to have achieved enormous things with a project called “e-seva” – essentially, public service offices where Internet access is available and transactions can be completed (paying bills such as electricity and telephone, where the service is still doubtless nationalised and so part of government – just like banks in the UK). It’s an impressive looking set of services and with only 119 centres they see 110,000 transactions a day. The population of this state is over 75,000,000 – not very much bigger than the UK, and certainly economically different. I believe that this total of transactions is greater than that carried out in the UK at present, assuming they’re all carried online in India (and I suspect they’re electronic at source – like a cheque process in a bank: the customer gives the paper, which is immediately rendered electronically and from then on the transaction is 100% online).

Loudcloud Client Event Slides - 28.11.2001.jpg

7. Proving. No scratch that. Defining the benefits remains hard. The UK is rightly given praise for trying to figure out a model for what an e-government project brings to the party. Whilst I was around when that was being done, it was nothing to do with me; but it needed to be done and was done pretty well. Other countries have tried (e.g. Australia) and found similar things – there’s a general sense of e-government is better but if you try and pin down whether the public sector is smaller or the economy has benefited, well, you’re going to struggle. Some of that is because figuring out the true transactional costs offline is already hard; and quantifying the saving and putting pound notes on it is even harder.

PSF Slides - 28.03.2003.jpg

8. It isn’t about the technology. There’s a great story about how the new mayor of the District of Columbia has shaken up process after process inside the operation of his city, tied in a single portal ( – hmmmm so much for that whole problem we had once with 2 letter domain names; in the UK he’d have been pushed down, splahed out on some plasma screens, used google apps (and thrown away all of their own servers) and are even looking at iPhones instead of police radios (do it – apple stock needs the juice). Alongside the technology, such as it is, they’ve rationalised the hiring process, stripped out every redundant step and made bureaucracy a dirty word. I just have no glib answer to that. It just sounds awesome. DC may be the heart of government for the USA but it has long been a poor and disenfranchised area with high crime rates (I think this year is the first time they’ve been invited to vote in the presidential primaries even). There is no slide that goes with this set of achievements.

9. e-Government is only the beginning. There is so much left undone from the original vision – join up, citizen focused, simple government, direct access government. It doesn’t matter which party conjured up the words, the intent was always the same. Make government more accessible to the citizen – in both directions.

IDeA slides - 28.09.2004-joinup.jpg

The online version of the story appears to be but a shadow of the offline version – the Economist (note, it’s, not is a subscription play (once it’s owned by the Murdoch family, they’ll unwind that I’m sure. What? It’s not going to be owned by them? One day maybe).

It’s a great article overall. It distills several years of learning into 16 easy to read pages. It has examples drawn from all over the world, although those from the UK tend not to be the most positive. Did we really need another story about the NHS and are ID cards and the NHS spine – not a £12.4 billion project by itself fact-checkers – as if it were the only global example of data aggregation?

IDeA slides - 28.09.2004-data.jpg

That said, it’s a backward look, i.e. mostly a catalogue of what people have done rather than what they could do. But by highlighting some real bright spots – the folks in DC for instance – the writers tantalise us with what we could do, if we got it together, crashed through the barricades and made it happen.

IDeA slides - 28.09.2004-simple.jpg

More Entourage 2008

bang.jpgTechnology will be the death of me. It’s the simplest things that go the most wrong. E-mail. Well, Microsoft e-mail.

In defence of Apple, this is plainly a Microsoft problem, I’ve had a pile of problems with Outlook 2008 on Windows – mostly where the whole PC would hang when receiving new email. I don’t think I was alone in seeing that problem.

Anyway, back to my world of MacBook Air and, to recap:

– I’d set up all of my email, migrating thousands of old mails from my PC, via Thunderbird and using all kinds of odd scripts. I just checked and I migrated around 25,000 mails for a grand total of 3.49GB

– My MacBook Air crashed, rendering the office database corrupt and unusable. Attempts to rebuild it all failed at stage 4 of 5.

– I tried re-installing Office 2008, to no gain

_ Eventually I found the “main identity” files (in your user folder, under “Microsoft User Data” and deleted everything. That cleared the problem and I was free to work with a blank slate on Entourage

– I installed Thunderbird on my Mac and moved all of my email into it. Then I killed off a pile of old email in sent and inbox (having secured a backup first) and exported just 3,000 email to Entourage compatible files

– I set up Entourage all over again. And did a database compaction (is that even a word in this context?) which reduced the mailbox size from 1.2GB to 327MB (and even handily took a backup) Surely that would be ok for Entourage to handle?

– Within an hour it had crashed. or, at least, it removed the “quit” option from the Entourage menu whilst letting me use other emails. The database utility popped up and said my database was corrupt again, but that it couldn’t quit Entourage so I needed to do something about it. I got the feeling its knickers were in a twist – kind of an error message tone thing.

– So, with great trepidation, I force quit Entourage. And fired up the database rebuild. In no time it had completed, without errors this time.

– I opened Entourage and found all of my email there, safe and sound. But when I tried to send an email, Entourage told me that I first needed to create an email account. It seems the rebuild had deleted all of my account set up details.

This thing gets one more chance. I’m a patient guy. I’ll give it 3 strikes. And then it’s back to thunderbird. Meanwhile, I’m leaving all of my emails on the server.

Proceed with absolute caution if you are considering Entourage as your main client. Proceed with real, real caution.

Entourage 2008 – Not Ready For Prime Time

Day three of MacBook Air ownership, day 0.01 of actually being able to use it. The effort to transfer is enormous – not just the email copying I wrote of in my last post.

First job is the “inevitable download” – bazillions of megabytes of patches and enhancements, not least Leopard 10.5.2. Second was software installation – Mac Office 2008, MarsEdit (to write this blog). Third was data file copying (all of my document folders). Fourth was “asset stripping” – getting rid of the things that just aren’t needed (Garageband, iMovie etc. Haven’t quite decided about iTunes so far – I might need it to sync my iPhone). Fifth and final was the great email conversion. Right there is a good 12 hours of effort.

But it was that last task that was a complete and utter bust, it turns out.

It had worked fine for all of about 60 minutes of use. Not even enough time to run down the battery (about 2 1/2 hours I think – forget all this 5 hour stuff).

I then got the Apple blue screen of death – a polite message in half a dozen languages saying that I’d need to press the power button to reboot. No problem I thought, I can do that. One crash in less than a day. Maybe I just had too much going on.

After the reboot, I tried to open Entourage. No can do said the unhappy programme. The database is kaboom. Or some technical equivalent. No problem it said, just rebuild it using this handy utility. 60 minutes later, the handy utility failed at step 4 of 5 of the rebuild. Reboot again. Rebuild again. Same failure, same place. Check the web. Dozens of instances of the same problem.

I imagine – and this is only me clutching at vague straws – that this is all to do with the size of my email inbox. I have 11,000 emails in my inbox. I don’t use folders. That’s what search is for. Why would I want to sort everything so that I could never find it again when search (especially in Windows Office 2008) works so well? Outlook handled the mail fine, why should Mac be any different? Sloppy code I assume.

I’ve now downloaded Thunderbird on the Mac (given I had to export from Outlook to Thunderbird to get to Entourage) and imported my email into that. Took about 20 minutes – far faster than the import into Entourage – and so far, doesn’t seem to have a problem with the mailbox size.

It’s a real shame. I thought Entourage, at least in the short time I’d been able to use it, was pretty good. I liked the “My Day” window a lot. Simple but effective.

So now I’m teaching another email programme what spam is, typing in my email account details again, setting up the preview pane the way I like it and learning some foibles.

There’s another problem since that crash and reboot. Opening any file in word or powerpoint or excel brings up an error telling me that the Office database is corrupt and that I need to rebuild it (tried that dummy, don’t you know that?). It doesn’t stop it working, it just tells me it’s bust and invites me to press “ok”. Can you imagine the conversation with a heart surgeon in this scenario? “I’m sorry sir, your heart is corrupt. You have 3 days to live. Press OK to continue”. Never mind cancel or reboot or “I can sort it”, just press ok to die in 3 days. I figured a quick reinstall of Mac Office would sort out the database corruption problem. No joy. Same error, same time, same place.

I killed everything I could find that had the Office label on it, using AppZapper and then reinstalled again from afresh. Still get the database error, still can’t rebuild it. I have no idea where this database is, how to delete it or how to resolve the problem. I’d even be prepared to use Entourage from scratch with a fresh and empty database if I could figure out how to get it to start.

The solution, at least I think it’s the solution, is to kill your “main identity”. Very Minority Report. Or is it 1984? I deleted the file – 3.5gb of it and restarted Entourage. A nice, shiny, empty, entirely blank email programme greeted me. What I thought I’d do now is copy just a few of my emails over, say the last 3 months. If I’m lucky, that will work and I’ll never see this problem again. Or at least not for a year or so.

Technology of the 21st Century? Microsoft have to be kidding. This is version 3 of Office for Mac I think. Normally it’s right by now. It obviously takes longer than it used to.

There are some famous last words about to be eaten sometime soon I fear. Linux might even start to look attractive at this rate.

Moving to Mac. The Entourage / Outlook Import and Conversion Story

Alongside moving to a new blog, I’ve also been working on moving to a new Mac, a Macbook Air to be exact. And when I say “moving”, I mean moving from a Sony Vaio and Vista to Mac and Leopard. I’ve tried this before, not from my the Vaio but from various other devices and always given up after a couple of weeks of effort. This time I think it’s different. And that’s not famous last words. Why?

– The MacBook Air is the first Mac I’ve seen with a decent screen that can compete for lack of weight against laptops such as my Vaio. The Vaio is about 1.7kg plus the power brick, call it 1.9kg. The MBA is about 1.3kg with the brick (previous Macs were about 3.3kg at best). Ok, it doesn’t have a DVD drive – but I never use it except for installing software, and that works fine using my iMac drive shared across the network.

– Office 2008 is a significant leap ahead versus previous Mac Office versions. It’s as good as Windows Office 2008 – some would say better (there are fewer interface changes which means fewer leaps of logic to understand where, say, paste special has gone)

– Third party tools such as Parallels or VMware fusion let you run the few Windows programmes (I have some stock charting software and my money management software) that you need to run. I say “need” – I’d like to replace them, but nothing so far works as well

Now, on to the things that are still hard:

– Moving email from Outlook on a PC to Entourage on a Mac. What a palava. Microsoft could not have made this harder if they’d set out to from the beginning. Some instructions that might help (I’ve found most guides on the web are near useless or, at best, confused).

So, how to import from Outlook (on a PC) to Entourage (on a Mac) via Thunderbird (on a PC)

1. Install thunderbird on your PC.

2. When you start Thunderbird it will helpfully bring up an import menu with only one option which is “don’t import anything.” Don’t worry about this and let it carry on.

3. Configure your email box if you must, but set it so that it doesn’t fetch new mail (otherwise you quickly find you’ve got yet another email inbox)

3. On the file menu, you’ll find that you can import from outlook. This function fetches everything – all the subfolders and so on – but it takes a while. I have about 3gb of email and it must have taken a good hour or so.

4. Now you’ll have all of your email in Thunderbird. Well done.

5. Go to this site and download the import script

6. Go to the tools menu and pick addons. You can then install the script you just downloaded.

7. You’ll end up with a file that is probably as big as your pst file in outlook. Mine was a 3gb folder with many sub-folders

8. Copy that folder to your shiny new Mac. There are a bunch of spurious files and folders in there with various extensions (such as .sbd) and sometimes there’s a file (inbox, for instance) as well as folder with the same name.

9. Select each file that you want to import to Entourage and rename it “.mbx” – I suspect you’ll find that none of the files have any extensions and show up with icons that look like TV screens (they do on mine)

10. Nearly there. Visit this page all about Entourage and download the script that converts files to text files. Select each file you want to convert (all the ones that you renamed as .mbx) and run the script. You can do it one by one or as a bulk convert – just keep track when you import with which ones you have and haven’t done.

11. Go to the Entourage file menu and select “import”. Pick “contacts or messages from a text file” and then “import messages from an MBOX-format text file” (note that whilst this says MBOX it means .mbx)

12. Choose the file – this time you have to do it one by one.

13. Entourage will go quiet for a while. You won’t be able to press anything but there will be no spinning beachball. You just have to wait.

14. Eventually – depending on how big your files are – it will say “finish”. Click that and you’ll have to wait again, but you will get a beachball this time. This seems to be a shorter wait, but a wait nonetheless.

15. The folder you have just imported will appear on the left hand side of the screen under the “on my computer” menu. You can then decide what to do with it. I copied all of the inbox file straight into the inbox, one for one, but moved the sub-folders under it. This actually copies all of the folders rather than moves them so, once you’re done, you have to delete the old ones.

I’m about to do my contacts now, but expect that’s a similar process. Sadly, all of my iphone contacts are synced with Outlook, but iTunes won’t let me sync them with Entourage – which would be a one step process. Like I said, they have made this too hard for no good reason. And don’t babble on about monopolistic suppliers and not wanting people to convert, this is just too hard for no good or defensible reason.

It can be done. And, like I said, this time I think it will stick. If only because half way through this process, my reliable (seriously, I’d had no issues) Vista laptop crashed. When it recovered it asked if I wanted it to search for a solution. It found one and said “Vista, by Microsoft, cause this problem. Download this patch” or something like that. I downloaded it and installed it. Only for the patch install to say “this patch is not suitable for your system”. Well, thank you, that was kind.

There are, though, some problems still. I don’t have an external DVD and I can’t figure out how to install Vista under Parallels via the Remote Disc (which installed Office without a hitch); Parallels doesn’t allow this to work (they say that the MacBook Air doesn’t but, as I said, it installs Office – but what do I know?); I know I don’t know anything about this kind of thing, but surely, given I have a valid licence key (which it verifies before telling me I can’t actually do the install), it could just get the image file off the web and download it?. Doubtless there are more of those to come.

I really, really hope that this isn’t famous last words.