The Novopay Technical Review - a bigger problem than I expected

When the Novopay Technical Review came out, I had some concerns about it, principally its vagueness on technical issues. So I put in an Official Information request, asking

1. How many people were engaged in the actual process of technical review?
2. What experience did these people have in software development and maintenance, actual hands-on-the-code experience, as opposed to management?
3. What experience did these people have in user interface design and testing for end users? Again, actual hands-on work experience rather than management is what I'm asking about.
4. Did any of the reviewers have hands-on programming experience?
5. Were the reviewers all normally employed by Deloitte, and if not, in what areas of the IT industry were they normally employed?
6. Was the final report submitted to all the review team for approval or should it be taken as one person's summary of their work?
7. Scope point 8 "The effectiveness of data entry into the system, including validation and quality assurance" is not clearly covered in the Final Report. What did the review team actually do to judge this effectiveness?
8. Were the review team allowed to inspect the actual source code of the Novopay system (at least the part specifically developed for Novopay)?
9. The Final Report refers repeatedly to Talent2 failing to provide requested information. For example, on page 31, "Solution architecture documentation." Now "application architecture" was stability and technical suitability point 2 on page 3 of the terms of reference. What reasons did Talent2 give for failing to provide this infomation or the "formal process documentation" also referred to on page 31? How were the reviewers able to judge the architecture or processes without this information?

They don't know

Here's an extract from the official reply.

I am advised that the Ministry appointed Deloitte to this role on the basis of the company's expertise, skill and reputation. The Ministry therefore relied on Deloitte to have all appropriate internal systems and processes in plae to adequately provide the services.

Because the system you have asked relate to the internal systems and processes that Deloitte would have used to produce the report, I do not hold the information you have requested. I am advised that the Ministry of Education does not hold it either. ...

The horror, the horror

Quoting the Ministerial Inquiry: "The former Secretaries of Education took an unwarranted level of comfort from the governance of the Novopay project. The lack of engagement with the project by the Ministry's Leadershop Team and Audit and Assurance Committtee... The Ministry ... demonstrated misplaced optimism ... Reporting to Ministers was inconsistent and at times duly optimistic, and sometimes misrepresented the situation. ... Ministers ... were comforted by their belief in ... scrutiny [by someone else]. Ministers, Secretaries and central agencies at varying times assumed that [PricewaterhouseCoopers] was providing Independent Quality Assurance across aspects of the project ... several parties ... believed that PwC was providing IQA and had been reassured by this. [But] PwC told us that the project did not have a formal IQA provided after October 2012 [and certainly not them.] [One official] told us that she could not remember whether there had been a formal decision to suspend IQA, and that she was not aware that the contract with IQANZ had ceased."

None of that has anything whatever to do with Deloitte, and I have not the slightest intention of saying anything against Deloitte. For the record, let me say clearly that I believe Deloitte's acted in good faith and with the highest competence compatible with the procedures they were required to follow.

My concern is the Government's strangely unbusinesslike conduct and what we an learn from it. On their own showing,

• The government have no idea who were the people who carried out the Technical Review and wrote the report, on the basis of which they threw millions more dollars at Novopay.
• The government have no idea what skills those people possess.
• The government have no idea what the reviewers actually saw or did.
• I think it is a fair summary that the Government exercised no oversight of the review process and relied on blind faith in Deloitte.

Let me repeat that I accept that Deloitte were in fact worthy of such faith. I trust the local branch of Deloitte to do my taxes.

But it doesn't matter that Deloitte are trustworthy. The Government should not be putting blind faith in any company. Heck, should the Archangel Gabriel have appeared with a Technical Review inscribed on golden plates, with a voice from heaven endorsing him, the Government would still have had a duty to grill him about his competences and processes.

There is an old proverb, "do not buy a pig in a poke".

In particular, what I trust is Deloitte's accounting and business skills. I have no information that would enable me to judge their software engineering skills, and no amount of business competence or good faith will automatically endow any person or company with such skills.

What do we learn?

You can outsource work but you can't outsource responsibility.

The development process as described in the Ministerial Inquiry clearly shows that so many activities were outsourced (starting with the core business activity of paying staff!) that too many people were trusting that somebody else would exercise effective oversight, and it never happened. The Technical Review is another case of outsourcing going hand-in-hand with an abdication of oversight.

As things stand, anyone who wants to trust the conclusions of the Technical Review must exercise the same utter blind faith that the Government do.