This article is an excerpt from The Art of Tendering: A Global Due Diligence Guide, which is available for purchase.
In its July 2015 decision in Problem Gambling Foundation of New Zealand v. Attorney General, the High Court of New Zealand struck down a contract award after finding flaws in the government’s evaluation and award process. The dispute dealt with a Request for Proposals (RFP) for treating problem gamblers. The Problem Gambling Foundation, a losing proponent, asserted that the government’s evaluators relied on undisclosed evaluation criteria and improper procedures during the evaluation process. As the Court noted:
The Foundation said that there were changes to the criteria in material respects and, had the Foundation known about the changes, different or additional information would have been provided. The Foundation contended further that these changes materially affected the weights of various sub-criteria and, in consequence, the weight given to the relevant main criterion in the final score. These changes altered weightings from between 2% and 7.5%. The figures are established in unchallenged evidence from Mr Mullins. Changes to this extent for a single sub-criterion had a material effect on the overall score for a proposal.
The Court also found that the government improperly conducted consensus scoring during what was referred to as a “moderation stage”. This had the effect of lowering the complainant’s score based on undisclosed “alignment with strategic plan” criteria:
The Ministry submitted that the Foundation should have realised that alignment with the strategic plan was an important sub-criterion in respect of the main alignment criterion because other proposals contained relevant information. I am not persuaded by that submission having regard to all of the other matters discussed up to this point, including the onus on the Ministry under the mandatory rules. The addition of this sub-criterion was not permissible having regard to the rules and the clear terms of the RFP. The error was material. It is again appropriate to digress from the detailed analysis of the documents to use this as another example of materiality. The alignment with strategic plan sub-criterion was given a sub-weighting in the main alignment criterion of 30%. This represented approximately 3% of the total evaluation of all criteria. And it represented approximately 4.3% of the total for the seven main quality criteria. The Foundation’s consensus score on this was zero for both its stand alone proposal and its lead agency proposal. Mr Ramsey said that, if the Foundation had been aware of the exact criteria that its proposals would be assessed against, it would have changed its approach to its proposals.
These evaluation problems were compounded by the government’s failure to maintain proper evaluation records. As the Court summarized, this irregularity was noted by the government’s process reviewers:
There was no contemporaneous document recording the panel’s discussions about this stage. The absence of any record in respect of the moderation stage was noted by PwC in its probity review as an exception. This related to a check list item as to whether appropriate records had been kept “to evidence the proceedings and the results of evaluation panel meetings”. PwC’s draft report, dated 29 November 2013, records the following:
Exception noted, but subsequently resolved: We noted that formal documentation of the proposal evaluations and the method employed in the Evaluation Panel’s moderation meeting needs to be more fully documented. We noted that although the proposal evaluations method is described in the Evaluation Pack and supporting documents provided to the Evaluation Panel members, there is no documentation for the method used by the Panel to moderate the proposal evaluation results and to arrive at the final recommended list of providers for thirteen Regions.
In particular, nothing had been formally documented to evidence how in seven out of thirteen Regions, some of the recommended providers were selected ahead of providers whose proposals had scored higher in the proposal evaluation results. This need was brought to the attention of the Senior Contract Manager and Chair of the Evaluation Panel who subsequently documented the process.
As the Court noted, efforts were then made to retroactively document the evaluation process to fill in the gaps in prior record-keeping:
The subsequent documentation of the moderation process, referred to by PwC, was initially contained in a redraft of the recommendation memorandum from Mr Levy to Mr Bartling. The final panel meeting had been on 15 October 2013. There were 13 regions in total. In the final report this was changed to read “in all regions”. The original memorandum appears to have been completed around 19 October 2013, after it had been sent to panel members, and there was a revision on about 2 November. The further revision, in response to PwC’s comments, was sent by Mr Levy to PwC on 28 November 2013. This included a draft of a new appendix to the memorandum, labelled appendix 7. This is the retrospective documentation. It is headed: “Appendix 7: Panel moderation methodology and process”. Appendix 7 does not appear to have been produced in its final form until 18 December 2013, when a further revised memorandum was sent to Mr Bartling. The memorandum as a whole was not produced in its final form until 27 January 2014.
The Court was unimpressed with the after-the-fact efforts made to correct the evaluation process deficiencies. It concluded that the government’s “moderation stage” breached the RFP rules:
I am satisfied that the moderation stage occurred in breach of the mandatory rules and in breach of the Foundation’s legitimate expectation as to both the general nature of processes and the criteria that would be used for evaluation. The moderation process and the moderation “principles” were not indicated in or through the RFP. This conclusion also seems to be implicit in what PwC said about moderation, although they referred expressly only to the absence of any reference in the panel’s evaluation pack. The “principles” used at the moderation stage involved the application of essential criteria that were not notified to proposers and which conflicted in material ways with criteria that were notified. The moderation process was in a material way disconnected from the process indicated by the RFP.
The Court concluded that the introduction of the “moderation stage” had the effect of nullifying the results from the prior stages of the government’s evaluation process:
What is indicated by the RFP is a structured process directed to the eight criteria, each of which had a weighting. This necessarily required a means of ranking proposals based on evaluation of each of the sub-criteria for each of the main criteria with the rankings derived from numerical weightings. A scoring system was the obvious means of doing this. That is what proposers were entitled to expect would happen. They did not have to know the detail of the scoring system to have a legitimate expectation in that regard. The process described by Mr Levy is quite unlike anything indicated by the RFP. He came close to saying in his evidence what is apparent: the results up to that point could be ignored.
The Court therefore determined that “the principles used to establish final rankings at the moderation stage involved a major deviation from what was expressly recorded in the RFP.” In its conclusion, the Court stated that government institutions must maintain lineal integrity throughout an evaluation process in order to ensure the defensibility of their ultimate contract award decisions:
The evaluation process was a carefully structured step by step process. It must be assumed that there was intended to be a logical and “probative” connection between one step and the next. It is plain from the Ministry’s evidence and submissions that this was intended.
At each step in the evaluation process there was a conclusion, commencing with the pre-score of each panel member, through to the new rankings at the end of the moderation process. Apart from the last conclusion, each of the preceding conclusions was provisional in the sense that it was not the final decision representing the recommendation of the Panel. But the conclusion at each step was nevertheless a conclusion which, given the design of the process, obviously had to be reached before there was a foundation for the next step in the process. Each of these interim conclusions had to provide a reliable foundation for the next step in the process. It had to provide a reliable foundation if there was any purpose in reaching the interim conclusion. The Ministry, understandably, did not suggest that there was no purpose in the various steps with the conclusions along the way.
The consequence of this is that, if there were material flaws of methodology in the process leading to a provisional conclusion which made that conclusion unreliable, the necessary foundation for a valid final decision is absent. The necessary foundation for a valid final decision is certainly absent if there was a series of unreliable conclusions at numbers of necessary stages of the process. None of this was cured by moderation, and there are the separate problems with moderation already discussed.
The Court therefore granted the complainant’s application and set aside the contract award decision.
As this case illustrates, the use of undisclosed criteria, unsound processes, and improperly documented procedures can prove fatal to the ultimate defensibility of government contract award decisions. However, the level of scrutiny that evaluation decisions face in legal challenges will turn in part on the level of intervention applied by the courts in scrutinizing those decisions. While in this decision the trial court applied an interventionist approach, that decision was ultimately reversed on appeal when the appeal court adopted a less interventionist approach to the review of the government procurement decision.
