This article is an excerpt from the The Art of Tendering: A Global Due Diligence Guide, which is available for purchase.
In its May 2018 decision in Lancashire Care NHS Foundation Trust & Anor v. Lancashire County Council, the England and Wales High Court – Technology and Construction Court ordered a re-evaluation after finding flaws in the original consensus scoring results of government evaluators. The dispute dealt with a procurement process for public health nursing services for persons aged zero to 19 conducted by a county council. The contract awards were challenged by two unsuccessful incumbent service providers.
As the Court noted, applicable UK and EU evaluation rules call for the use of transparent criteria and sub-criteria that enable proponents to understand what they need to respond to in their proposals:
The literature and authorities, both in the UK and of European origin frequently refer to criteria and sub-criteria. The use of these terms may be convenient on the facts of a given case or tender; but it is not desirable to try to fix them with immutable meanings for universal application. Typically, the terms are applied when tender documents identify a requirement that is to be addressed by tenderers and to which an aggregate mark may be allocated (“a criterion”), which mark in turn may be the product of a number of other requirements that fall within the overall scope of the criterion (“sub-criteria”), whether or not specific marks are to be allocated for individual criteria. What matters, in my judgment, is that the authority should identify (a) what the tenderer is required to address and (b) how marks are going to be awarded. Once it does that, it must (subject to exceptions that do not apply in this case) stick to what it has said it requires of tenderers and how it has said it will mark the tenders. Provided it does so, it does not matter whether the language of criteria and sub-criteria are used at all. On the other hand, where the terms are used, it usually denotes that the criteria (or sub-criteria) have been defined and are basic elements in the approach to be adopted and applied consistently by tenderers and the authority alike. Put another way, “potential tenderers should be aware of all the elements to be taken into account by the contracting authority in identifying the economically most advantageous offer, and their relative importance, when they prepare their tenders…”: see Case 532/06 Lianakis [2008] ECR I-251 at [36].
The purchasing institution used group consensus scoring in its evaluation process. As the Court noted, the Procurement Manager, Mr. Paul Fairclough, and the Category Manager who reported to him, Mr. Jig Parmar, maintained the evaluation records and moderated the consensus scoring sessions:
The moderation meeting took place on 14 November 2017. Mr Parmar chaired the meeting. Mr Fairclough took notes on his laptop. The evaluators could see that he was working on his laptop but could not see what he was doing, whether noting or deleting. In outline, the process that was followed was that the meeting would take the individual evaluators’ scores for a bidder’s answer to the question. There would then be a period when each evaluator was invited in turn to offer their comments on the answer, both negative and positive. Mr Fairclough took a note of the points that were made, though it was not verbatim. Analysis of his record shows that the order in which the evaluators gave their contributions was not constant: typically Mr Parmar would ask an evaluator with an outlying score to start the process. There would be a period of discussion of the points being made as the evaluators attempted to reach consensus on the score for the question under discussion.
However, since the group consensus scoring discussions resulted in changes to evaluator scores, the Court raised concerns that the initial evaluator notes were not reliable in documenting the reasons for the final adjusted scores since the evaluators failed to keep their own records of the reasons behind their changed scores and instead relied on the group notes taken by the moderators:
It is clear from the evidence of the witnesses (and I find) that, although the panel reached consensus on scores, there was not necessarily or even probably congruity of reasoning that led each evaluator to subscribe to the consensus score for the question. It is also clear (and I find) that the moderated discussion had the result that evaluators might be persuaded to change their minds on particular points so that, for example, a point that they had felt to be influential when carrying out their original evaluation became less influential than it had been or a point that they had not considered to be influential when carrying out their original evaluation assumed greater importance in the light of the discussion that took place. Shifting of ground and reasoning is of course to be anticipated in the course of a moderated discussion such as took place; but it means that the evaluators’ original score sheets are not a reliable guide to the reasons that ultimately caused the group to reach their consensus scores.
Furthermore, the Court found that the evaluation group failed to document whether it properly considered the complete list of sub-criteria for each evaluation category during their group discussions since they did not maintain an adequate comprehensive record of those discussions:
Neither Mr Parmar nor the evaluators instituted a system whereby each bullet point was discussed in turn. Rather, the discussion in relation to each question was initiated by reference to the comments (negative and positive) that the evaluators brought to the discussion as a result of their previous evaluations. There was no analysis at the end of the discussion to see whether all bullets had been covered either in the discussion or otherwise. The evidence does not establish that each bullet for every question was expressly discussed. It is not surprising that the procedure adopted at the moderation had the result that not each bullet point can be identified in Mr Fairclough’s notes as having been considered. The positive and negative points recorded in the notes can generally (but not always) be attributed as being relevant to particular bullet points. With some questions, the positive and negative points taken together appear to include reference to all of the bullet points for that question – but this is not always so. In the absence of a comprehensive record of the discussions at moderation, the absence of any recorded point that is attributable to a particular bullet point does not enable me to infer that the bullet point in question was not mentioned or discussed.
The Court determined that the lack of thorough records undermined the credibility of the evaluators when they gave evidence during the legal proceedings in defence of their evaluation decision. As the Court concluded, while some of that evidence was based on their evaluation notes, other testimony appeared to be a reconstruction of events aimed at supporting what was contained in the incomplete group evaluation records:
The evaluators gave evidence about the observations they had made on their original scoresheets, some of which was plainly reconstruction, some of which was equally plainly based on actual recollection, and some of which was a mixture of both recollection and reconstruction. When it came to giving evidence about the discussions and decisions at the moderation, their evidence was, to my mind, much less convincing for two main reasons. First, their evidence tended to be dominated by what they believed to have been their own particular position (whether or not it changed during the moderation); and, second, when trying to give evidence about what happened at the moderation, it was clear that their evidence was dominated by attempts to reconstruct what was signified by Mr Fairclough’s notes.
The Court also found significant gaps and inconsistencies in the final evaluation notes, which included contradictory comments made by different evaluators that were never reconciled to reflect how the evaluation panel reached its ultimate consensus scores:
Focus must therefore shift to Mr Fairclough’s note-taking and notes as a record, and the evidence of the evaluators as to what happened.
Mr Fairclough’s evidence was that he noted points that reflected the discussion that was held. Sometimes he would note a point but would subsequently delete it because the evaluator who had made the point no longer relied upon it after and in the light of the discussion that followed. It is evident that the views of individual evaluators in relation to a given answer might differ, so that one might think that the answer covered a particular bullet point satisfactorily while another did not. In such a case both positive and negative comments might be expressed in the course of the discussion and be noted by Mr Fairclough. I accept Mr Fairclough’s evidence that he would sometimes edit out points that he had initially recorded because of a subsequent change in position by one or more evaluators: but the notes of the moderation have a number of examples where conflicting points are recorded, with no apparent attempt to attribute them to individual evaluators, to reconcile them or to indicate whether or to what extent the panel reached agreement, even if that agreement was an agreement to disagree on the significance of a point while agreeing on the ultimate consensus score.
Examples of the recording of contradictory comments are provided at [11] of the Claimants’ closing submissions. I accept the evidence of the evaluators (specifically as summarised in [10] of the Claimants’ closing submissions) that the notes of the evaluation in their final form included points in relation to which the evaluators did not agree either that they were good points at all or as to the weight to be attached to them. Furthermore, even if Mr Fairclough set out to record all material points in his notes, it is apparent from what I have already set out and from his evidence that some were either omitted or edited out. Accordingly, although I accept that the points that have survived in Mr Fairclough’s notes are points that were identified during the discussion, they are not a complete record of the points that were made or even the points that were considered to be material at some stage during the discussion, as some were edited out at unspecified times and for reasons that are not themselves recorded.
The Court also found that the evaluation notes were changed retroactively after award and that there were no clear records of which notes were the original group notes taken during the consensus scoring sessions, which notes were after-the-fact revisions, and whether any original evaluation notes were permanently deleted when the retroactive notes were later inserted:
I am not satisfied that all points that were material in the course of or by the end of the discussion are recorded in the notes. Nor am I satisfied that the highlighted comments were the only points that led the panel to reach its consensus scores; and I am satisfied that they do not provide a full or accurate account of the reasons or reasoning that led either individual panel members or the panel as a whole to reach the consensus scores that were reached.
To make matters worse, the Court also found that the evaluators failed to follow the established procedure and never formally approved the final group consensus scoring notes. In fact, some of the evaluators never saw the final notes until they were later called to provide evidence during the subsequent legal proceedings:
On 24 November 2017 Mr Parmar wrote to the panel members saying that it “might be impractical” for panel members to sign the final moderated scoring sheet at that stage. Instead, he asked each panel member to send an email “which clarifies your satisfaction/agreement of the moderation process and final scores.” The notes were not sent to the panel members. Ms Green replied confirming her agreement with the moderation scores; Dr Slade confirmed she was “satisfied that the moderation process and generation of final scores had been fair and robust”; Ms Jones clarified her “agreement of the moderation process and final scores for Lot 1”; Mr Girvan confirmed “agreement with the moderation and final scores for lot 1-4….”. No one purported to agree the notes of the moderation. They were never agreed by the panel as an accurate record of the moderation. On the evidence I find that the panel members were not shown the notes of the moderation until they came to make their statements in these proceedings.
The Court also found that the town council misled the complainants by redacting dates and backdating evaluation notes in what was an extremely close scoring outcome that turned on two marks in the overall evaluation process:
Later, when the Trusts were pressing for information, the Council misled them by first redacting the dates and then backdating three of the individual members’ evaluation notes. To describe this (as the Council did) as merely “a regrettable episode of poor administration” is, to my mind, an unacceptable understatement.
The Trusts were informed that they had been unsuccessful on 27 November 2017. The limited information that was provided to the Trusts showed that the margin by which they had fallen short of the successful tenderer was 4.07%, with Virgin scoring 78.5% and the Trusts 74.43%. 0.07% of the difference was attributable to price. The 4.0% difference on quality evaluation represented a difference of two marks overall, one for Q3 and one for Q5.
As the Court stated, “a procurement in which the contracting authority cannot explain why it awarded the scores which it did fails the most basic standard of transparency.” The Court concluded that the “inconsistency in approach in the recording of the moderation of different questions in each tenderer’s bid means that it is not possible to identify a structure in the notes which reveals the reasoning process adopted by the panel that led to and explains their consensus scores on a given question.” Overall, the Court found that the evaluation notes did not provide a full, transparent, or fair summary of the discussions that led to the consensus scores sufficiently to enable the losing proponent to assert its review rights or to enable the reviewing court to exercise its supervisory jurisdiction. As the Court stated, there was “no sufficient account of the reasoning and reasons that led panel members to resolve their differences (if they did) so as to arrive at consensus scores.” Given the failure to create a proper reviewable evaluation record, the Court concluded that the resulting contract award should be set aside as unlawful.
As this case illustrates, the failure to maintain proper evaluation records can prove fatal for the defensibility and legal validity of any resulting contract award decision. Purchasing institutions should clearly define project roles to ensure that all evaluators understand their responsibility to maintain their own clear evaluation records in support of their final scoring decisions.
