Weak Evaluation Leads to Voided Award

By Paul Emanuelli

This article is an excerpt from the 2020 edition of The Art of Tendering: A Global Due Diligence Guide, which is available for purchase.

In its May 2019 decision in Word Perfect Translation v. Minister for Public Expenditure & Reform, the Supreme Court of Ireland upheld a lower court decision to set aside a contract award due to a manifest evaluation scoring error. The dispute dealt with an invitational tendering process conducted under a multi-supplier framework agreement for the provision of interpretation services.

As the Supreme Court summarized, the evaluation was out of 1,000 points. The winning bidder scored 885 while the complainant, Word Perfect, scored 870, a difference of 15 marks or 1.5 percent of the total available points. The marking scheme was weighted 75 percent for quality of service and 25 percent for cost. Word Perfect submitted the lowest price and received the full 250 marks available for cost, while its competitor only scored 166 points in the cost category. However, Word Perfect narrowly lost due to its scoring in the non-price categories of Service Delivery Plan and Quality Assurance Plan.

In the Service Delivery Plan category, Word Perfect lost points because its bid exceeded the mandatory word limit by 37 percent, so only the first 2,000 words of its bid were considered by evaluators. Word Perfect scored 197.5 points in that category, while the winner scored the full 250 points. That word limit proved material for scoring the Quality Assurance Plan category. The issue ultimately turned on the scoring rubric created to guide evaluators in their allocation of points in the Quality Assurance Plan category. While the scoring guide gave evaluators a wide degree of discretion in allocating between 80 and 100 percent of available points for an “excellent response”, that top-tier scoring band was only available for “fully meeting and exceeding requirements” in the category:

In particular, the reference to “fully meeting and exceeding requirements” is difficult in itself, but is particularly so when paired with a range of marks running from 80 to 100 per cent. Clearly, there is a wide variation possible between tenderers which rank just above the 80 per cent threshold, and those which achieve full marks. This is particularly important, since, given the automatic failure for any tender falling below the 60 per cent mark in any requirement, the outcome of the tender is largely determined by the marks in the top two categories. It may have been clearer if the top category had simply been described as an excellent response providing comprehensive and convincing assurance that the tenderer will deliver to an excellent standard, with the individual marks allocated indicating their relative merit within that category.

The full evaluation rubric that guided the evaluator’s allocation of points read as follows:

Word Perfect scored full marks in the Quality Assurance Plan category and was therefore allocated the total 200 available points. While the winning bidder scored lower, it still earned over 80 percent of the available points, scoring 170 out of 200 even though it failed to provide the requested “narrative” about quality assurance in text form and only provided diagrams to speak to the criteria. As the Supreme Court summarized, the Court of Appeal had previously concluded that the evaluators erred in awarding the winning bidder points from the highest scoring category given that it has only provided graphics as its quality assurance explanation and had failed to provide a “narrative” in written form. In applying a plain language interpretation to the “narrative” requirement, the Court of Appeal had concluded that this meant that bidders needed to provide more than just pictures, but also needed to describe their plan with words:

J. considered himself compelled to conclude that there was a manifest error in this regard on behalf of the evaluation team. The word narrative was a fairly ordinary English word which a reasonably diligent and well-informed tenderer would well understand as requiring a written statement. This was especially so in the context of the actual language of the tender document itself (“… a narrative summarising the information provided …”). Any doubts would be dispelled by the fact that the SRFT had stated just a few lines previously that the MI Reports “must use relevant visualisation tools” such as charts, graphs, etc. to display pertinent information graphically. He concluded that “[o]n any view, therefore, this relevant provision of the SRFT required the tenderer to supply both a visual display or depiction of the information by way of charts, graphs etc., and a narrative summary. Some might think that the narrative summary requirement was superfluous, but there is, I fear, no escaping the conclusion that, viewed objectively, this is what the SRFT actually required” (emphasis in the original). He concluded that there was no doubt that Translation.ie did not provide such a narrative summary in this case, and, accordingly, that the score of 85 per cent, which implied that Translation.ie had fully met or exceeded requirements, was therefore a manifest error.

The Supreme Court agreed with the Court of Appeal, finding that the evaluators failed to interpret a “narrative” as requiring a written text explanation. The Supreme Court determined that the evaluators scored the winning bidder improperly since that bidder failed to fully satisfy the requirement, but still scored in the top scoring bandwidth:

Decision in relation to the Quality Assurance Plan

As observed above, this issue is dependent upon the interaction between the interpretation to be accorded to the word “narrative”, and, just as importantly, the operation of the marking scheme. The appellant argued strenuously that the interpretation adopted by the Court of Appeal was too verbal and text-bound, and that it was possible to see the charts presented as satisfying the requirement of a narrative. In other contexts, there might be considerable merit in this submission. Lawyers tend to prefer information presented in the form of text, whereas others may be much more comfortable with the same information provided in a tabular or numerical form. If the information is provided effectively it would be an error to lay emphasis on the form, particularly if that may reflect an unarticulated preference for the familiar. However, in the context of the SRFT, the conclusion of the Court of Appeal that a narrative summary in text form was required is in my view compelling. This is what I consider to be the natural meaning of the words, but particularly when taken in their context, which refers separately to the use of tables. It is the information in such tables which is required to be summarised in the narrative. The appellant criticised the Court of Appeal for not affording a degree of “curial deference” to the exercise of “judgment calls” by the evaluators, or “judicial restraint” in relation to the actual marks awarded. Instead, the appellant contends the Court of Appeal failed to appreciate the “holistic approach” taken by the evaluation committee to the consideration of the criteria in section 4.3 of the SRFT in its totality. However, these appeals to deference and a holistic approach only confirm, in my view, that, if a clear and natural view is taken of the requirement of the SRFT, the Translation.ie tender must be considered to have failed to provide what was required in this component of the Quality Assurance Plan.

However, the evaluation marks in this regard, if taken in isolation from the detail of the marking scheme, might not themselves be capable of being a manifest error. If the evaluators, as they did, had given the same marks, and had explained that the lower mark given to Translation.ie was because of the absence of a narrative, I do not think it would be possible to argue that the particular marks given constituted a manifest error. However, Word Perfect points to the terms of the marking scheme and, in particular, the explanation that marks between 80 and 100 per cent could only be given for an excellent response that “fully meets or exceeds requirements”. As touched on above, the requirement that a tender fully meet requirements to achieve 80 per cent was perhaps not a fully intended and thought-through consequence of the methodology. It appears to create something of a straightjacket, which may lead inevitably to the conclusion that a response which is excellent in nearly every respect and fully meets or exceeds the requirements in all but one area, cannot achieve more than 79 per cent of the marks, even though it may be manifestly superior to a competing bid that did, in this example, provide some narrative text. While I do not think that this is a desirable conclusion, and, indeed, it may not have been intended, I consider that it follows from the language used in the SRFT methodology for calculating scores.

As the Supreme Court determined, the error in scoring the winning bidder’s Quality Assurance Plan could have materially impacted the final rankings since that bidder incorrectly received over 80 percent of the available marks from the top scoring band (170 out of 200) even though it failed to “fully meet” the requirements and should therefore have been assessed in the second-tier band and been awarded between 60 to 79 percent of the available points and scored between 140 to 158. As the Supreme Court observed, since the original scoring came within 15 total points, a proper scoring in the second band would have made the outcome “wafer-thin” since there was an 18-point range of available points within the second band:

Accordingly, I agree with the conclusion of the Court of Appeal in this regard that the award of marks in this band to Translation.ie was a manifest error, by reference not to the marks awarded or the quality of the submission, but rather because of the definition given of the submission which would attract marks in this band. It is necessary to consider finally what consequence follows from this finding. The fact that there was a margin of 15 marks out of 1000 between the parties, and that the error was in awarding 170 marks when 160 represented a submission that fully complied with the tender in this regard, means that, in theory at least, it is not inevitable that the error identified had a causal effect on the outcome of the competition. The evaluators could have awarded marks under 160 which would still have meant that Translation.ie was the successful party, although by a wafer-thin margin. I do not think this can influence the outcome, however, and the result of the competition must be set aside.

The Supreme Court therefore determined that the initial evaluation and award decision was the product of a manifest error and should be set aside. As the Supreme Court concluded “consideration of this case does emphasise the need for precision in the drafting of a tender and the construction of the tender process, care in the evaluation, and, not least, pragmatism and efficiency in any review.” The failure to properly draft criteria and scoring guidelines for evaluation teams puts their evaluation results in peril, particularly in situations where the scoring rubric gives evaluators wide discretion in allocating points within scoring bands and the final ranking of bidders falls within that margin of discretion.