This article is an excerpt from the 2020 edition of The Art of Tendering: A Global Due Diligence Guide, which is available for purchase.
In its August 2019 decision in Harris Corporation v. Department of Public Works and Government Services, the Canadian International Trade Tribunal determined that it is unnecessary to engage in group consensus scoring and that averaging individual evaluator scores instead is an acceptable evaluation practice. The dispute dealt with a bidding process for the provision of night vision goggles for the Royal Canadian Mounted Police (RCMP). While the complainant challenged the group evaluation of the competing binoculars as subjective and arbitrary, the Tribunal disagreed. The evaluation process was based on direct user testing of the proposed equipment. Rather than engaging in group consensus scoring after the individual evaluations, individual evaluator scores were simply collected and averaged to arrive at a final score. The Tribunal upheld this practice, finding that there is no obligation to conduct consensus scoring and that averaging out the scoring variations between evaluators was just as acceptable as addressing those variations through group consensus scoring meetings:
Again, the fact that there would be variations was inherent in the evaluation procedure disclosed in the RFSO. The RFSO expressly informed bidders that the evaluation would be based on user “perceptions”. When there are multiple individual evaluators and no consensus scoring, it is inevitable that there will be variation based on subjectivity. PWGSC tried to avoid these subjective perceptions amounting to “unbridled opinion” by structuring how they would be assessed. There would be multiple participants to avoid any individual bias. The individuals would be confined to the scoring form which asked set questions tied to task-oriented, quantifiable criteria. The scores would be added up to average out individual variations. The participants would then repeat the process under different trial environments.
In essence, PWGSC set up an evaluation procedure similar to a consensus scoring evaluation but without all the participants conferring on a final score. Instead, their individual scores were summed. Just as consensus scores are permitted to vary from individual scores, individual scores may vary amongst evaluators.
The Tribunal therefore dismissed the complaint.
As this case study illustrates, while evaluation results must be defensible, purchasing organizations are not required to engage in group consensus scoring to arrive at their final scores. To protect against the undue influence and interference challenges that have been brought against past group evaluations, the averaging of individual evaluator scores is an acceptable and defensible method of arriving at final scores in a more streamlined and manageable manner.
