By Paul Emanuelli

This article is an excerpt from The Art of Tendering: A Global Due Diligence Guide, which is available for purchase.

Many public institutions think that retaining external fairness monitors to oversee their group evaluations and relying on their procurement advisors to transcribe “official” evaluation records help enhance the defensibility of their evaluation process in major projects. However, these widespread practices are part of a losing strategy that exposes evaluations to successful legal challenges. This discussion surveys recent high-profile cases where the evaluation process went off the rails due to a failure to implement proper evaluation roles, responsibilities, and procedures.

Bid Dispute Panel Strikes Down $3 Billion Evaluation

In the June 2015 In the Matter of An Appeal by Dynalife, a bid dispute panel found significant irregularities in an Alberta Health Services (AHS) request for proposals (RFP) evaluation process.  The panel report nullified the results of the evaluation process that had initially resulted in the award of a $3 billion laboratory services contract. The Alberta Minister of Health subsequently announced the cancellation of the contract award.

The panel found that AHS failed to clearly document the process details, finding a need for greater transparency in AHS’s evaluation, ranking, selection, and award process path. The panel was also highly critical of the apparently subjective “Good Fit Question” used by AHS evaluators (referred to by the complainant as “choosing your own evaluation”). The ruling highlighted the need to avoid any criteria or process rules that appear to introduce an element of subjectivity or arbitrariness into an evaluation process.

The panel found that the lack of procedural clarity was exacerbated by the application of a “pencil and pen” process, whereby evaluators were directed to revisit and change their scores by an evaluation “advisory panel” that included an individual who was in a conflict of interest due to prior dealings with a particular proponent. The ruling was also critical of scoring conducted during a “due diligence” phase that relied on second-hand information that was not directly collected and assessed by the evaluators.  The panel found that the non-scoring assessors could unduly influence the evaluators and that an evaluator’s scoring could be undermined by a lack of direct involvement in assessing criteria that they were required to score. Notwithstanding AHS’s use of an external fairness monitor, the panel ultimately determined that the evaluation results were invalid.

Court Strikes Down Award Over Irregularities

In its July 2015 decision in Problem Gambling Foundation of New Zealand v. Attorney General, the High Court of New Zealand struck down a contract award after finding flaws in the government’s evaluation and award process. The dispute dealt with a Request for Proposals (RFP) for treating problem gamblers. The Court found that the government improperly conducted group consensus scoring and lowered the complainant’s score based on undisclosed “alignment with strategic plan” criteria during a non-transparent “moderation stage”. These evaluation problems were compounded by the government’s failure to maintain contemporaneous evaluation records. While its external procurement consultants advised the government to retroactively rectify those records, the Court was unimpressed with the government’s after-the-fact efforts to fill in those record-keeping gaps.

The Court concluded that the government’s “moderation stage” breached the RFP rules since it had the effect of nullifying the results from the prior stages of the government’s evaluation process and stressed that government institutions must maintain lineal integrity throughout an evaluation process to ensure the defensibility of their ultimate contract award decisions. While this decision was curiously overturned after New Zealand’s Court of Appeal adopted a non-interventionist approach to the review of government evaluations, the initial trial decision, which is more consistent with global standards, underscored how the defensibility of award decisions can be compromised by flawed group evaluations.

Critical Defects in Nuclear Bid Evaluation

The July 2016 decision of the England and Wales High Court in Energy Solutions EU Limited v. Nuclear Decommissioning Authority exposed the systemic defects in industry practices involving consensus scoring, record-keeping, and the oversight of evaluation teams. The Court found that the Nuclear Decommissioning Authority (NDA) improperly awarded a contract valued at £4.2 billion to a bidder that should have been disqualified based on threshold requirements. It also found that the winning bidder would have been outscored by the plaintiff if the evaluation criteria had been properly followed.

The Court found that the NDA failed to adequately resource the project, which put inordinate pressure on the evaluation team. The Court determined that the significant time pressures and excessive workloads resulted in mistakes. As the Court observed, the evaluators were “continuing their day jobs during the evaluation process”. However, the Court determined that NDA’s decision to impose a heavy workload on its employees did not absolve the NDA from its duty to run a proper evaluation process. The Court also found an “overly defensive” approach to record-keeping, which included discouraging evaluators from creating records and encouraging them to shred their notes. While the NDA retained an external law firm to assist in shielding some of the records, this strategy ultimately proved ineffective.

The Court also found flaws in the way that evaluators reached their consensus scores and noted that pre-established procedures were breached on “a number of occasions” since “scores were changed after evaluation was completed”. In fact, the Court found that “changes were made to the scoring and/or rationale following unrecorded conversations” with senior managers who were not part of the evaluation team. By interfering with the evaluators, the Court found that the NDA inappropriately introduced a hidden evaluation stage after consensus scoring had officially closed. As the Court stated, this “new, separate and informal stage” was “not part of the process of evaluation as it was designed” but was “an extra stage fraught with danger” since it “may not have been applied equally to the different bidders” and was “also not transparent because no records are available of it.”

The Court ultimately concluded that the NDA “fudged” the evaluation to avoid disqualifying the winning bidder for failing to meet a minimum threshold score. It also found multiple errors and inconsistencies in the scoring performed by the NDA evaluation team and that the plaintiff would have earned the top score if the criteria had been properly applied. The resulting damages award was reportedly in excess of £200 million.

$250 Million Claim in Group Scoring Challenge

In its November 2016 decision in TPG Technology Consulting Ltd. v. Canada, the Federal Court of Appeal upheld a trial court’s dismissal of the plaintiff bidder’s $250 million lost profit claim against the Government of Canada. The claim dealt with a federal government procurement process valued at approximately $428 million for the acquisition of information technology engineering and technical support services. While the trial court found flaws in the government evaluation committee’s consensus scoring process, it did not award damages because the evaluation flaws did not impact the final ranking. However, the case exposed the inherent weaknesses in many group evaluations.

The trial court noted that an averaging of the evaluation committee’s individual scores showed a significant divergence in TPG’s scores before and after consensus scoring. In one specific evaluation category, TPG’s average initial score was 80.6 out of 100 but its final consensus score was 49 out of 100. In another category, TPG’s average initial score was 32.8 out of 50 but its final consensus score was 22 out of 50. As the decision noted, confusion arose within the evaluation team about the method of scoring performance metrics. That confusion resulted in the evaluation committee abandoning its individual scores and re-scoring TPG’s proposal collectively.

Notwithstanding the use of external procurement advisors, the trial court found that the evaluation committee was inconsistent in its scoring process, since it failed to re-evaluate the winning bidder’s scores by consensus as it had done for TPG’s proposal. Notwithstanding this finding, the trial court ultimately rejected TPG’s claim since it found that the evaluation errors in the two noted categories had no impact on the ultimate proponent rankings. While the government was ultimately able to defend itself against the $250 million lost profit claim, it was only able to do so after protracted legal proceedings and after showing that the flaws did not change the results.

Court Derails Tainted Train Tender

In its July 2017 decision in Passenger Rail Agency of South Africa v. Swifambo Rail Agency (Pty) Ltd., the High Court of South Africa struck down a contract award after finding multiple irregularities in the tendering process. The case dealt with a contract award for the sale of 70 train locomotives. The Court found that the winning bidder’s tender was non-compliant and was inappropriately rescored to enable the tainted award.

The Court determined that the evaluators engaged in an inconsistent application of the scoring criteria. While the criteria were weighted in the Request for Proposals (RFP) according to their relative importance, the Court determined that the evaluators failed to follow these scoring rules in their actual evaluation process, which “resulted in an illogical evaluation”. The Court also found that the winning bid was inappropriately rescored after it failed to meet the minimum technical scoring threshold. The Court ruled that the re-scoring constituted an unfair manipulation of the evaluation process to the benefit of the winning bidder and to the prejudice of other bidders. In striking down the contract award, the Court stated that “there is simply no reason why the respondent should benefit from an unlawful award that was peppered with so many irregularities”.

Flawed Consensus Scoring Triggers Re-Evaluation

In its May 2018 decision in Lancashire Care NHS Foundation Trust & Anor v. Lancashire County Council, the England and Wales High Court – Technology and Construction Court ordered a re-evaluation after finding flaws in the original consensus scoring results of the government evaluators. The dispute dealt with a procurement process for public health nursing services conducted by a county council.

As the Court noted, the organization’s procurement department maintained the evaluation records and moderated the consensus scoring sessions. However, since the group consensus scoring discussions resulted in changes to evaluator scores, the Court raised concerns that the initial evaluator notes were not reliable in documenting the reasons for the final adjusted scores. As the Court noted, the evaluators failed to keep their own records of the reasons behind their changed scores and instead relied on the group notes taken by the moderators. The Court also found that the evaluation group failed to document whether it properly considered the complete list of sub-criteria for each evaluation category during their group discussions since they did not maintain an adequate record of those discussions.

The Court also found that the evaluators never formally approved the final group consensus scoring notes. In fact, some of the evaluators never saw the final notes until they were later called to provide evidence during the subsequent legal proceedings. The Court determined that the lack of thorough records undermined the credibility of the evaluators when they gave evidence during the legal proceedings in defence of their evaluation decision. The Court found significant gaps and inconsistencies in the final evaluation notes, which included contradictory comments made by different evaluators that were never reconciled to reflect how the evaluation panel reached its ultimate consensus scores.

The Court also found that the evaluation notes were changed retroactively after award and that there were no clear records of which notes were the original group notes taken during the consensus scoring sessions, which notes were after-the-fact revisions, and whether any original evaluation notes were permanently deleted when the retroactive notes were later inserted. As the Court stated, “a procurement in which the contracting authority cannot explain why it awarded the scores which it did fails the most basic standard of transparency.”

The Court concluded that the “inconsistency in approach in the recording of the moderation of different questions in each tenderer’s bid means that it is not possible to identify a structure in the notes which reveals the reasoning process adopted by the panel that led to and explains their consensus scores on a given question.” Overall, the Court found that the evaluation notes of the discussions that led to the consensus scores did not provide a full, transparent, or fair summary and therefore did not enable the losing proponent to assert its review rights or enable the reviewing court to exercise is supervisory jurisdiction. The Court concluded that the resulting contract award should be set aside as unlawful.

Fairness Advisors and Group Scoring Ruled Optional

In its February 2019 determination in Temprano and Young Architects Inc. v. National Capital Commission, the Canadian International Trade Tribunal confirmed that there is no duty to use fairness monitors in group evaluations and that averaging individual evaluator scores rather than conducting group consensus scoring sessions is an acceptable evaluation practice. The dispute dealt with a Request for Standing Offers (RFSO) issued by the National Capital Commission (NCC) for the provision of architectural consulting services.

The complainant challenged the evaluation process, arguing that, amongst other things, the NCC failed to ensure a fair and unbiased evaluation since it failed to conduct a group evaluation process overseen by a fairness monitor. The Tribunal disagreed, finding that the contracting authority conducted the evaluation in accordance with the criteria and essential requirements specified in the RFSO. More specifically, the Tribunal confirmed that even in instances where suppliers were required to meet a minimum scoring threshold in a specific evaluation category, averaging individual scores instead of conducting group consensus scoring sessions was an acceptable method of arriving at final evaluation scores.

The Tribunal also rejected the complainant’s assertion that the absence of a fairness monitor compromised the evaluation process. Rather, the Tribunal found that the contracting authority met its due diligence duties without using a fairness monitor and confirmed that there was nothing in the trade treaties that compelled the use of a fairness advisor. The Tribunal therefore determined that the complainant’s grounds of complaint were invalid and rejected the bid protest.

Tribunal Rejects Mandatory Consensus Scoring

In its August 2019 decision in Harris Corporation v. Department of Public Works and Government Services, the Canadian International Trade Tribunal once again confirmed that it is unnecessary to engage in group consensus scoring since averaging individual evaluator scores is an acceptable evaluation practice. The dispute dealt with a bidding process for the provision of night vision goggles for the Royal Canadian Mounted Police (RCMP).

The evaluation process was based on direct user testing of the proposed equipment. While the complainant challenged the group evaluation of the competing binoculars as subjective and arbitrary, the Tribunal disagreed. Rather than engaging in group consensus scoring after the individual evaluations, individual evaluator scores were simply collected and averaged to arrive at a final score. The Tribunal upheld this practice, finding that there is no obligation to conduct consensus scoring and that averaging out the scoring variations between evaluators was just as acceptable as addressing those variations through group consensus scoring meetings. The Tribunal therefore dismissed the complaint.

Court Dismisses $150 Million Claim Against NRFP

In its October 2019 decision in CG Acquisition Inc. v. P1 Consulting Inc., the Ontario Court of Appeal upheld the summary dismissal of a $150 million damages claim brought by a losing bidder in a negotiated Request for Proposals (RFP) process. The case dealt with an RFP for the Liquor Control Board of Ontario (LCBO) for the sale and subsequent redevelopment of land located along the Toronto waterfront.

The plaintiff, CG Acquisition Inc., was disqualified during that RFP process. It brought the claim for damages against the LCBO, Infrastructure Ontario, and P1 Consulting Inc., the fairness monitor, arguing that all three defendants breached their duty of fairness by not including the fairness monitor in the reconsideration of the initial disqualification decision. However, the lower court summarily dismissed the claim, finding that the fairness monitor was a consultant for the purchasing institutions and owed no duty to the proponents. The Court of Appeal upheld that decision after determining that there was no requirement to include the fairness monitor in the reconsideration decision, which was made by the LCBO after having received legal advice, since the fairness monitor was not a decision-maker or evaluator, and had no authority to provide legal or quasi-legal advice.

Future Considerations for Group Evaluations

As these cases illustrate, project teams should be careful to properly define the roles and responsibilities of various members of their team and should carefully adhere to the distinction between evaluators, decision-makers, procurement advisors, and legal advisors.

Purchasing institutions should clearly define project roles to ensure that all evaluators understand their responsibility to maintain their own clear evaluation records in support of their final scoring decisions since the failure to maintain proper evaluation records can prove fatal for the defensibility and legal validity of any resulting contract award decision.

When conducting group evaluations, the following should be instituted:

  1. evaluation teams should be adequately resourced and should not be subjected to excessive workloads or time pressures;
  2. proper records should be maintained of the entire evaluation process; and
  3. there should be no interference with evaluation team members or manipulation of their scores.

The use of undisclosed criteria, unsound processes, and improperly documented procedures can prove fatal to the ultimate defensibility of government contract award decisions. Project teams should ensure that all evaluation team members are properly briefed and trained to participate in the evaluation process, particularly when involved in major, high-profile projects. This would include the need to properly brief and train evaluators to avoid bias and to avoid making biased statements or taking biased or controversial written notes. The findings also underscore the importance of ensuring that all evaluators have all the required information before final scoring, and of ensuring that time pressures and scheduling logistics do not lead to variations from the planned evaluation process.

Recent cases also show that purchasing organizations are not required to use group consensus scoring to arrive at their final scores or to use external fairness monitors. Rather, public institutions can rely on the averaging of individual evaluator scores as the means of calculating and recording final scores, or can use a streamlined process where only those scores that fall outside a predetermined range of variation tolerance are addressed through group evaluation meetings.

While some purchasing institutions may elect to retain the services of external fairness monitors to supplement their process, those services should be in addition to, and not a substitute for, obtaining appropriate procurement and legal advice in connection with their major procurement projects. Before bearing the additional costs, time, and risks inherent in retaining external fairness monitors and in using group consensus scoring, purchasing institutions should first ensure that their complex projects are adequately staffed with properly trained procurement, legal, and technical advisors.

When employing consensus scoring, purchasing institutions should ensure that such processes are implemented with a high degree of diligence, since (i) scoring changes arising out of consensus scoring can be subject to a legal challenge; and (ii) tabulation errors or procedural irregularities arising out of such sessions, however innocuous or inadvertent, can be used against the institution to launch protracted legal proceedings and create significant potential legal exposure.

To help accelerate the evaluation process, while also addressing the risk of evaluation errors that could impact evaluation rankings, project teams should consider adopting a blended approach that uses averaging as the default scoring method, while allowing more limited group evaluation meetings that focus on addressing individual scores that fall beyond an acceptable predefined range of variation tolerance.

Implementing Enhanced Consensus Scoring

In recent years, group evaluations and consensus scoring have been the subject of numerous successful legal challenges that have exposed serious flaws in the evaluation practices of many purchasing institutions. The Procurement Office has developed the following Enhanced Consensus Scoring protocols, which allow evaluation teams to operate efficiently, while staying within the bounds of their due diligence duties.

  1. Scoping: Balancing Efficiency and Defensibility

Enhanced Consensus Scoring seeks to strike a balance between the mechanical averaging of individual evaluator scores, which forgoes the benefit of any peer review, and the collective group scoring of every evaluation score, which tends to lead to protracted evaluation sessions with questionable end results. To balance efficiency and defensibility, the objective of Enhanced Consensus Scoring is to set the balance between these two extremes.

Enhanced Consensus Scoring sessions occur after each evaluator is provided enough time to conduct an individual evaluation and to record his or her initial evaluation results. Only those scores that fall outside the pre-set variation tolerance are addressed during the consensus scoring sessions. Scores that fall within the pre-established range of variation tolerance are not discussed during consensus scoring and are instead simply averaged.

Instead of diluting the limited resources of the evaluation team, one of the main advantages of Enhanced Consensus Scoring is that it focuses the team’s efforts on the areas with the largest divergence in initial scoring since those areas are most likely to contain errors that would impact the ultimate rankings and undermine the fairness of the process.

  1. Setting the Variation Tolerance

The variation tolerance should be set before the Enhanced Consensus Scoring session. There is no single formula for variation tolerance since evaluation teams use different scoring ranges between, and even within, specific RFPs. For example, if evaluators are scoring a category on a 1-to-10 scale, the variation tolerance could be set for any scores that are more than three points apart. When using a 1-to-5 scale, it could be set at any scores that are more than two points apart. Those scores that fall within the acceptable range would be averaged at the end of evaluations and would not be raised for group discussion.

  1. Co-Ordinating the Enhanced Consensus Scoring Sessions

The outlier scores (those falling outside of the variation tolerance) should be identified by the group evaluation moderators (typically procurement advisors who are not members of the evaluation team) prior to the enhanced consensus scoring sessions. While manual spreadsheets can be used for this exercise, for more complex procurements, automated bid evaluation tools can help streamline this process, particularly for identifying the largest outliers for the prioritization of group discussions.

Where possible, those items with the greatest range of divergence should be addressed first during the Enhanced Consensus Scoring sessions. This is particularly important in situations where the evaluation group has a finite amount of time allocated to group scoring as this will better ensure that the items of greatest potential impact on the ultimate rankings are addressed by the group within the allocated time.

This is less critical in situations where the project team is prepared to allow for additional sessions if all outliers are not addressed in the initial session. Either way, this issue should be addressed prior to the first group session to confirm whether the sessions will be conducted based on: (i) prioritization within a finite amount of time (where the remaining outliers would default to averaging); or (ii) non-prioritized discussion of outliers with the contingency of additional sessions if time runs out before all outliers are discussed.

  1. Rules of Order for Enhanced Consensus Scoring

Enhanced Consensus Scoring sessions should be conducted in accordance with formal rules of order. They should be presided over by a moderator (not a member of the evaluation team, typically a procurement advisor) who chairs the meeting. The moderator should prepare the agenda ahead of time based on a pre-identified summary of outliers.

The moderator should brief evaluators before initiating the sessions to clarify the following Enhanced Consensus Scoring protocols:

  1. Only outliers will be raised for discussion during the session. If those items raise issues in related areas, the group is free to discuss those other areas as well.
  2. The evaluator with the lowest score for an item will be asked to speak first, followed by the evaluator with the highest score for that item.
  3. The process will continue with the next lowest evaluator, then next highest, until each evaluator is given the opportunity to address the item.
  4. While all evaluators will be given an opportunity to speak, evaluators will not be required to speak to the items under discussion.
  5. After each evaluator is given the opportunity to speak, the moderator will open the floor for any additional evaluator comments.
  6. Once there are no more speakers, the moderator will declare the item closed.
  7. Evaluators will then be given the opportunity to revisit their individual scores and to make any adjustments that, based on the benefit of the group discussion, they determine to be appropriate. Individual evaluator re-scoring can either be done during the course of the meeting or after the meeting, as pre-established by the moderator. Evaluators will not be required to disclose whether they intend to change any of their scores to other members of the evaluation team.
  8. Evaluators should be instructed to record the reasons for any changes to their initial scores.
  9. Once the evaluators have been given the opportunity to record any adjustments to their scores resulting from consensus scoring, the final individual scores will be averaged to arrive at the final overall scores.

Summary

In summary, by applying Enhanced Consensus Scoring protocols, project teams can:

  • streamline their evaluation processes to better deal with heavy workloads and tight time pressures, while optimizing the outcome by prioritizing high-risk areas;
  • maintain proper records of the evaluation; and
  • protect the evaluation team from inappropriate interference and protect the fairness of the process.

Since group evaluations have been subject to increasing scrutiny in recent legal challenges, purchasing institutions should revisit their group evaluation procedures to ensure that they meet prevailing due diligence standards. One way to meet those standards is by properly implementing the Enhanced Consensus Scoring protocols summarized above.

Guiding Principles for Government Evaluators

Whether it’s a formal bid protest, a public audit review, or a newsreel report, government evaluators often find themselves under a spotlight of scrutiny over their evaluation and award decisions. Given the glare of second-guessing that follows the public tendering process, this discussion offers six guiding principles for lowering the temperature in evaluation rooms.

  1. No Volunteers

In almost all cases, evaluation groups are made up of individuals who are moonlighted from their day jobs and assigned to evaluation teams; however, notwithstanding this double duty, evaluators don’t get a free pass on meeting fairness standards. Claiming “We did our best with limited resources” is not an adequate defense for failing to meet your due process duties. Evaluators need to set aside enough time to independently review, evaluate, and score each proposal. They also need to prioritize their participation in group evaluation sessions since “no-shows” are a no-go that cause rescheduling delays or trigger the need to remove evaluators.

  1. No Freestyling

Defending a legal challenge is no substitute for doing things properly the first time. The evaluation process is not a freestyle competition or an opportunity to second-guess established evaluation rules and criteria. Embarking on procedural deviations and employing hidden criteria can result in re-evaluation orders or the nullification of a contract award. Evaluators should therefore take the time to properly review the evaluation rules and criteria, and then stick to the script.

  1. No Conflicts or Bias

Conflicts of interest and bias are bad news for the defensibility of an evaluation. Evaluators should have no personal or financial interest in the result. Furthermore, while expert evaluators may come to the table with past knowledge of the various suppliers vying for the contract award, they should also come to the evaluation process with an open mind regarding the merits of each competing proposal. Potential evaluators who are unable to meet these impartiality standards should remove themselves from an evaluation to avoid tainting the integrity of the process.

  1. No Records Means No Defence

Keeping inadequate records can result in re-evaluation orders and voided contracts. The days of jotting down cryptic shorthand on sticky notes or showing up to evaluation meetings with incomplete work and hoping that someone else will prepare a consolidated evaluation record are over. Evaluators are not anonymous in the public procurement process. Team members are accountable for maintaining their own notes and scores and responsible for recording why they made any changes to their scores during group scoring sessions.

  1. Peer Review, Not Peer Pressure

Group scoring sessions should not be a forum for fudging results or pressuring evaluators into changing their scores. In fact, emerging enhanced consensus scoring procedures now typically include detailed protocols for ensuring that each group member exercises independent judgment as part of the evaluation. While group scoring sessions should enable peer review and discussion, each evaluator is a potential witness in a legal challenge and should therefore exercise unfettered judgment and be ready to stand behind his or her individual score, even under cross-examination.

  1. No Executive Privilege

No one involved in a government evaluation decision is above the law. When it comes to protecting the integrity of the tendering process, there is no executive privilege. This means that senior officials cannot simultaneously delegate responsibility for the evaluation process and then reserve the right to arbitrarily second-guess and override an outcome they disagree with. Senior officials should therefore either formally delegate the power to make evaluation and award decisions, or they should be prepared to actively participate according to the same due process standards that apply to everyone else.

Future Considerations

While your institution may have a long track record of taking liberties with the due process of its evaluation decisions and not being challenged, you should never confuse poor but untested past practice with proper practice. In the face of a legal challenge, denial of the rules is not a winning defense. When it comes to protecting the integrity of the evaluation process, it’s never too late to rectify your procedures to establish defensible practices.