By: Paul Emanuelli

This article is an excerpt from The Art of Tendering: A Global Due Diligence Guide, which is available for purchase.

In its May 2019 determination in BEM Systems, Inc., the New York Office of the State Comptroller rejected a contract award due to a flawed evaluation process. The dispute dealt with a Department of Transportation (DOT) request for proposals (RFP) for system implementation consulting services. A losing proponent challenged the evaluation process.

As the determination detailed, the DOT used a multi-staged best and final offer (BAFO) process, which short-listed finalists and allowed those finalists to submit revised proposals for the final evaluation. A losing bidder challenged the evaluation result after its debriefing:

Subsequently, DOT determined to provide the shortlisted offerors with the opportunity to submit best and final offers (BAFO) (see RFP, at Section 5.9). The BAFO process afforded the three shortlisted offerors the opportunity to revise their technical and/or cost proposals (see RFP, at Section 5.9). All three shortlisted offerors submitted a BAFO. Evaluators were allowed to revise their technical scores based on their consideration of any new or changed technical proposal information (Id.). After this final opportunity for evaluators to rescore offerors’ technical proposals, the technical proposal with the highest-rated raw score was adjusted to 1000 points (i.e. the maximum technical points available) and the other technical proposals were adjusted proportionately downward (see RFP, at Section 5.10).
The cost proposals resulting from the BAFO were rescored, and, as before, the lowest cost proposal was awarded the maximum 450 points and the higher cost proposals received proportionately lower scores (Id.). The RFP provided that the final perfected technical scores were combined with the final perfected cost scores to produce a final best value score for each offeror, and the offeror with the highest final best value score would be chosen for award (Jd.). On September 14, 2018, DOT informed BEM that it was the tentative awardee. On October 2, 2018, DOT informed BEM that DOT made an error and the tentative award was being made to Flairsoft. On October 3, 2018, DOT formally announced that Flairsoft was the tentative awardee. BEM thereafter requested a debriefing which was conducted by DOT on October 9, 2018.

The Comptroller found that the two-staged BAFO RFP process that allowed short-listed finalists to submit revised final offers was an acceptable practice under applicable procurement laws. However, the Comptroller found deficiencies in the manner in which the final evaluations were conducted, which included a lack of consistency in the evaluation of competing proponents, a lack of transparency regarding how the group evaluation would be conducted, inconsistency in the stated point weightings for different evaluation categories, unclear evaluation records, and a lack of clarity regarding the scoring of references:

As a threshold matter, the evaluation process set forth in the RFP satisfied the statutory requirements of SFL §163(9)(b) and SFL § 163(9)(c) which permits revisions of proposals from offerors susceptible to award as part of a BAFO process when the solicitation so provides. Here, in Section 5.9 of the RFP, DOT reserved the right to request a BAFO from shortlisted offerors. In the event DOT opted to request a BAFO, Section 5.9 of the RFP permits responding firms to submit revisions to their technical and/or cost proposals and describes the evaluation process with respect t the BAFOs received. Thus, DOT’ s determination to provide shortlisted offerors an opportunity to revise their technical and cost proposals as part of the BAFO was authorized under the SFL and the terms of the RFP.
However, our review of the procurement record identified various discrepancies between the evaluation process set forth in the RFP and. the more detailed evaluation instrument developed by DOT prior to the receipt of initial offers, and the evaluation actually conducted by DOT. These discrepancies include:
  • While not provided for in either the RFP or the evaluation instrument, when an evaluator was unable to comprehend the proposer’s technical solution to a given criterion, the evaluator was allowed to consult with a group of individuals, DOT’s Information Technology Staff (ITS), to interpret the proposal. In addition, the evaluators did not consult with ITS staff on the same criteria for all proposals, raising concerns as to the whether the evaluation was consistent. Additionally, five members of ITS, as a group, also served as one of the evaluators on the TEC. Neither the RFP nor the evaluation instrument describe how this group was to score proposals collectively.
  • The total number of available points listed throughout the procurement documentation was inconsistent, contributing to confusion as to the scoring of the proposals.
  • While the RFP required evaluators to document the reasons for changes to their evaluation scores, this did not always occur and the comments that were recorded were difficult to attribute to a specific score revision (see RFP, at Sections 5.4, 5.7 and 5.8).
  • Certain evaluators provided scores for references when no reference checks were conducted and no process for independently scoring reference checks existed in the technical evaluation plan.
In light of the discrepancies between the evaluation conducted by DOT and the evaluation process described in the RFP and the evaluation instrument, the evaluation and selection process did not satisfy the requirements of SFL §§ l 63(9)(b ), 163(7).

Furthermore, the decision detailed the lack of proper record-keeping during the evaluation process, which raised significant concerns about how the individual and group rescoring was conducted after the demonstration stage and after the final BAFO submissions:

E. Procurement Record Requirements
SFL § 163(9)(a) requires that “where the basis for the award is best value, documentation in the procurement record shall, where practicable, include a quantification of the application of the criteria to the rating of the proposals and the evaluations results, or, where not practicable, such other justification which demonstrates that best value will be achieved.” Furthermore, SFL § 163(9)(g) requires that “[a] procurement record shall be maintained for each procurement identifying, with supporting documentation, decisions made by the … state agency during the procurement process.”
The procurement record submitted to this Office by DOT with the proposed DOT/Flairsoft contract does not adequately support the evaluation and selection decisions made by DOT in the procurement process and, therefore, does not provide a basis for this Office to confirm that the evaluation process conducted by DOT was consistent with its evaluation plan and resulted in the selection of the “best value” proposal.
Specifically, the multi-stage evaluation methodology established by DOT provided for an initial technical score and three potential score changes during the evaluation process (see RFP, at Sections 5.4 and 5.6). These scoring stages set forth in the RFP consist of (i) the initial independent score, (ii) the after group discussion re-score (to determine initial best value), (iii) the after demonstration re-score, and (iv) the after BAFO re-score. However, the scoresheets provided to the evaluators did not have adequate spacing for evaluators to record the various score changes during the multi-stage evaluation process and related comments in support of the reasons for such score changes. Furthermore, even when the scoresheets contained evaluators’ comments, the comments could not be matched with the corresponding scoring stage of the process. As a result of this inability to confirm evaluators’ scores during each stage of the process, this Office was unable to discern the rationale of changes by the evaluators in order to verify DOT’ s selection of shortlisted offerors and contract awardee.
Additionally, there is insufficient documentation to support the significant increase in Flairsoft’s technical score as a result of its BAFO. All three shortlisted vendors offered certain BAFO technical clarifications, yet the evaluators only made changes to Flairsoft’s technical score. Flairsoft’s after-BAFO technical score increased in 14 out of the 31 rated criteria, resulting in an increase in Flairsoft’s raw technical score from 762.3 to 827.0 points out of 1000 possible points. In one criterion (System Training) Flairsoft’s score increased from 90 to 132 points out of 150 possible points. The BAFO resulted in enough additional technical points to substantially change the final award from BEM to Flairsoft, but the documentation supporting this change is vague and ambiguous. And when this Office specifically requested an explanation for the changes in Flairsoft’s technical score after the BAFO, the only further information provided by DOT was that Flairsoft significantly reduced the number of customizations required and committed to the addition of a search capability to allow users to access historic comparable sales.

The Comptroller therefore concluded that, based on its review of the evaluation records, it was “unable to confirm that DOT’s evaluation and selection decisions were in accordance with the evaluation methodology set forth in the RFP and the evaluation plan and whether the evaluation conducted resulted in a best value award.” The contract award resulting from that evaluation process was rejected due to those evaluation flaws. As this case illustrates, while multi-staged RFP evaluation processes may be acceptable under public sector procurement rules, those multi-staged evaluations must be conducted in a manner that is consistent with the criteria and process rules established in the RFP, and those evaluations must be well-documented in order to be defensible.