This article is an excerpt from The Art of Tendering: A Global Due Diligence Guide, which is available for purchase.
In its fall 2017 report entitled Report 1—Phoenix Pay Problems, the Auditor General of Canada found serious deficiencies in the government’s Phoenix project, a payroll modernization initiative. As the Auditor General summarized, the Phoenix initiative aimed to centralize the payroll function for approximately 70 percent of federal employees across 46 departments in a Public Services Pay Centre in Miramichi, New Brunswick, and to replace 1,200 pay advisor positions across those departments with 460 pay advisors and 90 support staff in the new pay centre.
A core part of that initiative included replacing a 40-year-old pay system with a new system that would be used by 101 federal departments and agencies, including the 46 departments to be serviced out of the new central pay centre.
As the Auditor General found, the payroll complexities flowing from the large number of collective agreements required significant customizations to the Phoenix system:
The implementation of Phoenix was complex. There were more than 80,000 pay rules that needed to be programmed into Phoenix. This is because there are more than 105 collective agreements with federal public service unions, as well as other employment contracts. In addition, many departments and agencies have their own human resource systems to manage employees’ permanent files that include basic information such as address, job classification, rate of pay, and types of benefits. Phoenix needs some of this information to accurately process pay, so the 34 human resource systems across the Government of Canada needed an interface to share information with Phoenix. As a result, to handle these pay rules and interfaces with human resource systems, Public Services and Procurement Canada added more than 200 custom-built programs to Phoenix.
The Auditor General found an exponential increase in the number of outstanding pay requests once the new system went live, which totalled 150,000 requests valued at half a billion dollars (CAD):
Overall, we found that a year and a half after the Phoenix pay system was launched, the number of public servants in departments and agencies using the Miramichi Pay Centre who had an outstanding pay request quadrupled to more than 150,000. Departments and agencies have struggled with pay problems since the launch of Phoenix. However, it took Public Services and Procurement Canada four months to recognize that there were serious pay problems, and it took the Department about a year after that to have a better understanding of the problems. During this time, the Department focused on responding to the growing number of pay requests. At the end of our audit, the Department had started to develop a longer-term plan toward a sustainable solution…[T]he problem grew to the point that as of 30 June 2017, unresolved errors in pay totalled over half a billion dollars. This amount consisted of money that was owed to employees who had been underpaid plus money owed back to the government by other employees who had been overpaid.
As the Auditor General observed, once the issues arose, “departments and agencies struggled to do the tasks they were responsible for in paying their employees under Phoenix” since “Public Services and Procurement Canada did not provide sufficient information and reports to help departments and agencies understand and resolve their employees’ pay problems.” The Auditor General noted that “the federal government has an obligation to pay its employees on time and accurately”, and that not doing so “has had serious financial impacts on the federal government and its employees.”
The Auditor General summarized the impact of the system failures as follows:
Since Phoenix was launched, there were only two months in which the Miramichi Pay Centre processed more pay requests than it took in, and the number of outstanding pay requests continued to grow. We found that employees had to wait an average of more than three months to have a pay request processed. At the end of our audit period, nearly 49,000 employees had been waiting for more than a year to have pay requests processed. Two thirds of these employees had a pay request that Public Services and Procurement Canada deemed to have a high financial impact, which it defined as over $100…[A]s of 30 June 2017, there was over $520 million in pay outstanding due to errors for public servants serviced by Phoenix who were paid either too much or too little. We found that the number of payroll payments that contained an error increased over the 2016–17 fiscal year. We calculated that about 51 percent of employees had errors in their paycheques issued on 19 April 2017, compared with 30 percent in the pay issued on 6 April 2016. These error rates were significantly higher than what Public Services and Procurement Canada targeted.
As this initial report concluded, the implementation failure had a significant adverse impact on government operations because the government was no longer able to reliably pay its staff.
In its spring 2018 report entitled Building and Implementing the Phoenix Pay System, the Auditor General of Canada followed up on a 2017 fact-finding report and focused on the causes for the federal government’s failed attempt to deploy its new Phoenix payroll system. As the Auditor General stated, the government initially planned to save $70 million a year in efficiencies through the deployment of the new Phoenix system:
The government expected the initiative to save it about $70 million a year, starting in the 2016–17 fiscal year. These savings were largely to be achieved through eliminating about 1,200 positions in departments and agencies for pay advisors—specialists who process pay, advise employees, and correct errors—which were replaced by 550 positions, including 460 pay advisors, at the Miramichi Pay Centre; automating many pay processes that were manual under the old system, using new software; and eliminating duplicate data entry and processing by integrating pay operations with the Government of Canada’s approved human resource management system.
However, the project did not go as planned. The Auditor General summarized the conclusions from its previous fall 2017 fact-finding report, as follows:
In our fall 2017 audit of Phoenix pay problems, we found that the system had problems immediately after it was put in place and that they continued to grow. Departments and agencies have struggled to pay their employees accurately and on time. In that audit, we found that as of 30 June 2017, there was over $520 million in pay outstanding due to errors for public servants serviced by Phoenix who were paid too much or too little. We calculated that about 51% of employees had errors in their paycheques issued on 19 April 2017, compared with 30% in the pay issued on 6 April 2016. We found that the number of outstanding pay requests— such as a request to make a change to an employee’s pay because of a promotion or to fix an error—had continued to grow to over 494,500 by June 2017.
The Auditor General’s spring 2018 follow-up report found that the inability of Public Services and Procurement Canada to properly manage the project led to the following significant operational failures:
Overall, we found that Public Services and Procurement Canada failed to properly manage the Phoenix project. Because of the Department’s poor management, Phoenix was implemented
without critical pay processing functions;
without having been fully tested to see whether it would operate as expected;
with significant security weaknesses, which meant that the system did not protect public servants’ private information;
without an adequate contingency plan in case the system had serious and systemic problems after it was implemented; and
without any plans to upgrade the underlying software application after it was no longer supported.
The Auditor General found that Public Services and Procurement Canada failed to properly consult the impacted departments and agencies and failed to properly test the system before deployment:
Furthermore, we found that Public Services and Procurement Canada did not fully consult and involve other departments and agencies during the development of Phoenix to determine what they needed Phoenix to do or to adequately help them move to the new system. The Department did not completely and properly test Phoenix before its implementation, which is contrary to recognized practices for developing a system. Phoenix executives cancelled a pilot project with one department that would have assessed whether Phoenix was ready to be used government-wide.
The Auditor General traced the root causes of the unsuccessful deployment to the project team’s failure to obtain an adequate budget for the large initiative, which resulted in a significant scaling back of system functionalities:
Large information technology projects require balancing cost, schedule, and scope. We found that Phoenix executives scaled back the project’s functions to save money or time. In the spring of 2012, after the planning phase of Phoenix, IBM told Public Services and Procurement Canada that Phoenix would cost $274 million to build and implement. The Treasury Board had approved a Phoenix building and implementation budget of $155 million in 2009. We found that Public Services and Procurement Canada did not consider asking the Treasury Board for more money to build and implement Phoenix. Instead, Phoenix executives decided to work with IBM to find ways to reduce the scope of work to fit the approved budget. As a result, Phoenix executives decided to remove some pay processing functions, not test some pay processing functions, shorten the project schedule by compressing the time between the two waves of Phoenix from seven months to two months, and reduce the number of IBM and Public Services and Procurement Canada employees assigned to the development and implementation of Phoenix.
We found that overall, Phoenix executives decided to defer or remove more than 100 important pay processing functions, including the ability to process requests for retroactive pay, such as acting pay, which is provided to an employee acting in a temporary role for a superior; notify employees by email of actions required on their part to process pay requests; and automatically calculate certain types of pay, such as increases in pay for acting appointments.
As the Auditor General observed, the project executives did not assess the impact of the scaled-down functionality before deciding to proceed with deployment. Instead, they planned to add missing functions after the system’s initial deployment:
Phoenix executives planned for these important functions to be added to Phoenix only after all 101 departments and agencies had been transferred to it. Phoenix executives did not re-examine the system’s expected benefits after they decided to significantly scale back what Phoenix would do. They should have known that such a significant change in the project scope could put the system’s functionality and projected savings at risk and undermine the government’s ability to pay its employees the right amount at the right time.
The Auditor General also found that the project team failed to properly test the system before it went live, resulting in many of the serious operational deficiencies subsequently encountered following deployment:
Testing Phoenix. We found that Public Services and Procurement Canada could not show that the Phoenix functions that had been approved as part of the February and April 2016 implementations were in place by those dates and were fully tested before implementation. The Department had identified 984 functions that it needed to include in the February and April 2016 implementations so that pay advisors could process pay. We reviewed 111 of them. We found that 30 of these 111 functions were not part of Phoenix when departments and agencies started to use it—the functions had been either removed or deferred. The decisions to remove or defer some of these functions, such as the processing of retroactive pay, led to increases in outstanding pay requests and pay errors. For the remaining 81 functions we reviewed, we found that 20% did not pass testing by Public Services and Procurement Canada before implementation. The Department did not retest the functions that failed the original testing.
The Auditor General found that weak oversight contributed to the failure since there were inadequate checks-and-balances placed on the project team when it improperly prioritized its scheduling and budget objectives over broader operational objectives:
Overall, we found that there was no oversight of the Phoenix project, which allowed Phoenix executives to implement the system even though they knew it had significant problems. There were no oversight bodies independent of the project management structure to provide independent advice to the Deputy Minister of Public Services and Procurement on the project status. This meant that the Deputy Minister did not receive independent information showing that Phoenix was not ready to be implemented or that the Miramichi Pay Centre and departments and agencies were not ready for Phoenix. Phoenix executives were more focused on meeting the project budget and timeline than on what the system needed to do.
Further, the audit stated that Phoenix executives failed to heed warnings that the system was not ready, proceeding to launch notwithstanding clear signs of pending system failure:
We found that before implementing Phoenix, Phoenix executives did not heed clear warnings that the Miramichi Pay Centre, departments and agencies, and the pay system itself were not ready. It was obvious even before Phoenix was in place that pay advisors in Miramichi were not handling the volume of files that the Department expected. Furthermore, several departments and agencies, along with third-party assessments, identified significant problems with the system. Phoenix executives did not understand the importance of these warnings and went ahead with implementing Phoenix.
In its subsequent October 2018 follow-up report, the Auditor General found that the government remained unable to rectify the outstanding payment errors, which at that point amounted to over $615 million in errors:
In 2017–2018, Phoenix processed approximately $25 billion in pay expenses. The government continued to have numerous challenges in accurately paying employees this fiscal year.
Due to these challenges, auditing pay expenses as part of the 2017–2018 annual audit of the Government of Canada’s consolidated financial statements remained labour intensive. We looked at an estimated 16,000 pay transactions across 47 of the 101 departments and agencies that used Phoenix. We found underpayments and overpayments made to employees. As a result of our testing, we estimated that the government owed employees $369 million (because the employees were underpaid) and employees owed the government $246 million (because they were overpaid). In other words, there were approximately $615 million worth of pay errors as at March 31, 2018.
The October 2018 report also found that despite the government’s ongoing efforts, these errors impacted well over half of all employees in its test sample, and that the situation was not improving:
We found that 62% of the employees in our sample were paid incorrectly at least once during the fiscal year (it was also 62% in the 2016–2017 fiscal year). For employees with pay errors, the number of errors for the year ranged from 1 to 19. As at March 31, 2018, 58% of the employees in our sample still required corrections to their pay—a similar percentage to the last fiscal year’s. These findings show that the situation for government employees has not improved.
As this case study illustrates, government institutions that fail to properly fund projects or establish appropriate project governance frameworks, run the risk of launching runaway projects that end up failing to meet operational objectives while also costing more time and money in the long run.