Case Study: $250 Million Failure, Who Are You Going To Sue?

This is one of two Case Studies on recent government project failures that I was asked to review. I am providing this assessment as a public service. The other assessment is on the Massachusetts Division of Unemployment Assistance.

Analysis

The in-depth analysis is restricted to the September 2012 Oracle contracts since that is the only set of original documents with enough data to make an analysis. Two audits reports were also read—one from Centers for Medicare & Medicaid Services (CMS) and the other from FirstData—and Maximus' two quality reports from November and December.

Figure 1: Discrepancy in PO Values

Lack of Clarity in POs

The September contract for Oracle was for $35.1MM with a very clear assignment to "assist" Oregon on a (T&M) basis to build the ORHIX system. This amount was for an estimated 118,477 hours of services totaling $35,094,504. There are, however, irreconcilable differences on this PO. The PO body has a narrative that defines the work is being contracted. The narrative list defines one extra area of work and the summary hours are inconsistent with those in the subsequent hourly estimates table (even though the grand total of hours for both sets are the same 188,477-hour total). When calculating the value of the PO using the labor rate and estimated hours tables found in the PO, the value should be $34,139,415.62—$955,088.38 less than what was paid to Oracle. Due to the lack of access to the Cover Oregon staff, this is unexplained.

Project Structure

The project had a number of root issues that gravely affected its ability to complete successfully. From my analysis, there were four major issues:

Figure 2: September SOW with focus on "Assist"

1. Hiring Oracle on a T&M basis. Time and materials contract placed all the risk for completing the project on the State of Oregon. Oracle's September 2012 contract has very little scope that they had to provide. A large majority of their service delivery lines start with the word "assist" (see Figure 2). They are clearly subservient to either the State of Oregon or a System Integrator (who ended up being the State). Oracle's contract should have been fixed-price for specific deliverables.
2. Failing to restrict the scope to what was essential to meet the go-live requirements for October 1, 2013. The scope of the project, according to the two independent audits, was not only to develop the ORHIX, but also to include the DHS Modernization project. This created the MaX project. The decision should have been to minimize scope by trimming it to a minimally viable product for the ORHIX go-live in October 1, 2013. The DHS Modernization should have been postponed to a subsequent year and a roadmap developed to define the phase-in goals for it after a successful go-live and system stabilization.
3. Failing to hire a system integrator with large-scale system integration experience. Projects to build systems of this size are rare. Their chances for success are low regardless of who builds them. Trying to be the integrator with little or no experience has almost no chance of success. A system integrator should have been hired on a fixed-price contract to run and manage the project.
4. Inadequate project governance and oversight. Numerous issues raise questions on the governance of the project from only looking at the minimal contractual documents that were available. These include:
   • Lack of reaction to the Maximus reports. It is unclear how the Maximus reports could go unheeded. This underscores, however, the most serious flaws in communication, cooperation, and understanding of reality of the project.
   • Hiring Oracle on T&M. As mentioned above Hiring Oracle on T&M removed all accountability. With a contract that primary had Oracle in an assisting role gave them less accountability.
   • Use of the Dell Contract process. It is not understood why the contracts were processed through Dell (these were service contracts processed through a hardware purchasing agreement). Accord to the other audits, the practice of putting these contracts through Dell continued. This only added cost to the project.

The CMS and FirstData audits point out many other areas where governance and communication fell short. This is a major issue that must be resolved prior to moving forward with this project or others.

Maximus Quality Reports

Figure 3: Maximus Quality Summary

Oregon hired Maximus to run monthly audits of the ORHIX project (Maximus refers to these as quality reports). These reports started in 2011, but only two reports were available to review—November and December of 2013. They show multiple areas where the project at high risk for protracted lengths of time. They clearly highlight and document issues early in the project. Figure 2 is one page from the December 2013 report that clearly indicates a history of issues on the project.

These reports underscore a lack of project control since they clearly show the project was in trouble from early 2013. Audits conducted by CMS and FirstData confirm that the project's oversight team failed to heed Maximus' warnings. According to FirstData, "multiple members of the [Legislative Oversight] committee told us they were completely unaware of the Maximus QA role and had not received any of the QA reports".Audit ReportsAs mentioned above, two different organizations performed audits on the project in 2014—after it was clear that the project was seriously missing its goals. These reports add to the findings here. Neither was commissioned to find root causes (CMS was a Federal audit to assess the state of the project, and FirstData was to answer seven specific questions). Both reports are very good, but stop at charting a course to prevent re-occurrence of the problem on this or other projects (this was not in their scope).No reports can be complete, and, although very valuable, FirstData's points out a couple inconsistencies that should be investigated. Three of these are:

• They report that the decision was made to not use a system integrator in 2011. However, the contract for services with Oracle in September, 2012, clearly states:

  "[Oracle will] Assist the SI Vendor with Architecture, Design Guidelines to build interfaces to DHS systems, Federal Systems, and third-party systems for the following: "1. Real time integrations using the Oracle SOA Suite. "2. Web Services "3. Batch interfaces"

  If the decision had been made a year earlier, why would Oracle say this when the rest of the document clearly talks about work with the State?
• The list of POs in their report does not include a PO for a Cloud Services contract signed on August 21, 2013. This is a three-year contract for $22MM of Cloud Services.
• The completion of a complete financial audit on the project. Aside from the discrepancy noted above, FirstData enumerates 43 POs that were issued to Oracle. There are references, however, to another contracted amount prior to September 2012 that is not reflected in their list. Notably, the detail in the Dell Quote BPB11111715-R07 (Addendum Two of Oracle License and Service Agreement US-TERM-GMA-5772-30-NOV-2011) refers to an original contract in the value of $7,241,836. The POs listed by FirstData cannot be reconciled to this amount.

Recommendations

In my opinion, although Oracle is culpable in the failure, they were on a T&M contract and it was the State of Oregon and Cover Oregon's lack of governance and controls that is the biggest issue. The investigation into what happened needs to move on to find the root causes and solve them. That must to be done before any further money is spent on completing the ORHIX project or defining an alternative option. The two steps are:

• A complete financial audit to show the actual and complete cost of the system. This should include money spent on ORHIX, DHS Modernization, and MaX (the combined projects). It should itemize all internal and external costs (not just Oracle).
• A root cause analysis focused on finding solutions, rather than blame. The FirstData analysis, although well done, was commissioned to answers to seven questions, which appear to be looking for blame. Blame does not solve the problem. It is only political fodder. To find answers an audit needs to look at the following:
  1. What were the processes for each of these roles and what changes need to be implemented to make the role effective:
     1. Scope definition.
     2. Project planning (Schedules, project methodology, risk assessment, mitigation planning, assumption tracking, communications, etc.).
     3. Inter-agency and project communication.
     4. Issue resolution and escalation.
     5. System architecture and validation.
     6. System and interface development.
     7. Project Quality: System testing, test definition and approval, and regression testing.
     8. System deployment and training.
     9. Vendor selection, due diligence, and contract review.
     10. Task/resource allocation.
  2. What was the plan around project governance and what is an appropriate plan to make it work?

Constraints on Analysis

According to the March 19, 2014 Assessment Report completed by FirstData, there were 43 POs issued to Oracle with a value of $132MM. There were other vendors and subcontractors involved in the Cover Oregon project. The value of their POs and the State's internal costs are unknown. My analysis was conducted using a small subset of all contractual documents along with assessments from Maximus, CMS, and First Data. No one from the State or Oracle was consulted in making this assessment or its recommendations. This is a serious constraint on this analysis and hence it only represents are a small part of what needs to be done.The POs reviewed include Oracle's September 2012 time and materials (T&M) services agreement and the August 2013 Cloud Services purchase orders (POs). The value of these POs is $56.2MM. The former references an initial PO for $7.2MM, but it was not reviewed. No POs were seen for the software purchase; however, I reviewed the Oracle Software License agreement that listed the packages and seat counts. Following is a summary of these items.Conflict of Interest DisclaimerThis was a non-paid engagement and neither eCameron nor myself has received any form of remuneration for this effort. The genesis of the investigation based on a request from a reporter at the Oregonian who asked for an interpretation of the material that he provided. This assessment is based totally on those documents.With these items completed and addressed, this and other large projects can proceed.