Sunday, 30 March 2014 00:00

Case Study: Division Of Unemployment Assistance, Massachusetts/Deloitte

Rate this item
(2 votes)

In May 2007, the Massachusetts Division of Unemployment Assistance (DUA) signed a contract with Bearing Point, Inc. to modernize the State’s unemployment processing system. The project was called the DUA Quality Unemployment System Transformation (QUEST) Project. Bearing Point filed for bankruptcy in February 2009 and Deloitte announced they would buy Bearing Point for $350MM in March of the same year.

The system was delivered in July 2013 for a cost of $46.6MM plus additional maintenance contracts for $4.5MM. It had over a 1000 reported bugs (no severity assigned) shortly after delivery. As of February 2014, there were reports of 100 bugs in the system (no severity assigned). Project cost, as compiled from SOW and addendums, is is shown in Table 1.

This is one of two Case Studies on recent government project failures that we were asked to review. I am providing this assessment as a public service. The other assessment is on the Oregon’s Cover Oregon Health Insurance Exchange.

Table 1: Bearing Point/Deloitte SOW and Addendums  
DescriptionDateFunctionality
Cost
Functionality
Chg. Req. Cost
 Maintenenace
Cost
Original Fixed Price May-07 $ 40,000,000      
Addendum 1 - delay only Sep-07 $ 40,000,000      
Addendum 2 - delay only May-08 $ 40,000,000      
Addendum 3 May-09 $ 40,594,880 $ 594,880    
Addendum 4 Missing        
Addendum 5 Aug-10 $ 41,815,560     $ 1,220,680
Addendum 6 Dec-10 $ 42,236,920 $ 421,360   $ 1,626,240
          $ 1,126,782
Addendum 7 Apr-11 $ 42,425,710 $ 188,790    
Addendum 8 Nov-11 $ 46,647,939 $ 256,375   $ 1,656,900
      $ 3,965,854    
Addendum 9 Feb-13 $ 46,647,939      
Functionality Total: $ 46,647,939 Maintenance $ 5,630,602
Contract Total: $ 52,278,541

Analysis

As with many failures, there are a number of factors that contributed. Some were out of everyone’s control, while others were due to lack of proper controls or process. From our analysis of the project three major factors were instrumental in the failure. Those were:

  1. The 2008 recession affecting the number of unemployment claims.
  2. Poor quality processes used by the vendor.
  3. Lack of large project experience by the State.

External Factors

The project serendipitously started about seven months prior to the start of the 2008 recession. Between January 2008 and December 2009 unemployment rates in Massachusetts soared from about 4.5% to 8.4%, doubling the number of unemployed people from 150,000 to 300,000.1 This severely taxed the DUA and surely restricted the amount of time people (especially subject matter experts) in the DUA could be assigned to the project.

Quality Processes

At delivery in July 2013, the DUA claimed quality was an issue. It appears that attention to and tracking of quality was not a focus by Bearing Point or, later, Deloitte. This was endemic from the start of the project. The original May 2007 contract shows major signs of that. Appendix A, titled “Deliverable Acceptance Criteria (Example)” is a blank page. Quality appeared to be a major focus of Addendum 8 of the contract (signed November 2011). In this addendum, Deloitte and DUA settled on a sampling plan for testing defects.

By definition, sample-plan testing allows for a given and determinant level of issues. For instance, one test allowed for 200 claims to be tested and 100% of them would need to pass. There is no mention of the test cases that were built, so it is assumed that the sampling was random and not testing edge cases. At approximately 250,000 claims per month (the number processed in February 2014), testing 200 with a acceptance rate of 100% would imply a statistical failure rate of about 0.89% or 2,250 failed claims per month. Expectations outside the DUA and Deloitte were that all claims would be processed by the system while the sampling plan was clearly designed otherwise.

Although I am not in a position to validate the claim, it appears that many of the bugs in the system are letting unreasonable issues pass through. The implication is that exception processing in the system as poorly designed. In other words, some processes are too difficult to handle programmatically and need human intervention. These should drop out of the system in a controlled and logical manner for people to assess and process.

Inexperience With Large-Scale Integration

In conversations with the Senate Committee on Post-Audit and Oversight, it was divulged that the state has little experience with large technology projects. A general lack understanding of how the deliverables were structured, basic governance to monitor progress and identify issues in the statements of work, and the use of a waterfall approach all added to the failure from the State’s side.

As mentioned above, traces of quality issues were at the beginning of the project and there did not seem to be a concern. These should have been addressed early in the project. Quality processes should have used industry standard acceptance test procedures that test edge cases. Sample testing should not have been used. Even as recently as February 2014, bugs were being reported without classifying their severity. It cannot be assumed that the 100 bugs reported in February were a mix of Severity 1 through 4 nor all Severity 1. Lack of notation indicates a poor understanding of testing and lax communication.

Bearing Point Appendix A Image

Figure 1: Bearing Point SOW Appendix A

Governance of the project was also deficient. Early in 2008, when the unemployment rate was on the rise, the DUA (or even BearingPoint) should have stopped the project instead of delaying it. It was evident that the DUA was going to be too busy to dedicate time to the project. This may have caused funding issues as money may not have been able to be carried forward, but under the circumstances the legislature could have made an exception.

Numerous other factors are also evident, but unconfirmed. For instance, the overview of the original SOW states that Bearing will “Re-engineer the UI [Unemployment insurance] business model.” This is a major undertaking and, in my opinion, would require far more than originally bid in both time and effort.

Other Factors

There were unconfirmed statements that the use of uFACTS (Deloitte’s Unemployment Framework for Automated Claim and Tax Services product) was to mimic the Minnesota implementation. If this were the case and the State had agreed with it, then it contradicts the prior mentioned commitment by Bearing Point to re-engineer the system and one would have to ask why the cost would be $40MM.

There were also claims that test procedures were to be developed by the DUA. This again points to a lack of understanding of how test procedures are built. The tests must match the requirements as documented by the integrator (Deloitte) and approved and run by the client (the State). Test procedures developed by an inexperienced client will surely not match the requirements and cause numerous false failures or improperly test edge cases.

Recommendations

The state should implement a project governance group outside of the legislative and executive branch. It should not be run by the state information technology group (i.e. not under the commonwealth chief information officers (CCIOs)), but rather a group of technically knowledgeable agency experts that are aware of the intricacies of their departments. They should be matched in authority with outside experts with experience in managing large technical projects. Strict conflict-of-interest rules should be in place to ensure that no one benefits from decisions made during the governance process.

Alternate methodologies should be used to reduce the size of the project. Iterative approaches, like agile, that allow interim deliverables would be best. However, phasing projects to produce smaller deliverables that show the solutions failings may be adequate.

Smaller projects with interim deliverables will open the bidding process to other very capable vendors that cannot bid on huge monolithic projects. This will most likely allow more in-state companies to bid and win contracts and keep the money with the State. Using local resources could also decrease costs by reducing travel expenses.

Quality and testing procedures should be run by an independent third party. The should not be part of the system integrator team (in this case Deloitte). This can be an outside party that works closely with the system integrator and the State with their incentive being cost effective testing that properly processes exceptions.

The challenge will be implementing these changes in a non-bureaucratic manner.

Constraints on Analysis

The analysis was completed by looking at initial contractual and subsequent change orders. Two conversations were held with Senate Subcommittee members. No one from the project (State or Deloitte) was consulted prior to making this assessment and recommendations.

Conflict of Interest Disclaimer

This was a non-paid engagement. Neither eCameron nor myself has received any form of remuneration, although I have asked for a thank you letter from Senator Creem’s office. The Senate Committee on Post-Audit and Oversight approached me asking for interpretation of the material provided.

Read 26062 times

Related items

  • Filling Execution Gaps: How Executives and Project Managers Turn Corporate Strategy into Successful Projects
    What Filling Execution Gaps Covers

    Filling Execution Gaps

    by Todd C. Williams
    ISBN: 978-1-5015-0640-6
    De G Press (DeGruyter), September 2017

    Project alignment, executive sponsorship, change management, governance, leadership, and common understanding. These six business issues are topics of daily discussions between executives, middle management, and project managers; they are the pivotal problems plaguing transformational leadership. Any one of these six, when improperly addressed, will hex a project's chances for success. And, they do—daily—destroying the ability companies to turn vision into value.

    Check it out on Amazon or the Filling Execution Gaps website

    Without the foundation of a common understanding of goals and core concepts, such as value being critical to success, communication stops and projects fail.

    Without change management, users fail to adopt project deliverables, value is lost, and projects fail.

    Without maintaining alignment between corporate goals and projects, projects miss their value targets and projects fail.

    Without an engaged executive sponsor, scope increases, goals drift, chaos reigns, value is lost, and projects fail.

    Without enough governance, critical connections are not made, steps are ignored, value is overlooked, and projects fail.

    Too much governance slows progress, companies cannot respond to business pressures, value drowns in bureaucracy, and projects fail.

    Without strong leadership defining the vision and value, goals are not set, essential relationships do not form, teams do not develop, essential decisions are not made, and projects fail.

  • Filling Execution Gaps: Building Success-Focused Organizations

    Executives define vision, strategy, and goals to advance the business. Projects enable companies to meet those goals. Between strategy and projects, there is a lot of work to be done—work that lays the foundation for project and operational success. Through experience and research, six common gaps exist in organizations that inhibit project success—an absence of common understanding, disengaged executive sponsors, misalignment with goals, poor change management, ineffective governance, and lackluster leadership.

  • Strategy-Execution Gaps

    The statistics on strategy execution are dismal:

    • 59% of middle managers fail at resolving conflicts in corporate strategy.
    • 45% of middle managers cannot name one of the top five corporate goals.
    • 64% of cross-department/functional issues are poorly resolved.

    And maybe as you could expect from this:

    • 53% of companies cannot react timely to new opportunities.

    You do not need to be a rocket scientist to know that this trajectory is not going to launch most companies’ latest strategic plans successfully. In fact, these data might make you feel that middle management would be better suited as test dummies for the next generation of manned space-vehicle. Granted, the data show there is a dearth of leadership in middle management, but the executive tier has a culpable hand.

  • Process Mapping

    Process is at the core of any business. It makes work predictable, repeatable, and transferable. Without it we cannot scale our businesses. However, process can be a bane to making progress. Processes that work for a $10 million company have difficulties supporting a $30 million company. Trying to scale them to a $300 million company will not only fail but not address the issues that larger companies have that were never dreamt of in a smaller organization. Processes need to be discarded, revamped, and built—all of that without creating an overburdening bureaucracy.

    Anytime you need to go someplace, you first have to know where you are. Processes are never static and your company's current state is probably far from where you think it is. Hence, the first step is mapping out you company's current state followed by defining the future state. This is more than a logical map of the process; it must also include physical maps. Whether your process is solely to provide a service (say, website development) or physical (say, manufacturing) there are logistical issues that complicate the process flow. Without fully understanding those nuances, future state processes will not reach the desired efficiencies.

    For more information about process mapping fill out the form to the left and we will get in touch with you.

  • Success vs Culture

    The other day a Latvian student contacted me for my views the connection between culture and success criteria—an important and intriguing topic. After working in Taiwan, Singapore, Korea, Japan, Israel, United States, and Canada, I wear many scars of both blatant and subtle cultural violations. I also know that within a culture one person's success is often another person's failure. So, after dispelling concerns about clicking on some random email link, I completed her survey (please feel free to take it yourself). In the process, I struck up a friendship with the student, Kristine Briežkalne, who is studying at Riga International School of Economics and Business Administration . She has some interesting views and presented me with a Venn diagram showing four frames to a project (business, client, project management, and growth perspectives) and how they intersected. As the diagram is part of her Master's thesis, I will let you ponder the how to label the overlapping areas (an eye-opening exercise).

Leave a comment

Filling Execution Gaps

Available Worldwide

Filling Exectution Gaps cover

Filling Execution Gaps is available worldwide. Below are some options.

 

PG DirectLogo
Limited Time Price $20.99
Amazon logo
Book or Kindle
Flag of the United States Canadian Flag Flag of the United Kingdom Irish Flag Deutsche Flagge
Drapeau Français Bandiera Italiana PRC flag
Japanese flag
Bandera de España
Flag of India
Bandera de México
Bandeira do Brasil
Flag of Australia
Vlag van Nederland
DeG Press Logo
Barnes and Noble Logo
Books a Million Logo
Booktopia Logo
Worldwide: Many other
book sellers worldwide.

Rescue The Problem Project

Internationally acclaimed

Image of RPP

For a signed and personalized copy in the US visit the our eCommerce website.

Amazon logo
Buy it in the United States Buy it in Canada Buy it in the United Kingdom
Buy it in Ireland Buy it in Germany Buy it in France
Buy it in Italy Buy it in the PRC
Buy it in Japan
Book sellers worldwide.

Other's References

More Info on Project Recovery

Tell me More!

Please send me more information
on fixing a failing project.

Upcoming Events

Sitemap