Sunday, 30 March 2014 00:00

Case Study: Division Of Unemployment Assistance, Massachusetts/Deloitte

Rate this item
(1 Vote)

In May 2007, the Massachusetts Division of Unemployment Assistance (DUA) signed a contract with Bearing Point, Inc. to modernize the State’s unemployment processing system. The project was called the DUA Quality Unemployment System Transformation (QUEST) Project. Bearing Point filed for bankruptcy in February 2009 and Deloitte announced they would buy Bearing Point for $350MM in March of the same year.

The system was delivered in July 2013 for a cost of $46.6MM plus additional maintenance contracts for $4.5MM. It had over a 1000 reported bugs (no severity assigned) shortly after delivery. As of February 2014, there were reports of 100 bugs in the system (no severity assigned). Project cost, as compiled from SOW and addendums, is is shown in Table 1.

This is one of two Case Studies on recent government project failures that we were asked to review. I am providing this assessment as a public service. The other assessment is on the Oregon’s Cover Oregon Health Insurance Exchange.

Table 1: Bearing Point/Deloitte SOW and Addendums  
Chg. Req. Cost
Original Fixed Price May-07 $ 40,000,000      
Addendum 1 - delay only Sep-07 $ 40,000,000      
Addendum 2 - delay only May-08 $ 40,000,000      
Addendum 3 May-09 $ 40,594,880 $ 594,880    
Addendum 4 Missing        
Addendum 5 Aug-10 $ 41,815,560     $ 1,220,680
Addendum 6 Dec-10 $ 42,236,920 $ 421,360   $ 1,626,240
          $ 1,126,782
Addendum 7 Apr-11 $ 42,425,710 $ 188,790    
Addendum 8 Nov-11 $ 46,647,939 $ 256,375   $ 1,656,900
      $ 3,965,854    
Addendum 9 Feb-13 $ 46,647,939      
Functionality Total: $ 46,647,939 Maintenance $ 5,630,602
Contract Total: $ 52,278,541


As with many failures, there are a number of factors that contributed. Some were out of everyone’s control, while others were due to lack of proper controls or process. From our analysis of the project three major factors were instrumental in the failure. Those were:

  1. The 2008 recession affecting the number of unemployment claims.
  2. Poor quality processes used by the vendor.
  3. Lack of large project experience by the State.

External Factors

The project serendipitously started about seven months prior to the start of the 2008 recession. Between January 2008 and December 2009 unemployment rates in Massachusetts soared from about 4.5% to 8.4%, doubling the number of unemployed people from 150,000 to 300,000.1 This severely taxed the DUA and surely restricted the amount of time people (especially subject matter experts) in the DUA could be assigned to the project.

Quality Processes

At delivery in July 2013, the DUA claimed quality was an issue. It appears that attention to and tracking of quality was not a focus by Bearing Point or, later, Deloitte. This was endemic from the start of the project. The original May 2007 contract shows major signs of that. Appendix A, titled “Deliverable Acceptance Criteria (Example)” is a blank page. Quality appeared to be a major focus of Addendum 8 of the contract (signed November 2011). In this addendum, Deloitte and DUA settled on a sampling plan for testing defects.

By definition, sample-plan testing allows for a given and determinant level of issues. For instance, one test allowed for 200 claims to be tested and 100% of them would need to pass. There is no mention of the test cases that were built, so it is assumed that the sampling was random and not testing edge cases. At approximately 250,000 claims per month (the number processed in February 2014), testing 200 with a acceptance rate of 100% would imply a statistical failure rate of about 0.89% or 2,250 failed claims per month. Expectations outside the DUA and Deloitte were that all claims would be processed by the system while the sampling plan was clearly designed otherwise.

Although I am not in a position to validate the claim, it appears that many of the bugs in the system are letting unreasonable issues pass through. The implication is that exception processing in the system as poorly designed. In other words, some processes are too difficult to handle programmatically and need human intervention. These should drop out of the system in a controlled and logical manner for people to assess and process.

Inexperience With Large-Scale Integration

In conversations with the Senate Committee on Post-Audit and Oversight, it was divulged that the state has little experience with large technology projects. A general lack understanding of how the deliverables were structured, basic governance to monitor progress and identify issues in the statements of work, and the use of a waterfall approach all added to the failure from the State’s side.

As mentioned above, traces of quality issues were at the beginning of the project and there did not seem to be a concern. These should have been addressed early in the project. Quality processes should have used industry standard acceptance test procedures that test edge cases. Sample testing should not have been used. Even as recently as February 2014, bugs were being reported without classifying their severity. It cannot be assumed that the 100 bugs reported in February were a mix of Severity 1 through 4 nor all Severity 1. Lack of notation indicates a poor understanding of testing and lax communication.

Bearing Point Appendix A Image

Figure 1: Bearing Point SOW Appendix A

Governance of the project was also deficient. Early in 2008, when the unemployment rate was on the rise, the DUA (or even BearingPoint) should have stopped the project instead of delaying it. It was evident that the DUA was going to be too busy to dedicate time to the project. This may have caused funding issues as money may not have been able to be carried forward, but under the circumstances the legislature could have made an exception.

Numerous other factors are also evident, but unconfirmed. For instance, the overview of the original SOW states that Bearing will “Re-engineer the UI [Unemployment insurance] business model.” This is a major undertaking and, in my opinion, would require far more than originally bid in both time and effort.

Other Factors

There were unconfirmed statements that the use of uFACTS (Deloitte’s Unemployment Framework for Automated Claim and Tax Services product) was to mimic the Minnesota implementation. If this were the case and the State had agreed with it, then it contradicts the prior mentioned commitment by Bearing Point to re-engineer the system and one would have to ask why the cost would be $40MM.

There were also claims that test procedures were to be developed by the DUA. This again points to a lack of understanding of how test procedures are built. The tests must match the requirements as documented by the integrator (Deloitte) and approved and run by the client (the State). Test procedures developed by an inexperienced client will surely not match the requirements and cause numerous false failures or improperly test edge cases.


The state should implement a project governance group outside of the legislative and executive branch. It should not be run by the state information technology group (i.e. not under the commonwealth chief information officers (CCIOs)), but rather a group of technically knowledgeable agency experts that are aware of the intricacies of their departments. They should be matched in authority with outside experts with experience in managing large technical projects. Strict conflict-of-interest rules should be in place to ensure that no one benefits from decisions made during the governance process.

Alternate methodologies should be used to reduce the size of the project. Iterative approaches, like agile, that allow interim deliverables would be best. However, phasing projects to produce smaller deliverables that show the solutions failings may be adequate.

Smaller projects with interim deliverables will open the bidding process to other very capable vendors that cannot bid on huge monolithic projects. This will most likely allow more in-state companies to bid and win contracts and keep the money with the State. Using local resources could also decrease costs by reducing travel expenses.

Quality and testing procedures should be run by an independent third party. The should not be part of the system integrator team (in this case Deloitte). This can be an outside party that works closely with the system integrator and the State with their incentive being cost effective testing that properly processes exceptions.

The challenge will be implementing these changes in a non-bureaucratic manner.

Constraints on Analysis

The analysis was completed by looking at initial contractual and subsequent change orders. Two conversations were held with Senate Subcommittee members. No one from the project (State or Deloitte) was consulted prior to making this assessment and recommendations.

Conflict of Interest Disclaimer

This was a non-paid engagement. Neither eCameron nor myself has received any form of remuneration, although I have asked for a thank you letter from Senator Creem’s office. The Senate Committee on Post-Audit and Oversight approached me asking for interpretation of the material provided.

Read 12165 times

Related items

  • Process Mapping

    Process is at the core of any business. It makes work predictable, repeatable, and transferable. Without it we cannot scale our businesses. However, process can be a bane to making progress. Processes that work for a $10 million company have difficulties supporting a $30 million company. Trying to scale them to a $300 million company will not only fail but not address the issues that larger companies have that were never dreamt of in a smaller organization. Processes need to be discarded, revamped, and built—all of that without creating an overburdening bureaucracy.

    Anytime you need to go someplace, you first have to know where you are. Processes are never static and your company's current state is probably far from where you think it is. Hence, the first step is mapping out you company's current state followed by defining the future state. This is more than a logical map of the process; it must also include physical maps. Whether your process is solely to provide a service (say, website development) or physical (say, manufacturing) there are logistical issues that complicate the process flow. Without fully understanding those nuances, future state processes will not reach the desired efficiencies.

    For more information about process mapping fill out the form to the left and we will get in touch with you.

  • Success vs Culture

    The other day a Latvian student contacted me for my views the connection between culture and success criteria—an important and intriguing topic. After working in Taiwan, Singapore, Korea, Japan, Israel, United States, and Canada, I wear many scars of both blatant and subtle cultural violations. I also know that within a culture one person's success is often another person's failure. So, after dispelling concerns about clicking on some random email link, I completed her survey (please feel free to take it yourself). In the process, I struck up a friendship with the student, Kristine Briežkalne, who is studying at Riga International School of Economics and Business Administration . She has some interesting views and presented me with a Venn diagram showing four frames to a project (business, client, project management, and growth perspectives) and how they intersected. As the diagram is part of her Master's thesis, I will let you ponder the how to label the overlapping areas (an eye-opening exercise).

  • Kill The White Knight

    There is a reason we do not teach classes on fixing failing projects. Many a cynic feels that we simply do not want to teach our trade, however, our reason is far nobler—we should be teaching prevention rather trying to create white knights to save the day. It is the same philosophy as building a fence at the cliff's edge rather than an emergency room at its base. Our language is replete with idioms telling us to look past the symptom and address problems at their root cause. 'An ounce of prevention versus a pound of cure' or 'a stitch in time saves nine.' Please, feel free to supply your own in the comments. Unfortunately, most of our businesses loathe this philosophy, waiting to address an issue until it is irrefutably broken.

  • Tales of an Expert Witness: Sex, Lies, and Video Tape (Part II)

    Trust relationships, certifications, and standards sound like such a safe harbor. These sound like such great words in a proposal or statement of work. How could you possibly go wrong building a trusted relationship with a customer by committing to follow a standard? In fact, this can burn you… in court.

    No one ever starts a project with the goal of ending up in court. In fact, litigation may never cross your mind; after all, you have built a trusted partner relationship. Taking a few cautionary steps, however, will make your life easier if you end up in that ill-fated litigious position. Your best chances for success come long before you enter the courtroom—even before the project starts.

  • Comparing Organizational Change Management Models

    A few weeks ago, I set out to write a post on the comparison of various organizational change management (OCM) methodologies and realized that would be a disservice to my readers. It would simply drag you down the path of implementation while failing to focus you on building the foundation. The pressure was too much and I have relented to numerous requests on making that comparison. The caveat is that juxtaposing these models is not comparing different varieties of oranges or even apples and oranges; we are surely comparing the peel to the fruit they contain. Hence, comparing methodologies like Kotter's model (the peel), Prosci's ADKAR (the core), and General Electric's Change Acceleration Process (the whole fruit) need a different approach.

Leave a comment

More Info on Project Recovery

Tell me More!

Please send me more information
on fixing a failing project.

Rescue The Problem Project

Internationally acclaimed

Image of RPP

For a signed and personalized copy in the US visit the our eCommerce website.

Amazon logo
Buy it in the United States Buy it in Canada Buy it in the United Kingdom
Buy it in Ireland Buy it in Germany Buy it in France
Buy it in Italy Buy it in the PRC
Buy it in Japan
Book sellers worldwide.

Upcoming Events

Other's References