Obamacare: descent into the maelstrom [UPDATED x2]

| October 9, 2013

maelstrom

As I have noted previously — here (pre-launch) and again here (post-launch) — the problems with the Healthcare.gov website (and, as a consequence, with the various state healthcare exchanges) are more than a few “glitches” or “bumps”. They appear to indicate serious, fundamental issues with the underlying architecture, design, and implementation of the Healthcare.gov systems. This latest report — from Sean Gallagher at Ars Technica — suggests how deep the problems go:

Amid all the attention, bugs, and work happening at Healthcare.gov in light of the Affordable Care Act, potential registrants talking to phone support today have been told that all user passwords are being reset to help address the site’s login woes. And the tech supports behind Healthcare.gov will be asking more users to act in the name of fixing the site, too. According to registrants speaking with Ars, individuals whose logins never made it to the site’s database will have to re-register using a different username, as their previously chosen names are now stuck in authentication limbo.

That’s pretty bad. And lame.

While Sean seems to make a few excuses for the website, he nevertheless chronicles not only the various problems the site has, as well as some of the factors behind its problems:

There were even earlier causes for concern. Back in March, concerns about the funding levels for the program prompted Baitman and HHS management to rate the program as “high risk”—giving it a score of 1 out of 5. In June, the Government Accountability Office, the nonpartisan auditing body that provides oversight reports to Congress, said that it was still a crapshoot as to whether the system would work on time. This uncertainty persisted because the hub being built by QSSI still hadn’t been completely tested (the hub is responsible for making automated decisions about eligibility). While the policies to govern how the hub works—and how various state systems were supposed to work—had been completed, there was still a lot of code to be written to make those policies into an actual system.

All of that pushed the development of the system closer and closer to the deadline. As one reddit user posted when the site ran into trouble on October 1, “My wife works on this project but not as a developer. Last night she said, ‘I have no idea how the site is going to go live tomorrow.'”

The result of the headlong rush to October 1 was a system that had never been tested at anything like the load it experienced on its first day of operation (if it was tested with loads at all). Those looking for a reason for the site’s horrible performance on its first day had plenty of things to choose from

Go read the whole thing. Meanwhile, this matches what I wrote years ago about the “thermocline of truth” pattern in large-scale IT projects:

Sometimes, even then [at the project deadline] management may not be willing to hear or acknowledge where things really are but instead insist on a “quick fix” to get things done. Or management will order the project to be shipped or put into production, at which point all parties discover (a) that the actual business drivers and requirements never successfully made it down through the thermocline to those building the system, (b) that there are serious (and perhaps fatal) quality issues with the delivered systems, and thus (c) that the delivered project doesn’t do what top management really requires.

What Sean does not address in his article are the reports from the insurance companies that many, if not most, of the applications coming to them are either incomplete or corrupted. In other words, even if people are able to finally get in with new passwords (and possibly new user names as well), there’s no guarantee that their applications — if they can finish applying — will make it intact and complete to the insurance provider they selected.

Pass the popcorn.

[UPDATE #1: 10/09/13]

Here’s a story I missed in yesterday’s Washington Post that talks about all the people who were waving hands and trying to tell the Obama Administration that the bridge ahead was out:

Major insurers, state health-care officials and Democratic allies repeatedly warned the Obama administration in recent months that the new federal health-insurance exchange had significant problems, according to people familiar with the conversations. Despite those warnings and intense criticism from Republicans, the White House proceeded with an Oct. 1 launch. . . .

Two allies of the administration, both of whom spoke on the condition of anonymity because of the controversy surrounding the rollout, said they approached White House officials this year to raise concerns that the federal exchange was not ready to launch. In both cases, Obama officials assured them there was no cause for alarm.

Robert Laszewski, a health-care consultant with clients in the insurance industry, said insurers were complaining loudly that the site, www.healthcare.gov, was not working smoothly during frequent teleconferences with officials at the Department of Health and Human Services before the exchange’s launch and afterward. “People were pulling out their hair,” he said.

As always, read the whole thing.

By the way, I know exactly how these people feel. As an enterprise-level IT consultant, I have reviewed majored troubled projects, made strong recommendations, and then often been largely ignored. The result has usually been project failure. Here’s a memo I wrote after reviewing a troubled project at a major US corporation for the third time in three years. I strongly suspect that a lot of the problems listed in this memo exist in the Healthcare.gov development effort. The important point is that you can’t just keep stubbornly pushing ahead — you need to step back, evaluate all the issues, then figure out what it’s going to take to fix the problems. There is a good chance that a lot of the “after hours” fixes being made to Healthcare.gov are actually introducing defects of their own. Meanwhile, to the extent that the current system is mangling data, the Healthcare.gov team faces the decision of either throwing that data away or doing an internal ‘data cleanup’ on their own systems.

Back in 1998, I testified three times before Congress on IT project management and failure (in connection with the Year 2000 issue); in one of those hearings, I said the following (emphasis mine):

Humanity has been developing information technology for half a century. That experience has taught us this unpleasant truth: virtually every information technology (IT) project above a certain size or complexity is significantly late and over budget or fails altogether; those that don’t fail are often riddled with defects and difficult to enhance. Fred Brooks explored many of the root causes over twenty years ago in The Mythical Man-Month, a classic book that could be regarded as the Bible of information technology because it is universally known, often quoted, occasionally read, and rarely heeded. Most publications and books on IT since then have debated, discussed, and deplored these same problems. And they are with us still. Their causes stem not from technology but from human frailties. Indeed, when asked why so many IT projects go wrong in spite of all we know, one could simply cite the seven deadly sins: avarice, sloth, envy, gluttony, wrath, lust, and pride. It is as good an answer as any and more accurate than most.  (Testimony before the Subcommittee on Government Management, Information, and Technology Hearing on “Year 2000: Biggest Problems, Proposed Solutions”, U.S. House of Representatives, June 22, 1998.)

Yep.

[UPDATE #2: 10/10/13]

A few different sources — including Andrew Couts over at Digital Trends — are reporting that the Healthcare.gov development effort cost over $600 million:

The exact cost to build Healthcare.gov, according to U.S. government records, appears to have been $634,320,919, which we paid to a company you probably never heard of: CGI Federal.  The company originally won the contract back in 2011, but at that time, the cost was expected to run “up to” $93.7 million – still a chunk of change, but nothing near where it apparently ended up.

I’m not sure how you spend $600 million in just two years on any IT project, much less a website. To say I’m staggered would be a gross underestimate, particularly since that (not-yet-final) cost is over 6 times the original top estimate.  Couts is an avowed supporter of Obamacare, but even he struggles with just how bad this project is:

Unlike some Americans, I actually want the Obamacare exchanges to succeed. I’ve given the state-specific options a try (there are 15 of them, including Washington D.C.’s) and they seem to greatly simplify the process of buying healthcare. And the rates do appear to come in far lower than what many people without health insurance from an employer have had to bear until now. It’s not government-run healthcare. There are no death panels. And, from what I can tell, the world will not end if more people have health insurance – quite the opposite, in fact.

What I cannot stand is a nation that has vast technological resources in its citizenry spending $600 million of our collective money to slap together a product that, thus far, has only managed to waste people’s precious minutes. So the next time our government comes up with any bright idea that relies upon a massive website, let’s all be sure to ask how they plan to build it. Because the standard operating procedure at the moment is just plain sick.

GRTHT (go read the whole thing).

Meanwhile, problems continue apace. I honestly believe that at some point soon, the Government is going to have to shut down the Apply Now section of Healthcare.gov for a period of weeks and possibly even longer. I think there’s at least a 50-50 chance that it may not re-open until January (or later), and that President Obama “waives” the individual mandate for 6 months or even a full year. ..bruce w..

P. S. Click here to see all my Obamacare posts.

Be Sociable, Share!

Category: Healthcare Reform, Information Technology, Main, Obama Administration, Obamacare

About the Author ()

Webster is Principal and Founder at Bruce F. Webster & Associates, as well as an Adjunct Professor of Computer Science at Brigham Young University. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at bwebster@bfwa.com, or you can follow him on Twitter as @bfwebster.