Obamacare is offline ‘for a few hours’ each night this weekend for repairs [UPDATED x4]

| October 4, 2013

healthcare.gov

Late news today (Friday, October 4th) that the Healthcare.gov site will be taken down for repairs:

Bedeviled by technology glitches that frustrated millions of consumers, the Obama administration is taking down its health overhaul website for repairs this weekend.

Enrollment functions of the healthcare.gov site will be unavailable during off-peak hours, the Health and Human Services Department said Friday. The department did not release a schedule for hours of operation, but an HHS spokeswoman said the site would be taken down at 1 a.m. EDT each night for a few hours.

The website will remain open for general information.

It’s already down tonight, well before 1 am EDT (the above snapshot was taken at 12:25 am EDT, 10/04/13).

If the problems could be solved with a few hours of work each night over the weekend, I believe that those problems would have been solved a day or two ago. I could be wrong (and often have been, sometimes spectacularly so), but I suspect we’ll see the enrollment function at Healthcare.gov suspended by early next week — after the fixes put in this weekend fail to solve the underlying problems — and it could remain suspended into November.

[UPDATE#1 — 10/04/13]

Just ran across this article that supports a lot of my suspicions about the root causes of problems at Healthcare.gov:

One possible cause of the problems is that hitting “apply” on HealthCare.gov causes 92 separate files, plug-ins and other mammoth swarms of data to stream between the user’s computer and the servers powering the government website, said Matthew Hancock, an independent expert in website design. He was able to track the files being requested through a feature in the Firefox browser.

Of the 92 he found, 56 were JavaScript files, including plug-ins that make it easier for code to work on multiple browsers (such as Microsoft Corp’s Internet Explorer and Google Inc’s Chrome) and let users upload files to HealthCare.gov.

It is not clear why the upload function was included.

“They set up the website in such a way that too many requests to the server arrived at the same time,” Hancock said.

He said because so much traffic was going back and forth between the users’ computers and the server hosting the government website, it was as if the system was attacking itself.

Hancock described the situation as similar to what happens when hackers conduct a distributed denial of service, or DDOS, attack on a website: they get large numbers of computers to simultaneously request information from the server that runs a website, overwhelming it and causing it to crash or otherwise stumble. “The site basically DDOS’d itself,” he said.

Be sure to read the whole thing.

I have to wonder if what I think of as “Henderson’s Syndrome” (after my co-blogger, Bruce Henderson) is (also) at work here. A few years ago, Henderson and I were reviewing a few troubled production systems at a large financial corporation. One failing system in particular that Henderson was looking at had to do with terminals at bank teller stations and the back-end servers they relied upon. What he found was that when a given server was getting overloaded with transaction requests from teller terminals, it would lock up, then automatically reboot to reset itself. The problem was, all the transaction requests from the terminals kept building up in the queue, so when the server came back on line, it would be immediately overwhelmed again by even more transactions, lock up again, reboot again. Rinse and repeat.

[UPDATE #2 — 10/05/13]

Another article on the problems with Healthcare.gov this morning, this one talking about the ‘incomplete’ and ‘corrupted’ data that insurance companies are getting from the system:

“We’re getting incomplete data—about half of the applications we haven’t been able to process,” said the source, who used the term “corrupted” to describe the batch of applications received.

That means the people in every batch who haven’t provided enough or correct information must be contacted for additional data to ensure they qualify for the insurance they want to buy, experts said. The federal exchange itself may have to seek that data from enrollees. . . .

“It doesn’t surprise me—I’ve heard similar numbers,” said Dan Mendelson, CEO of consulting firm Avalere Health, when asked about the 1-in-100 rate that Infogix cited.

“This is not a traffic issue,” Mendelson said. “Right now, the systems aren’t working.”

As always, go read the whole thing.

[UPDATE #3 — 10/07/13]

The Washington Post talks with a ‘techie’, who likewise says that the problems are systemic and not merely a lack of servers:

Based on my experience, the challenges look like glitches in software code. And the software code didn’t go through enough testing. It would take some time to find the bugs.

Then there are bugs in scalability, what happens when 100 people or more are trying to do the exact same thing. Those are the things that really need tuning at this point.

Most of the problems like these are in the software. Hardware is the easy part. You can add more hardware and do it easily. Software takes more time. In the rush of getting this out, it seems like testing wasn’t done completely. My expectations from this is that these problems should go away in the next few weeks. The site still won’t be as fast as something like Netflix, but it should work.

Go Read The Whole Thing.

[UPDATE #4 — 10/07/13]

Once again, thanks to Jim Geraghty for the pull-quote and link in his Morning Jolt e-mail newsletter! Go subscribe already. 

And now the Wall Street Journal reports that what any IT person with half a brain already knew: the problems with Healthcare.gov are due to internal defects in design and implementation, not just “lots of traffic” (as if 7 million hits over a few days represents “lots of traffic” these days):

Six days into the launch of insurance marketplaces created by the new health-care law, the federal government acknowledged for the first time Sunday it needed to fix design and software problems that have kept customers from applying online for coverage.

The Obama administration said last week that an unanticipated surge of Web traffic caused most of the problems and was a sign of high demand by people seeking to buy coverage under the new law.

But federal officials said Sunday the online marketplace needed design changes, as well as more server capacity to improve efficiency on the federally run exchange that serves 36 states.

You know what to do.

[UPDATE #5 — 10/07/13]

Megan McArdle, who has an IT background herself, debunks the new meme that Republicans somehow sabotaged the Healthcare.gov development project:

So prepare yourself for the next theory: This is the fault of Republicans. Had Republicans created state exchanges as they were supposed to, agreed to the Medicaid expansion and provided more funding, the reasoning goes, everything would be going swimmingly.

I blame the Republicans for a lot of things . . . .But I do not think that the Republicans can be blamed for this particular disaster. They did not force the administration to wait until late 2011 to begin awarding important contracts for implementation of the Affordable Care Act. Presumably, they were also not skulking around the Department of Health and Human Services, writing the memos that delayed, until February of this year, the deadline for states to declare whether they’d be running their own exchanges

Read the whole thing.

[UPDATE #6 — 10/09/2013]

My discussion of the Obamacare launch continues here.

Meanwhile, I stand by my prediction above that the ‘Apply Now’ portion of the website may well end up being shut down until November. In the meantime, it will be interesting to see what comes out of all this. ..bruce w..

P. S. Click here to see all my Obamacare posts.

Be Sociable, Share!

Category: Healthcare Reform, Information Technology, Main, Obama Administration, Obamacare

About the Author ()

Webster is Principal and Founder at Bruce F. Webster & Associates, as well as an Adjunct Professor of Computer Science at Brigham Young University. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at bwebster@bfwa.com, or you can follow him on Twitter as @bfwebster.