I’ll add a photo later, but our ship — the ms Nieuw Amersterdam — just left the Ft Lauderdale ship channel and is now at sea, a few miles off the Florida coast and headed for the Bahamas. I’ll try to make daily posts, largely about the National Review sessions on board. ..bruce..
[Final version -- flights are done and I'm on the ground in Florida]
“All the most important mistakes are made on the first day.”
- The Art of Systems Architecting (Maier & Rechtin)
Project Orca was the Romney campaign’s technological effort to track in near-real-time actual voting in precincts all over the United States, with the intent of using that information in combination with individual-specific demographic databases to increase or pull back phone-call and door-knock get-out-the-vote (GOTV) efforts based on what was happening there. Since it appears more and more that Romney lost largely because of a lack of Republican turnout, it’s safe to say that the Romney/RNC GOTV effort was a failure. What is less clear is how much of an impact Orca had on that failure.
Three important disclosures up front. First, I was a Project Orca volunteer in Colorado. Second, I have no direct knowledge of how Orca was envisioned, developed, and operated beyond my own experience as an end user. Third, I’ve worked in software engineering and information technology since 1974, and my major professional focus since 1994 or so has been on how and why IT projects fail or succeed — a subject on which I’ve taught seminars, published books and articles, consulted in large corporations, and testified in courts and before Congress.
That said, the flap over Orca has made it all the way to the Drudge Report, largely due to two key discussions. The first is by JohnE over at Ace of Spades, who wrote the initial scathing report on Orca and followed it up with a reply to some of the push-back from the Romney campaign. The second is by Joel Pollack over at Breitbart, who had a Romney worker in Colorado who spoke of some of the problems.
My own experience matches much of what JohnE (who was also in Colorado) stated. I participated in four training calls. The first three all gave the same information about how the system would work, though — as JohnE noted — the Romney people kept refering to the Orca software as an “app”, even though it was simply a website that you logged into. (From JohnE’s comments, it sounds as though the use of the term “app” confused at least some Orca volunteers.)
Unlike JohnE, I was very clear from both the calls and the e-mails I receive that I needed to pick up my poll watcher certification and bring it with me to my assigned polling place on Election Day, so I am curious about his confusion. On the other hand, for my particular county (Douglas County), it appears that the certifications weren’t ready until the day before Election Day, which strikes me as cutting it a bit close. But I did drive down to Castle Rock on Monday afternoon and picked it up.
There was one more call the night before Election Day, but that was just a very brief, rah-rah call. After that Monday night call, I tried logging into the Orca web site to test out my password, but the log-in failed. I found that puzzling, but just assumed that they had the system locked down ether for security or administrative purposes.
Since Colorado is one of the few states that does not allow any electronic devices (phones, tablets, laptops) in polling places, I was also very clear from the training that I would have to print out the list of registered voters, bring it with me, mark off voters as they identified themselves at the polling place, and then periodically run out to my car and enter voting information via my iPad or iPhone. I printed out the list — and immediately discovered what I considered the first, admittedly minor, glitch: the list, a PDF file over 60 pages long, was formatted in such a way that you could not three-hole-punch the sheets (to put in a binder) without punching through the check boxes (for tracking voting and reporting) for some of the voters on each page. As I said, minor, but my technical writing background made me frown a bit. Since I own Adobe Acrobat, I solved the problem via the kludge of adding headers and footers to each page and then telling the pages to shrink themselves to fit.
So, with two binders in hand (I printed two copies of the list), with iPhone, my iPad, and my laptop in the car — along with two Black & Decker power boxes, and a couple of folding camp chairs — I showed up at my polling place at 6:45 am. I also brought with me three dozen fresh donuts (a suggestion from one of the training calls) that were greatly welcomed and did much to smooth relations with the actual poll workers.
Everything went well until the first time I went out to the car to report voters. I tried to log into the Orca website, and once again I got a “login failed” error message. (Second minor but telling item: that error message had a typo in it. The fact that such a typo was in a high-probability error message in a live system with over 30,000 anticipated users speaks to haste and sloppiness.) I called the help number given in the error message, and got a support person at the Boston command center who said they would reset my password and gave me the new code, but that it would take a little while for the new code to go live.
When I came back out again an hour or so later and tried to log in, the new password didn’t work either. The Orca system also had a call-in option, so I tried that and was told that my PIN was invalid.
At this point, it was about 8:30 in the morning. The next eight or so hours were all more of the same — I would call, the password would be reset, I would wait a while, I would try to log in, it would fail. Rinse and repeat. I was told by some of the support people I spoke with that almost no one in Colorado was able to log in — a geographical uniqueness that I found puzzling, since the login screen just asked for username (an e-mail address) and password. Another support person mentioned that there were similar problems being reported for North Carolina. Finally, sometime around 4:30 to 5:00 pm, I was given a new pin for the dial-in line — and that finally worked. I reported all the voters who had voted so far that day and did updated reports twice more before the poll finally closed at 7:00 pm.
JohnE pushes back (with justifiable skepticism) against Zac Moffat’s claims that “the campaign had voting data from 91 percent of counties” — not that they is necessarily inaccurate, but that it could be largely irrelevant. As JohnE points out, there can be hundreds of precincts in a single county; beyond that, having voting data from a given county does not necessarily mean having actionable, timely, and sufficiently complete data throughout the day in order to make GOTV decisions for that county (or for a given precinct in that county).
There is still no evidence that there were significant Orca problems outside of Colorado; we’ll have to wait to see if someone on the inside of Orca development decides to leak. But here are some observations from an large-scale IT system development perspective.
“A complex system that works is invariably found to have evolved from a simple system that works. . . . A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.”
– Systemantics (John Gall)
First, the Orca folks basically released version 1.0 (or, perhaps more accurately, version 0.9 or lower) into a massive, live environment on the single most critical day, Election Day. This is commonly known in IT circles as the “Big Bang” approach, and it is a well-known harbinger of disaster. Even if the Orca developers were all geniuses, this still would be a grave error. We’re not talking about an iOS or Android app being used by a single person on his/her device for entertainment or productivity; we’re talking about a massive web-based system that has to accept data from (not just deliver content to) possibly tens of thousands of users simultaneously and in real time. There should have been days, if not weeks, of live tests and trials to see how the system performed and to get the kinks worked out. Instead, as far as I can tell, the first time most Orca volunteers actually used (or tried to use) the system was on Election Day. Even if the system worked perfectly, I doubt that the majority of users — who had no real training on the system — would have used it in an effective manner.
“There are products that you shouldn’t develop, companies you shouldn’t challenge, customers you shouldn’t win, markets you shouldn’t enter, recommendations from the board of directors you shouldn’t follow.”
– The Art of ‘Ware (Webster)
Beyond that — and as per the quote above (paraphrasing Sun Tzu), as well as the one at the start of this post — the real question is: should this have been attempted in the first place? There are anecdotal reports that the focus on Orca took away resources — especially feet-on-the-ground resources — for local GOTV efforts. And GOTV is exactly where the Republicans failed. If work on Orca had been started 3 or 4 years ago, and if it had been used on Election Day 2010 — both as a real-world test and to verify the expected benefits of matching actual voters with demographic information to predict results in a given precinct — then it might have been ready for prime time on Election Day 2012.
But to spend signficant time, money, and resources (including tying up some 30,000+ local volunteers) for an untested system and an untried concept was a recipe for disaster — a well-known recipe, I might add, for those of us who have studied large-scale IT projects (which, frankly, are prone to failure anyway).
“But,” you ask, “you signed up to volunteer.” Yes, I did. But I had no visibility into the project and no inherent reason in advance to assume that things weren’t being done well, beyond the few niggling questions I had prior to Election Day. And, to be perfectly fair, I have no evidence of the extent of problems beyond Colorado, aside from the acknowledged 90-minute downtime on Election Day morning. But, even if those are the only two failures, can you seriously argue that it was a success? A time-critical, mission-critical system whose entire purpose for existence is to be used in a single 12-hour window is down for over 10% of that time in all states and is down all day in one critical swing state?
And, finally, the bottom line: if the purpose of Orca was to provide real-time intelligence on actual voting trends and make GOTV more effective, then where are the results? By all accounts, the Romney campaigned was stunned by their loss; if Orca was providing accurate, real-time indication of voting, as the project directors claimed it would in the training calls (actual quote [from memory]: “We’ll know ahead of anyone else how the election is turning out.”), then Romney should have had his concession speech written well in advance. And if its goal was to trigger targeted GOTV efforts in swing states, then why did Romney lose almost every swing state? Measured by its stated goals, Orca was a failure, pure and simple. Worse yet, it may have diverted critical resources from more effective GOTV efforts.
“Start out stupid and work up from there.”
– Bruce Henderson
I’d love to be able to do a project post mortem on Orca, as I have done on so many troubled, failing, or failed projects. But even without access to people and documents, I can hazard a guess at some of the likely factors:
- A late start and an impossibly short schedule.
- Because of these above, a waterfall (single pass) approach to design, development and deployment, instead of an iterative approach.
- Lack of a single chief software architect, with resulting problems at subsystem interfaces.
- A large development team with no prior track record developing and releasing software as a single team.
- Along the same lines, key components divided up among separate teams with relatively late integration and testing.
- And possibly some turnover (departure) of key personnel through the project, along with new personnel added late.
- Insufficient resources (money, personnel, equipment, time) devoted to quality assurance — and by “quality assurance”, I mean not just a wide range of testing (including, as JohnE points out, stress testing), but also individuals and teams with proper expertise, agreed-upon standards and guidelines, reviews, metrics, defect management, and so on.
- A firm belief — without any real-world evidence — that the idea would actually work and that the results would actually be worth the diversion of resources (including volunteers).
I’m happy to talk with anyone from the project who wants to set me straight or explain what really happened. But, in the end, what really happened was that Orca failed to achieve its goals, and Romney lost what should have been a very winnable election.
[UPDATE 11/10/12 - 1751 EST]
This article bu Ben Howe over at Red State, if accurate, casts a different light on Orca — namely that those in charge were more interested in self-enrichment and promotion than actually delivering a feasible system:
Sources also said that arrogance played a big role, saying that the Romney campaign was a hostile battlefield of egos in which these consultants viewed any opposition to their world view as coming from an enemy. This apparently led to the ORCA program “receiving no stress test, no usage during super saturdays and no ability to have a Plan B or C when everything hit the fan.”
“The brain trust of the Romney campaign was so arrogant that they refused to change strategy. It was clear in June were SOL,” said one email.
Another source that closely studied the Obama campaigns GOTV efforts as compared to ORCA said bluntly that “the Obama training manuals made ORCA look like a drunken monkey slapped together a powerpoint” adding that we must duplicate and improve what they accomplished to have any hope for the 2014 & 2016 ground game.
But the failures in what was described as a “tightly wound consultant culture” didn’t stop there.
Stu Stevens of the Stevens and Schriefer Group was said to not be chasing poll numbers with the media buy strategy and appeared instead to be doing little more than “throwing darts at a dartboard.” At best using false numbers provided by ORCA; at worst milking the cash cow of the Romney campaign.
Go read the whole article. My initial assumption was that Orca was a sincere but misguided attempt to build a large complex system in a short period of time. Assuming that the inside sources are not (just) settling scores and destroying competition, it appears that venality and self-interest doomed the project from the start. ..bruce w..
No, no, no — just on a cruise. Or, more specifically, on the National Review 2012 Post-Election Cruise. Sandra and I went on the NR Cruise last year and had an absolutely wonderful time. This year, I suspect things will be a bit more pained, if determined. Maybe we’ll work to form a shadow cabinet, an English concept that I think could be useful here in the US. I’ll do my best to blog from the cruise, though internet access is slow and expensive. ..bruce w..
Among the election post-mortems I’m reading this morning, I see a frequent refrain about how Mitt Romney wasn’t a strong or ideal candidate. My immediate question is: compared to whom? Here are the GOP candidates who entered the race and stuck around long enough to participate in the initial debates. Who among the other seven candidates would have campaigned as well and drawn as much support as Romney? For those struggling to remember who all these folks are (top left to bottom right):
- Rick Santorum
- Rick Perry
- Mitt Romney
- Herman Cain
- Ron Paul
- Newt Gingrich
- Michele Bachmann
- Jon Huntsman
If Romney wasn’t the best candidate, who of the other seven would have fared better against Obama and the media? And if there were better potential candidates than these eight, then why didn’t they run? ..bruce w..
Romney did not win big. He did not win at all. The GOP did not gain control of the Senate and may have lost ground.
I was clearly wrong on what the American people wanted. At least 49% of them, in states that added up to an electoral college victory.
I was worried after I wrote my post Monday night that I might be channeling some inner intuition as to the outcome of today’s election. Now the best I can do is to continue to promote wise and good and honest candidates for public office, while taking steps to protect my family against what I fear are some pretty rough financial times ahead. As Jerry Pournelle is wont to say, things that can’t go on forever, won’t. Or to repeat Kipling’s closing stanza from yesterday:
And that after this is accomplished, and the brave new world begins
When all men are paid for existing and no man must pay for his sins,
As surely as Water will wet us, as surely as Fire will burn,
The Gods of the Copybook Headings with terror and slaughter return!
We will learn over the next four years just how cold the Cold Equations can be. ..bruce w..
[Thanks to the links from Instapundit (Glenn Reynolds), Chaos Manor (Jerry Pournelle), and John C. Wright's Journal (John C. Wright -- what, you were expecting Karl Rove?). In fact, if you follow the link below to my 2008 post, you'll see that I first read the poem over at Jerry Pournelle's blog, Chaos Manor.]
I posted this poem here four years ago during the last presidential election — it seems even more relevant now, though for different reasons. Rudyard Kipling was not so much a prophet as an astute observer of human behavior, particularly in a society.
As I pass through my incarnations in every age and race,
I make my proper prostrations to the Gods of the Market Place.
Peering through reverent fingers I watch them flourish and fall,
And the Gods of the Copybook Headings, I notice, outlast them all.
We were living in trees when they met us. They showed us each in turn
That Water would certainly wet us, as Fire would certainly burn:
But we found them lacking in Uplift, Vision and Breadth of Mind,
So we left them to teach the Gorillas while we followed the March of Mankind.
We moved as the Spirit listed. They never altered their pace,
Being neither cloud nor wind-borne like the Gods of the Market Place;
But they always caught up with our progress, and presently word would come
That a tribe had been wiped off its icefield, or the lights had gone out in Rome.
With the Hopes that our World is built on they were utterly out of touch,
They denied that the Moon was Stilton; they denied she was even Dutch;
They denied that Wishes were Horses; they denied that a Pig had Wings;
So we worshipped the Gods of the Market Who promised these beautiful things.
When the Cambrian measures were forming, They promised perpetual peace.
They swore, if we gave them our weapons, that the wars of the tribes would cease.
But when we disarmed They sold us and delivered us bound to our foe,
And the Gods of the Copybook Headings said: “Stick to the Devil you know.“
On the first Feminian Sandstones we were promised the Fuller Life
(Which started by loving our neighbour and ended by loving his wife)
Till our women had no more children and the men lost reason and faith,
And the Gods of the Copybook Headings said: “The Wages of Sin is Death.”
In the Carboniferous Epoch we were promised abundance for all,
By robbing selected Peter to pay for collective Paul;
But, though we had plenty of money, there was nothing our money could buy,
And the Gods of the Copybook Headings said: “If you don’t work you die.”
Then the Gods of the Market tumbled, and their smooth-tongued wizards withdrew
And the hearts of the meanest were humbled and began to believe it was true
That All is not Gold that Glitters, and Two and Two make Four—
And the Gods of the Copybook Headings limped up to explain it once more.
As it will be in the future, it was at the birth of Man—
There are only four things certain since Social Progress began:—
That the Dog returns to his Vomit and the Sow returns to her Mire,
And the burnt Fool’s bandaged finger goes wabbling back to the Fire;
And that after this is accomplished, and the brave new world begins
When all men are paid for existing and no man must pay for his sins,
As surely as Water will wet us, as surely as Fire will burn,
The Gods of the Copybook Headings with terror and slaughter return!
I’m part of the Romney campaign’s “Project Orca“. As such, I am now a certified poll watcher and will be at my local polling location for 12 hours tomorrow, logging everyone who comes in to vote (I have a printout of all registered voters for my polling location) and then transmitting that information back to Orca headquarters, where it will be correlated with demographic information to estimate what the Romney vs. Obama vote is looking like at my location. This is being done all over the United States, with some 20,000+ volunteers. The idea is that Orca HQ can assign in real time phone calls and door knocks to precincts around the country that can use them, while halting them for precincts that don’t need them.
I’m looking forward to the work and to hearing how well the whole thing works. ..bruce w..
Back before Joss drank the kool-aid.
It’s 1:47 am MST. I’m in a hotel room in Ogden, Utah, with plans to drive back to Colorado in the morning with my sweet wife. I woke up a while ago and can’t get back to sleep, so here I am posting.
Back in the 70s and 80s — before the Web, before the vast majority of Americans even knew what e-mail and the Internet were — a good friend of mine, A. Christine Baron, and her husband, Tom Porter, ran by physical mail a quadrennial contest among their friends to guess the electoral college outcome of the U.S. Presidential election. I believe we had to have our entries — giving electoral vote totals and anticipated state-by-state victories as tie-breakers — in to Chris and Tom 30 days ahead of the election itself. After the election, Chris and Tom would send the results out to everyone and name a victor (or victors). The reward was bragging rights for the next four years.
The one contest I remember was 1984. Believe it or not, Reagan’s re-election was not considered a sure thing, in spite of the economic recovery from the horrific Carter years. He was still seen as a buffoon, a war-monger, a grade-B actor in over his head, and his basic conservative principles were widely mocked by the Left and the mainstream media. I don’t remember the specifics of my entry into Chris & Tom’s contest; I did have Reagan victorious, but it was not by an overwhelming amount.
What I do remember nearly 30 years later, however, were the specifics of the winning entry. Said entry was submitted by Chris & Tom’s teenage son, and it was very simple: he picked Reagan to win every single state. Period. I don’t know if he did it as a joke or if he was tapped into some Zeitgeist that the rest of us just didn’t sense, but none of the rest of us were even close to the actual results:
Do I think that Romney’s Electoral College map will look like this? No.
Do I think that Romney’s Electoral College map could look more like this than a lot of the current projections? Yes.
As Mal says above: “No. They’re not going to see this coming.”
Oh, and the vote that dare not speak its name? Select text to see.
Romney Tsunami
You heard the phrase here first. Unless, of course, you didn’t.
Obligatory caution:
We’ve got less than 72 hours before this is done. Let’s misbehave. ..bruce w..
My sweet mother-in-law, Nora Soreson Anderson Hopkins, passed away on Monday after suffering the first of a series of strokes the prior Tuesday. Nora was nearly 90 and was more than ready to shuffle off her mortal coil (and had been for years, as she’d tell anyone who asked and some who didn’t). She was a wonderful woman who held up well under a tough life — her late husband, my wife’s father, built smokestacks all over the US and Canada for 30 years and lived in over 100 different places during that time. My wife and I are out here in Ogden, Utah, handling the various memorial, burial and legal arrangements and will be heading back to Colorado this weekend.
Come Tuesday, though, I will be serving a certified poll watcher for the Romney campaign at our local polling place. If you haven’t voted yet, vote now or vote on Tuesday, and encouraage others to vote, especially if they’re likely to vote for Romney. ..bruce w..
I posted this as an update to my post last Saturday on the coming liberal meltdown, but I think it deserves a post of its own.
First, one Wednesday Matt Bai over at the New York Times blog wrote a post on Wednesday (10/24) titled, “How Bill Clinton May Have Hurt the Obama Campaign“. Bai says:
…it was Mr. Clinton who forcefully argued to Mr. Obama’s aides that the campaign had it wrong. The best way to go after Mr. Romney, the former president said, was to publicly grant that he was the “severe conservative” he claimed to be, and then hang that unpopular ideology around his neck.
In other words, Mr. Clinton counseled that independent voters might forgive Mr. Romney for having said whatever he had to say to win his party’s nomination, but they would be far more reluctant to vote for him if they thought they were getting the third term of George W. Bush. Ever since, the Obama campaign has been hammering Mr. Romney as too conservative, while essentially giving him a pass for having traveled a tortured path on issues like health care reform, abortion and gay rights.
By an amazing coincidence, just the very next day (Thursday, 10/25), the Examiner published an article by Christopher Collins headlined, “Clinton asked for more security in Benghazi, Obama said no“, based on comments made by Ed Klein on the Blaze:
Last night, it was revealed that Secretary of State Hillary Clinton had ordered more security at the U.S. mission in Benghazi before it was attacked where four Americans, including U.S. Ambassador Christopher Stevens were murdered by Al-Qaeda but President Obama denied the request.
The news broke on TheBlazeTV’s “Wilkow!” hosted by Andrew Wilkow, by best-selling author, Ed Klein who said the legal counsel to Clinton had informed him of this information.
Klein also said that those same sources said that former President Bill Clinton has been “urging” his wife [Hillary] to release official State Department documents that prove she called for additional security at the compound in Libya, which would almost certainly result in President Obama losing the election.
So, who are you going to bet on: Chicago or Arkansas? I think it’s a tough call, but I have to go with Arkansas — Bill won two terms as President, Barack did not. ..bruce w..