San Diego Fires post-mortem: ourselves

| October 31, 2007

I’ve previously posted feedback from readers on how the media performed during the San Diego Fires as well as how the government performed. At the same time that I solicited that feedback, I also asked for suggestions on how our coverage here at “And Still I Persist” might have been better. Feedback and my own observations after the jump.

My biggest gaffe in all this was not anticipating the bandwidth issues — or, more precisely, expecting our web hosting firm to contact me before shutting the blog down completely, as they did at around 11 am PDT on Tuesday, October 23rd. At that point, we had over 1,400 people online simultaneously and had had over 3,000 hits the previous hour. Once the firm took us off-line, I called up with a credit card, let them know the critical nature of the information we were trying to get out, and asked them to get us moved up to a dedicated server ASAP. That took a little longer than they originally projected — about 3 hours instead of 2 — but that’s where the problems left my control.

Around the end of those three hours, the web hosting firm sent me the updated domain name server (DNS) information for the domain registrar. This DNS info clearly had problems with it (the IP addresses of both DNSs were identical, which Network Solutions promptly rejected), and I immediately replied (via e-mail) to the tech support group — but it would be another five (5) hours (and additional e-mails, plus at least one phone call) later before I got an “oops” e-mail from them with the correct DNS information. Even after I updated my domain name with the new DNS information, the blog still wouldn’t come up. I thought it might have to do with DNS propagation, but after a few more phone calls, it turned out the web hosting firm never re-enabled the PHP script that they had disabled to shut down the blog in the first place. Once they did that, the blog magically reappeared — at around 9 pm PDT, some ten hours after being shut down during the most critical day of the entire fire. In short, a good 7+ hours of our downtime was due to errors on the part of the web hosting firm.

I’m still hacked about this, in case you can’t tell, all the more so because at least one kind soul with plenty of bandwidth contacted us and offered to host our blog shortly after we went down on Tuesday. I considered it, but relied instead on the 2-hour estimate from my web hosting firm. Sigh.

Tremendous credit has to go to all the other web sites that were likewise doing their best to post information. The CalFire and San Diego County Emergency websites were struggling with the load on Monday and Tuesday, but had scaled up by Wednesday. The Sign On San Diego Fire Blog (run by the San Diego Union-Tribune) was my single best source of  timely information about fires, and I cited and liked to them in most of my fire-related posts. Bruce Henderson has already discussed at length the sources and methods he relied upon, particularly in doing his spectacular 3-D visualizations of the various fires. And, of course, there were the various blogs (bonbonio.org, Cat Dirt Sez, My Simple Life, and The Scratching Post, among others) also covering the fires, to whom we linked and who linked to us.

Most of the feedback we’ve received from readers has been positive to the point of embarrassment. You can see a lot of it in the public comments left on the SD Fires posts, but we’ve received e-mails as well. I’ve gone through and tried to extract comments that show why the blog was of use during the fires, particularly in comparison to other sources of information.

Now that you have the attention of so many of us, I would hope that you use your forum to educate people.  Different building and terrain conditions create different fire conditions. It would also be great to see more specific information on the “shelter in place” construction, vegetation and landscaping standards that Bruce H posted about.  The article referenced does not really give specifics.

I am from the Jesmond Dene area of Escondido but I have recently graduated college and moved to Texas…I found your website to be the most effective in keeping up to date with recent information on the fires and direction and there path of travel. I even let my parents know about your site even thought they are there they need to be in tune with info.

Your site has been an amazing resource for those of us who live in SD County.  Fortunately for me, the fire was never alarmingly close to my home (in Vista), but I didn’t know that until I saw your site.  You are doing something truly useful, and I admire that, partly because I aspire to do something truly useful myself, and know how much goes into it (both the direct effort, and the time it takes to cultivate the required skills to do something useful.)

I’m [from] a tiny town near Tacoma WA.  My sister lives in Coronado, CA, in San Diego Bay — she never appreciated the benefit of the sea-water moat around her island-town before this week.  I have a friend whose mom’s Rancho Bernardo home survived the fires by 1/2 mile.  And I love maps and was so frustrated at the lack of detail or map info from SD news sources early on (when signonsandiego.com wasn’t totally bogged down) and moreover from national TV:  all they showed was flames and more flames, no maps.  I’m absorbed and my heart is wrung by the unfolding story but I’m at work so haven’t had a lot of time to delve into every blogsite out there.  Yours put a human (and horse) face on the experience while providing so much map understanding!

I just want to say thank you for the hard work you all have done, the dedication and persistence, and for sharing it with all the rest of us.  This has been a valuable resource for keeping up from afar with developments down thataway for farflung folks like me, and I’m confident it was an anchor in the fog (smoke) for locals trying to parse out what their risk was and moves should be. So interesting to read references today to Bruce-the-map-guy’s intel outlook, and also the confirmation that’s what his training/experience at some point was.

By the way, it was an eerie experience when your website went black for all that time on … Tuesday?  The days muddle.  I’d found your great maps and site and was checking it every 4-5 hours;  sent links to a few folks for them to see too.  Then suddenly it wasn’t there, and I could only speculate that something bad had happened to wherever the server was — power outage, burnt up, something bad.  I thought that Horse Bruce might have had to evacuate but that wouldn’t have made the general site go down unless it was hosted at his house which was burning.  I was mystified and certainly was hoping for the best.  Had no idea it was the problem of pulling the plug because of overload of hits!  I had googled “San Diego fire map” and stumbled on your site.

First, I just want to express my gratitude toward you and the others; your efforts are deeply appreciated, much more so because it didn’t take a committee and a consensus; just expertise and good hearts.

For the future, there should be someone like you at the head of every disaster; an information specialist who sits at the top of the organization, but whose sole purpose is to gather and disseminate useful information.  Imagine if you had been sitting at the command center, hearing from the firefighters in the field, knowing where the hot spots were and able to disseminate the information almost in real time.

If there are legal and/or regulatory hurdles to such an effort, they should be analyzed and dealt with and proper legislation proposed.  E.g., you can’t be sued and taken for all you’re worth.

Perhaps you can market yourselves as consultants available in such an instance; start a professional organization for others of such ilk.  Consult with governmental agencies and get the structure set up ahead of time.  given the terrorist threat [ho-hum], this is a piece of the puzzle that would help a lot of people if it’s ever needed.

Thanks for your coverage of the SD wildfires.  I’m a San Diego native who’s currently in Nashville, TN.  My family is still in San Diego, so your blog has been a welcome source of information – especially with the flakiness of the CalFire website.  Thanks again for what you’re doing.  The coverage has been top notch.

You guys give the hard facts and info needed. Basically what the media is providing is pretty WEAK compared to you.

Your valiant efforts at ‘mashing’ fire info into concise statements and useful maps is greatly appreciated!

You guys could teach the emergency services a thing or six about coordinating and mashing – to the benefit of all.

Again… great work!  Be very proud of how much aid you provided to many worried folks!

Bruce, many thanks for your great website.   I live in NY and it’s been a tremendous relief to be able to follow breaking news, see evacuation areas and fire zones via your pictures and maps.  My sister and friends live in Encinitas/Rancho Santa Fe areas, some very close to edge of the fire.  I was distressed yesterday when you were down but happy now to be in touch again.  That list of homes sent chills down my spine this morning.  Good luck to all out there.

Thanks for the fire tracking on your blog.  Hard to get information better than this, even around here.

My mom is in PB at one of the hotels she owns.  She’s pretty certain that her house in Rancho Santa Fe is doomed based on what she’s seen on TV.  Do you know of any way to determine addresses or neighborhoods in specific areas that have burned?  Seems unlikely but, like Roseanne used to say, “You no ask, you no get!”

You get the idea.

It’s been quite an interesting run, but things are quickly returning to normal around here:

Still, if you go to Google and type in witch creek fires, our blog comes up at the top of the list. That is a credit to Bruce Henderson, who managed to create outstanding and useful visualizations of these fires while having to move his horse around and worry about possibly evacuating himself, his wife, and his cats.

We all hope nothing like this happens again any time soon.  ..bruce..

Be Sociable, Share!

Category: Emergency Preparedness, Information Technology, Main, SAN DIEGO EMERGENCIES, San Diego Fires

About the Author ()

Webster is Principal and Founder at Bruce F. Webster & Associates, as well as an Adjunct Professor of Computer Science at Brigham Young University. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at bwebster@bfwa.com, or you can follow him on Twitter as @bfwebster.

Comments (2)

Trackback URL | Comments RSS Feed

  1. bhenderson says:

    And again Webster shows us he is second to none at being able to quantify the complex and make it readable. Thanks for the “After Action” series, well done.

  2. ktcat says:

    You guys did an outstanding job. As far as how well the media did, well, no one got into journalism because they thought electrical engineering was too easy.

    Journalists are generalists of average intelligence. They understand English composition, entertainment and hairstyling. It’s easy and fun to be harsh with them, but all of your comments in the previous post discuss how the media failed to give an accurate Common Operational Picture (COP).

    Well, duhhhhh. What did you think they were going to do?

    There’s an organization or two in town that are global professionals at providing a COP. Where were they? Hiding behind their network firewalls and housed in offices looking at budget spreadsheets and PowerPoint presentations of schedules and wiring diagrams while the city burned all around them. Nero fiddled, they emailed.

    Well, they would have emailed if they hadn’t been forced to use NMCI. Since they used NMCI the best they could do was to beat their monitors with sticks and curse loudly.

    But I digress.

    Expecting bloggers and the MSM to provide an accurate and easy-to-understand COP while the COP professionals are in town seems a bit silly to me.

    I’m sorry you guys had to fill the breech. I’m hoping it will be better next time.