Introduction

Pitfalls of Modern Software Engineering [PMSE] by Bruce F. Webster

Copyright 1995, 2007 by Bruce F. Webster. All rights reserved.

Version 0.10 — Last update: 06/14/07

[Back to Bibliography] [up to PMSE] [Ahead to Part I: Organizational Pitfalls]

How The Original Pitfalls Book Came to Be

Somewhere into our third year of the Pages by Pages project — developing a full-blown, commercial object-oriented document processor — I was sitting in my office, rereading The Mythical Man-Month by Frederick P. Brooks. Dave Krich, our director of quality, wandered by, looked in, and quipped, “Isn’t it a little late for that?”

It was indeed. We were approaching our originally announced release date, and we knew we weren’t going to make it. But Dave’s comment and Brooks’ book set me to thinking about all that I wished I had known when I blithely started the Pages development effort some years earlier. I started making a list — mentally at first, and then on paper — of the pits we had fallen into along the way. I added those we had skirted and those we had avoided outright.

This was at the start of 1993. At about that same time, NeXT Computer, Inc. put out a call for session proposals for its NEXTSTEP Developers Conference to be held simultaneously with the NeXTWORLD Expo in May of that year. I sent in a proposal for a session titled “The Pitfalls of Object-Oriented Development.” The conference coordinators welcomed the session but were uncomfortable with the title — which was not surprising, given how NeXT had been touting the benefits of OOD. Being more interested in helping others than in proving a point, I changed the title to “Succeeding with Object-Oriented Development,” which the conference coordinators found more acceptable. But my materials for the session — a session I shared with Jayson Adams, a brilliant developer and (at that time) chief scientist of Millennium Software — stayed the same, focusing on OOD pitfalls and how to prevent them.

Encouraged by feedback from the conference attendees, I worked up a proposal for a book on the same subject. It gave a rough outline and listed a dozen or so sample pitfalls, based on the ones I had used in my presentation, but I indicated that there would be more. Even I didn’t realize what an understatement that would be. The proposal was accepted by M&T Books in fall 1993, and a contract was signed early in 1994. Shortly after that, I sent a more detailed outline with more than 30 pitfalls. As the year went along — and especially as I looked for new pitfalls in other sources — the list grew and grew.

The final total: 82 pitfalls were listed in my original edition of Pitfalls of Object-Oriented Development (M&T Books, 1995). Many were truly specific to object-oriented development, but — as several reviewers noted — many others were generic to software engineering and IT project management. The book was well-received in many circles, and it launched my initial consulting career after Pages Software closed its doors in mid-1995.

How PMSE Came to Be

I had been interested in doing a revised and enlarged edition of Pitfalls, particularly after some e-mail correspondence with (the tragically now late) John Vlissides, one of the Gang of Four whose Design Patterns had come out almost simultaneously. John pointed out that while he and his cohorts had documented key patterns in object design, I was likewise documenting key anti-patterns in OO development. But the editorial leadership at M&T Books had changed; they had little interest in maintaining the title, and within a few years, the book itself was out of print.

However, I found that my career had abruptly shifted from commercial software development (as a full-time employee) to corporate IT project review and rescue (as a consultant). The emphasis at first was on object-oriented development, including architecture and design. But it soon turned to fundamental issues with IT project management and software engineering. Time and again, I would go into a large corporation (usually with fellow consultants) to review a major IT project that was not going as planned. We would examine documents, deliverable, and source code, interview personnel at all levels, and then come back with our assessment and recommendations. I started out doing this on my own for a single client for most of a year, but in mid-1996 joined OSG Corp as its chief technical officer and worked with many of its clients doing the same thing.

During this time, I spoke at conferences on issue in IT project management and failure. For part of this period (1996-1998), I lived full-time in Washington DC and largely worked on-site at Fannie Mae, where OSG had a few dozen senior consultants working on object-oriented technology. Unexpectedly, I was drafted into Fannie Mae’s Year 2000 remediation effort, which not only led to my working with or interviewing dozens of professionals at all levels in Fannie Mae, it also led to my testifying three separate times before Congress regarding Year 2000 issues. I escaped that some — though not fully — when I moved in 1998 from DC to Dallas, where OSG has its headquarters.

In 1999, my career took yet another turn when PricewaterhouseCoopers recruited me to join its Dispute Analysis & Investigations (DA&I) group. PwC wanted to use me as a consulting and/or testifying expert in litigation involving information technology: failed or disputed IT projects, intellectual property infringement (copyright, trade secret, patent), and so on. As part of our work there, our IT group gathered information and documents on roughly 120 IT “systems failure” lawsuits over a 25-year period. I read all the documents and pleadings collected, then wrote a white paper identifying a half dozen classification patterns into which most of those 120 lawsuits fell. During this period, I also continued to do IT project consulting, both within PwC and with outside clients.

In 2001, I left PwC and set up my own consulting firm. I did corporate IT project reviews and consulting, either directly or through my old employer, OSG Corp. I also continued to do consulting/testifying expert work in lawsuits involving information technology (failed projects, intellectual property, and so on). And through this whole time — on my own, with OSG, with PwC, and on my own again — I got to look time and again at why organizations struggle so much with information technology, and why there is such a high failure rate among large-scale IT projects. I have done course-correction reviews of live IT projects costing upwards of $500 million; I have done post-mortem reviews (for litigation) of IT projects costing up to and over $1 billion. Along the way, I have reviewed hundreds of thousands of pages of project-related documents, interviewed hundreds of people, run analysis tools on millions of lines of source code, and through all this have had to explain (usually in written form) of where the problems were and why they existed. The analysis that I have done for litigation purposes I have further had to defend — under oath — through hours of deposition and cross-examination by opposing attorneys.

Along the way, my concept for a second edition of Pitfalls changed. Object-oriented development was no longer as ‘cutting edge’ as it had been in the early 1990s but had largely become the norm or at least a norm. To do a second edition covering just OOD would make little more sense than doing Pitfalls of Structured Systems Development or something similar. And, as noted, many of the pitfalls themselves were independent of whether OOD had actually been used.

And so my second edition transformed into Pitfalls of Modern Software Engineering, which I refer to as PMSE (pronounced “PimSee”). The title reflects the fact that the book assumes use of current methodologies, tools, development languages, and technologies — including object-oriented development and all its subsequent progeny. But having thrown the door open, I find that the list has gotten long indeed, as shown by the proposed outline. I fully expect that outline to change in the course of writing the book itself; the items in the outline represent topics that I believe may be of interested to readers.

The Need for This Book

I am not a curmudgeon. Back when I was a full-time software engineer, I struggled daily against the boundless optimism that gets so many developers into trouble and that has certainly landed me in trouble from on a few occasions during the first half of my career. All I have to do to thoroughly embarrass myself is to drag out my original development schedule predictions from the early days at Pages Software.

But the simple truth is that most organizations struggle mightily with information technology. A major theme in business analysis of IT over the past 10 years has been whether organizations actually gain any advantage from IT, or whether they merely avoid falling behind. Much of that analysis, I believe, is based on the high level of IT project delays and failures in most organizations, as well as ongoing issues with the quality of the systems that are actually deployed.

In my reviews of IT projects and issues over the past 12 or more years, I find that the same fundamental mistakes and issues come up again and again. Most of these issues have been documented for decades — books by Brooks, Weinberg, Youdon, De Marco, Lister, Gilb, Glass and Jones immediately come to mind. But I have also found that these books are seldom owned and even more seldom read by the key people in management — especially up to the CxO level — and so fools rush in, and all that. It is precisely that repetition of error that led me name my blog (and-still-i-persist.com) based on a quote from a short story (‘My Brother Leopold’) by Edgar Pangborn: “And still I persist in wondering whether folly must always be our nemesis.”

Who Should Read This Book

I wrote the original Pitfalls with three groups in mind:

• developers, those who do the actual architecture and implementation
• technical managers, those who oversee the developers
• upper management, those who run the company and make strategic decisions concerning direction and funding

Some pitfalls apply more to one group than another because of their likelihood to encounter the pitfall and their ability to understand why it’s significant. Nevertheless, it’s useful—and possibly vital—that all three groups have some degree of familiarity with all the pitfalls listed here. Because this book addresses such a wide-ranging audience, the technical level of the pitfalls varies significantly, starting off low in the first few chapters and gradually getting higher.

Those in upper management have three compelling reasons to understand these pitfalls above and beyond those they may need to avoid themselves. First, they’ll find this book a useful reality check on all they may have heard, read, or been told about object-oriented development. Second, they’ll be able to better evaluate reports from development groups, understanding the real problems faced by developers and recognizing when they’re being fed, ah, horse manure. Third, they’ll see that decisions they make—a change in specification, an added feature, a mandate on system configurations—can have a tremendous impact on development schedule or even feasibility, even when the change seems to be a small one.

Technical managers, who form the interface between upper management and developers, will need to watch out for every pitfall in this book. To them I say: You knew the job was dangerous when you took it. I’m probably talking more directly to you throughout this book than to either of the other two groups.

Developers may focus on the more technical pitfalls, but they need to understand the rest so that they can appreciate the battles that their boss, the technical manager, may have to fight.

By having all three groups familiar with the issues and dangers raised herein, you can work toward establishing a common understanding and terminology. That will make for better communication, better trust, and better results. This presumes that all three groups want to work toward that end, but if they don’t, you have problems even more profound than those addressed here.

How This Book is Organized

In the original book, each pitfall occupied two facing pages, which was a discipline that compelled restraint and brevity in some cases. In doing this initial web-based version, I will likely also impose a fixed length to each pitfall for the same reason.

For now, the web-based version will also follow the pitfall structure used in the book. Each pitfall is introduced with the reasons and causes behind it. Then the following items are detailed for the pitfall:

• Symptoms: your first clues that you may be headed for (or already mired within) this pitfall
• Consequences: what will happen if you don’t take action
• Detection: proactive steps to see if you are indeed facing this pitfall
• Extraction: how to get out of it once you’ve fallen in
• Prevention: how to avoid it in the first place

Unlike the book, any references will directly accompany the given pitfall; likewise, I may have a keywords section. One of my goals is to offer a front-end to the web-based version that allows you to enter symptoms and then presents a list of likely pitfalls.

Each section and subsection is meant to be self-contained, allowing you to browse and skip around at will. The goal is let you get in, find the information you need or want, and get out again.

[end of introduction for now]