(This article is part of the Big Rewrite series.)
While we’re all in the back creating the next revision of a product, who’s tending to the day to day issues of the existing product? Typically, it’s the domain experts and the original implementers of the product.
Regardless of our intentions, day to day life and in-your-face time-sensitive issues can very easily steal all of the attention from a Big Rewrite. Screaming customers need their problems solved. Outages and serious bugs need to be fixed. Enhancements have to keep rolling in if your project takes as long as projects tend to take. Somebody has to do these things. Training new people is hard and doesn’t seem to make sense. If we’re getting rid of a system, why would we train someone how to maintain it?
So, the experts keep the old system running while the new system is being built. So, who builds the new system? Not the experts, that’s who. Usually, it’s people like me: technology experts. And while we’re banging away at the existing system’s UI, trying to figure out what needs to be coded, the domain experts are doing their jobs. Unfortunately, this means the domain experts aren’t watching the Big Rewrite very closely. Regardless of how good the team, if communication is impaired between the domain experts and the technology experts, things are going to move slowly, and wrong software is going to be created.
(This article is part of the Big Rewrite series.)
To add to the stress of the Big Bang comes another, mostly people-related issue. Almost all technology rewrites are driven by some technologist. Behind almost every technologist pushing for a Big Rewrite is a business person saying “But, why?” The question is valid. The product already works. It’s successful enough to even consider re-plumbing it, so we must have already gotten something right, no?
So, then come the justifications. They start with the real reasons the software is being rewritten (but usually censored to avoid the technologist looking like he or she screwed up big time on the initial development of the product). The system will be more maintainable. It will be easier to add features. “Oh yea? So we can do more features faster?” “Uh, yea.” “How much faster?” And so on.
As those discussions get heated and prove unsatisfactory, the list of promises gets longer. The system will be more scalable. System response time will improve for our customers. We will have greater uptime. And so on.
It’s rare, in fact, that a technology rewrite can deliver on all these fronts. A J2EE Web application may not prove in practice to provide higher availability than a mainframe application. Rails might be a more flexible and productive environment for a developer, but Rails apps slightly underperform equivalent PHP apps. So, you don’t sell Rails as something that will be faster than PHP. You sell it as something that is more flexible and maintainable, and will perform reasonably compared to a PHP application.
The piles of justification lead to piles of additional work and/or piles of mismatched expectations and disappointment after release.
(This article is part of the Big Rewrite series.)
When you do a technology rewrite, you want things to be clean. That’s usually a major goal in a project like this. And at the beginning of a Big Rewrite, while you’re still wide-eyed and hopeful for your application’s ultra-elegant, scalable, maintainable future, you’re faced with a question: Should we deliver the Rewrite incrementally or all in one big release? Now, imagine your existing infrastructure is a home-grown Oracle Pro*C-based CGI framework with its own cookie-based authentication mechanism which relies on carnal knowledge of an aging mainframe ERP system. Incremental deliveries means making the new technology work within the dirty framework of the old system. One big release would mean we could just turn off the old system, turn on the new one, and keep our new efforts isolated and pristine.
In most cases, it’s the Big Bang approach that wins the argument.
Now picture yourself as a developer or project leader nine months into this project. The old system, still in production, has been patched and enhanced along side the new one as you’ve been developing it. You haven’t had time to keep up with each and every change that took place in the old system. As a result, on top of behavioral changes, you’ve got an ever-evolving database schema to port to the new platform. And the new system’s wish list has gotten so out of control, that there are major differences between the old and the new. To top it all off, working from the old system as a specification didn’t work, and you’re way behind schedule due to misunderstood requirements and rework.
The table has been set. The guests are on their third course. And now you have to come along and replace the table on which they’re eating without disturbing their meal.
On a big system with a lot of customers, data migration can be a huge problem. Not only do we have to keep track of what gets migrated when, but we have to actually perform the migration at some point. The Big Bang sounds like a lovely idea until you get to the actual event, and you realize it’s kind of like preparing for a world title boxing match when you know it’s the first and last time you’ll ever compete. The processes and software you have to create, the attention you have to pay before you can create an event like this is often as consuming, complex, and potentially disastrous as the system development effort itself.
But by making it a Big Bang release, you’ve maximized the chances that you’ll be behind schedule when you get to the end, and you’ve therefore maximized the chances that you won’t spend enough time preparing. This results in a bad time for both you and your customers.
Unfortunately, perhaps due to something intrinsic in human nature, this scenario is a cliche for Big Rewrite projects.
(This article is part of the Big Rewrite series.)
Imagine going to the hospital for a kidney transplant, and before and during the surgery saying to the surgeon: “Oh, and while you’re already in there digging around, I’ve had some problems with my lungs that could use a little attention. And, yes, I’ve been overeating terribly—-could you do one of those stomach reduction things I hear about? And on that note, how about a little plastic surgery since we’ve got the knives out?”
This is effectively what happens on The Big Rewrite. An existing product, no matter how successful, always has a few warts. The rewrite is seen by many people as the perfect opportunity to shave off the warts. If we’re going to do it over again, we might as well do it right this time.
Under the veil of a rewrite, the assumption is that the personality and capabilities of the software aren’t changing. So, what might start as just a few little tweaks will usually turn into an unbridled reinvention, with none of the usual checks and balances that go into new product development. With potentially many stake-holders involved and an uncontrolled process, I’ve seen little tweaks end up increasing the total effort and feature-set of a Big Rewrite by as much as 100%.
If that's the case, why was the customer involved all the time? They had a completely working specification in an already working system. Replacing it is more a matter of reverse engineering than gathering vision, customer feedback, use cases, stories, or any of the other crap the XP team used.
Here’s the problem: when does the label "Payroll System" become so broad that you don’t know if it’s an invention or an implementation? Could it be possible that at a huge company like Chrysler the payroll system was unlike any other payroll system that had ever existed? And, within the realm of this possibility, might it also be possible that Chrysler were redesigning the system, because through such changes as globalization and evolving international tax and labor laws, the system which was being replaced was no longer valid?
My point isn’t to say Zed is wrong. He makes some excellent points and may very well be right (though, knowing some of the guys on the C3 project, I’d guess they knew the difference between invention and implementation).
My point is that it’s not always clear cut which things are implementation and which are invention. Worse than being ambiguous, it’s often not clear that it is ambiguous. My experience says that most of the time, people doing the Big Rewrite will assume that they’re doing an implementation and will staff and estimate accordingly.
Most of the time, they’re wrong.
"Make it do what it already does." That’s a tempting and simple way to view software requirements on a rewrite project. After all, the system already exists. The question of "what should it do when…" can presumably always be answered with: "what it already does".
There are two major problems with this assumption. The first, and most disruptive, is that the programmers don’t know what questions to ask. This is especially true if the programmers weren’t the original developers of the system (most often the case on a major technology shift), but even a programmer who did the original implementation of a product won’t remember every nook, cranny, and edge case. What’s worse, with the fragile safety net of an existing implementation, programmers can easily oversimplify the interface, and assume they know the capabilities of the system. If a combination of drop-down selections results in a whole new corner of the system, how are they to know without stumbling onto it (or performing an exhaustive and expensive test cycle)?
If the software you’ve built is complex enough that it needs to be rewritten, it’s probably also so complex that it’s not discoverable in this way. This means that domain experts are going to have to be heavily involved. It means that requirements are going to need to be communicated in much the same way they are on a green-field project. And it means that, unless it’s only used as a supplement, the existing system is more a liability to the rewrite than an asset.
Optimistic programmers might think I’ve missed something important here. If you’re rewriting a system, you’ve already got the code. The code can serve as the spec, right? Probably not.
Based on my own experiences and conversations with thousands of software developers around the planet, I unscientifically conclude that almost all production software is in such bad shape that it would be nearly useless as a guide to re-implementing itself. Now take this already bad picture, and extract only those products that are big, complex, and fragile enough to need a major rewrite, and the odds of success with this approach are significantly worse.
Existing code is good for discovering algorithms—not complex, multistep processes.
This is the first in a series of articles, discussing why many software rewrite projects end badly and what to do to avoid some of the ways I've seen them go astray.You’ve got an existing, successful software product. You’ve hit the ceiling on extensibility and maintainability. Your project platform is inflexible, and your application is a software house of cards that can’t support another new feature.
You’ve seen the videos, the weblog posts and the hype, and you’ve decided you’re going to re-implement your product in Rails (or Java, or .NET, or Erlang, etc.).
Beware. This is a longer, harder, more failure-prone path than you expect.
Throughout my career in software development, I’ve been involved in Big Rewrite after Big Rewrite. I suspect it’s because I have an interest in learning eclectic computer languages, operating systems, and development environments. Not being just-a-Java-guy or just-a-Windows-guy has led to me becoming a serial rewriter. I’ve been on projects to replace C, COBOL, PHP, Visual Basic, Perl, PLSQL, VBX (don’t ask!) and all manner of architectural atrocities with the latest and greatest technology of the day.
In many cases, these Big Rewrite projects have resulted in unhappy customers, political battles, missed deadlines, and sometimes complete failure to deliver. In all cases, the projects were considerably harder than the projects’ initiators ever thought they would be.
This is not a technology problem. It’s not at all Rails-specific, but being in the limelight these days, Rails implementations are both very likely to happen and very risky right now.
So, why, in software rewrites, when you’re traversing exclusively familiar territory are the results so often unpredictable and negative?
For the next week, I’ll post specific reasons I’ve seen things go wrong. The following is a list which will eventually be made into links:
Stay tuned.