OK I GIVE UP

Phoenix Project: A Novel About It, Devops, And Helping Your Business Win

Book cover image

This book on IT is different from most others I have read on the topic. Instead of stringing acronyms, ideas and methods, it tells the story of a team lead who finds himself the head of IT in a big company. It would be difficult for me to call The Phoenix Project a novel; I would expect certain things from a novel, such as depth of characters, or a certain dialogue with literary tradition. The Phoenix Project has none of these, but it doesn’t need them. Its aim is to tell a parable on how IT can crash and burn, taking the rest of the company with it. Alternatively, through proper management, it can integrate with and serve the needs of the rest of the company. It feels like a documentary more than a novel, following a troubled IT head with a camera over his shoulder.

Our main character, Bill Pallmer, gets promoted from being a team leader to VP of IT as his employer, Parts Unlimited, is going through some dofficult times. The once-leading supplier of auto parts is trailing behind the competition on all fronts, but most importantly in profitability. It soon becomes clear that the main culprit for this is IT, and if the horrible situation the IT department is in doesn’t change, the company will be split, IT will be outsourced, and everyone working at IT will lose her job. As he tries to bring some order to the extremely chaotic situation, Bill finds out that since the IT department does not do proper maintenance, technical debt has risen to a scary degree, and fundamental pieces fail disastrously. At the same time, there is intense preasure to deliver business projects to the other departments, most importantly on the Phoenix Project. The Phoenix project is supposed to tie different sales channels together, add online shopping to them, and provide detailed data on customer behavior and inventary status. It is late and over-budget, and the prospects get even worse when a new version is deployed. The new code does not work as expected on production systems, grinds to a halt due to performance issues, and worst of all, leaks users’ credit card data.

As the drama of the initial screw-ups unfolds, we get to have a glimpse of how Bill’s promotion affects not only his relations with his colleagues, but also his family life. This is one of the strengths of this book; it doesn’t shy away from depicting how serenity in one’s professional life is very important for social well-being, and how this serenity is impossible to achieve if the necessary processes and infrastructure is not there.

Bill starts his crusade to turn the IT department into a functioning unit with establishing clear procedures to battle crisis situations, and mapping out all the work currently done and planned for the future. It turns out that IT is in a very difficult place; it is impossible to handle the project work with the current manpower, most of which is going to urgent work. The main bottleneck is one systems wizard named Brent who can’t concentrate on a single piece of work because he constantly gets pestered with requests from heads of other departments and has to do firefighting most of the time anyway. As things start looking bleak, Bill meets Dr. Erik Reid, a hippiesque and scruffy man who is considered for the board. Erik starts making regular trips with Bill to one of the production plants of Parts Unlimited, and instructs him on the challenges faced in the production process, which are disturbingly similar to those faced by IT. The production plant solved those challenges using lean methods, which have to be applied to IT management now. Erik’s lessons don’t come for free, though: Bill has to connect the dots together himself, making the connections between which part of IT corresponds to plants, units, and resources in the factory.

Erik’s lessons follow a methodology the authors call the Three Ways. These are three requirements a department has to satisfy in order to contribute optimally to the goals of the organization. The First Way is recognizing the goals which IT serves in the whole organization, and increasing the throughput to meet these goals. In order to achieve the First Way, a plant/IT department has to be aware of its resources and constraints, and organize work around those constraints so that they are utilized to the highest degree (as per Theory of Constraints). One enemy of throughput is WIP (work in progress), called ‘the silent killer’ by Erik. WIP causes distraction, lack of quality control and decrease in throughput due to clogging of critical resources. Erik tells Bill that he should start improving from the bottlenecks, because any other improvements will not have any effect unless the bottlenecks are removed. In order to do this, Bill installs a change management process to ensure that the flow of work through IT is organized and visible. He also makes sure that Brent is working on Phoenix, and not getting distracted by urgent requests, which have to go to other people now.

One of the most important issues faced by Bill’s team is that firefighting happens so frequently. Erik calls unplanned work ‘anti-work’, since it destroys the available time to do actual planned work. The main reason IT is doing so much anti-work is the technical debt that causes regular outages in critical systems. This anti-work also stops them from making the most out of their constraint, Brent. In order to get rid of firefighting, Bill has to rely on the change management process, and give priority to internal projects to clean up the infrastructure. His team also does regular fire drills in order to handle crisis situations better.

One of the more interesting lessons of this book concerns the time tickets spend waiting for someone. Conventional wisdom dictates that highest amount of work is done when everyone is busy all the time, working off tickets. If this is the case, however, work that has to change hands (most work, that is) will have to wait for a long time because the work queue of the person taking over is full. Accordingly, to decrease the amount of WIP and achieve high throughput, “Everyone needs idle time, or slack time. If no one has slack time, WIP gets stuck in the queue” (p.236).

The First Way is followed by, you guessed it right, The Second Way, which concerns how fast technological improvements and new features are presented to the user for faster feedback cycles. The lesson here comes from the time capital is locked up in R&D in the production process, and batch size in production. The longer it takes for a new product to get developed and brought to the user, the less of a payback it leads to for the business. Reduction of batch size in production is another good analogy for this; as the batch size gets closer to the theoretical ideal of one, throughput increases and variance decreases. To conquer the Second Way, IT and development work together to implement Continuous Integration, making Phoenix deployment automated, and removing the differences between production and testing systems.

The Third Way involves changing the culture to encourage people to take risks and learn from failure, trying new things but at the same time practicing to achieve mastery. Thanks to the success of the measures Bill implemented, he is picked as the future COO of Parts Unlimited at the end of the book, and it is insinuated that he will work to remove broders between the departments to work together on new projects and ideas to achieve the Third Way. The term DevOps is also mentioned, when Erik asks Bill to write a DevOps Cookbook as a way to pay hist debt back.

As I mentioned at the beginning, this book is more a parable than a novel. It doesn’t shy away from using cliche expressions like ‘flash a smile’, ‘his face turned scarlet’ etc. frequently. Some characters are also narrative-wise a bit problematic. Erik Reid is rather deus ex, for example, but you have to get the ideas on lean production in there somehow, right? It would be rather boring if we were told what Bill actually read, or which seminars he attended. Another problematic person is John the security officer, and how he changes from an unkempt and asocial security nerd to a cooperative team-worker overnight, thanks to some scolding by Erik. No one will read this book for psychological depth, though. There is also a villain in the book, one Sarah who is head of sales and marketing. Sarah sides with the board to split the company, actively undermining important projects. The weird thing about Sarah is that despite being a bit simplistic as a character, she appeared to me rather realistic; people like her really exist in the business world.

One of the strengths of the book is that the technical details, and the details of how real-life interactions in IT work are spot on. So for example when the name of a table on the live database is changed, the deployment of the Phoenix project fails because this table name is hard coded in many places, and fixing it takes a while.

The Phoenix Project is a very enjoyable read that imparts important lessons on IT management and software development. It is among a handful of books I would recommend to everyone working in a relevant field. I think it would be extremely useful even for people who rely on IT to get their daily business done.