As a newly minted CTO and Agile Manifesto Signatory thrust into managing over 80 developers in three countries, I firmly believe my then  decision (in 1999-2000) to run our fast-growing commertcial software firm as ‘all agile all the time’ likely saved me.

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Frankly it never ‘felt stange’ to do Agile as even then it was just common sense. I was amazed at the fallacies of logic which still are the domainant model of thought. In fact Agile is common sense! It’s a shame people still think it is some magical concept.
Also luckily for me the company flourished globally and we were able to sell the firm. Looking back I was young and arrogant enough to think I could do the job. Now I am… Well not young so much..

As you can imagine this created a deep commitment to agile as a way of life, as it just works if you are ‘allowed to succeed with it’. Those with experience know what I mean. So many failures I am asked to help on after the fact end up in this category (and I would say as an opinion that in most real results-producing initiatives in the domain of large corporations, the causation often is linked to internal constrains placed on those who would ’succeed if allowed’ and those constraints are often utter nonsense.

I am now of the blunt opinion that there is not much here that is not common sense. Rather then offer insight, it merely corrects a deeply flawed mental model help by the majority for many reasons (none of them logical or dependable in most cases at least in my opinion).

It’s never OK to operate knowingly harming to your stakeholders, regardless of if they could ever find out or not. Eventually they will as transparency into software is becoming assumed.

Indeed however, these same stakeholders can mandate their own demise on initiatives by applying incorrect meta-models of how software works (see outsourcing).

The answer of how to address this problem is at the very heart of moving software out of the operational ‘trade’ and into the strategic ‘craft’ producing assets of astounding beauty, quality and value.

Damon Wilder Carr
New York City
2008

As originally published April 26, 2005 (this is the first in what was a multi-part series)

Agile embraces changing requirements and does not even blink when it happens. Agile is about providing consistent and early working software - not prototypes.

Here is a quote that accurately describes the norm in software:

“Without knowing anything at all about your current project, I’ll bet even money that you’ll be late. After all, well over half of all projects deliver late or deliver less than was promised by the original deadline. It’s far worse when a project is on an admittedly aggressive schedule. Project people seem disconcerted when I proclaim that I’m willing to bet against them. They try so hard to believe that they’ll buck the odds. What usually happens is that everyone agrees that the deadline is very tight; everyone works very hard; and then, when people see that they won’t make it, they are shocked, disappointed, and deeply dismayed.”

—Tim Lister Reprinted by permission of Dorset House Publishing from WALTZING WITH BEARS (ISBN: 0-932633-60-9), p. 48. Copyright © 2003 by Tom DeMarco and Timothy Lister. All rights reserved.

Agile is not black or white, but a negotiated set of practices : Where is the essence lost?

You have probably heard of Extreme Programming and perhaps SCRUM, Evo, Crystal, and a host of others. Think of Agile as the ‘Abstract Base Class’ and each method as concrete implementations. I am often asked ‘what MUST you do to be called Agile’.

This is a tough question, and in my opinion, you must do perform the following steps to be considered ‘Agile’ there cannot be one master list, however there are a few immutables:

1) FEEDBACK LOOP: Coding is done best in my experience as ‘timeboxed’ (fixed-length) iterations (typically ranging from one weeks to four weeks). No extensions are allowed to this duration by design. Change that and you break the iterative model.

“But what If features cannot be completed when we promised?”

This is a good thing as you can manage it where the alternative would of been downstream chaos.

It is up to the project stakeholders to decide (if need be) if they would like those features moved into the next iteration (they could decide for whatever reason to cut the features for example – a rare occurrence but one that is at the heart of being Agile – responding to change).

2) It is nonsense to say ‘coding must be done using ‘Test Driven Development’ (TDD) and a ‘Unit Test Framework’ must be in place.

Why is that silly? Most do it wrong. You need more feedback loops. You need ‘verifiability’ over ‘testability’. I don’t care if it worked yesturday does it work today and do all the other items still work. VERIFY IT FOR ME.

Otherwise you have unmanageable multivariate hell. Even if you are an amazinf software designer, dependencies always exist that are more complex then any non-trivial system can manage by human mechaniations.

Here is a test. Complex software branches, then merges back into a mainline trunk, and most organizations have anywhere from 5-100 systems to deal with. Now make true non-trivial changes to that code base (and yes there should be effective reuse - oh no now your dependency management just became the #1 priority for success in many cases - which brings me to this: An architect’s job is dependency management.

He cannot do his job if changes are happeneing, causing unpredicatable results and placing your key software assets into a state of ‘null’ where null is ‘uh we don;t know if it works or not. It’s the lack of a value’.

Anyway not to be overly pedantic, but these concepts seems to be lost in favor ‘write a test, it fails, make it pass with easy code, refactor, repeat’. That logic and mindset is insulting and the opposite of what Agile means.

Personally I like to call this ‘Intention/Verification Development’ as programmers often psychologically do not respond well to the term ‘Test Driven Development’ anyway.

And ‘TDD’ is DESIGN, not testing. So call a spade a spade. You work through various iterations of your intention, settling on a good enough on that is verifiably ok in the face of change.

The main point (and one of the hardest to culturally get ingrained) is that software is not written until the SOME design is in place and it can be materially asserted to work or not work. Simple as that.

3) Plans are fine to create, but know they will change, and expect the change to be quite significant.

Agile is about admitting you cannot predict the future! If you have a culture that creates GANNT charts running out 12 months and believes it has any relation to what the reality will likely be, your organization is deluding itself.

The best you can do is statistically speak in terms of ‘probabilities’ and even that is pretty hard to do with accuracy (but I will show you how to at least vastly improve on the binary success/fail model so prevalent today in corporate America and likely one of the largest fallacies of logic still allowed to exist in our highly competitive business environment). Agile allows the Stakeholders (the CEO, the Business Experts, the Customers, etc.) to dictate the work to be done and when it is done. It is the rule, not the exception that you can expect your plans for the work you do (in other words the business requirements) in any future iteration to not be realized due to changes from your stakeholders. I am still astounded by the organizations that believe they can capture all of the requirements upfront, using a waterfall approach when it has been proven time and time again to be a recipe for disaster. Old habits die hard.Agile embraces changing requirements and does not even blink when it happens (and it will). Agile is about turning on a dime, responding to change without hesitation, and providing consistent and early working software (not prototypes) to the stakeholders to see, touch, and possibly even use.

Pair Programming

It’s hard to argue that this doesn’t work when done correctly but it just isn;t worth the fight all too often.

We know from the work of people like Dr. Laurie Williams in her book ‘Pair Programming Illuminated’, that this particular practice is filled with myths and false assumptions. When done correctly you can expect roughly the same amount of work produced as if the two individuals had worked alone. However you gain two critical elements:a) Collective Code Ownership. By using Pair Programming and switching your pairs consistently, you will create a shared knowledge of your system across your team. The typical model I see today is the ‘cowboy’ programmers pulling 100 hour weeks to get a system done, with the rest of the team struggling to add value on the far less critical elements of a project.

Studies prove that these ‘cowboy’ programmers are up to 2,800% more productive then their worst peers (see Robert Glass’s excellent ‘Facts and Fallacies of Software Engineering’ for more information) so it is no wonder these high bandwidth individuals tend to say “OK move out of the way and just let me do this. I can get this done in a few weeks”.

They are often viewed as ‘superstars’ and even promoted to management!This is another fallacy of logic as there is nothing about a high performing developer that would indicate they would be an effective manager, however I believe that is a main reason why we suffer in the US from astounding losses in software development in Corporations (up to $140,000,000,000 based on studies from the Standish Group). When performing pair programming, ‘cowboys’, often feel that they are doing more training then pairing, a key cultural problem to watch out for.b)

The quality gains in your code are significant, often as much as 200-400% when code is written in pairs. In turn, producing a productivity rate that is roughly equivalent to that of an individual, once the team learns how to pair program correctly. Yet why do so many organizations flat our refuse to embrace this obvious (and free) practice? The pair programming experience is highly social, with one person ‘driving’ (coding) and the other person watching intently and both individuals changing roles often. For the person watching and discussing the code, it is not so much syntax checking (we have compilers and IDEs for that). It is about keeping in mind the big picture, and asking for explanations where code doesn’t make sense.

Think of this as a traditional ‘code review’, done in real time. These two parties swap roles when it feels right to do so and they form a highly productive team. Thus, ensuring that all aspects of their project are accomplished: Strategic, Tactical and Operational. This is especially helpful when you have a developer who likes to write incredibly complex code that nobody can understand when a far simpler and ‘boring’ solution exists. The pair helps ensure this doesn’t happen right at the source (where in a more traditional code review process it would be to late). One problem in Agile I will address are the ‘cowboy’ programmers who are used to being the superstar and being greatly responsible for a successful project based on their immense talent and long work hours.One partner could say to the other:

“Do we really need to manually implement a recursive quick-sort when it is already provided for us in the framework we are working in?”

”Well I can optimize the sort I am sure as I read on a message board they didn’t optimize it properly”

”Well, the few CPU cycles we would save are not worth the time it would take (if indeed you are even correct). Remember we optimize last in Agile or we risk optimizing the wrong item, so lets use the framework and if we find it is a bottleneck we can always refactor to your solution later on. Deal?”

“OK you do have a point. In that case let me write this line, and we are done. OK let’s run the Unit Tests to make sure we are all green”

“Great Idea. If we are all green I’d like to drive and lets move on to the next task in our Story”.

NOTE: The term “green” is often used as the color of a successful unit test is green. It is more common then most developers ever predict. Later iterations of code are almost always written that break code from previous iterations. These broken segments are often items the developers were not even considering. The RED test is usually a fairly obvious problem that is quickly resolved. Once we are ‘all green’ code can be checked in (but that does not mean the build with all the developer’s code integrated was not broken. I will cover that in Continuous Integration, one of the easiest and most underutilized wins a team can implement (and the software is free).

‘War Room’ Environment

This is not really Agile in any way, Agile just points it out as ‘hey stupid this works almost all the time’. We’ve known this works for decades, but hey most still don’t know the GoF….

This is optimized by walls covered in white boards for impromptu ‘Agile Modelling’ sessions where all team members are extremely well versed in Object Oriented Design Patterns. This is nothing new (in fact there is very little new in Agile, It is the fact that these common practices have been brought together that makes Agile unique. Unfortunately, corporations often cannot dream of asking their people to leave their nice cubicles. Many managers cannot make this happen so they take a big hit in their probability of success to allow the core team to often be spread apart, have closed door offices, and suffering vast distances and time zones separating the team, making one of the most important aspects, face to face communication, hard if not impossible.

These processes are a negotiation, not a 100% mandate. Agile is almost always crafted to the organization. So how do we deal with these hindrances? How can we quantify the impact these poor decisions have on our likely outcome? Applying New “Best Practice” A process competency that has been sorely missing from Agile is a solid Risk Management Framework based on the concept of probability. One of the most influential books recently published is ‘Waltzing with Bears’ by Demarco and Lister, a must read for all Software professions (as is their classic ‘Peopleware’, in my opinion the single most important book for a manager to read (if I could only choose one).As we know software is often about predicting the future (which we know we cannot do) there are techniques that have proven applicability in many areas of business. It is astounding to me that these have not found their way into software. Perhaps people feel software is simply too complicated (and perhaps sometimes it is) but I would argue that it is light years ahead of the typical IT Manager committing to unrealistic dates and failing. In ‘Waltzing with Bears’ Demarco and Lister provide us not only a scathing analysis of the current state of managers predictive capabilities, they also provide us with something incredibly valuable: A Probability Distribution Curve for Software Projects. Armed with this (taken from thousands of projects) you are now armed to discuss all future events in terms of their probability, with the accuracy increasing over time.

About Damon W. Carr

Damon Carr is the CEO and Chief Technologist of agilefactor, based in NY, New York. The company continues its growth via Microsoft.NET Leadership, Software Engineering best practices Agile Process Evolution and the new book by Mr. Carr covering “Agile Software Factories” to be published in early 2006 by Addison-Wesley. Prior to joining agilefactor, Damon was the CTO and Co-Founder of Monetaire, a global company with offices in Europe and the US, where he spearheaded all technical innovation, including their flagship product which was named a ‘top three solution’ in Europe by Forrester Research. Damon continues to speak at the software industry’s most prestigious conferences including the upcoming ‘Better Software Conference’ (http://www.sqe.com/bettersoftwareconf/) where he is a featured speaker on ‘Agile Processes and Software Factory Evolution’, he also leads various high profile global user groups.