Monday, 30 March 2015

Scrum Team Size

One of the most important aspects of a Scrum team performing at their best is to get the size and composition of the team right. Size is simple, but it's importance is not generally recognized or explained. So I have taken it on myself to explain it here.

Composition is a bit trickier. Its not just about skill sets (which I already covered last year - see Agile Roles) but also about personalities and how members get along. Another consideration is how and how often teams should be reorganized. Due to lack of space and time I will leave the discussion of team composition until next week.

Scrum Team Size

I have worked in teams from a size of 3-4 to a team in the range of 14-18. In my experience a good Scrum team size is about five with a maximum of about seven. Depending on the people and the type of project you might be able to get away with a team size of nine.

I have read many books and articles on Scrum and the absolute maximum team size is always specified as either seven or nine. For example, I am currently reading an excellent book called Succeeding with Agile by Mike Cohn who says that the accepted rule is a team size of five to nine, but recommends (page 178) the less regimented rule of a team that you can feed with two pizzas. Mike mentions that this is a convention used at Amazon. (I'm not sure if this means that you can have a team larger than nine if you get larger pizzas or smaller developers:) Later on I will tell you how to split a team that is too big.

Scrum rules also give a minimum size of five, though I have also seen three mentioned. In an extreme case a small software company may have to make do with a team of three - two developers (one of whom acts as a part time Scrum Master) and a Product Owner (hopefully provided by the client).

Every Scrum article, book, expert and consultant emphasizes the importance of the team size, so don't question it! If you insist on questioning it, despite my advice, then I can give you several arguments which I explain in the next sections.

Social Loafing

Social loafing has been extensively studied starting with RingelMann about a century ago - see Ringelmann Effect.

I talked about this at length in my previous post.  In summary, social loafing is the effect that as more people work together to perform a task then the less effort each individual contributes to the task. The effect increases as the number of people increases but is measurable even with two people. Most studies show a pronounced drop when the group size reaches around seven to twelve people depending on the type of task.

Try to limit the size of Scrum teams if for no other reason than this.

Visibility

An important feature of Agile that counteracts social loafing is visibility.

Visibility was originally designed to assuage the fears of the customer. The PO (and other stakeholders) can, at a minimum, see that progress has been made at the end of each sprint. If they want even greater assuagement (is that a word :), they can examine the sprint board on a daily basis. Of course, the PO does not, and need not, know what each team member is doing but they can easily see progress as tasks move to "done" status.

peer pressure
counteracts
social loafing
However, visibility is also important within the team. In Scrum everyone in the team should know what everyone else is doing. This is essential if the team is to self-manage and self-organize effectively. It becomes very obvious when someone is not pulling their weight. The resulting peer pressure counteracts social loafing.

Obviously, as the team size gets bigger it becomes harder for team members to understand what everyone else in the team is doing. Studies have shown that even the most skilled multi-taskers can at most keep track of nine separate activities. My experience, and that of many others, is that a team of nine is the absolute maximum, while even seven is getting too big.

I have heard the argument that a Scrum team is like a sports team (such as a rugby team), and since a rugby team has 15 players you should be able to have a Scrum team of that size. (First, I would say that a rugby team is composed of two cooperating teams - the forwards and the backs - but I don't know enough about the sport to argue that further.) It's true that a Scrum team is a bit like a rugby team but the main difference is that individuals in sports team have much greater visibility - it is much more obvious what each player is doing. Another difference with sports, is that typically only a one or a few players are actively involved in the game at any time, whereas in a Scrum team everyone is working diligently all the time.

the major benefit
of the daily standup
is visibility
Scrum practices are important. In particular, the major benefit of the daily standup is visibility. This is why it is very important to make sure the standup is working properly as I discussed here a few years ago (see October 2012).

Remember, visibility is essential for counteracting social loafing. The team size should be small to facilitate this visibility otherwise team members can easily fade into the background.

CASE STUDY :)

A colleague once worked in a Scrum team that was far too big at a size of more than 20. As an "experiment" he tried several different strategies to avoid talking or saying anything useful at the daily standup. Due to the size of the team he found it very easy. Sometimes nobody would notice if he did not even attend!  This demonstrates how important a small team is for visibility.

Team Spirit

Having a good team spirit also counters social loafing. Many of the principles and practices of Agile encourage the team to work closely together which fosters team spirit.

To invoke another sporting analogy consider times in relay events such as swimming, running, etc. According to the Ringelmann Effect, times of athletes in relay events should be slower than their times in individual events. However, it is not unusual for athletes to perform as well or even better in a relay. This is entirely due to team spirit.

There are many aspects of Agile that promote team spirit.
  • focus on customer requirements
  • having a clear and common understanding of team goals
  • encouraging collaboration and having the team self-organize
  • rewarding the team, rather than the individual team members
  • encouraging open, honest communication
  • empowerment and self-management
  • motivation of team members to do their part
  • team members feel that they are making a valuable contribution
Notwithstanding Agile practices, team spirit may be affected by other factors, such as the personalities of the team members (something I talk about next week), competition with other teams, etc.

team spirit decreases
the larger
the team becomes
My point here is that team spirit diminishes the larger the team becomes which is another reason not to have a large team.

Competitive Spirit

Talking about team spirit I recalled that another way to get teams motivated is to encourage a bit of inter-team rivalry. I have actually only seen this work well once, where I was working with 12 to 16 recent university graduates. There was an interesting coincidence in that the group was almost exactly split between electrical/electronic engineering and software engineering graduates. I split the group into two teams based on their degree and tried to objectively compare the performance of each team, even displaying graphs of work completed. This worked well to motivate the teams, at least for a while.

However, I recommend caution in using this approach. First, you probably shouldn't do this if the teams need to work together in any significant manner - it does not encourage collaboration. Also you must keep it friendly and be entirely honest and open about it, otherwise it will be seen as manipulative.

Over-Specialization

Another problem with a large team is that team members may succumb to Production Line Mentality. With many developers there is too much opportunity to specialize in one particular aspect of the team's responsibilities. (The same can happen with entire teams as I discuss next week.) This leads to developers who are unwilling, even unable, to perform any tasks outside their area of expertise.

This has many undesirable consequences. First, there may be times when the specialization is not required and the specialist has nothing to do. Conversely, if the specialist leaves or has too much to do then there is often nobody with the experience in the area who can take over or help.

But the worst problem, in my opinion, is that specialists tend to concentrate on the their own narrow area of focus, rather than being aware of, and striving to achieve, the goals of the team.

Don't get me wrong. It is often very useful to have developers with specialist knowledge. However, they should be willing and able to work in other areas as required (On the other hand you should not waste their talents unless there is a very good reason.)

Coordination

So Scrum helps to counter social loafing, but there is another reason for the Ringelmann Effect besides social loafing. For complex tasks, such as software development, coordination overhead becomes an even more significant problem.
Complexity Grows Exponentially
as Team Size Increases

There are two aspects to coordination. First, there are technical issues to do with building the software - these are not addressed by splitting into smaller teams because there will still be a need for inter-team communication. It might be useful if there is a logical separation of tasks based on feature areas of the software so that the new teams can concentrate on different areas, but this does not reduce the need for coordination, just contains it within the separate teams.

IMPORTANT: Customer Focus when composing teams 

Note that it is essential to have a customer focus when deciding on the skills required for a team. Teams should be created to work on different areas seen from the customer's point of view. Hence a team should have all the skills needed to complete their tasks.

This is another good example of the importance of DIRE which prevents problems such teams being blocked by dependencies on other teams, over-specialization (as  above), lack of focus on what is important to the customer, etc.

I mention this now as it is a common problem in Scrum teams, and important to get right. I will elaborate fully next week.

The other issue of coordination is getting everyone on the team to work together to knock off the sprint tasks. This is where having two small teams instead of one large one can reduce coordination effort.

Some developers try to take this argument to the extreme, arguing for a "team" size of one. It's true that the most efficient way to produce software may be just to have one good developer go at it full-time, without interruptions. However, there are disadvantages to having too small a team as I will discuss next.

Minimum Size

So much for the maximum size - what about the minimum?

By extrapolation of the arguments above you might conclude that a team size of one or two is best. It's true that a lot of problems, especially of coordination, disappear when a single developer (or even two) works on a project. There is actually some validity in this argument - for example, there are many great pieces of software that have been created by one or two people.

But I think the main motivation for this argument is that most developers have the personality type that prefers to work alone. Moreover, industry training and years of behavioral conditioning have reinforced this preference. I don't have enough space to elaborate here but this is important enough to warrant a future post.

You probably already concede that for large projects using one very small team is silly. It might be most efficient to have one developer take five years on a project but in that time priorities may have changed. Even if inefficient, it is likely preferable, for example, to assemble a team of eight and complete it in one year. There is also increased risk due to the single point of failure.

On the other hand, if a large project is tackled by many small teams, then this is effectively the same as having one large non-communicative team. Unless you are able to split the work into many independent parts then you must encourage communication and collaboration by building teams of at least four.

So I recommend not having a team size smaller than four or five. If absolutely necessary you could have a team of two developers (one of which is a part-time Scrum Master) and a part-time Product Owner (possibly supplied by the client).

Splitting Teams

The obvious question now is if you have a large or growing team, how do you split it?

Once a team gets to a size of 8 then you should look at splitting the team into two. One way to do this with a team composed of one PO, one SM and 6 developers is to create two teams of 5 which share their PO and SM and have three developers each. If you already have teams with shared members (for example two teams of 7) then you can look at splitting into three teams of 5.

I have heard the argument that sharing a PO or SM between teams means more work for them. But working with two small teams should not really create much more work for the PO/SM than one large team. But it *will* save a great deal of time for the other team members if they do not have to attend long standups, planning meetings, etc (as well as having all the other advantages of small teams that I mentioned above).

Shared PO  

Having a shared PO between two teams should not result in more work than a PO with one big team.

However, I do not recommend having a PO shared by more than two or three teams. This would just be too confusing.
Another argument I have sometimes heard, for not splitting a team, is that there is no natural way to separate a large team as they are all doing similar work. It doesn't matter, split it anyway. Two teams can even work from the same product backlog, as long as they have a shared PO. Of course, it is probably best to have an equal mix of experience, familiarity with Scrum, etc on both new teams.

One common approach that you need to resist, when splitting, is creating "component" teams, ie a team that does not deliver capabilities to the customer but creates a component for other teams to use. Each team should be capable of delivering new features, that the customer can use, from start to finish. I will explain this next week when we look at team composition.

Conclusion

There are several effects detrimental to large teams. Generally these effects are seen in teams of any size but start to really kick-in once a team reaches about seven or eight people. These include:
  • increase in social loafing
  • reduction in team spirit
  • coordination overhead
  • inefficiencies due to over-specialization
At the other extreme, having very small "teams" of one or two people may address these problems, but to do this the work must be divided into many different tasks for each of the "teams". This introduces other problems:
  • some tasks are not easily divided
  • some tasks may take longer than others
  • increased coordination between teams
  • single points of failure
Moreover, when you have many small teams working together on a large project you effectively have one large loosely-coupled team, whence the large team problems start to rear their ugly heads again.

It seems that, even with their drawbacks, teams of size five to eight are the best size. The good news is that team spirit and many Agile practices will minimize these drawbacks. In particular, the visibility provided by Scrum's daily standup greatly reduces social loafing.

Monday, 9 February 2015

Scrum == Communism ?

I have heard it said, and have had the feeling myself, that Agile (Scrum in particular) is a bit like Communism. The follow-on from this is that Scrum is doomed to failure and/or that people won't accept it (presumably on ideological grounds). Let's explore the validity of this argument.

What is Communism?

First we need to decide what Communism is. I have always been a little vague on this but here are two simple definitions.
  1. The workers control the means of production.
  2. Classless society where you give what you can and you get what you need.
I will only discuss the first definition to say that it is not particularly relevant. First, the means of production in software development is the computer hardware and the brains of the developers. I don't think it's a big deal who owns the computers, and I don't think anyone would argue that the company you work for owns your brain. Of course, owning some of the company you work for (eg, stock options) may be a motivator but probably does not make much difference to the use of Scrum.
Please note that this article is in no way politically motivated or an attempt to promote Communism or any other political system.

I think the 2nd definition is more useful to our discussion. So the question is: Is there any similarity between a Scrum team and Communism as in the 2nd definition? Let's look at the definition in detail by splitting it into three parts...

Classless society

"all team members
are equal"
A scrum team is self-organizing. There are no bosses (within the team) to tell team members what to do. Of course, the PO determines the product backlog which gives direction to the team, but how the team accomplishes their goals is completely up to them.

Note that you shouldn't make the mistake of thinking that "self-organizing" means the team can do what they want - that would instead be anarchism. The team must work within the bounds set for them by senior management, the product backlog and the rules of Scrum. It is an advantage of Scrum that self-organizing teams usually organize themselves more efficiently and flexibly than an individual manager can.

Contribution (give what you can)

Just because all team members have equal rights does not imply whatsoever that they have equal abilities. Obviously team members will vary in what they can contribute depending on experience, knowledge and aptitude.

"everyone contributes according to their ability"
An advantage of Scrum is that team members, after time working together, grow to know of their colleagues' abilities and weaknesses far better than any outsider. This is one of the reasons that self-organization works so well. This, plus being empowered, means that the abilities of individuals are made far better use of. Further, the sense of camaraderie encourages the team to work towards the same end.

This is how Communism is intended to work but in practice "Communist" states fail because there is no incentive to work to your best ability. Scrum only works because in a small team everyone knows who is not pulling their weight and there is a team spirit and "peer pressure" to do your best.

Rewards (get what you need)

In Communism (in theory) there are no privileged individuals. Nobody has special benefits. Everybody is treated the same. Similarly, in Scrum the whole team is rewarded, regardless of the contribution of different individuals.

"everyone is rewarded equally"
This is another important advantage of Scrum. While rewarding on an individual basis encourages competition and hard work, it also promotes rivalry and virtually destroys collaboration.

A common argument against Scrum is that those in the team who make a greater contribution will be aggrieved because they get no better treatment than the worst performers. In my experience this rarely happens unless there are one or two team members not pulling their weight (in which case the team has the right to vote them out of the team). Generally, team members are committed to the team and not aggrieved when they know that everybody is doing their best.

Ringelmann Effect

So if Scrum is like Communism, the next logical conclusion is that it is doomed to failure like numerous Communist states. Wrong! There is a fundamental difference - a Scrum team is much smaller. In my opinion Communism (and Scrum) can work well for a small number of people. The ideal size is probably around five. Ten is stretching it. It can't work at all with a team size of 20 and there is no hope at all with a population of millions.

This is all due to the Ringelmann Effect, which has been known, and extensively studied for more than a century. The Ringelmann Effect essentially says that the larger the group performing a task the less productive each group member becomes.

The interesting thing is that when the size of the group exceeds about eight the
* The actual "plummet level" varies depending on the type of project. For a complex project it may be just 6 people. For a well-understood project it may be 10 or even 12.

My rule of thumb is to follow the Scrum guideline of a maximum team size of 9, but preferably around 5.
productivity, per person, plummets*. Studies have shown that a team of 14 people will produce less than a team of seven - that is total output not per capita output. Many would expect that by doubling the number of people in the team, the output, if not doubled, would at least increase, but it actually falls!

What is the cause of the Ringelmann Effect?

After much study it is generally recognized that there are two principal causes: social loafing and coordination overhead.

Social Loafing

Social loafing is the name for the phenomenon where people exert less effort to achieve a goal when working in a group than when working alone. Further, as the group size increases the average (per capita) effort decreases. The interesting thing is that this is often unconscious - the individuals may not even be aware that they are not trying as hard!

Social loafing is the main effect that Ringelmann discovered in his original experiments early last century. He found that, given the task of pulling a rope, the more people that are pulling the less was the average force generated. Importantly, when the size of the group reaches 7-8 then the average effort falls quickly.

What is the relevance of social loafing to this discussion?

First, I believe that social loafing is the principle reason for the failure of (large scale) Communism. However, social loafing in a Scrum team is reduced since team size is restricted to 9, and is typically 5 or 6. Moreover, social loafing is well managed by other Scrum practices such as the daily standup and is often counteracted by such things as team spirit.

One aspect of Ringelmann's experiments that we shopuld consider is they placed people in groups at random. I believe that if the groups had been given an incentive and were allowed to generate any sort of team spirit then the results may have been significantly different. For example, instead of simply pulling a rope with a group of strangers perhaps the groups could have been organized into "tug of war" teams that pulled against each other.

The benefit of "team spirit" is born out by the fact that in sporting events (swimming, running, etc) an individual will often perform at a higher level in a relay event than in an individual event (see Hüffmeier et al), though I suspect this depends on the specific individual and team involved.

Another important effect, which counteracts social loafing, is peer pressure. Peer pressure is often portrayed negatively but can have beneficial effects. The visibility of the daily standup (see Why It's Good in Scrum Standup) means that the team quickly becomes aware of any members who are not pulling their weight. I will mention (again) that for this to be effective the team must be of a small size (9 or less) otherwise the team will not be aware of what everyone is doing and individuals can fade into the background.

Coordination

Coordination refers to getting everyone working towards the same goal and simply communicating effectively with other team members when necessary. As the goals of the team, and the means to achieve them, become more complex, coordination rapidly becomes important. Once social loafing is under control (as discussed above) coordination become the more significant effect affecting the team's performance.

The problem of coordination affects any group endeavor so it should affect a Scrum team just as it would affect a Communist state. Again there is the obvious difference that a Scrum team has no more than nine team members so -- coordination is easier the smaller the team.

I will also mention in passing that some of the techniques of Scrum such as dividing a project into many small iterations avoid many of the problems of coordination. A proper discussion of this will have to wait for a later time.

Summary

All team members are equal, within the team. Team members contribute according to their abilities. The whole team is rewarded according to their achievements.

So is Scrum is doomed to the same fate as Communism? Not at all!

As was discovered by Ringelmann, social loafing can greatly affect performance of a team. This is a major problem for Communism and for Scrum. However, social loafing does not really kick in until the team size reaches eight or more, which is why Scrum teams should not be larger than 8 or 9 members.

Scrum practices also help. The visibility in the daily standup (again, this depends on the team being of small size) provides peer pressure which counteracts social loafing. Further, team spirit is enhanced by having the team work towards a common goal (ie, finishing the sprint backlog) which also overcomes loafing.

Coordination is another problem that is particularly important in software development due to the complexity of the work. Keeping the team small and adopting Agile practices can minimize this too.

It is generally agreed that Communism failed due to a lack of incentives -- people don't excel when they are given no reason to. This is often taken to mean that people are inherently greedy. But not all (or even most?) people are motivated by money/material possessions. A great motivator in a small team can be helping your colleagues and doing your part for the success of the team.

My conclusion is that Scrum is like Communism but on a small scale. Communism (and Scrum) can work well for a small group of people. It's just that it does not scale well.

Saturday, 31 January 2015

Lean Is Not Agile

Lean Software Development ostensibly tries to apply many of the principles of Lean Manufacturing to software development. I first remember reading about Lean Manufacturing, and the Japanese Quality Assurance movement, more than 20 years ago - before I had even heard of Scrum and well before the Agile Manifesto. In fact when I first read about XP (in '98 or '99) I remember thinking that a lot of ideas came from the quality assurance movement and even Lean.

However, in the last few years I have seen a lot of people saying or assuming that Lean Software Development is a "variety of Agile", like XP, Scrum, etc are. For example, I recently saw an advertisement for a course at a reasonably well respected university on SCRUM vs LEAN: Agile Methodologies Analysed and Compared.

Let's be clear on this, Lean is not a variety of Agile.
“ Lean is not
a variety
of Agile ”

Brief History

Lean manufacturing grew up as part of the Japanese Quality Assurance Movement around the middle of the last century. This movement was a major contributing factor in Japan's rise in a few decades to be the world's manufacturing powerhouse. The main focus of Lean Manufacturing is eliminating waste in its many forms.

The success of Japan's QA movement meant the ideas spread to other countries and many different industries. The ideas of QA and Lean Manufacturing found their way into SQA (software quality assurance) around the early 1990's. A great deal of them also found their way into Agile (perhaps via XP) in the late 1990's, but Agile is more than just a rehashing of the ideas of SQA and Lean.

A fundamental principle of QA/Lean is the concept of "Right First Time", which is trying to do things properly by eliminating mistakes. Agile outright rejected this idea, not in all things, but in the most crucial aspect of the software development process - how the software is specified and designed.

Agile is based on evolving the design based on responsive feedback. The assumption is that waste is unavoidable and that an emphasis on eliminating waste is counter-productive.

Lean Software Development

About 10 years ago Mary and Tom Poppendieck wrote a book called Lean Software Development: An Agile Toolkit, which apparently started a Lean movement "within Agile". I think I first heard of this book a few years ago but I have only just started reading it. I haven't had time to read all of it yet, but what I have read I agree with absolutely. It explains many of the principles and practices of Agile really well.

The only problem I have with the book is that it tries to apply "Lean Manufacturing" to justify Agile practices. The implication is that the core idea behind Agile is the lean philosophy of eliminating waste. This I find entirely fallacious.

The critical flaw in the premise of the book is demonstrated by this direct quote:

  Just as manufacturing processes are continuously improved by production workers, so must a software system be continuously improved by developers.
  - see Tool 19: Refactoring on page 137

The problem here is that it is confusing two different things: the manufacturing process and the product being manufactured.

A core principle of Lean Manufacturing and QA in general is to attempt to continually improve the process by which a product is created. But the actual product is not normally modified only the way in which it is made. For example, a watch manufacturer does not sell a watch (version 1.0) with only an hour hand, with the promise that later upgrades would add features such as a minute hand, increased accuracy, etc.

Applying this principle from Lean Manufacturing to software development means continually improving the way the software is developed. That is, Lean does not say to continuously refine the software (the product), only the process.

Rework is a QA term meaning work that has to be fixed or repeated because it was not done correctly the first time.
Hence the primary application of Lean ideas to software would be to improve that part of the process which far and away causes the most mistakes and rework - poorly understood, poorly communicated and continually changing requirements. Lean Manufacturing says to eliminate mistakes in the development process. In other words we must look at how to remove mistakes in the requirements.

By applying Lean Manufacturing principles to software development we must try harder to get the requirements right the first time. This is intended to avoid the enormous waste caused by repeatedly changing the design and the code. But this idea of BDUF is now generally regarded as futile.

BDUF?         

BDUF stands for Big Design Up Front and is the name given by Agilists for the way that software is normally developed using waterfall methodology. 

Decades of experience and studies have shown that, no matter how much effort is put into the initial design, there will be a disparity between what users need and what the software actually does.

I discussed this recently in Agile Design.
Agile Software Development

Agile takes the opposite approach to that of Lean Manufacturing. The assumption is that you can never get the requirements right the first time. There will always be mistakes.

The Agile approach is to start with the very basics of what you know, creating a minimal working initial version and then evolve the design based on feedback.

Software Development vs Manufacturing

Lean tries to take ideas from a manufacturing process and apply it to the software development process. However, these processes are very different - eg:
  • Manufacturing is about making many identical copies of a product. Every software product is different.
  • Manufacturing involves a repetitive process. Although there is some repetition in software development many things are unique to a project.
  • Eliminating even a small waste in manufacturing can make a large cost saving. The cost of eliminating waste in the software development process often outweighs the saving.
It should not be assumed that principles beneficially applied to manufacturing are similarly beneficial to the software development process.

Software Development vs Engineering

OK, software development is not like manufacturing, but isn't it like the design as opposed to the manufacture of products?

On the surface, this argument seems much more reasonable than the one above. A software developer is just another type of engineer; just as a civil engineer designs bridges or an aeronautical engineer designs planes, so a software engineer designs programs.

This is getting a bit off track but it seems relevant to clarify this here. Creating software is not like creating physical objects. There are important reasons when designing physical objects to get a whole lot of things right up front. For example, you don't want to be almost finished as skyscraper when you realise the foundations are inadequate. Software does not suffer (as much) from these sort of limitations and so can be easily evolved as required (subject to use of Agile practices such as Unit Tests). I discussed this at length in the Software is Different section of Agile Design.

The thing to remember: software developers are not engineers. The idea of "software engineering" has bugged me for many years and I will probably write about it in this blog soon.

Similarities

I should be clear that the QA (and Lean) share a lot in common with Agile. I touched on this in SQA and Agile Methodologies. Some examples are:
  • emphasis on JIT and deciding as late as possible
  • customer focus
  • automation of repetitive tasks
  • retrospectives and continuous improvement
  • importance of visibility of the process
  • quality is everyone's responsibility
  • empowering workers to make decisions
  • emphasis on training and learning
Despite that I do not believe some of the QA ideas typified by slogans such a "Right First Time" and "Quality Is Free" are always applicable. I might discuss this further in the future.

Conclusion

The first principle of Lean Manufacturing is to eliminate waste. This has been shown to work well for manufacturing. For a repetitive process designed to mass produce identical products, eliminating even the smallest waste can produce large savings.

However, this principle of Lean Manufacturing cannot be applied to the software development process. Every software project is different. The main type of "waste" that traditionally (and still commonly) occurs in software development is due to rework (of the design and the code) due to the requirements. To eliminate this waste the Lean philosophy says to improve the process - in other words try harder to get the requirements right from the start.

Agile takes the opposite tack. We start with the minimum requirements needed to get something working and evolve the design based on responsive feedback. In effect, Agile methods (especially Unit Tests) provide a means to deal with the "waste" in a far simpler and less costly manner than trying to eliminate it. This waste in unavoidable and trying to eliminate it is counter-productive.

This is not to say that Lean does not have anything to offer the software industry. In fact most of the principles of Lean are also shared by Agile. Even the philosophy of eliminating waste can be usefully applied to many areas of software development. But at their heart Lean and Agile are diametrically opposed.

Monday, 12 January 2015

Vendor Mentality

Last week I talked about production line mentality. Now it's time to consider vendor mentality.

How are they related? In essence:
  • production line mentality - I just do what the boss/job spec. says
  • vendor mentality - I just provide what the customer asks for

Note that I did not invent the term vendor mentality (though I did make up the term production line mentality).

It is apparently a well-known effect to which I was first introduced by a colleague - thankyou Rixt.
Vendor Mentality in Developers

In the software industry many developers and contractors suffer from vendor mentality. It is based on this simplistic approach: the client says what they want, the "software vendor" creates the software and the client pays for it. This works fine for (simple) manufactured goods. Creating software is usually much more involved than simply exchanging money for a product.

Here are some reasons:
  • the customer rarely knows exactly what they want
  • when the customer knows what they want, it may not be what they need
  • a customer will often specify what they see as the solution without really defining the problem or their objective
  • different customer representatives will have different understandings of requirements
  • what the customer wants will change as they understand their real objectives
  • what the customer wants will change due to external factors
  • what the customer wants will evolve as they try the software
  • what the customer wants can often change for no apparent reason

Vendor Mentality in Clients

Clients and customers can also suffer from vendor mentality - "just give me what I want". However, in my experience clients are far more amenable to feedback, negotiation and looking at the problem from a different perspective than developers believe. I have already discussed this problem on the customer side (see Customer Management) - which details how and why you need to educate your client if they have a tendency towards vendor mentality.

The Developer/Client Relationship

To overcome this problem both sides need to work together to establish a partnership rather than consider it a simple buyer/seller relationship.
  • developers need to ask questions rather than just take requests at face value
  • developers need to understand the business of the customer
  • developers often need to consider the customer's customer (see The Customer's Customer in Customer Management)
  • developers should not be afraid to say no to the customer
  • developers should identify ways to be of value to the customer
  • the customer needs a better understanding of the development process
  • the customer needs to actively guide the developers (via product owner)
  • the customer needs to be open about their problems and objectives
  • both parties should be honest and open with each other
  • both work together to come up with the most satisfactory solution

When Things Go Wrong...

Mistakes happen. A great indicator of the health of the relationship between a software supplier and their client is what happens when things go awry. In an unhealthy relationship a supplier will try to cover up their mistakes, or (if that is not possible) excuses will fly or attempts made to shift the blame elsewhere. On the other hand, mistakes by the client are often not pointed out and the price is grudgingly born by the supplier.

This situation is commonly attributed to a difficult customer. Indeed, the customer's representatives may make unreasonable demands, but this is usually due a lack of understanding on their part. First, they may not appreciate the difficulties of the development process. More often the problem arises due to a customer who does not really know what they want. In this case you need to clarify, for everybody's sake, exactly what they are trying to achieve.

However, rather than the customer it is more often the developers, and particularly managers, that suffers from vendor mentality. They will insist on staff bending over backwards to accommodate the client, and do anything to prevent the company (particularly themselves) from looking bad. In such a blaming culture mistakes are punished which results in a large amount of effort spent in trying to deflect blame and in trying to avoid getting caught up in other people's problems.

Better than adopting a vendor mentality (and trying to cover up mistakes or deflect blame) is to be open and honest with your customers. They (even large companies) are almost always less demanding and overbearing than their suppliers believe, especially when they fully understand the situation. It may be up to you, as the developer, to help them to fully understand the situation as already discussed. But if a customer cannot accept what you tell them and insist then you may be better off without them.

A major problem is that many developers do not have the skills and personality to effectively interact with customers. They may be afraid to question the client and point out mistakes in their thinking. This is exacerbated in a culture of fear which does not encourage risk taking.

Sometimes it takes courage to do what is right but in the long-term the customer will thank you. Here are some examples of what you as a developer can do:
  • never agree to unrealistic deadlines
  • never agree to take on too many projects at the same time
  • never take a request at face value - ask more questions
  • if what the customer is asking for makes no sense then try to ascertain their objectives (see below)
  • simply say no if it is not in the interests of the customer (or the customer's customer)
  • don't be afraid to tell the client of their mistakes
  • admit your own mistakes

5 Whys

OK, so how do you handle a client that asks for a software modification that makes no sense? Often a client will state exactly how they think the software should work, making many assumptions. Developers need to look beyond the request and ascertain the client's objectives. A particularly good technique I have found to arrive at the root cause of a problem is the 5 Whys technique. (Thankyou Inbar for introducing me to this technique!)

The 5 Whys technique simply says to ask "Why" to the initial request then to the subsequent answers until you can get no further. You can typically ask "Why?" up to five times (possibly more or less) before you get to the root cause of the problem.

Consider this example (where C stands for client or customer and D stands for developer):

C: We want to ask and record why the operator has chosen the recovery operation.
D: Why?
C: Because operators are selecting the operation when it is inappropriate.
D: Why?
C: Because they do not understand its purpose.
D: Why?
C: Because they have not been trained on its use.
D: Why?
C: We don't have time to give all operators training before they use the software.
D: Why?
C: Because training is expensive and we only have courses twice a year.
D: I understand. An alternative would be to move the "recovery operation" to the supervisor menu so only experienced users can select it. This assumes that all supervisors have been trained and understand its purpose.
C: Yes, all supervisors have undergone training. That sounds like a great idea!

Despite its usefulness there are things to watch out for. If you use the technique on a different client representative or group (or even the same representative at a different
The Wrong Approach :)   

C: I would like you to launch off a surface and into the air by using the muscles in your legs and feet.

D: What vertical dimensionality do you require?
time) you may get a completely different result. In my experience, it is quite surprising that a different set of answers, taking a completely different tack, often come to the same conclusion.

However, occasionally you can follow a different path and come to a completely different root cause. Of course, this is actually a good thing as it allows you to find different approaches to a problem, or even completely different problems.

Often asking the 5 Whys can reveal flaws in the client's understanding. The previous example may have instead gone like this:

C: We want to ask and record why the operator has chosen a recovery operation.
D: Why?
C: Because operators are selecting the operation when it is inappropriate.
D: Why?
C: Because it is too easy to select the operation.
D: Why?
C: Because it is presented as the first option on the sign-in menu.
D: Why?
C: Because that is the obvious place to put it.
D: Why?
C: We don't want people to sign in and use the system in a bad state.
D: Why?
C: That causes data corruption whence we have to restore the previous day's database.
D: That is not the purpose of the "recovery". How often do you have to restore the database?
C: It has recently been happening almost daily - on average once every 4 months per site.
D: We need to look into the reason for this!

Conclusion

As a developer you need to work with your customer(s) to find out what they are trying to achieve. Ask questions, take an interest in their business, and try to see things from their point of view (or even their customer's point of view - see The Customer's Customer in Customer Management).

Don't let the customer dictate the solution to a problem. Hopefully, you are much more familiar with different techniques and technologies, as well as any technical limitations. On the other hand, you may need to understand more about the customer's business in order to choose the best solution.

Conversely, the customer needs to better understand the development process (see What Every Customer Should Know in Customer Management). Some customers may need persuading to work with you in a partnership, but you will probably be surprised how amenable most customers are to developing a mutually beneficial relationship with their suppliers.

Above all, be frank with the customer, even at the risk of offending them (for example, by pointing out their mistakes due to lack of understanding). Of course, this should be handled tactfully, but it is in everyone's best interests to ensure the customer understands the truth.

Equally, you should not hide your own mistakes. This will help to build a relationship based on trust and mutual respect. All parties should be honest and open to work together to come up with the most satisfactory solution.

Wednesday, 31 December 2014

Production Line Mentality (and Silo Syndrome)

At the start of this month I mentioned, in passing, what I called Production Line Mentality. This is the propensity for individuals working in a large organization to concentrate solely on what they have been told to do, ie, their "job specification". This behavior is promulgated by a multi-layered, controlling management structure.

Two people pointed out that what I was describing already had a name - Functional Silo Syndrome or simply Silo Syndrome - a term ascribed to Phil Ensor. Though Production Line Mentality and Silo Syndrome are very similar I don't believe they are exactly the same as I discuss later.

Since the discussion of Production Line Mentality created so much interest I thought I should elucidate and mention a few things that I did not have room for last time. Later on I will talk about how the Agile approach addresses it.

Production Line Deficiencies

Some say the production line is the most important invention of the industrial revolution. However, it has well-documented problems such as being boring, demoralizing, demeaning and, if the truth be known, tends to generate product of low quality. Luckily, in manufacturing the traditional production line has almost vanished in the last few decades due to studies (eg, see next) demonstrating better approaches. Unfortunately, in the software industry elements of the production line remain.

To show the problem, and the solution, I will first mention a study of a manufacturing production line. This occurred at a Swedish washing machine maker in the 1980's. The company switched one production line to a system where different teams assembled the machines from start to finish.

Originally with a production line of about 20 people (six machines per day), they  reorganized into four teams of five people. (Of course, there were costs involved in re-tooling, training etc.) Each worker was given the skills and knowledge to assemble a major component or even the whole machine.

In brief, they found that productivity quickly increased to about 7-8 machines per day (and even more over time). Morale improved and less supervision was needed as individuals were motivated to get the work done and even suggest improvements that boosted their own efficiency. Importantly, the product had a marked improvement in quality. There were many other documented advantages such as less conflict and much greater flexibility when someone was absent.

Production Line Mentality in Software

OK, so the production line has problems, but what has that got to do with software development? Actually, quite a lot.

First, many developers are quite happy to potter along on their own problem with little or no interaction with others. By focusing on a small narrow task they lose sight of the bigger picture. To be productive all team members need to communicate and understand the organization's objectives.

Second, traditional software development methodologies encourage this behavior The Waterfall model divides the development process into a series of phases (much like a production line). The work is divided into specialized areas: business analysts, system analysts, designers, coders, various flavors of testers, technical writers, etc.

Finally, many large IT organizations are placed under a large amount of pressure by senior management. This often leads to a blaming culture and a controlling management style which is probably the largest contributing factor, as I discuss below.

Silo Syndrome

Although the Silo Syndrome has essentially the same causes and effects as the production line mentality it is usually described in terms of the behavior of departments or "groups" within a larger organization rather than individuals. In the interests of self-preservation a group becomes protectionist of their knowledge and uncooperative with other parts of the organization.

The original article on Functional Silo Syndrome by Phil Ensor beautifully explains this (see Functional Silo Syndrome). The following discussion of causes and effects of production line mentality generally also apply to the silo syndrome.

How Does It Happen?

The behavior starts as a response to an autocratic, controlling management style. As a defense mechanism the individual (or group) withdraws from any activity that involves risk. Some also restrict access to information, since knowledge is power. Of course, this is detrimental to the organization as a whole by stifling creativity, knowledge sharing, and collaboration.

There are many reasons for this poor management style. It could simply be due to an individual manager. More often it is due to the culture of the organization -- coming from the top. Autocratic managers are generally attracted to an organization with such a culture (but they can pop-up anywhere).

I can think of a few different scenarios:

Egotism

A common problem is a technically gifted individual who has been promoted to a position of power and attempts to micromanage all subordinates. Often they believe they know better than anyone working for them how to do the work. They only have people working for them because they don't have enough time to do everything themselves.

Their enormous ego leads them to believe it, but they generally do not know better than those actually doing the work. Even if it were true it is still demoralizing and demeaning to be micromanaged in this way.

Insecurity

A related problem is a manager that is insecure in their own ability. Surprisingly this may occur in concert with the above since the manager may project confidence but are worried that eventually others will see through the façade. By controlling information they can more easily cover up their mistakes or find someone to blame.

Self-preservation

One of the most basic instincts is for self-preservation. If a manager gives up control then their role may be seen to be less important or even redundant. By embedding themselves in the work processes they make themselves an essential cog of the machine. Also, as we saw above, an insecure manager tightly controls information to hide their own ineptitude.

This is perhaps the biggest impediment to addressing the problem. In a top-down, authoritarian structure, many layers of management are required to monitor and control everyone. If the people who actually do the work are empowered to make decisions then many of the layers can be reduced or even eliminated. Unsurprisingly, there is a huge amount of resistance to this sort of change.

Transference

If a culture of fear emanates from the top then it will permeate the entire organization. There is no advantage, and significant disadvantage and difficulty, in treating your staff well if you are treated badly yourself. In other words a manager cannot empower their subordinates if they are not themselves empowered.

Megalomania

It is rare but some managers simply enjoy wielding power to the point that it is almost sadistic. Luckily, I have only ever encountered one such manager who would deliberately inconvenience or even humiliate those that worked for him.

Ignorance

Finally, there are those managers who don't know any better. Either, they are inexperienced and are following the lead of others or this is the way they have always worked and are not open to any other options.

How Does It Work?

In most organizations, production line mentality results in a large hierarchical management structure. Each manager is given strict control of a group which is split into sub-groups each with their own manager, and so on. This hierarchy can contain many layers until you get down to the people at the bottom, who actually do the work.

Information flows vertically. If you need information from someone in another section this is done through your supervisor. Information is distributed by a manager on a "need to know" basis.

Rewards are doled out by management based on the perceived value of each individual. Inevitably this fosters competition and rivalry, which kills any sort of collaborative spirit. Individuals expend effort making themselves look good (and even try to make their co-workers look bad) to their boss, rather than focusing on the goals of the organization.

When something goes wrong it's easy to point the finger at the person responsible. However, the root cause of the problem is usually that the person who made the mistake was not given enough information or assistance to do their job properly.

Negative reinforcement (ie, punishment for mistakes) is the primary motivator, rather than positive reinforcement. This leads to an aversion to risk where nobody wants to take responsibility for anything beyond their small area of influence.

Indicators

Many people working in this sort of environment are not even aware of it or just accept it. This is often because they have have never worked for a more enlightened company. Here are some possible clues based on the work environment:
  •  there are more chiefs than Indians - ie more managers than workers
  •  monetary rewards are given based on individual performance reviews
  •  each position has a detailed written job description
  •  you often have to sign legal documents such as non-disclosure agreements
  •  requirements documents must be signed off by several people
  •  developers are not allowed to talk to customer representatives
Here are some clues that your boss is the problem:
  •  you don't really know what your boss does
  •  you can never find your boss or arrange a meeting
  •  you are not given enough information to do your work properly
  •  you have to ask for a (deserved) pay rise
  •  you cannot send emails to the client without clearing it with your boss
  •  your boss threatens to fire you when you express any sort of dissent
  •  you get into trouble for swapping tasks with a co-worker
  •  when you book a meeting with your boss she postpones or is late
  •  if your boss books a meeting with you and you are late she hits the roof
  •  all emails you send must be CC'd to your boss
  •  you cover up a problem rather than tell your boss
  •  you discover there was a meeting (on a subject crucial to your work) to which you were not invited
  •  you are asked to work overtime when there is no reason to do so
  •  your boss keeps reminding you about the poor job market
  •  your boss never admits they made a mistake
  •  your boss keeps reminding you of your own past mistakes
Here are a few general clues about an organization with a poor culture:
  •  you have never met anyone who uses the software that you write
  •  everyone knows about a problem that needs fixing but nobody is willing to tackle it
  •  there is often conflict in the workplace
  •  many problems occur - the same problems happen over and over
  •  people in the organization have stress-related health problems
  •  you are regularly given large design documents about projects you have never heard of
  •  two developers discover that they have both fixed the same bug
If you are a boss, here are some clues that you might be the problem:
  •  you get the feeling that you don't know the full story about what is happening
  •  the people who work for you show no initiative
  •  the people who work for you are incompetent
  •  the people who work for you are risk-averse
  •  you don't know the name of everyone who works for you
  •  there are people that you have not talked to in over a month
  •  you have an "open door" policy but nobody takes advantage of it
  •  the client asks for a simple change to which you give an estimate, but are shocked when the team cannot deliver

Consequences

By now I hope you are convinced of the detrimental effects of production line mentality (and silo syndrome). To reiterate it makes for an organization with little collaboration and creativity to solve problems. Insular thinking results in inefficiencies such as poor processes, redundant processes and bad decision making.

The effect on morale results in poor employee engagement, absenteeism and sub-optimal productivity. Moreover, because managers build themselves into the system they become a single point of failure which affects flexibility and resilience.

Prevention

So how do we prevent this from happening? The simplest way is to follow an Agile methodology. For example, in Scrum the team is self-organizing, so resist the urge to "help" them get organized. Another piece of good advice comes from XP (Extreme Programming) to "eliminate fear".

The most important point is for managers to empower their charges. Don't assume they will get it wrong. If they don't do it the way you would do it, it may actually be a better way. If you trust people they will usually honor that trust.

Some simple tips are
  • encourage spontaneous feedback, eg don't react badly to a mistake or oversight.
  • reward the team rather than individual to encourage collaboration
  • eliminate formality & the need to "go through channels"
  • have more than an open door policy - go out and talk to people
  • use instant messaging - but also talk face to face sometimes
  • encourage people to interact as necessary with whoever they need to

Does Scrum Promote Silos?

Scrum is great for counteracting "production line mentality". Allowing the team to self-organize promotes interaction (within the team) which results in better organization, processes and products. It is also good for morale due to team members having a feeling of making a contribution and working together toward a common goal. In particular, the daily standup allows everyone to know what everyone else is doing, with improved coordination and less mistakes.

But does Scrum inhibit collaboration between teams? The simple answer is - it shouldn't but it can.

When different groups do not collaborate this is known as the "silo syndrome" as we saw above. Sometimes Scrum teams have this problem due to a misunderstanding of the roles on the product owner and scrum master. Let's first look at the problems these roles are designed to overcome.

Product Owner

Even in (or especially in) an open, collaborative environment there can be a problem of conflicting and volatile requirements. There are often many people (users and other client representatives, managers, business analysts, etc) with input into how the software is to be modified. One manager might insist on all bugs being fixed as a high priority, another might want to sneak in their favorite feature or improve an existing feature -- you have probably seen this sort of thing yourself. Different people have different ideas and priorities; and these ideas and priorities tend to change often.

In Scrum, having a product owner and sprint backlog avoids this problem by having all requirements come through one person. This ensures that the developers are not pulled in many different directions at once. Further, because the sprint backlog is unchanging they know that things will not suddenly change, at least until the end of the sprint.

However, having a product owner does not in any way preclude the team from talking to people outside the team. In fact this sort of collaboration is encouraged. It simply means that all requirements come through the PO and are prioritized in the backlog.

Scrum Master

The scrum master is in many ways an interface between the team and higher-level management. The main task of the scrum master, outside the team, is to remove roadblocks. This involves them using their influence to obtain resources, help or information. Some scrum masters take it upon themselves to intercept interference from outside the team, such as a senior manager who wants to control the team. Their are always managers who think they can organize the team better than it can organize itself.

In summary, an over-zealous scrum master can turn a scrum team into a "silo".

Finally, a Scrum Team can be unfairly perceived as a silo. A team should be composed of all the persons required to accomplish their tasks from start to finish. This means that they may not require a lot of communication with people outside the team.


Conclusion

Production line mentality (and silo syndrome) begins with a culture where personal interest (or the interests of a group within the organization) is not aligned with the organization's best interests. Fear prevails and individuals are driven by self-protection (or protection of their department) rather than desire to do the best for the organization as a whole.

This culture is characterized by many layers of bureaucracy which is a classic example of the DIRE principle. Employees show no initiative and just wait to be told what to do. Risk-taking is discouraged since mistakes are punished.

The solution to this problem is to empower employees to take control of their own destiny. Encourage collaboration and discourage rivalry by rewarding the team and not individuals. Share the vision of the organization so that everyone is working towards the same end.

Generally, following the ideas behind Agile will encourage collaboration and work against the production line mentality (and silo syndrome).

[Note: I was also going to talk about the closely related problem of vendor mentality, but that will have to wait for next week.]

Monday, 1 December 2014

Agile Roles

Back in May I wrote about how software design is done in Agile. I received some feedback which I promised to respond to in a future post (this one). Sorry for the delay.

Someone asked about the role of designers in Agile. They thought I was implying that specialist designers were not needed. Well, just to be clear, I will say it outright: No dedicated designers! All or most of the team should be involved in the design though typically some will have more input than others.

"No dedicated designers!"
When moving to Scrum people who just do "design" need to stop it. Designers traditionally are either technical (sometimes called architects) or business-oriented (sometimes called analyst/designers). I talk below about how these (and other) roles may change when moving to agile but in brief designers can move in two different directions - either to more development work or to more of a PO type role. Architects often become more hands-on - spending more time coding and reviewing. Analyst/designers can move back to development or if not from a technical background will need to move into a role involving mostly acceptance testing and customer interaction.

Another comment was: "If design, programming (and testing) are to be done together then should they all be done by the same person?"  In essence: YES. But before I get into that I need to talk a little about roles in Agile teams.


Multi-skilled Teams

Everyone agrees that Agile development uses multi-skilled (or cross-functional) teams. The problem is nobody really agrees (or even pretends to know) exactly what that means. Does it mean each of the team members have the skills to do all, or at least many, of the required tasks? Or does it mean that altogether the team has the required skills but each specializes in their own skill-set?

In my experience most Agile "experts" use the latter definition. For example, I was recently involved in the implementation of Scrum in a large company, and one of the things that the Scrum consultants emphasized is that a Scrum team (ie, "pigs" not "chickens") is made up of members with a wide variety of skills. However, I get the strong impression that this is mainly to placate those employees who feel that they may be redundant in the new order of things.

In reality, in an organization suffering from production line mentality (see below) adoption of an Agile methodology will mean that there will be less need for some existing roles. Less BDUF (see my post on Agile Design) means that there is no need for dedicated designers as design is performed continuously by the developers. Also, developers take more responsibility for testing so there is less need for dedicated testers doing black-box testing (like regression tests). This means that developers have more responsibilities (rather than just coding), but this is a good thing as it promotes DIRE (see DIRE) but also makes the work more rewarding and interesting.

If you need less people for other roles, such as designers, does this mean you need more developers? The simple answer is no. In my experience the same developers can actually be more productive, so you may even need less developers too!


Here I should note that I am talking about using Scrum in a pure software development environment.

I know that Scrum has been used successfully in other environments and that is great. However, whatever the environment all team members must have enough shared context to understand what everyone else in the team is doing.
Scrum Team Composition

Ideally all members of a Scrum team should be developers. That is, team members (apart from the Product Owner and perhaps the Scrum Master) should have the ability to write code. To understand why I will first recap on the advantages of Scrum.

Scrum (and Agile in general) has many benefits such as promoting team-work, working towards a common goal, quick response to change, etc. This is achieved by good communication and empowering the team to make their own decisions. The daily standup is particularly important as it allows everyone to know what everyone else is doing.

When everyone in the team can (and does) understand what everyone else is doing this avoids issues like:
  • differences in understanding of what is to be done
  • poor division of work
  • duplicating work - such as two developers fixing the same bug
  • forces pulling in different directions and introducing conflicting changes
  • not realizing when someone needs help
  • not being able to have someone else take over a task due to illness
  • not being able to have the whole team focus on a critical problem
  • inability to find another team member who understands what you are doing well enough to perform a review
Unless the team members are all developers they cannot really understand what everyone else in the team is doing (and how, why, etc). This is essential for Scrum to work (see Why Scrum Fails).

Of course, different individuals will have different levels of experience and different aptitudes. Over time, team members become familiar with each other's abilities. It is easy and natural for team members to take each other's skills into account when allocating tasks during the sprint planning.


Production Line Mentality

The production line was a major invention in the history of industrialization, since even unskilled workers can become very good at a simple repetitive task. However, its drawbacks were quickly discovered. Workers suffer from frustration and alienation. Once a task is mastered boredom sets in and mistakes are made, even for simple things.

Quality and productivity are greatly improved when workers are given a variety of challenging tasks. Performing different jobs as part of a small team working towards the same goal can be a tremendous motivator.
Another problem with production line mentality is that it usually results in post-production testing - ie someone inspecting the product coming off the end of the line. 

This sort of "Black-box testing" is not a good way to add quality particularly for software (see Unit Tests - White Box Testing).

What I call "Production Line Mentality" is where everybody has a specific well-defined job specification. This is common in an organization with a top-down management style where a worker's purpose is to impress their boss rather than work with others. In other words it does not encourage people to work together to accomplish their team's goals.

It is caused by a controlling (rather than empowering) management style where each role is kept simple to be easily understood and hence controlled. However, it results in a large number of specialized positions and a multi-layered management structure to be able supervise everybody in the organization.

This discussion actually ties in nicely with some of my recent posts. First, multi-layered management is a classic example of the Layer Anti-Pattern which is a pattern of bad software design which I have found can be applied in other domains.

Moreover, these layers form an excellent example of the DIRE principle. First, the managers at the top of the hierarchy are typically completely out of touch with the people actually doing the real work in the organization as they are isolated from them by all the layers. Perhaps even worse is that it encourages the workers themselves to concentrate on their own specific task without regard for the goals of the broader organization.

In an Agile development environment the developers are empowered to do whatever they need to do (within their bounds) to finish their tasks by the end of the sprint. For a traditional controlling manager it may be hard to deal with an environment where you can't even understand what everyone is doing let alone control it.

A controlling manager can be a real problem for the Agile approach. On the other hand, having developers who are happy to (or used to) being controlled like this is another major obstacle for the adoption of Agile. Years of manipulation (and even threats) means that many are happy simply to be given work as outlined in their job spec.


Traditional Roles

Let's look at traditional roles and how they change in an Agile world.

Team Leader

Scrum books simply say that the team leader role is no longer necessary. The team self-manages so the need for a manager within the team ceases to exist.

I disagree with this. In fact most of the same books describe a role for the Scrum Master which is almost exactly what I have always thought a team leader should be. (Perhaps they like the different title to emphasize that many team leaders are using a bad style.)

The team leader is now called the Scrum Master. Many team leaders also write code. That's OK too - you can have a part time scrum master who also codes.

However, if you think a team leader/scrum master is there to tell the team members what to do, then forget it. One of the main advantages of Agile is that teams are self-organizing and self-managing (within the bounds set for them by the PO and higher management). Small self-organized teams generally work much more efficiently and harmoniously than "centrally-organized" teams.

Designer

There is no longer any room for analysts/designers who spend weeks or months understanding a problem and trying (unsuccessfully, of course) to come up with the perfect, well-documented design. In an Agile world analysis and design is performed "just in time" (see JIT). The developers design, code (and test) all at once. This is not to say that what is designed is left to the whim of the developers - in fact how the design evolves is more closely controlled through the customer representative (product owner).

Traditional designers can move toward a role with industry knowledge, in particular how the software will help the business (or even the customers of the business). Perhaps they will be the actual PO (product owner), though I prefer someone even closer to the customer than that. Probably they will become a "business analyst" assisting the PO. Depending on the software (and despite the Agile preference of JIT design) there may be a need to mock up designs slightly ahead of time to garner feedback from the PO and/or customer.

Alternatively, or also, they may also do a lot of Acceptance Testing (see below). Whatever they do, their main focus (as for any Scrum team member) should be on completing the sprint backlog before the end of the current sprint.

Architect

There are just two points I want to mention about software architects using quotes from the excellent book 97 Things Every Software Architect Should Know.
" [the architect] should not sit in an ivory tower dictating the way forward but should be on the ground working with the team "
- from 19. Architects must be hands on by John Davies
"As well as being prone to incidental complexity and inertia, designing large systems upfront means larger projects, which are more likely to fail, more likely to be un-testable, more likely to be fragile, more likely to have unneeded and unused parts, more likely to be expensive, and more likely to have a negative political dimension. ...
"Design the smallest system you can, help deliver it, and let it evolve towards the grand vision. Although this might feel like giving up control, or even shirking your responsibilities, ultimately your stakeholders will thank you for it."
- from 97. Great software is not built, it is grown by Bill de hOra.

In brief: architects work with the team to evolve the design, spending most of their time coding and reviewing.

Programmer

In an Agile environment programmers now do everything that is required of the team (analyze, design, code, test, etc). They do whatever it takes to complete the sprint tasks by the end of the sprint.

This is in stark contrast with some traditional environments where programmers have just been coders - they take a design from the analyst/designer and code it for "QA" to test. This is demeaning for programmers who are usually capable of a lot more. Even if you care not for the morale of developers giving the programmers free reign to think about the design and even experiment at little will produce a much better design than any analyst/designer (no matter how clever) can invent in their head.

Of course, this does not mean the programmers can do whatever they want. They are closely guided by the product owner who, after all, prioritizes the backlog effectively deciding on the sprint task(s) that are the focus for the team.

Tester

If waterfall development is disempowering to programmers it is much worse for testers. As compensation, testers are often given the impressive title of quality assurance experts, despite the fact that quality assurance has little to do with testing.

You may not believe this but using Agile development (especially Unit Tests) there is a greatly reduced need for dedicated testers. How great this reduction is depends to a large extent on how much testing the developers do already. At one extreme I recently worked in an environment where there were twice as many testers as developers! The software contractor had there own tester(s) who worked closely with the coders. Then the customer had several testers, plus an "offshore" test team to perform regression testing, as well as an automated testing team.

There are many different names for the types of testing (and testers) that are used in the software industry. In Agile there is just two general types of testing:

  1. Unit Tests: automated tests written by the developers of the code to test individual (low and high level) modules
  2. Acceptance Tests: performed by the PO, customers, or anyone who wants to try the software

Note that Unit Tests are always automated, but acceptance tests may be automated too. Often the developers will add a simple scripting system to make it easy for the PO and others to create test scripts.

This is useful for testing the current changes. But don't go overboard and add lots of tests for existing functionality as they will most likely fail when changes are made (and even act as a disincentive to change). Leave the regression testing to the Unit Tests.
To understand this better consider the testing that an aircraft manufacturer does. First, individual parts are developed (or purchased). The developers of these parts (for example, spark plugs) will design, build and test them themselves and provide a hopefully reliable product. Some parts may be made from a combination of other lower-level parts, but it is still up to the developer of any part to thoroughly test it. Of course, larger, more complicated, parts may require more elaborate testing - testing of an aircraft engine may require a huge test rig. In the end it is always up to the maker of a part to ensure it performs to the requirements of the plane manufacturer. Testing of parts is the equivalent in software development of Unit Testing.

When the first plane (or a prototype) is finally built it is then up to a test pilot to perform various flights and manoeuvres to ensure that the plane performs as expected. This is the equivalent in software development of Acceptance Testing. This analogy is not perfect though, since Acceptance Testing of software should usually be performed much earlier than test flights.

If a Scrum team requires a dedicated tester at all then perhaps one or two members (at most) could be required for Acceptance Testing on behalf of the PO. They should spend much of their time understanding what the software is to do by discussing it with PO, business analyst, customer and any other stakeholders - something that an analyst/designer traditionally does. They might also attempt to automate much of their work using acceptance test scripts.

Whatever, they do their main focus should be producing working software at the end of the sprint. This means giving the users what they need as opposed to testing for conformance to any written specification.


Conclusion

If you are about to, or have been trying to, move toward an Agile development environment then you should ignore the advice given by many Agile coaches and experts that most roles will not change. There is no longer any need for software designers and less need for testers. Instead the whole team is involved in analysis, design, coding and testing - and this is all done "just in time" (see JIT). (Of course, if they are willing to adapt a role can be found for anyone -- it may be more challenging but will probably involves better use of their abilities.)

Most importantly, managers have less control of the team, which is now self-organizing. The major benefit is simply that a creative team can organize themselves much better than any manager. It also makes for a more rewarding and enjoyable working environment. The developers concentrate on creating good software and overcoming hurdles rather than making themselves look good for their annual performance review or trying to shift the blame when something goes wrong.

It may be hard to overcome the unthinking production line mentality (I do what the boss tells me to do) or vendor mentality (I do what the customer tells me to do). But please try!