Sunday 13 April 2014

DIRE

The most useful ideas in software design can be applied in different ways and in different settings. They can save a lot of time and trouble if you can learn to recognize when they are appropriate. Otherwise, you can try to keep them in mind when designing and building software.

One such idea is the DRY principle (see the DRY section in Software Design). It simply says that keeping an eye out for duplication (in its many subtle forms), and trying to eliminate it, will save you a lot of time and worry in the long run. The main advantage is that it is easy to remember and, with practice, easy to identify and apply. The DRY acronym was invented by Thomas and Hunt (see The Pragmatic Programmer). Of course, the idea behind it has been around a long time but Thomas and Hunt elucidated and promoted it well.

Unfortunately DIRE is a 4 letter acronym. I know the fashion nowadays is for TLA's but this was the best I could do.
DIRE is an acronym that I invented for another idea. It stands for Don't Isolate Related Entities. It basically means that you should make every effort to keep things together that belong together. There is a close relationship between DRY and DIRE -- DRY is really a special case of DIRE where the "related entities" are so closely related that they can be partially or totally combined.

DIRE may seem obvious but many things conspire to defeat it. It is not always apparent how or why "entities" are closely related. Often related entities are separated for technical (and even non-technical) reasons, or simply by accident. I give examples below to show the broad range of areas and how you might apply it.


Origins


I guess everyone has had the experience of being interrupted in a sequence of tasks. When you come back later you've forgotten most of the context. Continuing where you left off is tedious and error-prone. In particular, interrupting developers a lot is a great way to generate bugs.

Like everyone I quickly discovered this when I started programming professionally (about 3 decades ago). However, it was not until about 2 decades ago that I began to think of this as an example of a more general principle. This was precipitated by my first encounter with JIT.

I discussed JIT in detail last week but in brief JIT is a QA technique where things are done "Just In Time". JIT is particularly useful in software development to:
  1. Increase visibility making it easier to find problems in the development process
  2. Reduce costs of delay (such as interruptions as discussed above)
  3. Reduce costs of rework (such as redoing design and testing)
SPACE vs TIME

JIT is an example of not isolating related activities in time. I guess in that case DIRE actually stands for Don't Isolate Related Events.

There are also many examples of not isolating related things physically (ie in space) - from the order of lines in source code to the seating arrangements of developers.
The cost of rework has by far been the biggest problems in traditional software development. For example, a large up-front design can mean a lot of rework as the design needs to change. But typically, to avoid the cost, we somehow find a way to live with a bad design, with many undesirable consequences which I explained a few months ago (see Change).
JIT
is an example
of DIRE

Gradually developers are realizing the advantages of JIT, and hence DIRE, since JIT is an example of DIRE.  For example, many Agile practices (such as JIT-design and TDD) avoids the costs of rework by applying the idea of JIT, and performing related activities more closely in time.


Examples


Coding

A good example of DIRE as applied to code is the declaration of variables. Separating the declaration of a variable from its use is a common source of bugs (as I mentioned in point 8 of this Code Project Article). This is why C++ improves on C by allowing variables to be declared anywhere not just at the start of a compound statement.

Many Coding Standards work against the advantages of DIRE, by prescribing:
  • order of declarations in a header file
  • order of functions in a source file
  • order of members in a class

Design

Most design practices and patterns do not intentionally contravene DIRE. However, they almost invariably promote decoupling in some form or another. Decoupling by it's nature tends to separate things. Though essential in almost every design, decoupling can be overused or misapplied.

Often a better design can decouple in a different way and not isolate closely related things - a good example is provided by the Layer Anti-Pattern. Sometimes a particular design pattern is appropriate but a cumbersome interface can make communication difficult between two closely related objects. Finally, sometimes complicated design patterns are simply not necessary but are used due to a misunderstanding or zealous desire to try something new.


Management

Managers essentially do two things:
  1. help employees to achieve the goals of the organization
  2. show that the work is being done
The first of these is very much about using the DIRE principle; but the second can work against DIRE.

First, helping employees is mostly about removing roadblocks. Roadblocks take many forms but are essentially things that prevent people doing their job efficiently and effectively. Removing roadblocks simply means getting rid of something that isolates related entities - such as a developer and the information they require to do their job.

Another part of helping employees is to get them working towards a common goal. This is achieved by fostering communication between them - another good example of DIRE where the related entities are the team members.

The problems occur when a manager allocates tasks and tries to monitor the work being performed. To do this a project must be divided into smaller and smaller tasks. The problem is that dividing tasks, by definition, creates divisions. If the way that tasks are divided is badly done then related tasks are separated in time or done by different people when they should not be. The classic example of this is the Waterfall development methodology as discussed in the next section.

Finally, some managers directly contravene DIRE for their own self-interest. This is why developers are not allowed to communicate directly with users or why they are only given just enough information to do their job. By controlling access and knowledge managers retain power which reduces the likelihood that it will be discovered that they are unnecessary, or not as important, to getting the work done as they like to believe. Of course, this sort of behavior is bad (but still common in the software industry) since it has DIRE consequences.


Process

Development methodologies are one of my pet topics (and the reason for this blog). Here also there are good examples of the benefits of DIRE.

As I mentioned above the best (or worst, depending on how you look at it) example of this is the Waterfall development methodology, which tries to control and monitor the development process, but in the process isolates closely related tasks. The processes of designing, coding and testing of a particular features or piece of software are very closely related. The Waterfall methodology separates them (and not just in time) by splitting the project into phases. This contravention of DIRE is the reason for all the problems with the Waterfall methodology and the main reason for the invention of Agile (as explained below).
the Layer Anti-Pattern in design is analogous to the Waterfall model of development
It's only when you think of these things in terms of DIRE that an interesting analogy between software design and the development process becomes apparent. Splitting development tasks in the wrong way causes DIRE problems. In software design, as we saw above, splitting the design into modules in a bad way causes similar problems. A specific example is the analogy between the Layer Anti-Pattern in software design and the Waterfall model of development - splitting a design into a stack of layers is like splitting the work into a series of phases. In both cases the separation causes problems by isolating closely related entities.


How Do Related Entities become Isolated?


The whole point of DIRE is to bring together things that should never have been separated. But was there a reason for this separation in the first place? Sometimes things are separated by accident but usually it is done for a reason. Often there is a trade-off, where DIRE is sacrificed for some other perceived benefit such as reduced complexity (as in the Layer anti-pattern) or improved monitoring/control (as in the Waterfall methodology).


Divide and Conquer

The most common reason for isolating things is to reduce complexity, using the principle of divide and conquer. This is fundamental to software design and expressed in such terms as modularity, information hiding, decoupling, etc (see Software Complexity).

Although important, it can be used improperly. For example, how software is divided into modules is often done poorly. On the other hand a split *may* be appropriate but the method with which a module communicates (ie, its interface) may be too limiting. And sometimes different parts of code are separated prematurely or for no good reason at all.


Planning

Talk to almost any manager and they will say "You can't achieve anything without a plan". In fact many will believe this with every ounce of their being. I guess this is natural as planning is their raison d'etre. However, it is an exaggeration at best. In fact, there have been many worthwhile achievements in human history which had no plan.

Don't get me wrong - I think a little bit of planning can be a good thing. However, the tendency is to divide a project into many small tasks. Each such task has a well-defined deliverable so that the manager can closely monitor progress to determine if the project is behind schedule.

Unfortunately, detailed planning can interfere with actually achieving team goals. By dividing the project into tasks the implication is that each task can be completed independently, which can result in closely related activities being isolated. Even at a high level dividing a project into stages means that work stops after each major milestone, then gradually builds up before the next deadline. This stop-start work-flow isolates related tasks and leads to poor continuity.


Playing Safe

A common reason for separating related entities is avoiding risk. JIT is a perfect example -- when JIT is used in manufacturing the process becomes crucially dependent on input parts arriving on time. Hoarding inventory is safer but introduces the problems I discussed last week.

In software development a good example of the disadvantages of playing safe is demonstrated in the traditional way of designing software. Here the design is supposed to be fully specified and signed off by the customer before coding is begun. This allows for the developers to protect themselves if  when things go wrong and the customer becomes unhappy. Their usual refrain is "It's not our fault, you signed off on the spec."


Much better results are invariably obtained using the Agile approach of JIT-design (see below). Doing things the right way usually involves more risk which is the reason XP promotes the value of courage (see Extreme Programming). 

Retaining Control

Another, more insidious, reason that managers like to isolate, is to retain control. If developers start talking directly to users then the work may begin to get done without even involving the manager. And since knowledge is power then information is doled out on a "need to know" basis. If you have a boss like this then my advice is simply to look for a new job.


Over-design

Similar to the problem of over-planning is the problem of over-design, exemplified by the Gas Factory Anti-Pattern. Martin Fowler gives a very reasoned account of this in Is Design Dead?

Accidental

Sometimes things are separated for no good, or apparent, reason. This is a problem to especially watch out for as a design evolves.

Agile


DIRE  is at the core of the Agile methodology. Consider the four elements of the Agile Manifesto and how they exemplify DIRE...

1. Favor Individuals and Interactions over processes and tools.


I have always found this rule a bit vague, but I think it is about getting the team working together, rather than following some safe, bureaucratic procedure. Breaking down barriers and encouraging communication and brainstorming is a great example of DIRE.


2. Favor working software over comprehensive documentation.


This is a critical part of Agile and nowhere is it better demonstrated than in the use of Unit Tests. Unit Tests are working software. Moreover, they can work as a substitute for massive amounts of design documentation. In fact they have several advantages over design documentation (see the Documentation section of Unit Test Advantages).

How does this relate to DIRE? Well the problem with a design document is that it is isolated from the "real" design - ie the actual implementation. The only link between the two is through the mind of the developer(s). This has forever been a major source of problems since people misinterpret and misunderstand.
Unit Tests avoid this problem. Using Unit Tests as a substitute for design documentation allows you to take advantage of the DIRE principle. As documentation Unit Tests are directly connected to the code they are designed to document and test.

3. Favor customer collaboration over contract negotiation.


This is similar to rule 1. It is about getting developers to communicate and work with the users. As I said above, contract negotiation is symptomatic of playing safe.


4. Favor responding to change over following a plan.


Responding to change is at the heart of Agile development. Most Agile principles and techniques use DIRE to quickly and more easily respond to change:
  • JIT-design brings the design process and corresponding coding together
  • continuous integration is about bringing different modules together early to detect problems earlier
  • TDD brings coding and corresponding testing together to find bugs faster
  • getting developers and users/customer to work more closely together allows quick response to change
  • giving users working software earlier finds problems faster

Disadvantages of DIRE


There are no inherent disadvantages of DIRE that occur to me. The main problem is that it may be a matter of opinion the best way to split up a task or problem, such as the design of some software.

For a large problem, we necessarily have to split it up to reduce complexity. Using divide and conquer we can usually isolate and solve sub-problems one at a time until the whole problem is solved. This is the most fundamental principle in software design, and in most technical problems.  See my first post.

As an example, consider the Waterfall development methodology, which splits the process of creating software into phases (design, coding, etc). There are obvious advantages into doing the design of all the parts of the software at the same time, then all the coding together, etc. Doing all the testing at the end should avoid a lot of regression testing. However, experience (and awareness of DIRE) tells us that isolating the design of particular part of the software from the coding, and isolating the coding from the testing causes many problems.


Waterfall Methodology
The Agile approach is to divide the problem up in a very different way. First, quickly get a minimal working version of the software, then add to it over time. I have previously discussed the advantages of this evolutionary approach, ad nauseum, so I won't repeat them here.

Agile Methodology
The disadvantage of the Agile approach is the separation of related activities that occurs.

It's easier to design all the features at the same time since when you go back to design the next part you may have forgotten a lot of the original design. Similarly, most programmers will tell you that it is faster to implement Features 1a and 1b (of the above diagram) together. Finally, there will be a great deal of testing rework, since testing Feature 1b will probably entail retesting feature 1a and the core system again. 


Unit Testing

Fortunately Agile techniques, particularly Unit Testing, come to the rescue in alleviating all these disadvantages:

  • Unit Tests act as design documentation making it easy to do design and re-design at any time
  • Unit Tests make it is easy to add new features without fear of breaking existing behaviour
  • Much testing is automated with Unit Tests which means the re-testing is automated

Summary


DIRE is a simple principle that helps to create better software. It may seem obvious but keeping it in mind allows you to find better approaches to many problems. For example, DIRE is behind most of the principles of Agile.

Using a divide and conquer approach is natural and essential to most tasks and problems, but sometimes how something is divided may cause undesirable consequences. However, by careful application of the DIRE principle it is often possible to reduce the "isolating" effects of the division. Further, looking out for things that should not be separated may allow you to see a different way to do the division. A good example of this is the Layer Anti-Pattern.

Sunday 6 April 2014

JIT

I first encountered JIT (short for Just In Time) when doing a course on SQA in 1993. JIT is an idea that originated in the Quality Assurance movement in Japan, in the area of inventory control in manufacturing (Toyota?). JIT was an idea that went against conventional thinking, which was to ensure you always had more than enough inventory (eg, of inputs into a manufacturing process). JIT took the opposite approach which was to have as little inventory as possible (or none at all) by relying on product arriving when needed (ie, just in time).
DIRE   

JIT is an example of the broader principle of DIRE which I will discuss one day soon.

The obvious advantage is that inventory costs are reduced. But there are other, not so obvious ones so let's list all those that I can think of:
  1. Reduced storage costs for inventory
  2. Greater visibility in the supply chain - this allows problems to be quickly found and fixed - see Visibility (below)
  3. Parts are less likely to have deteriorated (rusty, stale, etc) - see Cost of Delay
  4. Parts are less likely to be obsolete when needed - see Cost of Rework
In the software industry the first advantage is not at all important - a little bit of disk space costs a lot less than a warehouse full of car parts. However, the others are significant so I will give some examples.

Visibility

Traditionally software is created by building low-level components then gradually assembling them into more and more grandiose structures. Though design is usually done top-down, implementing the design is done bottom-up.

There are several problems with this approach. The main one, in this context, is that the creation of a low-level component will be isolated from its first use. Moreover, the initial use is unlikely to exercise all the features of the module. [Another problem is that the low-level module is typically over-designed and far more complex than is necessary for its intended purpose - see Reusability Futility.]

In other words a module is not even tried in a realistic environment until some time later, typically never fully used as intended. Eventually it's used and realized that it is too slow, has many bugs, or just doesn't do what's required. By then it is much harder to fix. In some cases it is easier just to start again.

A better way is to build everything at once. How can you possibly do that? It is usually fairly easy to build the minimal piece of software that actually works, without modules and just a basic design. Later, we can split parts off into modules and evolve the design. If it is truly minimal then every bit of the code will be used to satisfy the initial acceptance test. That way you know there are no surprises lurking in some piece of code that has not been exercised yet.

Cost of Delay

In manufacturing the JIT approach avoids the costs associated with parts deteriorating over time - what I call a "cost of delay". In software development the main cost of delay is due to forgetting how things work. As we saw above, introducing a delay between two "tightly-coupled" tasks interrupts one's train of thought.

This is a major problem in debugging. It's much easier to fix a bug if it's discovered while you are still working on the code. Fixing bugs months, weeks or even days later is tedious, time-consuming and error-prone. Further, you might have to do thorough regression testing to ensure that the change does not introduce new bugs - which is a cost of rework (see below). Also see (JIT Testing) for more on why it is better to test ASAP.

Cost of Rework

The design of manufactured goods changes occasionally to address design flaws, for reasons of fashion, and even to add enhancements. For example, if it is found that two parts do not fit together reliably then a change might be made to the way they are joined, such as adding a longer thread to joining two pipes. If the manufacturer has thousands of one of the parts in their warehouse then these will have to be thrown away and remade to the new specification.

In software development the problem of rework is much worse because things tend to change often. If you do something before it's needed you will probably have to redo it. I will now use the example of designing software (but the problem of rework can be found in other areas of software development such as the need for regression testing).
the "best" solution
to the problem
evolves
as developers
have more time
to consider alternatives

One of the most serious problems with the Waterfall methodology is the separation of the design and coding phases. Usually when it comes to implement the design it is no longer even close to what is really required. There are many reasons for this which I discussed in detail last October (see Change). In brief, the problem to be solved (ie, what is required) evolves over time as the customer gains more insight. Furthermore, the "best" solution to the problem evolves as developers have more time to consider alternatives.

Proponents of the Waterfall model say that the design is not cast in stone. Even so, the problem is that to maintain the design document according to the current best solution requires a continuous rework. Moreover, it is human nature for the designers not to want to throw away all or part of their original design upon which they labored so intently.

So with a large up-front design the alternatives are:
  1. put a lot of effort into maintaining the design which requires to a lot of rework
  2. put up with a bad design which causes major problems as the code is modified
Generally, in traditional waterfall projects the second option is used, not by any conscious choice, but because it is easier. This means that the design is obsolete before the code is written. In fact parts of the design may be obsolete before the design is finished. In Agile development this is addressed by postponing design until it is required - sometimes called JIT-Design.

Summary

The advantage of JIT is in bringing together activities that are related. In the manufacturing industry (where it was invented) this means bringing the activity of producing a product closer to the activity of producing its component parts.

In software development it is similar but it can be applied in more ways. One of the most important is taking the Agile approach of not having a separate design phase followed by a coding phase but instead using JIT-Design. A similar one is to test code as soon as possible, of which TDD is the extreme example.

JIT is one example of a more general principle that I have called DIRE (Don't Isolate Related Events or Entities). I will discuss DIRE in detail next week.