Software Architecture Notes

A Return On Investment Model for Consultant-Written Software Projects

Here is a simplified picture of a "value cycle" for a software project done by a consulting company for a client. How the project is set up can theoretically alter the return by a factor of as much as 300X (that's TIMES, not %). Practically, lower leverages are achieved (more in the range of 30 - 100X).

This is the picure on the day that the software is delivered and put into production:

What is this trying to illustrate?

The return on a project is the difference in the net present value (NPV) of the benefits of the delivered software (increased revenue, decreased costs) and the costs of the project (direct project costs and on-going costs of ownership). When the project is delivered the cost of ownership is unknown. Cost of ownership is usually ignored in Return On Investment (ROI) calculations. But, the cost of ownership will end up being half of the total cost of the project (See Set up reference to Glass here We will return to this later.

Rates Leverage

The arrow from direct project costs to effort illustrates that the thing that the client's direct dollars buy is staff hours. Really the only way a client can get a better deal is by getting a better rate; that is, more effort per dollar spent. If that is the focus, a consulting company with distributed or off-shore capabilities can change this by a factor of 4X.

Productivity Leverage

The line from Effort to Countable Deliverables illustrates that the point of excerting effort is to produce some sort of tangible deliverable : working code, reports, screens, tests, help systems or whatever. The general term for the rate at which effort is turned into deliverables is productivity. There are three things that are known to affect this form of productivity:

• Skill level : The best developers are more than twice as good as average developers, and an order of magnitude better than the worst. Careful hiring can raise the raw talent level of the developer pool by a factor of 2X
• Schedule compression and team size : For a given project scope, the minimum time schedule costs twice as much as a schedule that takes 20% longer. A further 20% increase in schedule reduces costs by an additional factor of two. Put in reference here Ever since The Mythical Man-Month we have known that the trade-off between people and months is not linear. But very few project planning trade-off discussions take full advantage of this. From a raw productivity standpoint, just extending the schedule and shrinking the team can result in a 4X improvement in what is delivered to the client.
• Software Practices : Improved practices are worth an additional potential 2X productivity improvement.

If the stars could all be aligned correctly (small, highly skilled team with reduced schedule pressure and experience with development best practices) the maximum productivity leverage is 16X

Functionality Leverage

The line from Countable Deliverables to Business Functionality illustrates that the goal of the project is to deliver business value, not just raw poundage of technical artifacts. Here is another fun fact to know and tell, courtesy of Ron Glass : reference here adding 25% to the complexity of a function increases the cost by 100%. This is why Agile and eXtreme Programming practices like You Aren't Going To Need It (YAGNI) and constant access to an on-site customer are so important. Defering complexity that does not pay its own way increases the rate that deliverables are converted into usable business functionality (aka "done stories".) Its too early to know what the actual impact of this is in the field, but assume that this results in another leverage factor of 2X.

Value Leverage

All business functionality is not created equal. The least important third of the functionality does not contribute a third of the value to the business. For rough numbers, the most important two thirds of the functionality delivers 90% of the value. Due to non-linearities of scale (smaller things are easier and cheaper to build than large things), it probably only costs half as much to build the most important two thirds of the system. Release planning allows the client to get at least another 1.5X leverage by focusing on the more important stuff first, and stopping the project when it gets to things of lesser value. This also counts on incremental delivery so that the client can judge for themselves when the project has delivered value.

Time to Value Leverage

Net Present Value calculations are boring. They compute what the value of a dollar next year is in terms of dollars today, based on a discounting interest rate. Yawn. Usually some sort of prevailing interest rate is used; something like the prevailing prime rate plus something. Maybe its 5% or 10%. Since a lot of projects hardly last a year this would not seem to be something to get excited about.

But wait. If you tell the client that the project is going to cost the same but be delivered two months later than originally expected, how do they react? If it was all just NPV to them, then the benefits would be out another couple of months, and the benefit of the project would be reduced by a few percent. Nothing to get excited about. Right?

Well, actually, not right. At least, not always. Sometimes people get very excited when the date slips. Sometimes meeting a date means the difference between hitting a market window or missing it. Or being ready for the Christmas processing runs, or the new tax laws, or whatever. In those cases a few months difference means a lot.

Why do traditional NPV calculations miss this urgency? Because NPV assumes that the scare resource is money. The discount rate is the cost of money. It assumes that if we need to get more money, we can get it at that price. But money is not the scare resource in a time-sensitive project. Time to delivery is.

To capture the value of time in an NPV calculation requires that we use a discount rate that is appropriate for the money value of time, rather than the time value of money. For a project that has an external time constraint, the appropriate discount rate may be several hundred percent, rather than the traditional five or ten percent.

Suppose that a project is time sensitive in this sense. Also, suppose that the project can be delivered in increments, and that value is derived from the time that the functionality is first delivered, rather than at the end of the project. Using release planning and incremental delivery, the time to value leverage factor can reasonably be assumed to be as much as an additional 2X.

Practical Limitations

The theoretical maximum leverage would be 384X = 4 * 16 * 2 * 1.5 * 2. That would consist of:

1. A project staffed at a low rate site, such as India or Eastern Europe.
2. A small, highly skilled team with reduced schedule pressure and experience with development best practices
3. Full use of Agile practices, including maximum bandwidth communication with the customer.
4. Value-based release planning and incremental delivery so that the project delivers value in order of business value.
5. Incremental delivery of functionality for a time-sensitive project

Enter Reality

This theoretical project has some important contradictions in it.

• For instance, it assumes a completely off-shore team to reduce rates, while at the same time assuming high bandwidth Agile practices like on-site customer and hiring practices that ensure best-of-breed developers.
• It also assumes a small team / relaxed schedule staffing profile while delivering a time compressed project.

So, what are realistic combined leverages for combinations that do not have built-in contradictions?

Effective Rate Leverage for Distributed (Mixed) Teams

The maximum leverage from rates is actually less than the 4X difference between American and off-shore rates implies, assuming that some on-site staff (account manager, project manager, business analyst, QA lead, one developer pair for integration and installation) are required. The actual rate leverages for project teams of different total sizes, assuming a fixed on-site presence of six, and raw rates of $100/hr for on-site and $25/hour for distributed staff are:

It seems unlikely that much above a total team size of 30 we could keep the on-site team fixed at six. So a working number for the maximum rate leverage is roughly 2.5X. This assumes that the productivity multipliers are the same in on-site and distributed staff. However, the hiring market in some distributed locations has been tight enough that the same hiring criteria cannot be applied in all locations. Assuming that it requires 1/3 more distributed resources to balance the hiring differences between locations and the combined rate and hiring leverages look like this:

Schedule Impacts

Assuming that each of the teams above gets the maximum 2X leverage for hiring and another 2X for advanced practices and tool use, the next multiplier is the 4X for using a smaller team for a longer time. This is something that is part of the client's situation, so we have to consider the two extreme cases: a maximumally time constrained projecet and a maximally budget constrained project. Keeping the team sizes of ten and thiry and varying by time sensitivity gives the following table of maximum leverlages:

Applying Remaining Factors

Assuming that the 2X for functionality leverage and 1.5X for value leverage apply equally in all cases, but that the 2X for time to value only applies in a time constrained project, the final table of maximum leverages is:

Comparing Competitors

These best leverage figures are compared to a pretty low base. Let's call them Clueless Consultants and figure out their leverage for a ten person, non-time constrained project.

With Clueless as the baseline, how does the local development staff fare?

In this comparison the in-house development staff is almost an order of magnitude more effective than Clueless. How does a consulting firm do if it as access to really good people at international rates, but staffs to maximize rates and works according to a non-Agile model?

Interestingly enough, this variation is closer to how the in-house staff performs but is somewhat less effective than the in-house staff. Suppose that the same consulting firm has started to adopt Agile practices, but due to inexperience and difficulty with the client corporate culture, only delivers about half of the benefits of Agile.

This combination is roughly competitive with, and actually is expected to be marginally better than using in-house staff for the same project. But we have left a lot of money on the table (10.2X vs 67.2X.) Suppose that we turn the benefits of Agile all the way up to their maximum value?

At this point the consulting firm is expected to be almost twice as effective as the in-house staff. This is a great deal for the client, as they are getting the functionality at 1/2 price compared to their normal costs. And the consulting firm has kept as big a team as possible billing. However, this was not a time constrained project, so the client could have gotten the project for 1/4 or 1/8 their normal price if the project had been structured to minimize costs rather than time to delivery.

But maybe that example was unfair. After all, Agile is best when time to delivery is a driver. Let's reconsider the partial and full Agile teams in the case where the project is time constrained.

With international staff to draw on and the best people and best practices, even with only partial implementation of Agile, the consulting team is able to deliver a time constrained project to the client at about half the cost that would have been required if the in-house staff had been available to execute it. Full execution of Agile brings the value up to the full 33.6X -- a bit less than 1/4 the cost of doing the project with traditional in-house staff.

Now let's compare the different thought experiments, keeping the original leverage numbers, but also comparing them against the leverage of the in-house development staff:

So, Was There a Point in Here Somewhere?

There are several actually:

• The client probably did not come to us with a project that would have fit in the current staff's regular workload. They came to us either because they did not want to hire up for a temporary peak, or because they were approaching a new technology that the existing staff did not have skills for (or both.)
• The existing staff will compare their existing steady-state productivity to the consulting team's productivity. While this comparison is unfair, most consulting firms will not be able to convincingly demonstrate to the existing development staff that the consulting developers are better than the measurements that the in-house staff have for themselves.
• It is possible for the consulting team to be both more cost-effective than the in-house team would have been if it had been forced to staff up, and more effective than the in-house staff image of itself, but this requires that everything work.
• If the consulting team cannot actually get 2x better people who are 2x with their practices and tools, it is going to be a rough go with the in-house staff, even if everything else goes well.

What are our Numbers? Really?

How Does Cost of Ownership Affect Return?

Up to now we have been considering ROI on the day that the project is delivered. On that day the cost of ownership is still at zero. What happens to the ROI calculation over the life of the project, as the second half of the cost of the project is incurred by the client?