Entries from January 2008 ↓

Natural philosophers throughout history have struggled with myriad alternative theories for how the universe works. When your view of reality is incomplete, how can you possible make reliable choices between competing theories? Often, the answer to this dilemma was to apply what is now known as Occam’s Razor – choose the alternative which explains all the facts without adding unnecessary complexities. In many branches of science, philosophy and in every day life, Occam’s Razor has become a useful heuristic for choosing between alternatives.

The Enterprise Architect is often presented with myriad alternative choices when designing solutions for the business. Occam’s Razor is a useful tool here as well – don’t make a solution more complex than it needs to be.

In the case of Enterprise Architecture I think there is also a good argument for refining the Razor to the principal of Cost versus Benefit. When faced with different alternatives for a solution, pick the one where the benefit most outweighs the cost. And you can usually eliminate a host of alternatives by getting rid of those where the benefit is not justified by the cost.

One example that often crops up is when to expose some IT functionality as a re-usable service? I’m often asked by solution designers – “should I make this a service or not?” The simple answer is to look at the cost versus the benefit. Developing a Service always come with additional cost. There is additional design time cost to ensure that the Service will support all the anticipated requirements across multiple different business processes. There is additional runtime cost as we need to wrap the Service in platform neutral technologies such as SOAP and WSDL. There is additional administrative cost in monitoring and managing the Service at an enterprise level. This additional cost may not be justified if the functionality is only ever to be used in one business process and there is no reasonable justification for re-use in the future. In order to justify the additional cost to turn a point-to-point function/integration into a re-usable service, you need to have some understanding as to the additional lifetime cost that service-enablement entails and whether the value of that service to the enterprise (usually in the form of re-use) outweighs the additional cost.

Another example where the cost vs. benefit razor can be applied is in the realm of error handling for integration solutions. A common mistake is to over-complicate integration solutions by over-automating error handling. Quite often, the simplest solution is to manually handle errors when they occur. The main effort should be expended in providing a solution which will detect errors and flag them to administrators for manual resolution. It is only worth automating recovery for those errors when you know that the benefit of automation outweighs the cost. For example, if I have an error that occurs once a month and it takes someone 5 minutes effort to remediate the error, then it is probably not worth automating that fix. Alternatively if an error occurs five times a day and it takes someone 3 hours to fix each error, then that is crying out for an automated solution. The trick here is that you often don’t know the full cost vs benefit equation for errors until the integration solution is in place and working.

All this seems obvious, but it is surprising how often simple heuristics like cost vs benefit can be overlooked – especially when design decisions can be heavily influenced by fashion.

Don’t design a solution that is more expensive than the benefit it provides. 

I recently found another set of SOA metrics on IT-Director.com. There is a gratifying degree of overlap with the metrics that I discussed recently. I particularly like the categorization into different metrics for different roles in the organization.

From his observation of innovative organisations, Smith has come up with 10 possible service delivery metrics, grouped by organisational area:

Corporate Metrics:

1. Revenue per service;
2. Service vitality index (the amount of revenue from new services over the last 12 months as a proportion of total service revenue);

Management Metrics:

3. Number of new services generated and used as a percentage of total services;
4. Mean time to Service Development (MTTSD);
5. Mean time to Service Change (MTTSC);
6. Service Availability;

Project Metrics:

7. Service reuse;
8. Cost of not using or stopping a service;

Service Development Metrics:

9. Service complexity;
10. Service quality assurance based on systems-level tests that examine the behaviour of service-oriented use cases across possible choreographies.

The original post contains some further discussion of the metrics.

I suggest that Service Reuse also has a “management” role since that is related to return on assets which is a key measure of success for your SOA initiative.

Revenue per Service is another interesting one and I’m not clear if this refers to “external revenue” or “internal revenue”. There is an argument for measuring both.

External revenue reflects the value of services to supporting overall revenues to the business. I suspect it would be difficult for most organizations to attribute external revenues to the service level. Revenue would map to a particular business process which in turn is supported by services. There would then need to be some way to allocate a contribution to each service on this basis.

Internal revenue would reflect the value of the service to internal business operations and could be counted via some metric on how often a service was used.

Following from my last entry on metrics I do think it is important for metrics to be used to monitor and demonstrate continuous improvement as part of an SOA initiative (or any IT initiative, for that matter).

Key metrics should be built into your SOA organizational infrastructure so that they are maintained and reported as a by-product of your day-to-day SOA activities. So even though I don’t believe there is (currently) any absolute measure of SOA success, what is important is the relative measure and how that changes over time – “am I getting better at this?”

So what are some good metrics? ROI is the most fundamental metric. The reason we embark on an SOA and the reason that the business pays for it is to improve the bottom line – make IT more cost effective and more responsive to business needs. ROI is important to your CEO, your CFO, your CIO and your shareholders. The problem with ROI is that it can be difficult to measure…especially when dealing with less tangible measures such as opportunity cost around time-to-market (agility). What we need are some good proxies for ROI which can be easily measured on a day-to-day basis.

Cost to build a new service is a good metric. If services are provisioned as a project, then it should be relatively easy to measure the total cost of delivering a new service to the business. You should normalise this metric against the complexity of a service. Split services into simple, intermediate and complex if that makes sense. The total cost per new service should decrease over time.

Elapsed time to build a new service is also a good metric to monitor agility. This is a simple form of Nick Malik’s agility metric. Basically we want to ensure that we are delivering new services faster by leveraging off our SOA infrastructure. One of the commonest complaints I hear from businesses everywhere is that IT takes too long to deliver. As with the cost metric, you probably want to normalise against the complexity of a service in some way.

Service utilization is another good measure. Once a service is provisioned, how many business processes make use of that service? What is the daily throughput for the service? These can be objectively measured by service monitoring tools. But note that counting business processes requires some thought to meta-data in your service infrastructure.

Services with a high utilization will indicate what is of most value to your organization. Conversely, services with a low utilization may indicate a low value service or a function which shouldn’t have been made into a service in the first place. Perhaps your governance processes around service identification and prioritization could be improved here. Alternatively, services that used to have high utilization and now have low utilization may indicate obsolescence where it makes sense to retire that service.

Average cost per service, Delivery time for a Service and Service Utilization are easily measured, fundamental metrics for your SOA. There are other SOA costs that should also be accounted for.

Average cost to run a service is important. Add up the BAU costs (hardware, software, maintenance, monitoring, administration, power consumption etc) of your services infrastructure and divide by the number of services you are running. This measure will often be high when you start out and have few services, but should reduce over time.

Governance cost should also be tracked. What is the cost of the governance processes that you put in place to evaluate service candidates, prioritize and plan your services road-map. Usually this cost is made up of time and effort for the people involved in SOA governance. The governance cost per service will initially be high, but should hopefully reduce over time. [This raises an interesting question…how do you measure the effectiveness of governance in order to justify the additional cost? It is difficult to separate out the contribution of governance to the overall SOA benefits.]

These measures can be combined to give an overall ROI index for your SOA. If you can estimate the total business value of your processes and divide by the total cost of the services that support those processes, then you get an aggregate ROI measure

Finally a warning to be careful about how you define a service for inclusion into these metrics. True re-usable services come with a higher price-tag in terms of planning, governance and infrastructure, but the idea is that re-use amortizes the cost over multiple business processes ending up with a positive ROI for the service. There are probably many integration points/projects which do not justify treatment as a re-usable service. These “point integrations” should not be included in the SOA metrics because you are not comparing like for like.

In Measuring the agility of an SOA approach Nick Malik proposes a metric to verify the “agility” claims for SOA. While I applaud Nick’s efforts to create some objective measures for the business case around SOA, I think that Nick’s proposed methodology falls down in step 2.

Nick proposes you apply the agility metric to two projects – one SOA project and one non-SOA project – and then compare the results. Hopefully the SOA project would demonstrate a higher agility measure.

The basic problem that I see is that “SOA-ness” is not a property of a project, “SOA-ness” is a property of an organization.

SOA is more about the general organizational approach to delivering projects and involves things like:

• The organizational structure that defines and delivers IT projects (including the way projects are planned and funded).
• The technical standards, frameworks and best practices that have been put in place and refined over time.
• The SOA assets (Services) that have already been delivered and are available for re-use on any given project.

If we wanted to compare metrics from two different populations to determine if SOA truly does provide benefit, then we would need to compare at the level of organizations. Find two different organizations – one that has adopted an SOA approach for IT management and one that has not. Then find which is the most agile or which has the better ROI or whatever favourite metric you have.

In practice I don’t think this is realistic. Embarking on a true SOA approach requires significant organizational change and that requires something of a “leap of faith” for the early adopters. Over time, hopefully the benefits of SOA will become apparent. Until then, I keep my eyes open for studies which can demonstrate SOA benefits and best practices – but I won’t hold my breath because as with many of these things, “your mileage may vary”.