Following from my last entry on metrics I do think it is important for metrics to be used to monitor and demonstrate continuous improvement as part of an SOA initiative (or any IT initiative, for that matter).
Key metrics should be built into your SOA organizational infrastructure so that they are maintained and reported as a by-product of your day-to-day SOA activities. So even though I don’t believe there is (currently) any absolute measure of SOA success, what is important is the relative measure and how that changes over time – “am I getting better at this?”
So what are some good metrics? ROI is the most fundamental metric. The reason we embark on an SOA and the reason that the business pays for it is to improve the bottom line – make IT more cost effective and more responsive to business needs. ROI is important to your CEO, your CFO, your CIO and your shareholders. The problem with ROI is that it can be difficult to measure…especially when dealing with less tangible measures such as opportunity cost around time-to-market (agility). What we need are some good proxies for ROI which can be easily measured on a day-to-day basis.
Cost to build a new service is a good metric. If services are provisioned as a project, then it should be relatively easy to measure the total cost of delivering a new service to the business. You should normalise this metric against the complexity of a service. Split services into simple, intermediate and complex if that makes sense. The total cost per new service should decrease over time.
Elapsed time to build a new service is also a good metric to monitor agility. This is a simple form of Nick Malik’s agility metric. Basically we want to ensure that we are delivering new services faster by leveraging off our SOA infrastructure. One of the commonest complaints I hear from businesses everywhere is that IT takes too long to deliver. As with the cost metric, you probably want to normalise against the complexity of a service in some way.
Service utilization is another good measure. Once a service is provisioned, how many business processes make use of that service? What is the daily throughput for the service? These can be objectively measured by service monitoring tools. But note that counting business processes requires some thought to meta-data in your service infrastructure.
Services with a high utilization will indicate what is of most value to your organization. Conversely, services with a low utilization may indicate a low value service or a function which shouldn’t have been made into a service in the first place. Perhaps your governance processes around service identification and prioritization could be improved here. Alternatively, services that used to have high utilization and now have low utilization may indicate obsolescence where it makes sense to retire that service.
Average cost per service, Delivery time for a Service and Service Utilization are easily measured, fundamental metrics for your SOA. There are other SOA costs that should also be accounted for.
Average cost to run a service is important. Add up the BAU costs (hardware, software, maintenance, monitoring, administration, power consumption etc) of your services infrastructure and divide by the number of services you are running. This measure will often be high when you start out and have few services, but should reduce over time.
Governance cost should also be tracked. What is the cost of the governance processes that you put in place to evaluate service candidates, prioritize and plan your services road-map. Usually this cost is made up of time and effort for the people involved in SOA governance. The governance cost per service will initially be high, but should hopefully reduce over time. [This raises an interesting question…how do you measure the effectiveness of governance in order to justify the additional cost? It is difficult to separate out the contribution of governance to the overall SOA benefits.]
These measures can be combined to give an overall ROI index for your SOA. If you can estimate the total business value of your processes and divide by the total cost of the services that support those processes, then you get an aggregate ROI measure
Finally a warning to be careful about how you define a service for inclusion into these metrics. True re-usable services come with a higher price-tag in terms of planning, governance and infrastructure, but the idea is that re-use amortizes the cost over multiple business processes ending up with a positive ROI for the service. There are probably many integration points/projects which do not justify treatment as a re-usable service. These “point integrations” should not be included in the SOA metrics because you are not comparing like for like.
