Entries from January 2008 ↓

In my last post on this topic I talked about the concept of an ESB. Here I talk about why you would want one.

There are plenty of whitepapers, analyst reports and vendor statements about the features and functions of the various ESB products. In my experience, the key advantages of using an ESB are less about features and functions and more about how you use it.

Standardization

One of the primary advantages of an ESB is that it gives you a standardized platform for integration. When everyone is using the same tools you can develop enterprise-wide frameworks, patterns and best practices for building re-usable services. Without a unifying platform, you get a divergence of integration methods which leads to inconsistency and higher cost of management and change. So an ESB platform helps with design-time governance. Note that this is not the same as standardization in the sense of using web-services standards. The important thing is that you use the ESB to support your own enterprise standards. These may be based on external standards – but that may be of secondary importance.

Loose Coupling

The bus architecture of an ESB encourages you toward a loosely coupled architecture.

• Physically decoupled by making use of message passing mechanisms (e.g. JMS) versus direct network connections (e.g. HTTP).
• Semantically decoupled by use of message transformation so that services are exposed in a system-neutral format which reduces application lock-in and reduces the cost of change.

Scalability and Reliability

Physical loose-coupling provides scalability advantages such as high-availability, fault-tolerance and load balancing. The messaging layer in the ESB directs messages between service endpoints to the appropriate instance of the endpoint. For example, in the event of a service provider failure, messages will be redirected to a backup provider – thus supporting high availability. In the case of load balancing, messages are distributed between redundant providers (or consumers) to handle high volumes of message traffic. You could say that physical loose-coupling supports change at the “micro” level where short term changes in the system topology can be compensated for via real-time message redirection.

Routing and mediation

Message routing supports scalability and fault tolerance. An ESB can also be used to support business-level routing and mediation. For example content-based routing allows services to be invoked based on the content of a service request. A business example would be routing of a customer enquiry to the branch where that customer account is located. A technical example would be the routing of a service request based on the version of the service being invoked.

Complex message exchange patterns

Traditional HTTP-based services support only one-to-one request-reply MEPs. An ESB supports more complex MEPs such as asynchronous one-way messaging and to multiple subscribers using topic-based messaging. Asynchronous publish and subscribe mechanisms support new ways of intermediating service consumers and subscribers – such as auditing, service monitoring – which are extremely useful for runtime management and governance of your services. Beyond mere governance, higher level business functions such as complex event processing (CEP) and business activity monitoring (BAM) are supported by this ability to “listen in” to service traffic on the ESB.

The benefits of an ESB that I’ve described above stem largely from the architecture of an ESB and in particular from the use of a message bus as the primary underlying transport. But it is important to understand that these benefits don’t automatically come “out of the box”. Your solution architecture (and your architects) must recognise and utilise the architectural principles underlying the ESB.

This week I had to travel for a few days. When rushing out of the house that morning I forgot my mobile! On top of that my flight got canceled and I ended up hanging around the airport for a few hours – sans mobile.

Its times like these you really feel how dependent we are on these gadgets. The main problem is that people have an expectation that they can get you instantly on the phone. This is extending now into the email arena where the expectation is that you are instantly addressable via email on your blackberry.

Fortunately, my laptop, wi-fi and web-based services came to the rescue (somewhat). My phone carrier allows me to set call diversions from their web site. So I could divert my number to a nearby landline. For outbound calls, I was able to buy a prepaid calling card online. An access number and PIN was emailed to me and voila any nearby landline becomes a gateway to the world. I had effectively provisioned a telephone account over the web!

I suppose the really smart thing would have been to use Skype, but my laptop lacks a microphone…

A question I often get is “what is an ESB and why do I need one”? This question is motivated by a number of concerns; non-technical people have heard the term but don’t understand the concept, semi-technical people are trying to figure out conflicting vendor definitions, and technical people are confused by the debate between different service enablement approachs – RESTful versus ws-* versus middleware-supported hybrids.

The Elevator Pitch

A service bus provides a uniform and consistent platform to allow service providers and service consumers to interoperate. An ESB provides benefits such as:

• standardization
• loose coupling
• resilience and high availability
• monitoring and intermediation

Hardware

I’m not sure of the provenance of the word “bus” as it is applied in the technical domain (I’m sure there is some interesting etymology there) but you can confidently trace it back to the concept of a computer hardware bus. The idea of a hardware bus (or backplane) is that hardware components – such as sound-cards, video cards, floating point accelerators, tape-drives, barcode scanners etc – can all slot into and interoperate through a shared infrastructure. By supporting a standard hardware interface and a standard software protocol, the hardware bus abstracts the details of each individual hardware component. The key features of the harwdare bus are:

• standardized hardware connectivity to the backplane
• standardized software protocol between each component and the backplane
• hardware components can operate independently without having to know details about each other
• a single infrastructure replaces multiple point-to-point connections between components (i.e. does away with a lot of ad hoc soldering).

Software

Networked systems arrived in the seventies and grew out of control in the eighties. Early network infrastructures such as Unix sockets were hard-wired point-to-point affairs with little or no abstraction of the the two programs that were working together.

The idea of a software bus is that software components can work together – yet independently – via a standardized message passing mechanism that would abstract away the need to create individual network connections between components. The software bus would take care of routing messages to the required location and also take care of all that hard stuff like quality-of-service, reliability and scalability. This is equivalent to standardizing the “hardware connectivity” in the hardware bus. TIBCO’s predecessor – Teknekron – articulated the concept of the software bus in the early nineties

The Service Bus

So the hardware bus standardizes hardware connectivity and the software bus standardizes software connectivity. The Service Bus has refined the concept of the software bus by taking a more service-oriented approach and adding support for the XML stack underlying web services and transport connectivity (e.g. bridging HTTP to JMS).

So why do you need an ESB? More on that anon…

Yesterday I noted the weak case-studies at SOA Consortium. Here is a great post from Richard Veryard about the poor quality of SOA Case Studies in general.

Richard is from CBDI Forum which I can say from experience is one of the most comprehensive and independent sources of SOA best practices and information. Perhaps CBDI Forum can add an SOA case study to it’s journal?

An important force helping to push SOA up the slope of enlightenment is the sharing of knowledge, experience and validation from SOA “doers” rather than marketers. The SOA Consortium is one body set up by the OMG to try to help this along (although caveat emptor the consortium is sponsored by a bunch of vendors).

I read through the SOA Executive Insight Report which reports on the discussion between a number of CIOs who all apparently have extensive SOA experience. There are a couple of interesting comments of note.

The CIOs…

“expressed concern regarding the current industry focus on wire protocols and products, rather than business value generation and the necessary business and IT changes for sustainable SOA success. The industry – vendors, press and practitioners – must “elevate out of the technical weeds” in order to engage the business on SOA.”

This was written in April 2007, but is still so true today. In particular the arguments of ws-* versus REST are not about SOA and should not be confused with the viability or otherwise of service oriented approaches to providing functionality and value to IT users.

The second interesting comment is:

“CIOs noted the lack of common practices for

• service versioning,
• the complexity of testing shared services with multiple consumers,
• the challenge of identifying mission critical services,
• the reinvention of chargeback procedures,
• blind spots in capacity planning, and
• the need for ‘real-time releases’.

The CIOs want best practices for these issues. Specifically, they mentioned an ‘ITIL for SOA’.”

These are issues that I see creating many challenges for SOA adopters and there are no simple solutions. These are also issues I have run into in my own SOA work and I hope to write about them in the future.

One other comment on the SOA Consortium web site. There are a number of SOA “case studies” which is great to see, but very disappointing once you drill into them. They lack any real “meat.” We need more than just a few bullet points…details are needed on what challenges they faced and how they overcame them. As they stand the case studies look like a selection of vendor power-points.