Whither CEP?

An interesting post by Colin Clark lamenting the inability of CEP to live up to earlier growth expectations. The article is definitely worth reading in full, but if I can pull out a few cogent points, I believe Colin ascribes lack of CEP growth to:

  • CEP is mostly a marketing phrase.
  • Vendors have focussed tightly on High Frequency Trading and neglected other areas of opportunity.
  • Vendors have diss’ed developers by forcing them to learn new and arcane language(s).
  • Vendors have neglected business users by neglecting visualization requirements.

In broad terms I agree – although I’m not sure languages are an impediment given the explosion of new language interests in the mainstream. I think the fundamental problems are two-fold:

CEP products haven’t yet crossed the chasm from toolbox to platform. They are still very technical and incomplete. Most CEP products concentrate on the “engine” and neglect two really important areas – visualization (as pointed out by Colin) and context. Event processing relies on understanding events within the broader business context which requires no barriers between the event stream and other systems of record or operational data stores. This is an extremely challenging technical problem – how to marry real-time data streams with large volumes of data “at rest”.

The business value for CEP is not always obvious. Unless you’re involved in a high stakes, low latency arms race moving at ever more relativistic velocities, batch will usually work for you. Most organizations don’t yet operate in realtime. Those outside of HFT that do or plan to operate in real-time are doing some work with CEP (e.g. Telcos, Utilities and Logistics) but there the challenges are around my first point – integrating existing/legacy network optimization applications with the event stream. In such situations, it’s the optimization technology that drives the implementation, not the event processing technology.

So whither CEP?

Ultimately CEP has three pre-requisites: the business need to operate in real-time, the IT infrastructure to support this and the ability to analyse events within the context of all relevant data assets. The CEP “product” comes at the end of a long line of dependencies.

A Crap Customer Experience

Customer self service is usually just secret code for pushing cost and effort onto the customer, but sometimes it can be “win-win” where the provider saves money and the customer avoids dealing with the dreaded call centre. But I recently had an experience where poor integration leads to a “lose-lose” situation.

Qantas airways offers codeshare flights with its budget subsidiary Jetstar. The booking is with Qantas, the flight number is a Qantas flight number, but the plane is operated by Jetstar. Sometimes these can’t be avoided and I recently had to take this option.

Overall I like online web check-in. It seems to save me time and gives some autonomy over seat selection. When I try this with my codeshare flight, I naturally go to the Qantas website to check-in. The “manage my booking” page offers me the usual facilities, except the “check-in now” button is very hard to find. There is no indication that I’m in the wrong place, perhaps I’m just not looking hard enough, or I’m just dumb.

A call to the Qantas helpdesk confirmed that indeed I need to check-in on the Jetstar website. Thanks for telling me. But – get this – I have to use a special booking reference number that the operator gives me over the phone. The reference number supplied on my Qantas ticket won’t work.

Ok, so over to the Jetstar website and nothing works! I can’t even get in, but apart from blinking at me when I hit “enter” there is no indication of the problem. I call the Jetstar helpdesk (yes, I’m stubborn) and am told to enter my surname in uppercase. Apparently when Qantas makes the Jetstar booking my surname is entered in uppercase and the surname field is case sensitive (why?).

The integration fiasco is now clearer. It seems that when I make a Qantas booking on a codeshare flight, Qantas makes a “proxy” booking in their system and a real booking in the Jetstar system. But all the details given to me refer to the fake booking, not to the real booking. I’m not even aware there are two bookings until I persist with online check-in (which most wouldn’t).

When faced with an integration problem most people take one of two approaches: either make the user experience seamless or allow the stitches to show, but give users the tools to navigate the business process. Qantas/Jetstar just ignores the whole problem and leaves its customers dangling. The result is wasted time and frustration for anyone wanting to check-in early, plus additional cost to Qantas/Jetstar in helpdesk calls.

The solution to this is mind-bogglingly trivial. The “manage my booking” web page could offer me the usual “check-in now” button hyperlinked to the Jetstar site and containing the Jetstar booking reference as a parameter. Hell, they could even uppercase my surname on the way!

Web 2.0 Discovers Integration 2.0

My last entry was about Gnip and I take this subject up again following on Dare Obasanjo’s blog entry on the same subject where Dare grapples with the problem of how to share social activity streams across the web. The fundamental problem is not new to anyone dealing with the problems of integration inside the firewall, a plethora of different apis and protocols leads to spaghetti architecture. What we seem to have now in the social web is JBORS – Just a Bunch of RESTful Services.

This is only a problem if people want these services to collaborate, but there certainly seems to be sufficient desire and opportunity to make this happen. I also think that the social web – chock full of personal and dynamic information, collaboratively maintained by an army of “ordinary people” is a good testbed for the techniques which may be applicable in other applications such as Electronic Health Records.

The Gnip approach to the challenge is to provide a type of ESB in the cloud. The basic functions of an ESB – transport and mediation – is applied to the various protocols which are exported or imported by popular social applications.

Dare worries that this approach puts Gnip in the middle of the social network as a “single point of failure”. This is the classic problem with the ESB and the source of much derision about “ESB in a box”. You’re not solving the problem of protocol/api/schema proliferation, you’re simply pushing the mappings into a central logical component which – depending on its architecture – may be a single point of failure. So what are the alternatives?

One approach would be to standardize the interfaces so as to minimise or completely eliminate the need for mediation. One might refer to this as a service-oriented architecture, but I hesitate to use such an unfashionable term. But there’s the rub. The social web is a dynamic, anarchic frontier on the bleeding edge of information technology. What effect would standardization have on that eco-system? Would it slow down or block off avenues of innovation? Would new business models be choked off, trapped in a proprietary cul de sac? Maybe! These are the dangers of premature standardization which are best applied to mature technologies and processes.

Even if you could standardize to the required level of completeness, how do you coordinate dozens of different companies to support the standards? It’s pretty much impossible. We’ve struggled with this inside the firewall for decades. For example how do you get Siebel and Metasolv and Retek and Primavera all standardizing around the same set of services? Well in this case, they all get acquired by Oracle and it becomes Oracle’s problem.

The first step in the standardization of the social web has been taken by Google with OpenSocial. If standardization of the social web ever happens I think it will be a long time before the requirement for mediation disappears. In the meantime we have Gnip, and Friendfeed and…watch this space.

Thylacines and Wolves

If you look closely – right now – as we speak – a new ecological niche is opening up in the web.

It is just over eight years since we first saw the read-write web and a little more than three years since we heard about web 2.0. In that time, everyone has been gradually building up more and more valuable assets in the web: emails, photos, blogs, collaborations, videos, social networks. For some of us, a large chunk of our lives now has an independent existence in the web.

But something about this has started to become problematic. Our social assets are splattered across dozens of different sites and platforms. Multiple social networking sites vie for our attention. The result is increasing fragmentation of information and its associated problems – duplication and inconsistency. The world-wide-web has rediscovered that old enterprise bogey-man – integration! (or the lack thereof).

Some recent examples include John Udell asking where is SOA when you really need it, and Loic Le Meur lamenting the fragmentation of his social map.

At the same time a raft of new applications is attempting to address these issues:

  • Google Open Social aims for social network interoperability.
  • OpenID has now achieved broad support (if not success) as a way of managing distributed identity and authentication.
  • Ping.fm unifies message posting while FriendFeed aggregates the receiving side.

The key thing about these initiatives is that they all start at the edge of the integration problem. They attempt to support interoperability by unifying the interfaces to these web 2.0 platforms.

The new player that caught my attention recently represents the genesis of “web middleware” in the form of Gnip which bridges the “air gap” between the Producers and Consumers of the social web. And in a beautiful example of parallel evolution, Gnip makes use of wholly web protocols such as XMPP and Atom to provide the functions which are familiar inside the enterprise as JMS and SOAP. Gnip provides connectivity, message delivery and mediation between different data formats. Pinch me if that doesn’t sound just a little like an ESB. But it lives in and has evolved entirely from the web! This is the IT equivalent of discovering the Thylacine in the new world as an evolutionary parallel to the Wolf in the old world.

The funny thing is that while some middleware vendors are trying to figure out how to colonise the cloud (e.g. here and here), the natives are already evolving into that niche.

Hand me that ESB to your left….

In 2002 Sonic Software invented the term ESB to differentiate their middleware from everyone else’s middleware. This is an old marketing tactic when you can’t differentiate your product from the others – just invent a new category. To lend credence, you need the collusion of an analyst and Gartner was there to knock this marketechture into the hype-stratosphere. We can also thank Sonic for the crap notion that an “ESB is a prebuilt SOA“.

ESBs have been back in the news again with the usual attendant vitriol and this has revived the perpetual question – “what is an ESB and why do I need one“?

I’ve seen ESBs referred to in two rather broad contexts. The first context refers to a distributed infrastructure that allows your Service Providers and Service Consumers to find each other and work together in a scalable and loosely coupled manner. This is often referred to as the “ESB as design pattern” and I think it is what many people would like ESB to mean. This “Service Bus” is effectively an extension of the “Message Bus” concept and is what I had in mind when I wrote about this subject a while ago.

The second context is ESB as a tool or a “thing you buy from a vendor” and this seems to be the terminology that calls forth all manner of vituperations. In this context ESB is secret marketing code for what used to be called “integration middleware.” This is also the thing that Jim Webber calls the “Erroneous Spaghetti Box.”

In this context, the eternal ESB question becomes – “in an SOA world, do I really need integration middleware?” I really think you can make more sense of the ESB debate by recasting into this terminology. The SOA pragmatists (including myself) see that there is a role for “integration middleware” as tools within your SOA infrastructure.

Eric Roch thinks of an ESB as a “productivity tool for integration” and points out that The legacy integration problem hasn’t gone away yet.

ZapThink say that an ESB could play the role of a Service Intermediary where transformations and content-based routing are essential requirements.

Todd Biske characterises five different types of ESB product each supporting different functions such as legacy integration, orchestration, mediation and transport, SOAP wrapping and gateways. The key insight here is how the label “ESB” has come to be applied to so many disparate products. “ESB” has come to mean everything and nothing and should be banished from our vocabulary.

So I agree with the pragmatists – design your SOA infrastructure to support your business requirements. Consider the so-called ESB products as tools and if you need mediation, or orchestration or transformation or integration to legacy systems then consider using the scads of tools – both proprietary and opensource – out there that support this stuff. And whatever you do, don’t try to author XSLT using Notepad.