An interesting post by Colin Clark lamenting the inability of CEP to live up to earlier growth expectations. The article is definitely worth reading in full, but if I can pull out a few cogent points, I believe Colin ascribes lack of CEP growth to:

• CEP is mostly a marketing phrase.
• Vendors have focussed tightly on High Frequency Trading and neglected other areas of opportunity.
• Vendors have diss’ed developers by forcing them to learn new and arcane language(s).
• Vendors have neglected business users by neglecting visualization requirements.

In broad terms I agree – although I’m not sure languages are an impediment given the explosion of new language interests in the mainstream. I think the fundamental problems are two-fold:

CEP products haven’t yet crossed the chasm from toolbox to platform. They are still very technical and incomplete. Most CEP products concentrate on the “engine” and neglect two really important areas – visualization (as pointed out by Colin) and context. Event processing relies on understanding events within the broader business context which requires no barriers between the event stream and other systems of record or operational data stores. This is an extremely challenging technical problem – how to marry real-time data streams with large volumes of data “at rest”.

The business value for CEP is not always obvious. Unless you’re involved in a high stakes, low latency arms race moving at ever more relativistic velocities, batch will usually work for you. Most organizations don’t yet operate in realtime. Those outside of HFT that do or plan to operate in real-time are doing some work with CEP (e.g. Telcos, Utilities and Logistics) but there the challenges are around my first point – integrating existing/legacy network optimization applications with the event stream. In such situations, it’s the optimization technology that drives the implementation, not the event processing technology.

So whither CEP?

Ultimately CEP has three pre-requisites: the business need to operate in real-time, the IT infrastructure to support this and the ability to analyse events within the context of all relevant data assets. The CEP “product” comes at the end of a long line of dependencies.

Happy New Year! Asynchronicity is busting out all over the web and my prediction is that 2010 will be the year of “events”:

• Of course Twitter has brought The concept of publish/subscribe messaging to the masses and we enjoyed their journey of discovery to the heights of scalability in 2009.
• XMPP has been embraced by the real-time web crowd, most publicly in Google Wave but also in other “back-web” contexts such as Gnip.
• Web sockets is an experimental feature of HTML 5 which enables push messages directly to web pages.
• New frameworks for event-driven programming are emerging such as EventMachine, Twisted, Node.js.
• In 2009 every major software vendor had a CEP product.

In the meantime, SOA has become so damn synchronous. But it doesn’t have to be!

One of the fundamental tennets of SOA is that reducing coupling between systems makes them more scalable, reliable and agile (easier to change). SOA goes a long way to reducing coupling by providing a contract-based, platform independent mechanism for service providers and consumers to cooperate. However I still think we can improve on current SOA practices in further reducing coupling.

Coupling still intrudes into many aspects of how SOA is practiced today:

• HTTP transports tie us to a regimen of synchronous request-reply with timeouts which creates tight couplings between provider and consumer. Even though one-way MEPs were an original feature of SOAP, message-oriented transports remain the forgotten orphan of web-services standards.
• Many SOA services are conceived, implemented and maintained as point-to-point entities…providers and consumers forced into lock-step due to inadequate versioning and lifecycle management.
• Process orchestration layers often form a bridge between service providers and consumers, which on the face of it provides some level of indirection. But in many cases orchestration provides limited value and may actually serve to increase the overall system coupling.

In many cases we can achieve the benefits of service orientation to much greater effect by exercising a little scepticism toward some of these shibboleths of the web services world and embracing a more asynchronous, event-oriented way of building processes. So this year, embrace your asynchronous side and do something to reduce your system coupling: build some pub/sub services, learn about Event Processing or Event-Driven Architecture, try one of the technologies I pointed to above.

Just as developers should embrace multiple languages to broaden their skills, so should architects embrace and be fluent in multiple architectural styles.

In my opinion, coupling is the most fundamental attribute of a system architecture and tight coupling is probably the most common architectural problem I see in distributed systems. The manner in which system components interact can be a chief determinant of the scalability and reliability of the final system.

So I really like Ian Robinson’s post on Temporal and Behavioural Coupling where he uses two coupling dimensions and the inevitable magic quadrant to classify systems based on their degree of temporal and behavioural coupling.

See Ian’s post for the slick professional graphics, but to summarise – event-oriented systems with low coupling  occupy the “virtuous” third quadrant of the matrix. Conversely the brittle “3-tier” applications that many of us struggle with, occupy the “evil” first quadrant where coupling in both dimensions is high.

However I’m a little miffed to see no mention of my favourite “document-oriented message” in Ian’s diagram. As Bill Poole writes; document messages have lower behavioural coupling than command messages, but more than event messages. So would you put document-oriented messages near the middle top of the matrix between command-oriented and event-oriented messages? Unfortunately that would break the symmetry. But it also highlights another problem.

Any type of message – document, command or event-oriented could temporally be tightly or loosely coupled. Temporal coupling is more a property of the message transport than of the message type. So I suggest that the two coupling dimensions are characterised as follows:

• Temporal coupling – characterised by message transport from RPC (tight coupling) through to MOM (loose coupling).
• Behavioural coupling – characterised by the message type from event-oriented (tight) through document-oriented to event-oriented (loose).

It so happens that distributed 3-tier systems generally employ both command-oriented messages and RPC transports – hence making them inherently “evil”. Whereas events (being asynchronous)  are naturally virtuous by typically being carried over MOM transports (it’s difficult to request an event notification).

Between heaven and hell, it is in the murky mortal realms of SOA where we need to be constantly mindful of the interactions between message type and transport – lest our system ends up in limbo.

I will be manning the booth tomorrow afternoon at the TIBCO SOA Online Summit. The topic of discussion is Events and Realtime Business Intelligence which is an active part of my portfolio these days. Two of the guys from my blogroll will be presenting – James Taylor from Smart (Enough) Systems and Paul Vincent from TIBCO. Please come along and say “hi”.