CEP and Reflex

Imagine you are walking along a jungle trail and you see a distant orange-black furry shape (kind of cuddly) heading your way. You infer it’s coming closer because it gets larger and that growling noise gets louder. Those large yellow toothy things look sharp and…and…well you’re gone!

On the other hand, imagine you’re walking down a jungle trail and you spot a tiger in the distance. Now you’ve got a chance to escape.

The difference between these two scenarios is the processing time taken to recognize a threat and then respond to it and I frequently use this analogy to illustrate the value of Complex Event Processing (CEP) and how CEP complements traditional Business Intelligence (BI).

The current state of BI art – data warehouses, analytical tools and reporting – is akin to the first scenario. Here you identify a threat or opportunity by analyzing facts from the world around you. This is a useful activity and there is nothing inherently wrong with it. The problem arises when you rely on BI techniques to drive a quick response.

Stimulus-Response Cycle for Traditional Business Intelligence

As the figure above illustrates, the problem with BI is the lead time required to derive first a result and then a reaction. Many BI systems use a combination of batch ETL and data-marts to migrate transactional data into a data warehouse. Then most analytics tools are designed for periodic post-hoc analysis by a small cadre of specialists. This means that the lead time for recognizing a threat or opportunity can be on the order of days, weeks or even months.

We can however short circuit part of this BI cycle by taking advantage of CEP within the transactional data stream. We effectively decouple analysis from recognition and assign those functions to the most efficient component within the solution.

bicep-cycle1

The new CEP+BI lifecycle is shown in the next figure. BI techniques are still used to classify and understand the opportunities or threats, but we don’t rely on BI to drive a reaction. Instead, opportunities and threats are parameterized into a set of rules which CEP can apply to transactional data in real time.

In other words, analytics tells you that tigers are dangerous, CEP allows allows you to spot the tiger before it eats you!.

Gartner’s Top 10 Strategic Technologies for 2009

Gartner has nominated their top 10 strategic technologies for 2009. In priority order they are:

1. Virtualization
2. Business Intelligence
3. Cloud Computing
4. Green IT
5. Unified Communications
6. Social Software and Social Networking
7. Web Oriented Architecture
8. Enterprise Mashups
9. Specialized Systems
10. Servers – Beyond Blades

My primary interests in SOA and CEP are represented in items 7 and 2 respectively (with some applicability to 8).

Business Intelligence. Business Intelligence (BI), the top technology priority in Gartner’s 2008 CIO survey, can have a direct positive impact on a company’s business performance, dramatically improving its ability to accomplish its mission by making smarter decisions at every level of the business from corporate strategy to operational processes. BI is particularly strategic because it is directed toward business managers and knowledge workers who make up the pool of thinkers and decision makers that are tasked with running, growing and transforming the business. Tools that let these users make faster, better and more-informed decisions are particularly valuable in a difficult business environment

(emphasis added).  Complex Event Processing is a key enabling technology for real-time business intelligence. The ability to abstract “rules” out of data using analytics and then have those rules executed in real-time to match against “events” running around your enterprise bus is what will make BI achieve the needs of business to make informed decisions “faster”. Traditional BI where you feed data into a mammoth DWH and then analyse the results at some point later suffer from high latencies and reduced information currency. CEP can reduce these latencies from weeks or months down to seconds.

Web-Oriented Architectures. The Internet is arguably the best example of an agile, interoperable and scalable service-oriented environment in existence. This level of flexibility is achieved because of key design principles inherent in the Internet/Web approach, as well as the emergence of Web-centric technologies and standards that promote these principles. The use of Web-centric models to build global-class solutions cannot address the full breadth of enterprise computing needs. However, Gartner expects that continued evolution of the Web-centric approach will enable its use in an ever-broadening set of enterprise solutions during the next five years.

Recognition that SOA is more than just “Web Services”and that at least one other architectural paradigm is available to SOA implementers.

Interesting that BPM has dropped from the list after featuring in the previous two years. And SOA itself has not been mentioned since the list for 2006. It looks like SOA has become more business as usual – part of the IT “furniture” – as I predicted some time ago.

Using Events to Add Real-time Intelligence

I will be manning the booth tomorrow afternoon at the TIBCO SOA Online Summit. The topic of discussion is Events and Realtime Business Intelligence which is an active part of my portfolio these days. Two of the guys from my blogroll will be presenting – James Taylor from Smart (Enough) Systems and Paul Vincent from TIBCO. Please come along and say “hi”.

The benefits of an ESB

In my last post on this topic I talked about the concept of an ESB. Here I talk about why you would want one.

There are plenty of whitepapers, analyst reports and vendor statements about the features and functions of the various ESB products. In my experience, the key advantages of using an ESB are less about features and functions and more about how you use it.

Standardization

One of the primary advantages of an ESB is that it gives you a standardized platform for integration. When everyone is using the same tools you can develop enterprise-wide frameworks, patterns and best practices for building re-usable services. Without a unifying platform, you get a divergence of integration methods which leads to inconsistency and higher cost of management and change. So an ESB platform helps with design-time governance. Note that this is not the same as standardization in the sense of using web-services standards. The important thing is that you use the ESB to support your own enterprise standards. These may be based on external standards – but that may be of secondary importance.

Loose Coupling

The bus architecture of an ESB encourages you toward a loosely coupled architecture.

  • Physically decoupled by making use of message passing mechanisms (e.g. JMS) versus direct network connections (e.g. HTTP).
  • Semantically decoupled by use of message transformation so that services are exposed in a system-neutral format which reduces application lock-in and reduces the cost of change.

Scalability and Reliability

Physical loose-coupling provides scalability advantages such as high-availability, fault-tolerance and load balancing. The messaging layer in the ESB directs messages between service endpoints to the appropriate instance of the endpoint. For example, in the event of a service provider failure, messages will be redirected to a backup provider – thus supporting high availability. In the case of load balancing, messages are distributed between redundant providers (or consumers) to handle high volumes of message traffic. You could say that physical loose-coupling supports change at the “micro” level where short term changes in the system topology can be compensated for via real-time message redirection.

Routing and mediation

Message routing supports scalability and fault tolerance. An ESB can also be used to support business-level routing and mediation. For example content-based routing allows services to be invoked based on the content of a service request. A business example would be routing of a customer enquiry to the branch where that customer account is located. A technical example would be the routing of a service request based on the version of the service being invoked.

Complex message exchange patterns

Traditional HTTP-based services support only one-to-one request-reply MEPs. An ESB supports more complex MEPs such as asynchronous one-way messaging and to multiple subscribers using topic-based messaging. Asynchronous publish and subscribe mechanisms support new ways of intermediating service consumers and subscribers – such as auditing, service monitoring – which are extremely useful for runtime management and governance of your services. Beyond mere governance, higher level business functions such as complex event processing (CEP) and business activity monitoring (BAM) are supported by this ability to “listen in” to service traffic on the ESB.

The benefits of an ESB that I’ve described above stem largely from the architecture of an ESB and in particular from the use of a message bus as the primary underlying transport. But it is important to understand that these benefits don’t automatically come “out of the box”. Your solution architecture (and your architects) must recognise and utilise the architectural principles underlying the ESB.

What is an ESB and why do I need one?

A question I often get is “what is an ESB and why do I need one”? This question is motivated by a number of concerns; non-technical people have heard the term but don’t understand the concept, semi-technical people are trying to figure out conflicting vendor definitions, and technical people are confused by the debate between different service enablement approachs – RESTful versus ws-* versus middleware-supported hybrids.

The Elevator Pitch

A service bus provides a uniform and consistent platform to allow service providers and service consumers to interoperate. An ESB provides benefits such as:

  • standardization
  • loose coupling
  • resilience and high availability
  • monitoring and intermediation

Hardware

I’m not sure of the provenance of the word “bus” as it is applied in the technical domain (I’m sure there is some interesting etymology there) but you can confidently trace it back to the concept of a computer hardware bus. The idea of a hardware bus (or backplane) is that hardware components – such as sound-cards, video cards, floating point accelerators, tape-drives, barcode scanners etc – can all slot into and interoperate through a shared infrastructure. By supporting a standard hardware interface and a standard software protocol, the hardware bus abstracts the details of each individual hardware component. The key features of the harwdare bus are:

  • standardized hardware connectivity to the backplane
  • standardized software protocol between each component and the backplane
  • hardware components can operate independently without having to know details about each other
  • a single infrastructure replaces multiple point-to-point connections between components (i.e. does away with a lot of ad hoc soldering).

Software

Networked systems arrived in the seventies and grew out of control in the eighties. Early network infrastructures such as Unix sockets were hard-wired point-to-point affairs with little or no abstraction of the the two programs that were working together.

The idea of a software bus is that software components can work together – yet independently – via a standardized message passing mechanism that would abstract away the need to create individual network connections between components. The software bus would take care of routing messages to the required location and also take care of all that hard stuff like quality-of-service, reliability and scalability. This is equivalent to standardizing the “hardware connectivity” in the hardware bus. TIBCO’s predecessor – Teknekron – articulated the concept of the software bus in the early nineties

The Service Bus

So the hardware bus standardizes hardware connectivity and the software bus standardizes software connectivity. The Service Bus has refined the concept of the software bus by taking a more service-oriented approach and adding support for the XML stack underlying web services and transport connectivity (e.g. bridging HTTP to JMS).

So why do you need an ESB? More on that anon…