soabloke

Progressive Data Constraints

2008-11-21T05:23:34Z

As a follow up to my last post, Richard Veryard described the concept of Post Before Processing whereby rules are applied once you have safely captured the initial information. This is a good way of managing unstructured or semi- structured data and it reminded me of other cases where data is progressively constrained and or enriched during processing.

Content management systems use this technique at the unstructured end of the spectrum where source and draft content is pulled into the system from the “jungle” and then progressively edited, enriched, reviewed etc until the content is published. Programmers will be familiar with this in the way that source code is progressively authored, compiled, unit tested and integrated via source code control (plus a bunch of QA rules and processes).

Many computer-aided design applications also provide this ability to impose different rules through the lifecycle of a process. Many years ago I worked on a CAD application for outside plant management for Telcos which had a very interesting and powerful long transaction facility. Normal mode representing the current state of the network enforced an array of data integrity and business rules - such as what cables could be connected to each other and via what type of openable joints etc.

In the design mode, different rules are in place so that model items can be edited and temporarily placed into invalid states. This is necessary because of the connected nature of the model (a Telco network) and the fact that the design environment reflects the future state of the network which may not correctly “interface” to the current network state. The general mode of operation was multiple design projects created by different network designers who managed the planning and evolution of the network from it’s current state to some future state. And multiple future states could potentially exist within the system at any point in time. Design projects follow a lifecycle from draft through proposed and into “as built”. This progression is accompanied by various rules governing data visibility, completeness and consistency.

This is a useful model of how to manage data which may be complex, distributed or collaboratively obtained and managed. Effectively building a process around a long transaction which manages transition of data between states of completion or consistency.

Master Data Management is another topical example of this type of pattern. In this scenario data is distributed across systems or organizations. Local changes trigger processes which manage the dissemination of changes to bring the various parts of the data into consistency. During this process different business rules may be applied to manage the data in transition.

I think these concepts can be more generally applicable to SOA design-time governance. For example in the collaborative design of enterprise schemas or services contracts.

Talk to the Hand

Talk to the Hand

2008-11-10T22:15:23Z

A common issue with any kind of BPM implementation or distributed application is the problem of what to do with errors. The most important thing is to direct errors to the correct system or person. “Correct” meaning a system that can actually do something about the error.

Take our internal timesheet application as an example. I’m sure your system is similar, where an administrator sets up a project along with various business rules such as who can enter time against a project, or what is the valid date range for the project. When I enter my time records, I invariably run afoul of one of these rules - who could have guessed the project would run late?

The problem arises when I get a (usually very obscure) error message about one of these constraints. I can’t actually do anything about incorrect dates or ad hoc resource assignments because I don’t own the project. So it’s no good erroring out to me, the user. The really brain-dead thing is that I have to discard the timesheet entries, send an email to the project owner (whoever that may be) and enter the data later…if at all. It certainly doesn’t help collect accurate and timely information.

A better approach would be to accept the entered data and refer the errors to the project owner who is in a position to do something about it. This person can review the errors and correct the project rules accordingly. Alternatively they can reject the entry because I may have actually broken some valid rules.

This process flow is much more user friendly and streamlined. It requires a couple of important facilities:

You need to suspend business rules for part of the process. My timesheet entries initially break the business rules but we don’t know if that is because I have err’d or because the rules/reference data are in error. We need to accept the entries in an invalid state until the right authority can decide. This represents an example where data validity rules are not absolute, but depend on the process.

Secondly you need an error service…a facility whereby errors when detected can be routed to the appropriate person or system for resolution.

Our timesheet system is based on the old database paradigm - a monolithic data-centric application with only one place for rules to be applied. The “workflow-based” solution I describe is slightly more complicated than the monolithic application but the payoff in terms of a better user experience leads to more accurate and timely records. And if your a services organization with thousands of billable timesheet records each day, then you know what takes priority.

Gartner 2008 SOA User Survey

2008-11-05T23:13:38Z

The Gartner 2008 SOA User Survey is a good read with some surprising insights into SOA adoption.

One interesting development is that the rate of SOA adoption has slowed in 2008. About half of respondents last year who were planning SOA adoption, now have no plans for SOA adoption. The two main reasons for not pursuing SOA are a lack of SOA expertise, and the perceived lack of a business case. These reasons may be correlated in the sense that lack of SOA expertise makes it difficult to build an SOA business case. But the fundamental conclusion is that SOA doesn’t make sense for everybody.

SOA Adoption shows considerable geographic disparity. Europe has almost universal adoption (70% currently using), followed by North America (55% currently using) then Asia (25% currently using). The majority of organizations in Asia have no plans to adopt SOA. The report doesn’t really analyse why Asia has such low SOA adoption. My guess would be a combination of factors including lack of SOA expertise in the region, the characteristics of Asian companies being late technology adopters and the preponderance of manufacturing in the region which the survey shows has overall low SOA adoption compared with other sectors.

Organisation size correlates strongly with SOA adoption and the range of SOA deployment. There is a sweet spot for mid-size companies with current SOA adoption high in companies with employees between 1000 and 10,000. Large companies obviously struggle with the governance processes required to adopt SOA enterprise wide.

A big surprise for me was the correlation between SOA adoption and primary development language. Forty percent of current SOA adopters use Microsoft .NET. There is also a clear trend over the last 3 years away from Java toward Microsoft .NET and “other” languages such as dynamic languages. Correlation doesn’t mean causality so there is a lot of wiggle room in how you interpret this but clearly there is a move away from Java for SOA development. Harkening back to the COM/CORBA wars of the 90’s one of the key factors was that the Microsoft development environment made COM so easy to develop versus the complexities and diversities of CORBA that eventually COM came to dominate the component world. Is history repeating itself?

Web Services are the dominant SOA model, but a significant minority uses POX and REST approaches. About one third of existing SOA adopters already use or are planning to adopt EDA. The report also claims significant plans for WOA adoption, but I’m not convinced by the data. An eyeball comparison between Figure 14 (current WOA adoption) and Figure 15 (planned WOA adoption) doesn’t show a great deal of difference to me, except for Figure 15 looking a little more “peaky” around the 50% mark. So WOA adoption will increase, but I’m not convinced the data shows this is “dramatic” as stated in the key findings of the report.

The State of SOA Adoption

Gartner’s Top 10 Strategic Technologies for 2009

2008-10-25T11:56:30Z

Gartner has nominated their top 10 strategic technologies for 2009. In priority order they are:

1. Virtualization
2. Business Intelligence
3. Cloud Computing
4. Green IT
5. Unified Communications
6. Social Software and Social Networking
7. Web Oriented Architecture
8. Enterprise Mashups
9. Specialized Systems
10. Servers – Beyond Blades

My primary interests in SOA and CEP are represented in items 7 and 2 respectively (with some applicability to 8).

Business Intelligence. Business Intelligence (BI), the top technology priority in Gartner’s 2008 CIO survey, can have a direct positive impact on a company’s business performance, dramatically improving its ability to accomplish its mission by making smarter decisions at every level of the business from corporate strategy to operational processes. BI is particularly strategic because it is directed toward business managers and knowledge workers who make up the pool of thinkers and decision makers that are tasked with running, growing and transforming the business. Tools that let these users make faster, better and more-informed decisions are particularly valuable in a difficult business environment

(emphasis added). Complex Event Processing is a key enabling technology for real-time business intelligence. The ability to abstract “rules” out of data using analytics and then have those rules executed in real-time to match against “events” running around your enterprise bus is what will make BI achieve the needs of business to make informed decisions “faster”. Traditional BI where you feed data into a mammoth DWH and then analyse the results at some point later suffer from high latencies and reduced information currency. CEP can reduce these latencies from weeks or months down to seconds.

Web-Oriented Architectures. The Internet is arguably the best example of an agile, interoperable and scalable service-oriented environment in existence. This level of flexibility is achieved because of key design principles inherent in the Internet/Web approach, as well as the emergence of Web-centric technologies and standards that promote these principles. The use of Web-centric models to build global-class solutions cannot address the full breadth of enterprise computing needs. However, Gartner expects that continued evolution of the Web-centric approach will enable its use in an ever-broadening set of enterprise solutions during the next five years.

Recognition that SOA is more than just “Web Services”and that at least one other architectural paradigm is available to SOA implementers.

Interesting that BPM has dropped from the list after featuring in the previous two years. And SOA itself has not been mentioned since the list for 2006. It looks like SOA has become more business as usual - part of the IT “furniture” - as I predicted some time ago.

Gartner 2008 SOA User Survey

Using Events to Add Real-time Intelligence

2008-10-20T02:07:10Z

I will be manning the booth tomorrow afternoon at the TIBCO SOA Online Summit. The topic of discussion is Events and Realtime Business Intelligence which is an active part of my portfolio these days. Two of the guys from my blogroll will be presenting - James Taylor from Smart (Enough) Systems and Paul Vincent from TIBCO. Please come along and say “hi”.

No related posts.

Climate Change Update

2008-09-24T15:30:14Z

The ABC “Big Ideas” series has a great talk from Tim Flannery on the current status of climate change thinking and science:

The response to climate change has focused largely on what we can do to reduce the production of emissions. But leading environmentalist Dr Tim Flannery reminds us that we should not lose sight of the tree fix. Farms and forests could become ‘enormous engines of planetary cleansing’. Join us for an update on the science of climate change and some solutions based on the force of nature.

There is a sobering assessment of the challenge facing us in terms of reducing carbon pollution, but Dr Flannery demonstrates true leadership with some positive and pragmatic suggestions.

Design for Change

The Architectural Role of Messaging

2008-09-20T12:13:46Z

JMS has brought messaging more into the mainstream which is a good thing. But just like any new technology there is the danger that the first implementations will reflect older paradigms. I remember when I made the move from FORTRAN to C, for a while I wrote a lot of FORTRAN programs in the C language syntax until I got more familiar with C features and idioms. The same goes for my more recent ventures into Ruby with many years of Java thinking under my belt.

Coming back to JMS, I find when I review distributed applications that have been designed by people with a strong client-server or web background I see a lot of rpc message semantics. While rpc (or synchronous request/reply) has it’s place, this is not always the best approach. A common mistake is to regard messaging as simply a way to get messages from point A to point B…treating JMS as a simple transport such as a TCP socket or HTTP.

Messaging originated as the concept of a distributed queue. Most programmers are familiar with queues from GUI frameworks where communications between widgets are mediated via an event queue. The event queue supports a number of functions such as decoupling widgets from each other…allowing each widget to do what it needs to in it’s own time, and supporting event driven interactions between widgets. The event queue along with multi threading is key to giving user interfaces the responsiveness and robustness that you expect. In this way queueing provides more than just a communications mechanism but is key to the architecture of a GUI framework.

The same is true of distributed messaging systems. In their original conception distributed queues do more than just provide a way for data to pass from one system to another, they provide an important element of isolation.

A fundamental difficulty in building distributed systems is that the different components have different performance characteristics. In addition the uptime of your total system is the product of the uptime of individual components. To ensure maximum uptime you want your components to be independent of each other and, in the event of failure you want to be able to restart from where you left off. This is where message queues work really well. Component A puts a message onto a queue and doesn’t care if or when component B takes that message off the queue. This is known as the fire and forget message pattern and it provides the best isolation between your system components.

If instead we make Component A wait for an acknowledgment from Component B before it proceeds then we are building a tight coupling into the system. Any performance difficulties or failure experienced by Component B could spread back to Component A and thence to other components up the chain.

So the role of messaging in distributed systems goes beyond just getting a message from point A to point B. It also acts as a kind of expansion joint for your system allowing individual components to vary in their performance characteristics - or even fail totally for short periods - without breaking adjacent components.

Without these messaging expansion joints, your system is tightly coupled and prone to system wide failure originating from a single component. Messaging - using the fire and forget pattern - allows these issues to be locally absorbed and managed within normal system operations.

Push versus Pull

2008-08-19T05:10:20Z

From OSCON via O’Reilly Radar here’s a good case study of an architectural decision driven by the system requirements rather than the usual religious considerations that pollute the bloggosphere.

FriendFeed needed update info from Flikr but a REST-based “pull” approach is highly inefficient in this case. Instead the solution architects opted for a “push” approach using xmpp as the message transport. This is a really good presentation because it goes into the architectural choices and implications of “push” versus “pull”.

I characterize this as “pull vs push” rather than “REST vs xmpp” (or “REST vs *” or “why REST is crap”) because fundamentally it comes down to the best choice of how to synchronize changes between systems. You make this choice based on the usage characteristics of the different systems, the likely traffic volumes this will result in and the consequential resource impacts. Having made the choice between push or pull you then choose the appropriate message transport.

The web doesn’t do a lot of “push” and consequently there is not a lot of discussion about push and REST. Dare Obasanjo characterises it nicely:

Polling is a good idea for RSS/Atom for a few reasons

there are a thousands to hundreds of thousands clients that might be interested in a resource so the server keeping track of subscriptions is prohibitively expensive

a lot of these end points aren’t persistently connected (i.e. your desktop RSS reader isn’t always running)

RSS/Atom publishing is as simple as plopping a file in the right directory and letting IIS or Apache work its magic

The situation between FriendFeed and Flickr is almost the exact opposite. Instead of thousands of clients interested in document, we have one subscriber interested in thousands of documents. Both end points are always on or are at least expected to be. The cost of developing a publish-subscribe model is one that both sides can afford.

Inside the firewall, the situation is often more akin to that between FriendFeed and Flikr. This is why messaging is more common inside the firewall than outside - not because of any universal superiority between REST versus messaging, but because the system requirements are different and often favour a push approach rather than pull.

While your over at Dare’s excellent Blog, be sure to also check out his discussion of push versus pull in the context of scaling Twitter and MS Exchange. These are important considerations for designers of federated systems such as federated databases or federated messaging systems. The example of FriendFeed to Flikr could be considered as the first incremental step toward a federation.

The Architectural Role of Messaging

ILOG Drinks the Microsoft Kool-Aid

2008-08-11T05:23:26Z

In addition to having an affinity for IBM, ILOG has jumped in bed with Microsoft with ILOG Rules for .NET.

I am not easily impressed, however I must say that this product earns a lot of points in my book. It took several weeks of in depth research for this article, but I came away with a very positive impression of the product and the .NET platform in general. If Microsoft’s WCF performance figures are to be believed, anyone building enterprise applications on .NET need look no further. They even eat their own dogfood.

The copy editor (a new one) had a heavy hand with this one, but the message is still clear. You can read the review of ILOG Rules for .NET and it’s applicability to the enterprise over at InfoWorld magazine.

It is certainly ‘interesting times’ in the BRMS space. With the union of JRules & Filenet through the IBM acquisition, one wonders if this shotgun marriage isn’t the final step on the BRMS/BPM/SOA consolidation. FairIsaac seems to be on the ropes, with little choice but to accept an acquisition. That leave only a handful of commercial vendors. Perhaps we’ll see the rise of open source BRMSs? I’ve already seen more uses of JBoss Rules (Drools) in the field this year that I’ve seen in the past three, and it’s improving by leaps & bounds with each release.

Interesting times indeed.

No related posts.

Hand me that ESB to your left….

2008-08-10T09:29:42Z

In 2002 Sonic Software invented the term ESB to differentiate their middleware from everyone else’s middleware. This is an old marketing tactic when you can’t differentiate your product from the others - just invent a new category. To lend credence, you need the collusion of an analyst and Gartner was there to knock this marketechture into the hype-stratosphere. We can also thank Sonic for the crap notion that an “ESB is a prebuilt SOA“.

ESBs have been back in the news again with the usual attendant vitriol and this has revived the perpetual question - “what is an ESB and why do I need one“?

I’ve seen ESBs referred to in two rather broad contexts. The first context refers to a distributed infrastructure that allows your Service Providers and Service Consumers to find each other and work together in a scalable and loosely coupled manner. This is often referred to as the “ESB as design pattern” and I think it is what many people would like ESB to mean. This “Service Bus” is effectively an extension of the “Message Bus” concept and is what I had in mind when I wrote about this subject a while ago.

The second context is ESB as a tool or a “thing you buy from a vendor” and this seems to be the terminology that calls forth all manner of vituperations. In this context ESB is secret marketing code for what used to be called “integration middleware.” This is also the thing that Jim Webber calls the “Erroneous Spaghetti Box.”

In this context, the eternal ESB question becomes - “in an SOA world, do I really need integration middleware?” I really think you can make more sense of the ESB debate by recasting into this terminology. The SOA pragmatists (including myself) see that there is a role for “integration middleware” as tools within your SOA infrastructure.

Eric Roch thinks of an ESB as a “productivity tool for integration” and points out that The legacy integration problem hasn’t gone away yet.

ZapThink say that an ESB could play the role of a Service Intermediary where transformations and content-based routing are essential requirements.

Todd Biske characterises five different types of ESB product each supporting different functions such as legacy integration, orchestration, mediation and transport, SOAP wrapping and gateways. The key insight here is how the label “ESB” has come to be applied to so many disparate products. “ESB” has come to mean everything and nothing and should be banished from our vocabulary.

So I agree with the pragmatists - design your SOA infrastructure to support your business requirements. Consider the so-called ESB products as tools and if you need mediation, or orchestration or transformation or integration to legacy systems then consider using the scads of tools - both proprietary and opensource - out there that support this stuff. And whatever you do, don’t try to author XSLT using Notepad.