pushing soa up the slope (with a pointy stick)

I’ve recently joined the talented people at Sixtree and consequently am blogging over there.  Please check it out.

(Not much to see here in the meantime)

This is a few months old, but Werner Vogels republishes the Amazon.com Shareholder Letter. A summary of how the quintessential 21st century company views its competitive advantage:

1. Service Oriented Architecture
2. Distributed state management
3. Decision management

Regarding Service Orientation:

Our technologies are almost exclusively implemented as services: bits of logic that encapsulate the data they operate on and provide hardened interfaces as the only way to access their functionality. This approach reduces side effects and allows services to evolve at their own pace without impacting the other components of the overall system. Service-oriented architecture — or SOA — is the fundamental building abstraction for Amazon technologies. Thanks to a thoughtful and far-sighted team of engineers and architects, this approach was applied at Amazon long before SOA became a buzzword in the industry. Our e-commerce platform is composed of a federation of hundreds of software services that work in concert to deliver functionality ranging from recommendations to order fulfillment to inventory tracking. For example, to construct a product detail page for a customer visiting Amazon.com, our software calls on between 200 and 300 services to present a highly personalized experience for that customer.

A couple of years ago I read a book – “Javascript: The Good Parts” – which changed my mind about a language that I’d struggled with back in the nineties. I realise now that much of my struggle then was with the DOM and horrible browser incompatibility. Nevertheless the scarring was bad enough to keep me on the server side for the next decade.

I recently found a series of videos featuring Doug Crockford expounding on Javascript and its place in history. This series provides an interesting and entertaining perspective on computers and languages beyond just what runs in the browser.

• Volume 1 – The Early Years
• Chapter 2 – And Then There Was Javascript
• Act III – Function The Ultimate
• Episode IV – The Metamorphosis of Ajax
• Part 5 – The End of All Things
• Scene 6 – Loopage

The whole lot is a bit of a marathon but you can sample some of the main themes in a couple of episodes. I especially recommend episodes 1 and 4.  Episode 6 stands well on its own as a compressed version of the whole series. Episodes 2 and 3 are very specific to Javascript the language if that’s what you’re after.

I like episode 1 because it has nothing whatsoever to do with Javascript but is an interesting tour through the history of computing, describing how we got to the languages and architectures that we work with today. This is essential context for any working programmers today. If you view only one episode, I recommend this one. The historical perspective is reduxed in Episode IV where Ajax is discussed in the context of HTML and its origins.

The important take away from these two talks is how the history of computing is anything but linear and deterministic. We got to the place we are via  a series of steps, mis-steps, loops, accidents and fashions. And the random-walk continues. Crockford says:

“…important new innovations are received with contempt & horror and are accepted very slowly – if ever.

Often it is necessary for the previous generation of technologists to “die off” before major progress is made – especially in programming languages.

A key problem is that it usually requires many years of hindsight to determine which innovations really are useful and innovative.

You’ve probably heard that Amazon AWS had some problems recently. A question on Stackoverflow recently pointed out a detailed summary of the problem posted on the AWS message board.

Obviously every distributed system is different and every outage is unique so it is difficult to generalise. Some takeways I have are:

1. Outages happen to even the best guys on the block…so you better plan for yours.
2. Building distributed systems is hard…so you need experience and experienced friends.
3. Manual changes are a common cause…not said explicitly in the AWS writeup, but strongly implied.
4. Outages are often “emergent” phenomena whereby a simple error causes many systems to interact in a way which grows exponentially. The AWS writeup refers to this as a “storm” and I have witnessed similar “storms” in large distributed systems. The degree of coupling and simple aspects like backoff parameters can make the difference between a disturbance that grows exponentially or decays exponentially. Think of the Tacoma Narrows bridge – perhaps the analogy is a stretch, but tuning of a few simple parameters can avoid destructive resonances.
5. One of the responses pointed to the Netflix Chaos Monkey as being vindicated by the outage. The “Lean” guys have taught us that if something is difficult (like testing or deployment) then you should do it often until it aint difficult any more. Perhaps system failure/resilience is the next frontier for this approach.

A few notes (mostly to self) about running the latest ruby 1.9.2 and rails 3.0.3 environments on Windows (yeah I know all the kool kidz have Mac OSX):

Ruby Installer for Windows is great: http://rubyinstaller.org/

Even better is the DevKit (available on http://rubyinstaller.org/downloads/) which gives you an MinGW compile environment for building native gems.

If you want to run in debug mode in NetBeans, then check out this patch: http://noteslog.com/post/netbeans-6-9-1-ruby-1-9-2-rails-3-0-0-debugging/

And finally if you get this error with sqlite:

no such file to load — sqlite3/sqlite3_native

then make sure you’ve got sqlite3.dll in your ruby/bin directory.