This is a few months old, but Werner Vogels republishes the Amazon.com Shareholder Letter. A summary of how the quintessential 21st century company views its competitive advantage:

1. Service Oriented Architecture
2. Distributed state management
3. Decision management

Regarding Service Orientation:

Our technologies are almost exclusively implemented as services: bits of logic that encapsulate the data they operate on and provide hardened interfaces as the only way to access their functionality. This approach reduces side effects and allows services to evolve at their own pace without impacting the other components of the overall system. Service-oriented architecture — or SOA — is the fundamental building abstraction for Amazon technologies. Thanks to a thoughtful and far-sighted team of engineers and architects, this approach was applied at Amazon long before SOA became a buzzword in the industry. Our e-commerce platform is composed of a federation of hundreds of software services that work in concert to deliver functionality ranging from recommendations to order fulfillment to inventory tracking. For example, to construct a product detail page for a customer visiting Amazon.com, our software calls on between 200 and 300 services to present a highly personalized experience for that customer.

In my last post I showed how to send a Simple Notification Service (SNS) message to an email endpoint. Now I show how to easily add a WebHook endpoint. WebHooks are a design pattern using an HTTP POST to send a notification to a URL which the “subscriber” has registered with the service. WebHooks are being used in an increasing number of web APIs and there is an interesting interview with Jeff Lindsay on this topic at IT Conversations.

A useful test platform for WebHooks is PostBin.org. Simply click on the “Make a PostBin” button and you will be presented with a new URL for your notification messages – something like “http://www.postbin.org/1hf0jlo“. This is the URL you register with SNS.

Turning to the SNS dashboard, add a new subscription to a topic that you’ve already configured in SNS. Specify protocol “HTTP” and enter the PostBin URL as the endpoint. SNS will post a confirmation message to this URL before you can send through messages.

Go back to your PostBin URL and you should see the confirmation message.

Buried in the message is the SubscribeURL which you need to hit in order to confirm the subscription. I pasted it into notepad and edited “cleaned up” the URL before pasting it into a browser. This confirms the subscription with SNS.

Now back in the SNS Dashboard you can send a new message. In my case, since I still have my email endpoint, the same message is sent to both the email and the WebHook endpoints…thus:

…and thus:

AWS is the latest to the join the cloud pub/sub hubbub with the (beta) launch of its Simple Notification Service. I took it for a spin this morning while waiting for my flight.

Once I signed up for the service in my AWS account, it was a simple matter of opening up the AWS Console and selecting the SNS tab.

You create a topic (I called mine “MyPersonalTweeter”).

Then create a subscription.

Subscribers can be HTTP/HTTPS WebHooks, or an email address or an SQS queue. I selected email.

AWS sends a confirmation to the nominated email address (damn, there go the SPAM opportunities).

Then just publish a message to the topic consisting of a subject and a message body (for some reason my first message never arrived, so I’m showing message number two sent later that day).

Digging a bit deeper, topics have policies which allow reasonably fine-grained control over who can publish and subscribe to your topic.

The recent brouhaha about Twitter scalability has highlighted the growth of the latest spectator sport in the blogosphere – “armchair architect”. Everyone’s a Monday morning expert on which language/database/framework is/isn’t the secret to extreme scalability. The real secret is the architecture and organizational maturity. Here are some case studies to prove it:

A Conversation with Werner Vogels where Werner talks about using service-orientation to scale out massively distributed services which power the Amazon e-commerce platform. This is one of my favourites because it covers organizational as well as technical aspects of scalability. One of the unique attributes of Amazon is that service-orientation pervades everything – even their organizational structure. Developers are responsible for running their own services.Werner characterises the adoption of services as a challenging and major learning experience, but it has become one of their main strategic advantages. Key lessons learned:

• service-orientation is an excellent technique to achieve isolation and high levels of ownership and control
• prohibiting direct database access allows scaling and reliability improvements without affecting clients
• a single unified service access mechanism supports service aggregation, routing & tracking
• service orientation improves development and operational processes leading to more agility
• giving developers operational responsibility enhances the quality of the services.

The eBay Architecture (PDF) covers the evolution of eBay from 1998 to 2006. It’s a great example of how continuous reinvention is needed to keep up with rapidly growing scaling requirements. Frank Sommers writes a good summary and discussion where he argues that organizational capability is just as important as technical architecture for scalability. Another key ingredient of the eBay story is the ability to discard “conventional wisdom” when required. This is covered in an interview with Dan Pritchett, revealing some of the “rules” that eBay bends in order to scale.

• eBay.com doesn’t use transactions – mainly for scalability and availability reasons
• different data is treated in different ways – so best effort suffices for some
• references the CAP theorem – consistency, availability, partitioning – pick any two
• many have arrived at the same idea – and transactions are the first to go

Scalable Web Architectures is a great presentation on scalable web architectures in general and Flickr in particular. Also check out Cal Henderson’s list of his other presentations.

Architectures You’ve Always Wondered About provides slides from QCon London 2009 presentations. Case studies about eBay, Second Life, Yahoo!, Linked-In and Orbitz.

Avoiding the Fail Whale is a video in which Robert Scoble interviews architects from FriendFeed, Technorati and iLike.

Improving Running Components at Twitter – Evan Weaver describes how Twitter learned to scale by moving to a messaging architecture.

Real Life Architectures at High Scalability – provides a huge collection of pointers to architecture case studies from around the web.

Everyone agrees that 2008 was the year that Cloud Computing Hype took off, but here is the “proof” that it was sometime around October 27 2008.

Click here for larger image

The two graphs show the Google Insights data for search terms related to “cloud computing”, “azure”, “ec2” and “google cloud”. The lower diagram shows the search popularity for each term normalised to a range of 0 to 100. The upper diagram shows the rate of change of these search terms with respect to the overall category of “Computers and Electronics”.

In absolute terms “ec2” had the lions share of search popularity reflecting perhaps its more mature status. “Azure” didn’t appear until September – leading up to its launch on October 27th. To represent Google I had to use a compound term such as “Google Cloud” because “Google” by itself simply swamps the search results and would skew it to unrealistic levels, so it is likely that the Google representation in these “cloud searches” is lower than reality. Using a different term such as “Google App Engine” doesn’t change the conclusions.

The interesting part is the growth in search interest. “Cloud Computing” shows very strong and steady growth throughout the year, peaking around October 27. This is the rate at which search interest was growing, so although search interest will continue to grow in the future, it may never grow as fast as it was in October 2008. Azure may be a late player in the field, but it seems to have sparked a lot of general interest.

Try the analysis yourself at Google Insights for Search