The recent brouhaha about Twitter scalability has highlighted the growth of the latest spectator sport in the blogosphere – “armchair architect”. Everyone’s a Monday morning expert on which language/database/framework is/isn’t the secret to extreme scalability. The real secret is the architecture and organizational maturity. Here are some case studies to prove it:
A Conversation with Werner Vogels where Werner talks about using service-orientation to scale out massively distributed services which power the Amazon e-commerce platform. This is one of my favourites because it covers organizational as well as technical aspects of scalability. One of the unique attributes of Amazon is that service-orientation pervades everything – even their organizational structure. Developers are responsible for running their own services.Werner characterises the adoption of services as a challenging and major learning experience, but it has become one of their main strategic advantages. Key lessons learned:
- service-orientation is an excellent technique to achieve isolation and high levels of ownership and control
- prohibiting direct database access allows scaling and reliability improvements without affecting clients
- a single unified service access mechanism supports service aggregation, routing & tracking
- service orientation improves development and operational processes leading to more agility
- giving developers operational responsibility enhances the quality of the services.
The eBay Architecture (PDF) covers the evolution of eBay from 1998 to 2006. It’s a great example of how continuous reinvention is needed to keep up with rapidly growing scaling requirements. Frank Sommers writes a good summary and discussion where he argues that organizational capability is just as important as technical architecture for scalability. Another key ingredient of the eBay story is the ability to discard “conventional wisdom” when required. This is covered in an interview with Dan Pritchett, revealing some of the “rules” that eBay bends in order to scale.
- eBay.com doesn’t use transactions – mainly for scalability and availability reasons
- different data is treated in different ways – so best effort suffices for some
- references the CAP theorem – consistency, availability, partitioning – pick any two
- many have arrived at the same idea – and transactions are the first to go
Scalable Web Architectures is a great presentation on scalable web architectures in general and Flickr in particular. Also check out Cal Henderson’s list of his other presentations.
Architectures You’ve Always Wondered About provides slides from QCon London 2009 presentations. Case studies about eBay, Second Life, Yahoo!, Linked-In and Orbitz.
Avoiding the Fail Whale is a video in which Robert Scoble interviews architects from FriendFeed, Technorati and iLike.
Improving Running Components at Twitter – Evan Weaver describes how Twitter learned to scale by moving to a messaging architecture.
Real Life Architectures at High Scalability – provides a huge collection of pointers to architecture case studies from around the web.
