Werner Vogels, CTO at Amazon.com, posted a paper on Amazon’s internal called “Dynamo” about two year ago that was selected to be presented at the 21st ACM Symposium on Operating Systems Principles. Dynamo is Amazon’s internal cluster software platform. It is really their operating system since they add “hot swappable” commodity hardware to the system in plug-and-play fashion as needed.
I know I’m late to finding this particular paper since it was first published about two years ago. Better late than never!
I found it interesting that Amazon uses a service-orientated-architecture of about 150 services to compose any one page. I just drew a few boxes on this page to illustrate some obvious ones that the end user sees. I’m sure there are other services working in the background that track how many times you’ve viewed the page, what parts of the page you scroll to and what buttons you click on to name just a few.
The paper goes on to describe four “other key principles” which I think are important architectural keys when building a SOA-based system.
- Incremental scalability – As new systems are added to the overall fabric of Dynamo, it increases the overall capability of the system.
- Symmetry – Except for processing capacity and hardware vintage, all nodes act the same.
- Decentralization – Peering instead of master-slave. The “master” will always become a bottleneck!
- Heterogeneity – Basically the Dynamo system allows the workload to scale out onto whatever type of server is available as long as the platform is the same. I bet that allows Amazon the flexibility of “purchasing and forgetting” hardware. The commodity hardware spans several generations of hardware improvements as servers are added and retired.
Some interesting solutions for large-scale systems:
- DHTs - Distributed Hash Tables
- Consistent Hashing
- Vector clocks
- Quorum
- anti-entropy based recovery – Couldn’t find a link for this concept.
Thanks to Werner Vogels for publishing the paper!