Data Infrastructure at Airbnb


Part 1: Philosophy Behind our Data Infrastructure


> Leave some headroom: we oversubscribe resources to our clusters in order to foster a culture of unbounded exploration. It is easy for infrastructure teams to get wrapped up in the excitement of maximizing resources too early, but our hypothesis is that a single new business opportunity found in the warehouse will more than offset those extra machines.

Part 2: Infrastructure Overview



> Gold is our source of truth and we copy each bit of data from Gold down to Silver. Data generated on the Silver cluster is not copied back to Gold, and so you can think of this as a one way replication scheme that leaves Silver cluster as a superset of everything.

Part 3: Detailed Look at Our Hadoop Cluster Evolution