You decoupled your APIs from their implementations and put them behind RPC interfaces. You build and deploy services independently. You code health is impeccable. You put your user data in a persistent, replicated, and consistent store, where it belongs. Your developer velocity has skyrocketed.
Now we have new problems. We’ve got N independent services with M edges of interaction between them. That’s N services that need to be built, tested, and deployed on the infrastructure that expected you to have one service whose mess of entanglement was a secret you had with the compiler.
How do we deploy N binaries with N sources of static configuration and M sources of runtime configuration safely without losing our collective minds? In this talk, I’ll share some of how we grew that aforementioned N from 1 to many in Gmail. Specifically:
- Consistent naming schemas for services, environments
- Maintaining lightweight, easy-to-change production configuration abstraction layers
- Release early, often
- Canary everything by sharding into more A/B environments than you'd think you’d need
- Encourage backwards compatibility in all APIs
- Validate and test all configuration before changing global state
And, of course, some of things we (Gmail) learned by breaking things along the way.
