When we set out to rebuild the engine at the heart of our managed Apache Kafka service, we knew we needed to address several unique requirements that characterize successful cloud-native platforms. These systems must be multi-tenant from the ground up, scale easily to serve thousands of customers, and be managed largely by data-driven software rather than human operators. They should also provide strong isolation and security across customers with unpredictable workloads, in an environment in which engineers can continue to innovate rapidly.
We presented our Kafka engine redesign last year. Much of what we designed and implemented will apply to other teams building massively distributed cloud systems, such as a database or storage system. We wanted to share what we learned with the wider community with the hope that these learnings can benefit those working on other projects.
