Scaling Azure Cosmos DB for ChatGPT in Habitat

OpenAI published a detailed look at how its application storage platform, Habitat, manages online storage for over 1 billion ChatGPT users across 40 geographic regions. The post outlines the second part of a series covering the evolution of Habitat into a globally distributed storage platform serving over 500 petabytes of data.

Part II focuses on scaling the underlying Azure Cosmos DB storage component through a working partnership with the Azure team. The engineering team navigated three distinct phases in managing multi-tenancy at scale.

Initially, the platform relied directly on Azure Cosmos DB primitives, utilizing the established 10K RU/s limit per logical partition key. Subsequent phases explored workload isolation, single-tenancy, and eventually returned to a default multi-tenancy model featuring data movement capabilities, co-managed containers, and integrated caching.