S3 from scratch: Mail.ru Cloud Storage on In-Memory database

For over three years of production, Mail.ru Cloud Solutions S3 storage has grown to a cluster of tens of terabytes in RAM, which consists of metadata about billions of user objects.

The architecture of a cloud database with such scale requires a non-trivial approaches. For example, it is highly important for it to be almost linearly scalable.

I, as a creator of this system, will tell you the history of architecture of our solution, tell about specific sharding implementation, aboud live data migration between clusters without downtime, and the important role the Tarantool database play in the development of our clone of S3.