Muli Ben-Yehuda's journal

March 1, 2019

Data center vignettes #3

Filed under: Uncategorized — Muli Ben-Yehuda @ 10:36 AM

It’s Friday. You’ve been hacking on this cool bit of code for awhile. It will be such a pleasure to deploy and see the user engagement numbers go up. The CI is green. The code is tight. You take a deep breath and deploy.

Ten minutes later, everything is fine. Ten minutes after that, still good. An hour passes. You check out the Grafana dashboard, and everything looks OK, except… why does the size of one of your key data stores continually increase with the new code?

This is not yet an emergency but it will become one if it keeps up. Each of your servers is limited to two SSDs. The infrastructure guys wanted to keep SKU sprawl to a minimum, and most of the CPU cycles are used for computation anyway, so they decided to “right size” the storage on each server to two SSDs per node. You crunch some numbers and realize that the data store is going to stop growing and stabilize — exactly 100GB after it exhausts all available space on your servers. You curse and roll back the code until they can install more SSDs, sometime next decade.

Wouldn’t it have been nice if there was no limitation on the amount of storage your application could use, while still enjoying the benefits of direct attached SSDs? Enter LightOS, coming soon from Lightbits Labs.

Blog at