Coping with x2,000 data volumes growth

Experience my pain as I try to build an Azure Data Platform that scales.


I worked in a startup and our volumes started small, really small. I was the data scientist but I needed to actually build a data platform first. By the end of 6 months, daily volumes were 2,000 times the size. This talk takes you through my 6 months of learning how to build a high-scale data platform whilst requirements, scale, and technology were all changing. We’ll look at the various bits of Azure I used, how they started off or became wrong for my situation, how I designed the system to cope with the constant swapping and changing of technologies, and then we’ll look at the system I ended up with that could cope with the scale and at then some.


View the presentation full screen or view it below. Hit the Space bar to navigate through the slides.

Presentation history