Simon Eskildsen

Simon Eskildsen on scaling Shopify, building turbopuffer, and the future of databases

The interview discusses the scaling challenges faced by Shopify from 2010 to 2020, particularly with MySQL databases. One humorous incident involved an unexpected PHP cron job running every hour as part of Percona utilities, causing a soft lock-up due to its dependency on PHP, which was continuously tracking open files and sessions. The team spent considerable time optimizing systems like Memcached, Redis, and MySQL by reducing connection counts and employing various tricks since there were no open-source proxies available at the time. The infrastructure team evolved from a traditional operations-focused structure into one capable of handling multiple severe incidents (SEVs) during flash sales. Flash sales occurred when high-profile influencers promoted new products on platforms like Instagram or Facebook, leading to sudden surges in traffic and inventory contention that could overwhelm systems. Shopify had to manage these by ensuring robust inventory reservation processes and preparing for the influx of demand from millions of users trying to purchase limited stock simultaneously.