KubeCon + CloudNativeCon North America 2024

What has Kubernetes got to do with a High Energy Physics experiment collecting one million physics events per second at a data rate of 5 TB/s? That is what we would like to show you! The ATLAS experiment at CERN filters one million complex collision signatures per second provided by the Large Hadron Collider in quasi real-time, using a mixture of custom electronics and a large computing farm (the Event Filter – EF – farm) consisting of up to 5000 commodity servers. In this talk, we will tell you how we are going to exploit Kubernetes to orchestrate the ATLAS EF computing farm. In particular, we will focus on the strategy and optimizations we put in place in order to start more than 25000 PODs over more than 2500 worker nodes in about 50 seconds. We will also show the impact of the Kubernetes Scheduler and Controller Manager QPS values on POD start and stop throughputs and we will report about how custom scheduler profiles allow us to schedule PODs at an average rate of about 500 Hz.