Loading…
In-person
November 12-15
Learn More and Register to Attend

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for KubeCon + CloudNativeCon North America 2024 to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.

Please note: This schedule is automatically displayed in Mountain Standard Time (UTC -7). To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date." The schedule is subject to change and session seating is available on a first-come, first-served basis. 
Wednesday November 13, 2024 4:30pm - 5:05pm MST
Nothing is foolproof. Everything fails eventually. Observability tools help predict and lessen the impact of those failures, as the watchers of your software systems. But who watches the watchers? At Grafana Labs, we're not immune to production incidents. Just like any company, we still sometimes move too quickly. We run complex, microservices-based systems ourselves, so we have to eat our own dogfood on a daily basis. In this talk, I reveal: - how we solved a years-long mystery that cost us $100,000+ - how we got our internal Mimir clusters to reliably hold 1.3 billion time series for metrics - what we've had to do to scale our Loki clusters to handle 324 TB of logs a day - what our Grafana dashboards to monitor Grafana Cloud look like Sometimes, it's easier to learn from failures in observability than from successes. This talk is a confession of some of our worst sins as well as a realistic look under the hood at how we're improving the continuous reliability of our stack.
Speakers
avatar for Nicole van der Hoeven

Nicole van der Hoeven

Senior Developer Advocate, Grafana Labs
Nicole is a Senior Developer Advocate at Grafana Labs and a performance engineer with over a decade of experience in breaking software and learning to build it back up again. She has lived in the Philippines, the US, Australia, the Netherlands, and Portugal, helping teams all over... Read More →
Wednesday November 13, 2024 4:30pm - 5:05pm MST
Salt Palace | Level 1 | Grand Ballroom B
  Observability
  • Content Experience Level Any

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Share Modal

Share this link via

Or copy link