KubeCon + CloudNativeCon North America 2024

As the demand for LLMs continues to soar, the need for secure, cost-conscious, and content-aware control over its usage is paramount. In this talk, we explore why Envoy Proxy is the optimal choice for building an LLM gateway, leveraging its unique architecture and capabilities. Unlike traditional proxies (e.g. NGINX), which rely on scripting languages for customization, Envoy Proxy stands out due to its extensibility features: filter architecture, callout architecture (ext-proc, ext-auth), and ability to dynamically load libraries. Combined with its high-performant, async core ( C++), Envoy can run as an ingress, egress and mesh gateway. We'll look at using Envoy proxy for LLM credential management, prompt guarding/decorting, analyzing content safety, usage controls, context-aware failover, and observability. Ideal for developers, architects, and tech enthusiasts looking to solve challenges around LLM usage and picking the right technologies for their platform infrastructure.