KubeCon + CloudNativeCon North America 2024

In the era of supporting AI and large language models, acquiring GPU supplies from major cloud providers is challenging and expensive. Meanwhile, a significant supply of affordable GPU resources is emerging from various third-party providers. Hybrid Kubernetes clusters are the perfect solutions to integrate these GPUs into existing services built on large cloud providers. At our startup focusing on cloud infrastructure, we have created numerous hybrid Kubernetes clusters based on K3s and P2P VPN using the Wireguard protocol. With this setup, we have successfully integrated more than ten sources of GPUs from different geographical locations, achieving nearly unlimited on-demand GPU resources while reducing GPU costs by 2x-4x. In this talk, we will discuss the architecture, pros and cons, requirements, and limitations of pure hybrid Kubernetes clusters for GPUs. We will also share lessons learned during the building and management of such true hybrid Kubernetes clusters.