Over the last year, a lot of the Kubernetes AI work can look like Kubernetes itself learned to understand GPUs, distributed training and model state. In my observation, the model is more modular. Kubernetes has evolved rapidly to define the interfaces and scheduling primitives, while specialized components provide the domain knowledge. This has generally been the principle Kubernetes has followed.
Physical topology is one example. For AI workloads, PCIe, NVLink and InfiniBand topology can directly affect throughput. Dynamic Resource Allocation gives Kubernetes a richer resource model through ResourceClaims, DeviceClasses and ResourceSlices. The hardware-specific knowledge still comes from pluggable drivers such as NVIDIA’s DRA driver (similarly Google's TPU driver), which describes the devices and their capabilities to Kubernetes.
Distributed workloads are another example. Large models often span multiple nodes and need to operate as one coordinated unit. LeaderWorkerSet provides a Kubernetes-native way to manage a leader and its workers together for lifecycle, scaling and recovery. It is implemented as a CRD and controller on top of existing Kubernetes primitives. More native gang scheduling support is starting to move into kube-scheduler but still in alpha.
Inference routing in my opinion follows the same pattern. KV cache and model weights can't be instantly reconstructed after a failure, and routing to the wrong replica means redoing expensive work. The Gateway API Inference Extension addresses this by adding InferencePool as a new backend type using extension points (backendRefs, policy attachment) Gateway API already had. But the actual routing decision runs in a pluggable Endpoint Picker sitting entirely outside Kubernetes' own data path.