Responsibilities

Design and implement a versioned provisioning state machine covering physical-host discovery, inference bring-up, GPU driver and CUDA setup, health validation, and decommissioning or RMA.
Build declarative self-service APIs and a Kubernetes-native control plane for requesting, scaling, and tearing down inference clusters.
Automate detection, draining, repair, replacement, and reintegration of degraded or failed nodes.
Ensure production reliability through idempotency, retries, rollback, drift detection, testing, code review, versioning, and CI/CD.
Partner with the inference and ML platform teams to encode topology, interconnect, and scheduling requirements as platform abstractions.
Develop defragmentation, rebalancing, scheduling, bin-packing, health, remediation, and utilization systems for efficient GPU capacity.

Requirements

Strong software engineering experience using Go, Python, Rust, or a similar language.
Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent.
Experience building software control planes or orchestration systems using state modeling and reconciliation, such as Kubernetes controllers, operators, custom reconciliation loops, or workflow engines.
Experience designing event-driven systems using message queues, event streams, or publish/subscribe systems rather than polling or cron-driven scripts.
Experience building internal platforms or APIs for other engineering teams with attention to developer experience.
Preferred experience with bare-metal provisioning technologies such as PXE, iPXE, Redfish, IPMI, or BMC.
Preferred networking fundamentals including VLANs, BGP, or fabric design, and/or GPU or accelerator infrastructure.
Preferred experience with GPU cluster software stacks including NCCL, CUDA, and InfiniBand or RoCE.
Experience at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization is preferred.
Systems programming experience in Rust or Go is preferred.