Datadog adds Kubernetes inference workload monitoring in Preview

One view for distributed inference resources

Inference frameworks distribute serving work across Kubernetes resources, which can require engineers to compare model configuration, pod status, and scheduling events through separate commands and dashboards. Datadog's inference workload is a virtual resource that groups the Kubernetes resources supporting a serving model.

In Kubernetes Explorer, engineers can use model, framework, and cluster context to identify a workload and inspect related pods, deployments, replica sets, and custom resources. Datadog says the feature connects the custom resource with the Kubernetes objects it manages and the inference engine, such as vLLM running inside serving containers.

The feature supports vLLM-backed workloads using KubeRay, NVIDIA Dynamo, and KServe. Datadog says 66% of organizations hosting generative AI models reported using Kubernetes for some or all of their inference workloads in the CNCF's 2025 survey.

Correlating readiness, telemetry, and scheduling

Datadog automatically detects pods running supported inference frameworks and assigns inference_workload and inference_workload_component tags. Metrics, traces, and logs inherit those tags when they can be linked to the relevant pods, allowing engineers to correlate telemetry for a workload or component across Datadog products.

The Components tab shows pod health within a framework's serving structure, including Ray head and worker components or Dynamo frontend and worker components. Engineers can review readiness and pod counts, then inspect pod status, restarts, node placement, and scheduling details. This can help distinguish pods waiting for resources from running pods that are unhealthy.

The Metrics tab provides runtime performance and resource usage information. For Dynamo workloads with paid GPU Monitoring access and the relevant collection configured, available metrics include GPU memory, activity, and occupancy, along with container CPU and memory usage. Engineers can select View in GPU Monitoring to investigate device health.

Finding GPU capacity constraints

Datadog illustrates the feature with a Dynamo model that has separate prefill and decode components. If a decode worker pod is stuck in Pending, its scheduling details can report insufficient capacity for the required GPU type. That evidence can explain why the requested serving capacity has not become available.

Engineers can filter the Inference Workloads page by GPU type and examine other workloads using that capacity. In the example, a lower-priority Dynamo model consumes capacity needed by a critical workload. Reviewing its model demand and resource configuration can help teams assess whether scaling it down is appropriate.

Datadog also says Investigate with Bits uses Kubernetes MCP tools to gather context about pending pods and recommend next steps. In the example, reducing the lower-priority model's replicas could release GPU capacity for the critical model's decode worker, provided the released resources meet its scheduling requirements.

What the Preview means for Kubernetes teams

The feature brings orchestration configuration, serving readiness, pod state, scheduling evidence, and, where configured, GPU telemetry into a connected investigation path. Datadog says this context can help teams move from a model-level symptom to the pods and capacity constraints behind it, then evaluate possible workload changes or GPU reallocation.

Inference workloads are also exposed through Datadog's Kubernetes MCP tools. Through an MCP-compatible assistant, users can discover inference workloads, identify their model and framework, and inspect underlying resource details, parent relationships, and manifests while investigating pod state and configuration.

The Kubernetes inference workload feature is available in Preview. Datadog directs users to its Container Monitoring and GPU Monitoring products for more information and offers a free 14-day trial for people new to Datadog.