GPUFleet

The problem

GPU clusters are expensive to operate, and scheduling strategy has a direct impact on utilization, cost, and job latency. But the tradeoffs between strategies are poorly understood because they depend on workload mix, cluster topology, and placement constraints.

GPUFleet provides a controlled environment to test these tradeoffs. Run the same workload through different schedulers, compare the outcomes, and inspect the decision trace to understand why.

Approach

A discrete-event simulation engine processes job arrivals, scheduling decisions, and completions using a min-heap event queue. The engine jumps between meaningful events rather than iterating through empty time. Time-stepped simulation becomes expensive and introduces artificial scheduling artifacts under sparse workloads; a discrete-event model keeps complexity proportional to actual system activity while preserving correctness.

Four pluggable scheduler strategies compete on the same synthetic workload. Each produces a full decision trace (JSONL) recording every scheduling pass: what jobs were pending, what placements were feasible, and what the scheduler chose and why.

FIFO

First-come first-served, first-fit placement. Simple baseline.

Bin Packing

Largest jobs first, pack onto fullest nodes. Reduces fragmentation at the cost of latency.

Cost-Aware

Evaluates all feasible placements, chooses the cheapest. Advantage grows with GPU cost variance.

Priority

High-priority jobs first. Maximizes throughput but starves low-priority work.

Results

Showcase scenario: 64 GPUs (8 nodes, mixed A100 + H100), 150 jobs with Poisson arrivals, 50% same-node placement requirement.

No scheduler dominated. Priority delivered the best throughput and lowest mean latency, but at the cost of fairness. Bin Packing improved fairness by preserving larger contiguous placements, but increased latency. FIFO remained the simplest middle ground, while Cost-Aware behaved similarly to FIFO because placement constraints dominated cost differences in this scenario.

Key findings

Utilization over time

Priority and FIFO keep the cluster busier for longer, but the utilization chart alone hides fairness and fragmentation effects. Higher sustained utilization does not automatically imply better scheduling outcomes.

Latency distribution

Priority lowers median latency but creates the worst fairness score, showing how responsiveness for urgent jobs can come at the cost of starvation for lower-priority work.

Why FIFO stalls despite free GPUs

Every scheduling pass is recorded as a JSONL record. This makes it possible to answer questions like "why did FIFO fragment here?" with concrete evidence rather than intuition.

{
  "schema_version": "1.0",
  "time": 12398.6,
  "scheduler": "fifo",
  "cluster_state": {
    "total_gpus": 64,
    "idle_gpus": 5,
    "idle_gpus_by_node": { "node-5": 1, "node-6": 4 }
  },
  "candidates": [
    {
      "job_id": "job-15",
      "feasible": false,
      "reason": "requires 8 GPUs with >=40.0GB; only 5 available"
    },
    {
      "job_id": "job-25",
      "feasible": false,
      "reason": "requires 8 GPUs with >=40.0GB; only 5 available"
    }
  ],
  "decisions": []
}

Outputs

• comparison.json — stable structured metrics for all schedulers
• decision_trace_{scheduler}.jsonl — per-pass decision logs
• report.md — markdown comparison summary
• PNG charts — utilization, queue depth, latency, and cost

Stack

Python 3.10+  ·  heapq-based discrete-event simulation  ·  matplotlib for charts  ·  PyYAML for scenario config  ·  stdlib-only simulation core for deterministic behavior  ·  124 tests, 98% coverage