MLX Engine Benchmark Showdown

About the benchmark process.

This report and process was inspired by the reporting originally done at https://www.largitdata.com/blog/mlx-inference-benchmark-apple-m5-max/

I used the exact same benchmarking code made available at https://github.com/ywchiu/mlx_benchmark_lab

Where this data differs from the original is that I ran the benchmark on an Apple M3 Pro Macbook Pro with 36GB of RAM, instead of a M5 Mac with 64GB of RAM. I also did not test dflash-mlx or mtplx, as the range of models available for those engines is narrow and specialized and does not seem like a fair comparison entrant for this test. I did include vllm-mlx. Rapid-MLX is a fork of vllm-mlx.

Each engine was run locally using https://github.com/joemaddalone/inference-lab

Contents

1. Overview
2. Setup & method
3. Decode throughput
4. Prefill throughput
5. Time to first token
6. Stability
7. End-to-end latency
8. Key findings

1. Overview

Four MLX-based inference engines were benchmarked serving the same Qwen3.6-35B-A3B-4bit model over the OpenAI-compatible chat API, streaming. Prompt context was swept from 64 to 32,768 tokens (7 sizes × 5 runs each), and three primary metrics were recorded per run:

• Prefill tps — prompt tokens processed per second before the first output token.
• Decode tps — output tokens generated per second (streaming).
• TTFT — time to first token, in milliseconds.

The headline result: omlx separates from the field as context grows. Below ~2,000 tokens the four engines are within noise of each other. Beyond that, omlx’s prefill throughput scales to roughly 5,400 tps while the other three plateau around 550–680 tps, producing a TTFT and end-to-end latency gap of roughly one order of magnitude at 32k context.

2. Setup & method

Test configuration (from run logs)

• Same prompt template for every run: /no_think prefix + a fixed text to summarize (the /no_think prefix is the Qwen3 convention to disable thinking; no reasoning tokens were observed).
• Context sizes: 64, 512, 2,048, 4,096, 8,192, 16,384, 32,768 prompt tokens; 5 timed runs each after a same-size warm-up.
• max_tokens=256, temperature=0, streaming with stream_options.include_usage.
• A best-effort prefix-cache clear was attempted before each run (/v1/cache/clear).

Data notes & caveats

vllm-mlx reports prompt_tokens=0 on every run. The harness computes prefill_tps = prompt_tokens / prefill_time, so vllm-mlx shows a flat 0 and its prefill throughput cannot be compared directly. Its TTFT is still measured, however, and tracks rapid-mlx / mlx-vlm almost exactly at every context size — strong evidence that its real prefill cost is comparable to those two, not to omlx.

omlx streams in larger chunks. For a 256-token completion, omlx emitted ~37–43 content deltas while the other three emitted ~256 (one token each). Completion token counts are identical (~256), so decode-tps figures remain directly comparable, but this is an implementation difference worth knowing when parsing the raw JSONL.

All figures below are medians of 5 runs per context size, recomputed from the JSONL in data/. Best value per row is highlighted in green.

3. Decode throughput

Legend: rapid-mlx · omlx · mlx-vlm · vllm-mlx

Decode tps (median) vs prompt context length, log scale on x.

Median decode tps by context size

omlx posts the highest median decode speed at every context size. The other three engines swap the 2nd/3rd/4th places depending on context.

Decay with context length

Decode tps as a percentage of each engine’s own baseline at its smallest tested context.

Every engine loses speed as the context grows, and the shape of the decay is remarkably similar across engines:

• % of own baseline retained at 32,768 tokens: omlx 77.7%, rapid-mlx 77.6%, mlx-vlm 77.2%, vllm-mlx 74.6%.
• The most expensive step is between 4,096 and 8,192 tokens, where all four take a visible step down — decode falls furthest for rapid-mlx (54.3 → 45.6) which then partially recovers at 16,384 (49.1).
• vllm-mlx degrades the most overall; omlx retains the most.

4. Prefill throughput

Prefill tps (median) vs prompt context length, log scale on x. vllm-mlx reports 0 (prompt tokens unmeasured).

This is where the benchmark separates. omlx’s prefill throughput scales with context: roughly 137 tps at 64 tokens rising to ~5,400 tps at 8k–32k tokens. rapid-mlx and mlx-vlm stay roughly flat at ~560–680 tps regardless of context length.

Median prefill tps by context size

* vllm-mlx reported prompt_tokens=0; prefill tps is therefore 0 / not meaningful (see Section 2).

Interpretation: at small contexts all engines prefill at a few hundred tps. omlx is the only engine whose prefill scales up with batch/compute parallelism as the prompt grows, reaching its ceiling around 8k–16k tokens. The practical consequence is that omlx’s time-to-first-token stops growing with context in the mid range — see the next section.

5. Time to first token

TTFT (ms, log-y) vs prompt context length.

Median TTFT (ms) by context size

At 32,768 tokens, omlx reaches the first token in ~4.9 s versus ~47–48 s for every other engine — a 9.6–9.7× gap. At 8,192 tokens the gap is ~8×. Only at the smallest context (64 tokens) does omlx trail anyone: mlx-vlm’s 339 ms is the fastest TTFT recorded in the whole benchmark.

Notably, omlx’s TTFT is non-monotonic: 2,501 ms at 2,048 tokens, 2,327 ms at 4,096, then 1,249 ms at 8,192 — it drops even though the prompt quadruples. This is the direct consequence of the prefill scaling shown in Section 4; once parallelism kicks in, longer prompts cost less wall time per token. rapid-mlx, mlx-vlm and vllm-mlx instead show the expected linear-ish growth in TTFT, which is why their curves climb steeply on the log-y chart.

6. Stability (run-to-run variance)

Decode tps stddev per context — lower is more consistent across the 5 runs.

Per-context distribution of decode tps across the 5 runs (boxplots).

Decode tps stddev by context size

Median decode tps used elsewhere; stddev computed over the 5 runs per cell.

• omlx is the most consistent engine at every context ≥ 2,048, with stddev ≤ 0.2 tps — essentially zero spread across the 5 runs at large contexts.
• rapid-mlx is the most volatile at 8,192 (stddev 5.6, range 36.9–52.4): one run dropped to 36.9 tps while the other four sat near 52 — a ~30% swing. It also shows a 48.0 tps outlier at 4,096.
• mlx-vlm has cold-start outliers at small contexts — stddev 10.7 at 64 (a 34.1 tps run vs ~56 median) and 9.1 at 512 (a 36.5 run). Its large-context behavior is stable.
• vllm-mlx is generally stable (≤ 0.9 everywhere except a 48.9 tps outlier at 512, stddev 3.2).

Practical takeaway: small-context medians for mlx-vlm and 8,192-token medians for rapid-mlx are inflated/depressed by single outlier runs; the 5-run medians in Sections 3–5 should be read with those outliers in mind.

7. End-to-end latency

Median total request time (seconds) — TTFT + decode

At 32,768 tokens, a full request takes 10.6 s on omlx versus 53.0–53.8 s on the other three — a ~5× end-to-end difference. Because all engines decode at comparable rates (~43–45 tps at that size), the entire advantage traces back to prefill/TTFT. The three non-omlx engines are essentially indistinguishable end-to-end at large contexts (within ~0.8 s of each other at 32k).

8. Key findings

1. omlx wins the moment context grows past ~2k tokens

Prefill throughput scales to ~5,400 tps (vs ~560–680 tps plateau for rapid-mlx / mlx-vlm), making its TTFT up to ~10× lower and total latency ~5× lower at 32,768 tokens — while also delivering the highest decode tps at every size.

2. Decode is the crowded middle — everyone lands in a 44–58 tps band

Decode differences are small in absolute terms (omlx leads by 1–8 tps), and all four engines degrade by a comparable ~22–25% by 32k context. Decode throughput is not the differentiator between these engines.

3. The prefill gap is the story, and it compounds

At 32,768 tokens omlx’s advantage is 9.5× on prefill tps, 9.6–9.7× on TTFT, and ~5× on total time — because prefill time is the dominant term in a request once the prompt is large.

4. Small-context behavior flips the leaderboard

At 64 tokens, mlx-vlm has the fastest TTFT (339 ms) and fastest prefill (233 tps); omlx’s prefill starts slow (137 tps) and only wins from 512 tokens up. For short-prompt, latency-sensitive workloads mlx-vlm is competitive at the very smallest sizes.

5. Stability favors omlx at scale; watch the outliers elsewhere

omlx is near-deterministic at large contexts (stddev ≤ 0.2 tps). rapid-mlx showed a ~30% decode drop in one 8,192-token run, and mlx-vlm had two cold-start outliers at 64/512. For capacity planning, rely on medians over multiple runs.

6. Data caveats to carry forward

vllm-mlx’s prefill tps is not measurable in this harness (prompt_tokens=0 reported); its TTFT aligns with rapid-mlx / mlx-vlm, implying similar real prefill cost. omlx also streams ~37–43 (not ~256) content deltas for a 256-token completion — larger SSE chunks, identical token counts. Neither affects the decode comparisons.