Skip to content
Aixy·

Gateway benchmarks

Because speed matters.
We like it fast.

Your AI deserves a fast path. See how Aixy compares in a controlled gateway latency benchmark.

Latest benchmark run:

Less time in the gateway.

Added latency at p99Milliseconds · Lower is better
  1. Aixy 0.476 ms
  2. Bifrost 2.149 ms Aixy −77.84%
  3. Portkey OSS 7.417 ms Aixy −93.58%
  4. LiteLLM (Python) 15.551 ms Aixy −96.94%
Download results

Aixy configurations

Real controls. Measured overhead.

Compare Aixy with guardrails, budgets and rate limits across input sizes and response protocols.

Controlled Linux benchmark · Lower is better

Mean block added p99Milliseconds · Lower is better

JSON · 1 KiB · Six configurations

  1. Default0.465ms95%: 0.462–0.468
  2. Guardrails0.515ms95%: 0.513–0.517
  3. Rate limits0.723ms95%: 0.716–0.730
  4. Hard budgets0.948ms95%: 0.939–0.957
  5. Guardrails + hard budgets1.001ms95%: 0.987–1.016
  6. All controls1.337ms95%: 1.306–1.368

Mean of eight block p99s. Intervals are per-condition 95% block intervals.

Download configuration results
View all measurements

8 blocks · 36 comparisons · 384 cells · 28,955,691 measured requests, including direct controls.

Mean block added p99, in milliseconds. Eight blocks per condition.
ConfigurationJSONSSE
1 KiB16 KiB64 KiB1 KiB16 KiB64 KiB
Default0.4650.5330.7160.5250.5740.749
Guardrails0.5151.0752.8310.5751.1162.843
Hard budgets0.9480.9891.1740.9891.0021.192
Guardrails + hard budgets1.0011.5523.2681.0471.5363.290
Rate limits0.7230.7980.9830.7920.8531.025
All controls1.3371.8763.6681.3861.9083.718
Sample counts and intervals

All core blocks passed the harness-saturation and direct-control drift checks. Intervals are per-condition 95% block intervals.

Actual measured requests in each of eight blocks. Point values and intervals use milliseconds.
WorkloadConfigurationp99 (ms)95% block interval (ms)Requests per block
JSON · 1 KiBDefault0.4650.462–0.468120,431 · 118,942 · 119,486 · 119,815 · 118,495 · 120,086 · 121,068 · 120,727
JSON · 1 KiBGuardrails0.5150.513–0.517108,525 · 107,889 · 108,218 · 108,315 · 107,682 · 107,965 · 109,377 · 108,109
JSON · 1 KiBHard budgets0.9480.939–0.95775,204 · 74,695 · 76,155 · 75,883 · 76,150 · 76,395 · 76,297 · 76,001
JSON · 1 KiBGuardrails + hard budgets1.0010.987–1.01669,999 · 70,260 · 70,611 · 70,826 · 70,194 · 71,627 · 71,268 · 71,888
JSON · 1 KiBRate limits0.7230.716–0.73083,318 · 83,224 · 84,677 · 83,630 · 84,448 · 84,558 · 84,596 · 83,579
JSON · 1 KiBAll controls1.3371.306–1.36855,093 · 55,040 · 55,166 · 56,096 · 55,724 · 55,853 · 56,544 · 56,730
SSE · 1 KiBDefault0.5250.521–0.52875,728 · 76,018 · 76,275 · 76,211 · 76,678 · 76,864 · 76,194 · 76,456
SSE · 1 KiBGuardrails0.5750.573–0.57871,068 · 71,205 · 71,309 · 71,449 · 71,220 · 71,262 · 71,310 · 71,726
SSE · 1 KiBHard budgets0.9890.971–1.00754,997 · 55,609 · 55,717 · 55,881 · 55,562 · 56,343 · 56,198 · 56,258
SSE · 1 KiBGuardrails + hard budgets1.0471.032–1.06152,570 · 52,160 · 52,808 · 53,085 · 53,238 · 53,590 · 53,177 · 53,488
SSE · 1 KiBRate limits0.7920.779–0.80558,970 · 58,850 · 60,085 · 59,977 · 59,973 · 60,330 · 60,401 · 59,620
SSE · 1 KiBAll controls1.3861.364–1.40950,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
JSON · 16 KiBDefault0.5330.529–0.53797,986 · 97,610 · 98,225 · 98,193 · 98,412 · 99,670 · 99,060 · 99,239
JSON · 16 KiBGuardrails1.0751.069–1.08251,312 · 51,195 · 51,307 · 51,473 · 51,612 · 51,665 · 51,650 · 51,701
JSON · 16 KiBHard budgets0.9890.971–1.00665,959 · 65,863 · 67,019 · 66,918 · 67,315 · 67,224 · 67,274 · 67,094
JSON · 16 KiBGuardrails + hard budgets1.5521.530–1.57450,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
JSON · 16 KiBRate limits0.7980.783–0.81372,726 · 72,695 · 72,769 · 72,353 · 73,446 · 73,021 · 73,334 · 73,969
JSON · 16 KiBAll controls1.8761.854–1.89750,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 16 KiBDefault0.5740.571–0.57867,898 · 67,213 · 67,782 · 67,842 · 68,295 · 67,862 · 68,327 · 68,547
SSE · 16 KiBGuardrails1.1161.112–1.12150,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 16 KiBHard budgets1.0020.981–1.02350,799 · 50,731 · 51,038 · 51,077 · 51,187 · 51,416 · 51,509 · 51,373
SSE · 16 KiBGuardrails + hard budgets1.5361.519–1.55350,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 16 KiBRate limits0.8530.847–0.85953,899 · 53,819 · 54,100 · 54,101 · 54,274 · 54,911 · 54,654 · 54,766
SSE · 16 KiBAll controls1.9081.882–1.93350,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
JSON · 64 KiBDefault0.7160.711–0.72165,416 · 65,168 · 65,717 · 65,451 · 65,814 · 65,647 · 65,857 · 65,679
JSON · 64 KiBGuardrails2.8312.792–2.86950,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
JSON · 64 KiBHard budgets1.1741.166–1.18150,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,008
JSON · 64 KiBGuardrails + hard budgets3.2683.223–3.31350,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
JSON · 64 KiBRate limits0.9830.968–0.99752,871 · 52,476 · 53,101 · 52,929 · 53,167 · 53,179 · 53,264 · 53,115
JSON · 64 KiBAll controls3.6683.626–3.71050,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 64 KiBDefault0.7490.745–0.75350,405 · 50,639 · 50,817 · 50,630 · 50,682 · 51,036 · 51,019 · 50,924
SSE · 64 KiBGuardrails2.8432.824–2.86150,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 64 KiBHard budgets1.1921.179–1.20550,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 64 KiBGuardrails + hard budgets3.2903.251–3.32950,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 64 KiBRate limits1.0251.016–1.03450,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
SSE · 64 KiBAll controls3.7183.688–3.74850,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000 · 50,000
What each configuration includes

Default includes authentication, pricing and usage tracking. Guardrails enables inline detectors and PII redaction; fixed text inputs exercise the benign path, with enforcement and redaction checked separately before timing. Hard budgets and rate limits exercise organization, project and API-key scopes through Redis. The “All controls” configuration combines guardrails, hard budgets and rate limits.

Core inputs use a fixed text corpus. KiB counts message-content bytes, not tokens or the full HTTP body. SSE measures a streamed response; p99.9 is a descriptive point estimate. CPU covers the response window, with gateway and Redis measured separately.

Workload benchmarks

More demanding workloads.

Compare Default and All controls across larger prompts, conversation history, tool-call payloads and streaming responses.

Exploratory results · Three blocks · No confidence intervals

Large input · 256 KiB · JSON

Mean block added p99Milliseconds · Descriptive results
  1. Default1.418 ms
  2. All controls10.641 ms

Measured requests per block

Default
30,274 · 30,059 · 29,630
All controls
10,000 · 10,000 · 10,000

Mean of three block p99s · No confidence intervals

Descriptive full-response diagnostic; the client/loopback baseline is retained.

Large input · 256 KiB · SSE

16 stream frames · 0 ms provider frame interval · 0 ms client read delay

Mean block added p99Milliseconds · Descriptive results
  1. Default1.414 ms
  2. All controls10.551 ms

Measured requests per block

Default
26,683 · 26,690 · 26,571
All controls
10,000 · 10,000 · 10,000

Mean of three block p99s · No confidence intervals

Descriptive full-response diagnostic; the client/loopback baseline is retained.

Conversation history · 64 KiB · JSON

Mean block added p99Milliseconds · Descriptive results
  1. Default0.730 ms
  2. All controls3.579 ms

Measured requests per block

Default
63,381 · 62,994 · 63,189
All controls
16,607 · 16,476 · 16,496

Mean of three block p99s · No confidence intervals

Descriptive full-response diagnostic; the client/loopback baseline is retained.

Conversation history · 64 KiB · SSE

16 stream frames · 0 ms provider frame interval · 0 ms client read delay

Mean block added p99Milliseconds · Descriptive results
  1. Default0.764 ms
  2. All controls3.594 ms

Measured requests per block

Default
49,361 · 49,168 · 49,245
All controls
15,402 · 15,391 · 15,412

Mean of three block p99s · No confidence intervals

Descriptive full-response diagnostic; the client/loopback baseline is retained.

Tool-call payload · 64 KiB · JSON

Mean block added p99Milliseconds · Descriptive results
  1. Default0.754 ms
  2. All controls3.590 ms

Measured requests per block

Default
60,722 · 60,577 · 60,476
All controls
16,483 · 16,433 · 16,441

Mean of three block p99s · No confidence intervals

Descriptive full-response diagnostic; the client/loopback baseline is retained.

Tool-call payload · 64 KiB · SSE

16 stream frames · 0 ms provider frame interval · 0 ms client read delay

Mean block added p99Milliseconds · Descriptive results
  1. Default0.791 ms
  2. All controls3.656 ms

Measured requests per block

Default
47,914 · 47,535 · 47,583
All controls
15,351 · 15,301 · 15,289

Mean of three block p99s · No confidence intervals

Descriptive full-response diagnostic; the client/loopback baseline is retained.

Slow client (1 ms read delay) · 1 KiB · SSE

64 stream frames · 0 ms provider frame interval · 1 ms client read delay

Mean block added p99Milliseconds · Descriptive results
  1. Default47.295 ms
  2. All controls48.457 ms

Measured requests per block

Default
1,249 · 1,249 · 1,248
All controls
1,234 · 1,232 · 1,230

Mean of three block p99s · No confidence intervals

Includes intentional client-read pacing. These results do not isolate gateway latency or establish capacity. Reduced minimum: 100 requests per block. Tail percentiles remain exploratory; inspect the actual sample counts.

Slow client (5 ms read delay) · 1 KiB · SSE

64 stream frames · 1 ms provider frame interval · 5 ms client read delay

Mean block added p99Milliseconds · Descriptive results
  1. Default161.559 ms
  2. All controls162.605 ms

Measured requests per block

Default
263 · 263 · 263
All controls
262 · 262 · 262

Mean of three block p99s · No confidence intervals

Includes intentional provider-frame and client-read pacing. These results do not isolate gateway latency or establish capacity. Reduced minimum: 100 requests per block. Tail percentiles remain exploratory; inspect the actual sample counts.

Synthetic PII redaction · 64 KiB · JSON

Mean block added p99Milliseconds · Descriptive results
  1. Default0.721 ms
  2. All controls4.263 ms

Measured requests per block

Default
65,717 · 65,058 · 64,924
All controls
14,189 · 14,131 · 14,003

Mean of three block p99s · No confidence intervals

Synthetic email and phone data. Default forwards the original input; All controls applies the configured inline redaction.

Download workload results
What these workloads exercise

Input sizes count message content, not the full HTTP body. Conversation history uses six fixed messages, with most input text in the system message.

Tool-call payloads use five fixed messages containing a prior function call and its result, plus one tool schema. No tools or MCP servers are executed.

View all workload measurements
Three blocks per condition. Descriptive added p99 in milliseconds; actual measured requests per block.
WorkloadConfigurationp99 (ms)Requests per blockDiagnostic notes
Large input · 256 KiB · JSONDefault1.41830,274 · 30,059 · 29,630Descriptive diagnostic. Measured requests per block: 30,274, 30,059, 29,630.
Large input · 256 KiB · JSONAll controls10.64110,000 · 10,000 · 10,000Descriptive diagnostic. Measured requests per block: 10,000, 10,000, 10,000.
Large input · 256 KiB · SSEDefault1.41426,683 · 26,690 · 26,571Descriptive diagnostic. Measured requests per block: 26,683, 26,690, 26,571.
Large input · 256 KiB · SSEAll controls10.55110,000 · 10,000 · 10,000Descriptive diagnostic. Measured requests per block: 10,000, 10,000, 10,000.
Conversation history · 64 KiB · JSONDefault0.73063,381 · 62,994 · 63,189Descriptive diagnostic. Measured requests per block: 63,381, 62,994, 63,189.
Conversation history · 64 KiB · JSONAll controls3.57916,607 · 16,476 · 16,496Descriptive diagnostic. Measured requests per block: 16,607, 16,476, 16,496.
Conversation history · 64 KiB · SSEDefault0.76449,361 · 49,168 · 49,245Descriptive diagnostic. Measured requests per block: 49,361, 49,168, 49,245.
Conversation history · 64 KiB · SSEAll controls3.59415,402 · 15,391 · 15,412Descriptive diagnostic. Measured requests per block: 15,402, 15,391, 15,412.
Tool-call payload · 64 KiB · JSONDefault0.75460,722 · 60,577 · 60,476Descriptive diagnostic. Measured requests per block: 60,722, 60,577, 60,476.
Tool-call payload · 64 KiB · JSONAll controls3.59016,483 · 16,433 · 16,441Descriptive diagnostic. Measured requests per block: 16,483, 16,433, 16,441.
Tool-call payload · 64 KiB · SSEDefault0.79147,914 · 47,535 · 47,583Descriptive diagnostic. Measured requests per block: 47,914, 47,535, 47,583.
Tool-call payload · 64 KiB · SSEAll controls3.65615,351 · 15,301 · 15,289Descriptive diagnostic. Measured requests per block: 15,351, 15,301, 15,289.
Slow client (1 ms read delay) · 1 KiB · SSEDefault47.2951,249 · 1,249 · 1,248Descriptive diagnostic. Measured requests per block: 1,249, 1,249, 1,248. Reduced sampling minimum; exploratory tail percentiles.
Slow client (1 ms read delay) · 1 KiB · SSEAll controls48.4571,234 · 1,232 · 1,230Descriptive diagnostic. Measured requests per block: 1,234, 1,232, 1,230. Reduced sampling minimum; exploratory tail percentiles.
Slow client (5 ms read delay) · 1 KiB · SSEDefault161.559263 · 263 · 263Descriptive diagnostic. Measured requests per block: 263, 263, 263. Reduced sampling minimum; exploratory tail percentiles.
Slow client (5 ms read delay) · 1 KiB · SSEAll controls162.605262 · 262 · 262Descriptive diagnostic. Measured requests per block: 262, 262, 262. Reduced sampling minimum; exploratory tail percentiles.
Synthetic PII redaction · 64 KiB · JSONDefault0.72165,717 · 65,058 · 64,924Descriptive diagnostic. Measured requests per block: 65,717, 65,058, 64,924.
Synthetic PII redaction · 64 KiB · JSONAll controls4.26314,189 · 14,131 · 14,003Descriptive diagnostic. Measured requests per block: 14,189, 14,131, 14,003.
How we measured

Each gateway received one core and 4 GiB of memory on the same arm64 Linux host. Eight balanced blocks used direct-to-provider controls before and after each gateway, with at least 50,000 requests and 60 seconds per cell. Added latency is each client duration minus its matching provider duration, including the client and loopback baseline. Results are the mean of eight block p99s, not a pooled p99; Aixy’s 95% block interval is 0.474–0.479 ms.

Aixy’s native fixture exercises forwarding and usage components. This JSON test compares the configured gateways, including standard Python LiteLLM; it does not measure production or streaming performance. Enabled controls differ by gateway; the download includes their profiles.

The separate Aixy configuration matrix compares six native configurations across fixed 1, 16 and 64 KiB text inputs, JSON and SSE. Its eight near-balanced blocks use grouped direct controls around fresh configuration lifecycles; core results are means of block percentiles. The fixture uses native controls and bounded local usage acknowledgements. The four-gateway comparison above remains a separate campaign. Extended workloads are descriptive three-block diagnostics.