Load Testing

Glubean runs load tests — concurrent, sustained traffic against an API to measure latency percentiles, throughput, and error rate under pressure — as a first-class primitive alongside test, contract, and workflow. Load plans live in *.load.ts files, import from @glubean/sdk/load, run with glubean load, and upload as kind=load runs that produce a rich load artifact (per-endpoint/step breakdown, latency distribution, timeline, threshold pass/fail, bounded failure samples).

This is distinct from recording a single-request metric inside a functional test. Load testing measures aggregate behavior under concurrency, not one request in isolation.

When to reach for it

Write a load test when you’re asking about performance under load, capacity, throughput/RPS, p95/p99 latency, “can this handle N concurrent users,” soak/stress behavior, or gating a performance SLO in CI. Load is opt-in — it generates real concurrent traffic, so don’t add it as part of routine functional/contract coverage.

The load generator — what it is and isn’t

Glubean’s local load engine is a single Node.js process driving all virtual users on one event loop (the artifact records this as runtime.processModel: "single-process-async-producer-slot"). Know what that model buys you and where its limits are — a saturated generator produces numbers that look exactly like real measurements but describe the generator, not your API.

IO-intense: yes. An event loop excels at concurrent IO — thousands of in-flight HTTP requests on one core is the normal regime. High concurrency values are fine as long as each iteration spends its time waiting on the network.

CPU-intense: no. Every millisecond of synchronous compute in a scenario step — heavy crypto, large JSON transforms, compression, image work — runs on the same single core that schedules all virtual users. CPU-heavy iterations stall the event loop, which throttles your own ingress and inflates every measured latency: the recorded p95 silently becomes generator queueing time, indistinguishable from backend latency in the report. If an iteration genuinely needs heavy computation:

• pre-compute data once at module load and feed it through feeders, instead of generating per iteration;
• move the computation behind a service and call it over the wire — that turns CPU work into IO the generator handles well;
• keep scenario bodies thin: request, assert status/shape, record, done.

One core today. Concurrency (in-flight requests) can go high, but total generator throughput is bounded by a single core. Multi-core and distributed load generation are on the roadmap; until then, treat very-high-RPS targets as a generator capacity question, not just a backend one.

The host matters too. Load generation leans on the whole machine, not just the CPU:

• Memory — every in-flight iteration holds request/response buffers; RSS scales with concurrency × response size. Watch it on soak runs.
• File descriptors — each concurrent connection is an open fd. High concurrency on default limits exhausts them (EMFILE); raise ulimit -n before blaming the backend for connection errors.
• Network path — ephemeral ports, keep-alive reuse, VPN/proxy hops, and plain bandwidth all cap what the generator can deliver. Run it close to the system under test, not through a corporate VPN.

Rule of thumb: keep an eye on the generator host (CPU, memory, fd count) during a run. If the generator is saturated, fix that before reading the numbers.

Define a load plan

A load plan is a scenario (the workload, reused across runs) plus a runner (concurrency, duration, thresholds). Import load authoring APIs from @glubean/sdk/load:

import { loadScenario, loadRunner, feeder } from "@glubean/sdk/load";
 
// 1. Scenario — the steps each virtual user repeats. Steps run in order;
//    each step's requests are attributed to its endpoint + step in the report.
const shopScenario = loadScenario<{ q: string }>("shop")
  .step("browse", async (ctx) => {
    const res = await ctx.http.get("load/catalog", { searchParams: { q: ctx.input.q } });
    ctx.expect(res).toHaveStatus(200);
  })
  .step("checkout", async (ctx) => {
    const res = await ctx.http.post("load/checkout", { json: { items: [{ sku: "A1", qty: 1 }] } });
    ctx.expect(res).toHaveStatus(201);
  });
 
// 2. Runner — concurrency + duration + thresholds. Export it so the CLI finds it.
export const shopLoad = loadRunner("shop-load", {
  scenario: shopScenario,
  concurrency: 25,                 // virtual users in flight
  duration: "20s",                 // "60s" / "2m" / ms — total run length
  rampUp: "3s",                     // optional: ramp concurrency up over this window
  feeders: { term: feeder.fromArray([{ q: "pen" }, { q: "notebook" }], { key: "q" }).roundRobin() },
  input: ({ feed }) => ({ q: (feed.term as { q: string }).q }),   // per-iteration input
  thresholds: {
    transaction: { errorRate: "<5%", p95: "<400ms" },             // whole-scenario gate
    endpoints: {
      "GET /load/catalog":  { p95: "<120ms", p99: "<400ms", errorRate: "<0.5%" },
      "POST /load/checkout": { p95: "<180ms", errorRate: "<4%" },
    },
  },
  report: { failureTraces: 10, slowTransactionSummaries: 10 },   // bounded sample caps
});

LoadContext

A load step’s ctx is a LoadContext, not the full TestContext from a functional test — some APIs don’t make sense under concurrency.

Kept: http, vars, secrets, session, expect, assert, skip, fail, log, warn, pollUntil, setTimeout.

Added for load: input (per-iteration input from feeders), iteration ({ id, index }), producerSlot ({ id, index }), now(), report (checkpoint / primaryComplete), and metrics (custom metric handles declared on the runner).

Removed: validate, trace, metric, action, event, getMemoryUsage, retryCount — these don’t fit concurrent load. Use a functional test if you need ctx.metric() for a custom business metric.

Config keys

Thresholds — the pass/fail gate

Thresholds turn a load run green or red, and are what gates CI. They apply at three scopes:

thresholds: {
  transaction: { errorRate: "<1%", p95: "<2000ms", p99: "<4000ms", throughputPerSec: ">500" },
  endpoints: { "GET /api/foo": { p95: "<200ms", errorRate: "<0.5%" } },
  steps:     { "checkout":    { p95: "<800ms" } },
}

Expressions are comparison strings: "<400ms", "<5%", ">500". Each evaluated threshold lands in the artifact’s summary.thresholds[] with its actual value and pass verdict.

A threshold-failed load run is still valid data. Failing a p95 gate turns the run red, but the numbers it measured are real and comparable — Performance reads it like any other completed run.

Async scenarios that declare a primary/continuation boundary get three more scopes — primary, endToEnd, and continuation — explained next.

Async workloads — primaryComplete and the continuation phase

By default a load run is a closed loop: each of the concurrency producer slots runs iterations back-to-back, and a slot only starts its next iteration when the previous one finishes. For request/response APIs that’s the right model — but for async APIs (submit a job, get a 202, poll until the result is ready) it silently breaks in two ways:

1. One number conflates two different SLAs. The iteration’s latency is the submit plus the whole poll wait, so p95 measures the backend job’s end-to-end time — not the submit endpoint’s latency under pressure. You can’t gate the two separately.
2. The test under-pressures the system. While a slot sits in a poll loop, it generates no new ingress. The slower the backend drains jobs, the less submit traffic the run produces — pressure adapts to the system instead of stressing it, which inverts what a load test is for. (The runtime detects this shape — a long poll with no declared boundary — and emits a summary.advisories[] warning.)

ctx.report.primaryComplete(id) fixes both. It declares the measurement boundary between the primary phase (the load you’re actually pushing — the submit) and the continuation phase (everything after — typically the poll-for-result loop):

const jobScenario = loadScenario("async-job")
  .step("submit and await", async (ctx) => {
    const res = await ctx.http.post("jobs", { json: { kind: "export" } });
    ctx.expect(res).toHaveStatus(202);
    const { id } = await res.json<{ id: string }>();
 
    // Primary load delivered — split the metrics here, and free this slot
    // to submit the next job while this iteration keeps polling.
    await ctx.report.primaryComplete("submitted", { releaseProducerSlot: true });
 
    await ctx.pollUntil({ timeoutMs: 30_000, intervalMs: 500 }, async () => {
      const status = await ctx.http.get(`jobs/${id}`, {
        context: { glubeanRoute: "GET /jobs/:id" },
      });
      return (await status.json<{ done: boolean }>()).done;
    });
  });

It has two effects, deliberately separable:

1. Metric split — always. From the first primaryComplete in an iteration, the runtime tags every request, assertion, and step with phase: "primary" | "continuation" (this is a runtime switch, not a user tag). The artifact then reports summary.primary (producer side: submit throughput and latency) and summary.endToEnd (the full logical iteration) separately, and thresholds gain matching scopes:

thresholds: {
  primary:      { p95: "<150ms", throughputPerSec: ">200" },  // the ingest SLA
  endToEnd:     { p95: "<30000ms", errorRate: "<1%" },        // the job SLA
  continuation: { backlog: "<500", backpressureMs: "<1000" }, // test-health gate
}

2. Slot release — opt-in. Pass releaseProducerSlot: true to also free the producer slot immediately: the runner back-fills a new primary iteration on that slot while the continuation finishes in the background. This turns the closed loop into a bounded-open model — submit pressure stays constant even when completions are slow, which is exactly the regime where an async system saturates.

“Bounded” is enforced by the runner’s continuation config. Without a bound, a backend that stops completing jobs would accumulate unbounded in-flight polls:

This is also why primaryComplete must always be awaited: under block-producer with a full backlog, it is the backpressure point. It resolves with a receipt — { measuredPrimaryComplete, releasedProducerSlot, duplicate, backpressureMs } — and duplicate calls within one iteration are ignored for scheduling (the first one wins).

When producer release is used, the artifact adds summary.continuation (backlog peak, queue wait and backpressure percentiles, release coverage) so a red run can be attributed: was the system slow, or was the test starving itself?

Rules of thumb:

• Request/response scenario — don’t call it. The closed loop and plain transaction thresholds are the correct model.
• Async scenario, you care about end-to-end time — call primaryComplete without release: you get the metric split; scheduling stays closed.
• Async scenario, you’re asking a capacity/saturation question — release the slot and set continuation.maxOutstanding, so ingress pressure stays constant and bounded.

(ctx.report.checkpoint(id) is the boring sibling: pure report attribution, no effect on phases or scheduling.)

Stable endpoint keys

Performance and Benchmarks group load evidence by endpoint key. Static paths are easy, but dynamic load URLs can otherwise fragment into many literal endpoints.

When a load step hits a templated endpoint, pass the canonical route through the non-wire context.glubeanRoute option:

const itemScenario = loadScenario<{ id: string }>("item")
  .step("get item", async (ctx) => {
    const res = await ctx.http.get(`items/${ctx.input.id}`, {
      context: { glubeanRoute: "GET /items/:id" },
    });
 
    ctx.expect(res).toHaveStatus(200);
  });

The route key is used only by Glubean reporting; it is not sent to the system under test. Contracts add this metadata automatically. Load scenarios are raw runtime workloads, so declare it when you want clean per-endpoint comparisons.

Custom metrics

Use load custom metrics when the SLA is not only HTTP latency. Declare metrics on the runner, record them in scenario steps through ctx.metrics, and gate them with thresholds.customMetric:

import { counter, loadRunner, rate, trend } from "@glubean/sdk/load";
 
export const asyncJobLoad = loadRunner("async-job", {
  scenario: asyncJobScenario,
  concurrency: 50,
  duration: "60s",
  metrics: {
    pollOk: rate(),
    e2eLatency: trend({ unit: "ms" }),
    retries: counter(),
  },
  thresholds: {
    customMetric: {
      pollOk: { rate: ">99%" },
      e2eLatency: { p95: "<2000ms" },
      retries: { sum: "<100" },
    },
  },
});

Inside the scenario:

ctx.metrics.pollOk.add(true);
ctx.metrics.e2eLatency.add(ctx.now() - startedAt);
ctx.metrics.retries.add();

Custom metric summaries are folded into the load artifact and appear alongside ordinary p95, throughput, and error-rate evidence.

Traffic mix & table expansion

Run multiple weighted scenarios in one plan:

import { loadMixEntry } from "@glubean/sdk/load";
 
export const mixedLoad = loadRunner("mixed", {
  scenarios: [
    loadMixEntry({ id: "checkout", scenario: checkoutScenario, weight: 70 }),
    loadMixEntry({ id: "refund",   scenario: refundScenario,   weight: 30 }),
  ],
  concurrency: 300,
  duration: "60s",
});

Or expand one plan per data row (e.g. per region):

export const regionLoad = loadRunner.each(REGIONS)("region-{region}", (row) => ({
  scenario: scenarioForRegion(row.region),
  concurrency: 100,
  duration: "30s",
}));

Run & upload

Ad hoc, by explicit target (file, dir, or glob):

glubean load                             # Discover and run *.load.ts under the project
glubean load tests/load/shop.load.ts     # Run one plan ad hoc
glubean load tests/load/                 # Run every *.load.ts under a directory
glubean load tests/load/ --upload        # Upload each completed load artifact
glubean load tests/load/ --upload-target tgt_abc   # Override the upload target

Or repeatably, through glubean.yaml profiles. Declare named plans in a top-level load: block and reference them from a profile:

load:
  plans:
    shop:   { target: tests/load/shop.load.ts }
    search: { target: tests/load/search.load.ts }
 
profiles:
  perf:
    load: { plans: [shop, search] }      # a load-only profile needs no suites
    envFile: .env.staging
    upload:
      enabled: true                      # auto-upload, no --upload flag needed
      projectId: prj_abc123
      targetId: tgt_abc123
      tokenEnv: GLUBEAN_TOKEN_STAGING

glubean load --profile perf              # Run every plan in the profile
glubean load --profile perf --plan shop  # Narrow to one named plan

The profile carries envFile and the upload directive (destination project / target / token env var), same as glubean run --profile; explicit CLI flags still override. Overlapping plan targets are deduped, so a file matched by two plans runs once.

Division of labor: the loadRunner(...) owns the traffic shape — virtual-user concurrency, duration, mix, thresholds, and the emitted LoadArtifact. glubean.yaml owns which plans a profile runs and where their evidence goes. CLI flags override per invocation.

See the CLI Reference for command flags and Upload Your First Run for auth/upload setup.

What a load run produces

A load run’s artifact (schemaVersion: "glubean.load.v1") is what the Cloud dashboard renders:

• summary — pass (threshold verdict), totalIterations, successfulIterations, failedIterations, errorRate, throughputPerSec, latency ({ p50, p90, p95, p99, max } ms), latencyDistribution[] (fixed-ladder histogram), thresholds[] (each { scope, metric, expression, actual, pass }), advisories[].
• endpoints[] — per endpoint (routeKey, method): requestCount, errorRate, latency, throughputPerSec, latencyDistribution, statusCounts.
• steps[] — per scenario step: invocationCount, errorRate, latency.
• scenarios[] — per scenario aggregate.
• matrix[] — scenario-step × endpoint (which endpoint each step hit).
• timeline — fixed-width windows over the run: { offsetMs, requests, errors, throughputPerSec, latency, iterations, peakInFlight }.
• summary.customMetrics[] — folded rate, trend, and counter metrics when declared.
• samples — bounded failureTraces[] (failed iterations) and slowTransactions[] (slow-but-passing), capped by report.

These surface in the target’s Performance tab (capacity, trend, thresholds, and run-vs-run comparison), the Runs tab (per-run detail: config, KPI cards, over-time timeline, latency histogram, per-endpoint table, transaction breakdown, threshold table, samples), and Benchmarks (cross-source comparison).

Performance vs Benchmarks

Both Cloud surfaces read the same uploaded kind=load evidence. The difference is the comparison boundary:

Use Performance for drift, baselines, capacity, and run A vs run B inside one target. Use Benchmarks when the source itself changes: Express vs Bun, two deployments, or several services that implement the same API.

Benchmarks in Cloud

The loadRunner id is also the anchor plan id for Cloud Benchmarks. If you want to compare several implementations of the same API, upload load runs from separate source targets using the same authored runner id:

export const shopLoad = loadRunner("shop-load", {
  scenario: shopScenario,
  concurrency: Number(process.env.LOAD_CONCURRENCY ?? 500),
  duration: "60s",
});

Then create a benchmark in Cloud with anchor plan shop-load and attach the source targets, such as Express API and Bun API.

For cleaner benchmark deltas, keep the load shape consistent across sources: same runner id, same duration or iteration bound, same scenario mix, and comparable environment labels. Cloud still renders warned comparisons, but it surfaces the warning and the compared run configs instead of silently hiding or equating them.

Best practices

• Start small. Default to modest concurrency and a short duration (e.g. 25 virtual users / "20s") and scale up from there — don’t open with 1000 VUs.
• Always set thresholds so the run has a pass/fail verdict and can gate CI. Base them on your SLOs, or start conservative and tighten over time.
• Reuse functional knowledge. A load scenario makes the same http calls as a functional test, just without per-request assertions beyond status/shape.
• Keep iterations IO-thin. No heavy synchronous compute inside scenario steps — see the generator model for why CPU work poisons the measurements.
• Don’t call ctx.metric() in a load step — it isn’t available there. For a custom business metric, use a functional test instead.

Next

• Performance — read one target over time: trend, capacity, thresholds, run comparison
• Benchmarks — compare several source targets under the same load plan
• CLI Reference — glubean load command and flags
• Upload to Cloud — token, project, and target setup