Example: LLM gateway (SQLite)

examples/llm-gateway: an LLM gateway on SQLite. Postpaid; commit never checks the balance.

bunx vitest run examples/llm-gateway

examples/llm-gateway/src/gateway.ts

/**
 * An LLM gateway on SQLite.
 *
 * - Tokens are priced per million with `cumulative` rounding, so a month of tiny calls never drifts.
 * - Calls are buffered per (account, model) and flushed once a minute as one `observe` (ref type "batch").
 * - Output tokens feed a credits pool with a per-model weight; credits are priced by volume with `on_crossing`.
 * - Every account gets its first 1M input tokens per model free, for life (usedSoFar over all time).
 * - A CRM discount from the context is applied in getRate. Postpaid: commit never checks balances.
 */

Tariffs

examples/llm-gateway/src/gateway.ts

const perMillion = (price: number, id: string): Rate => ({
  id,
  model: "graduated",
  tiers: [
    { from: 0, unitPriceMicroUsd: 0, per: 1_000_000 },
    { from: LIFETIME_FREE_INPUT, unitPriceMicroUsd: price, per: 1_000_000 },
  ],
  policy: { rounding: "cumulative", tierPeriod: "lifetime" },
});

/** Credits: $0.02 per 1k credits, $0.015 once the month passes 10M credits — for the whole month. */
export const CREDIT_RATE: Rate = {
  id: "credits-2026",
  model: "volume",
  tiers: [
    { from: 0, unitPriceMicroUsd: 20_000, per: 1000 },
    { from: 10_000_000, unitPriceMicroUsd: 15_000, per: 1000 },
  ],
  policy: { adjustmentTiming: "on_crossing", rounding: "cumulative", tierPeriod: "month" },
};

• Input tokens are priced per million (per: 1_000_000) with cumulative rounding: rounding each call up would make every tiny call cost at least 1 micro-USD; cumulative rounding keeps the month’s total exact (B2).
• The first 1M input tokens per model are free for life: usedSoFar is counted over all time (tierPeriod: "lifetime", B16).
• Output tokens are only counted (price 0) and feed the credits pool with a per-model weight (opus 5, sonnet 3, haiku 1). Money flows through credits.
• Credits are priced by volume + on_crossing: once the month passes 10M credits, the whole month is at the cheaper tier, and the difference for earlier credits is a separate …:adj item (B6).

Catalog and getRate

examples/llm-gateway/src/gateway.ts

const metering = buildMetering()
  .context(Context)
  .refs(["request", "batch"])
  .meter("llm/credits")
  .meter("llm/input", { model: z.enum(MODELS) })
  .meter("llm/output", { model: z.enum(MODELS) }, { feeds: { "llm/credits": (d) => CREDIT_WEIGHT[d.model] } })
  .getRate(async (meter, dims, ctx, at) => {
    const discount = (r: Rate) => (ctx.discountPct ? applyDiscount({ rate: r, percent: ctx.discountPct }) : r);
    if (meter === "llm/input") {
      // lifetime quota: the counter spans all time
      const usedSoFar = store.usedSoFar({ account: ctx.accountId, meter, dims });
      return { rate: discount(perMillion(INPUT_PRICE[dims.model], `input-${dims.model}`)), usedSoFar };
    }
    if (meter === "llm/output") {
      // output tokens only count; money flows through credits
      return { rate: { model: "graduated", tiers: [{ from: 0, unitPriceMicroUsd: 0 }] } };
    }
    const { from, to } = monthWindow({ at });
    return { rate: discount(CREDIT_RATE), usedSoFar: store.usedSoFar({ account: ctx.accountId, meter, from, to }) };
  })
  .commit(store.commit);

The CRM discount comes from the context and is applied to every tariff with applyDiscount inside getRate (B19). The monthly credits counter is read over the monthWindow({ at }) window.

Per-minute buffer

examples/llm-gateway/src/gateway.ts

/** Flush every minute that ended before `now`. Safe to call repeatedly: batch refs are idempotent. */
async flush(now: number) {
  const results = [];
  for (const b of buffer.drain(now)) {
    const r = await metering.observe(
      [
        { meter: "llm/input", dims: { model: b.model }, quantity: b.inputTokens },
        { meter: "llm/output", dims: { model: b.model }, quantity: b.outputTokens },
      ],
      { type: "batch", id: `${b.ctx.accountId}:${b.model}:${b.minute}` },
      b.ctx,
      { at: b.minute },
    );
    if (!r.ok) buffer.restore(b); // try again on the next flush
    results.push({ batch: b, result: r });
  }
  return results;
},

Calls are buffered per (account, model, minute) and written as one multi-line observe when the minute is over. The ref is { type: "batch", id: "{account}:{model}:{minute}" }, so flush can run again: the same batch is a no-op in commit. A failed batch goes back into the buffer. In production the buffer is a Durable Object or Redis; the pattern is the same. Buffering and scheduling are the application’s job, not the library’s.