Instrumenting a Fail-Closed License Server

A commercial PDF library refuses to write a file when its license server says no. That makes server downtime a customer outage — so the telemetry had to answer "is anyone being blocked right now?" on two small VMs, with no SaaS bill and no public network path.

Why a licensing service raises the stakes on telemetry

The product is an accessible-PDF generation library. Every time a customer's process serializes a document, a baked-in guard calls POST /v1/authorize to reserve page quota and verifies an Ed25519 signature on the response. The guard is deliberately fail-closed: no valid grant, no PDF.

That single design decision sets the observability requirements. A web app that flickers loses a page view. Here, a wedged connection pool or a rate limiter collapsing into one shared bucket stops customers producing documents in their own pipelines — and they experience it as our library is broken, not as our server being down. Four questions had to be answerable in seconds: are both instances serving, is Postgres reachable, are grants failing, and is the background job that expires trials still running?

Stage 1 — Instrument: a metrics registry with no dependencies

The API is Hono on Node behind nginx. Rather than pull in a Prometheus client, the whole registry is ~120 lines of module-scoped counters rendered as exposition text on demand. Two reasons: the metric set is small and known, and a licensing server is exactly where I don't want another transitive dependency tree in the request path.

The one part worth care is latency — fixed millisecond buckets, made cumulative at render time, so histogram_quantile works over it without a library owning it.

// Latency histogram: fixed millisecond buckets (cumulative at export).
const LATENCY_BUCKETS_MS = [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000];
const latencyCounts = new Array<number>(LATENCY_BUCKETS_MS.length + 1).fill(0); // +Inf
 
export function recordRequest(status: number, durationMs: number): void {
  requestsTotal++;
  const cls = `${Math.floor(status / 100)}xx`;
  if (cls in statusClass) statusClass[cls]!++;
  latencySumMs += durationMs;
  let placed = false;
  for (let i = 0; i < LATENCY_BUCKETS_MS.length; i++) {
    if (durationMs <= LATENCY_BUCKETS_MS[i]!) { latencyCounts[i]!++; placed = true; break; }
  }
  if (!placed) latencyCounts[LATENCY_BUCKETS_MS.length]!++;
}

Counters are per-instance by construction. Nothing tries to be clever about that — aggregation is Prometheus' job, and every dashboard query either sums across the host label or breaks out by it on purpose.

Two of the eleven series are not request counters, and they are the two I actually watch:

• license_pg_pool_connections{state="total|idle|waiting"} — read straight off the node-postgres pool at render time. waiting > 0 means requests are queued behind connections, which on a fail-closed path is a customer-visible stall.
• license_sweep_age_seconds — seconds since the last successful run of the hourly notification sweep, from a sweep_runs table. This is the one metric that reports on a cron job rather than on the process exposing it, which is the only way a job that silently stops running becomes visible.

Health endpoints that mean different things

/healthz is liveness with no dependencies: the process answers. /readyz runs SELECT 1 and returns 503 when the database is gone, so the load balancer drains an instance whose Postgres connection is dead instead of feeding it traffic. Sweep staleness rides along in the readiness body but deliberately does not fail readiness — a stale sweep means notifications are late, not that the API should stop issuing grants.

app.get("/readyz", async (c) => {
  try { await pool.query("SELECT 1"); }
  catch { return c.json({ ready: false, db: "down" }, 503); }
 
  // Staleness is reported for alerting, not enforced as readiness.
  const sweep = await sweepFreshness();
  const sweepStale = env.sweepStaleSeconds > 0
    && sweep.ageSeconds !== null
    && sweep.ageSeconds > env.sweepStaleSeconds;
  return c.json({ ready: true, db: "up", sweep: { ageSeconds: sweep.ageSeconds, stale: sweepStale } });
});

Stage 2 — Log: structured lines that can never leak a license key

Request logging is one middleware emitting JSON lines. The constraint that shaped it: this service authenticates with bearer license keys and signs grants with an Ed25519 private key. A log line that echoes a request body or an Authorization header is a credential leak with a 30-day retention policy attached.

So the logger touches five fields — method, path, status, duration, request id — and nothing else. Not the body, not headers, not the query string (paths only, since query strings can carry identifiers).

// Prometheus scrapes /metrics every 30s; the LB hits /readyz every ~10s.
// Infra plumbing, not application events — excluded so the stream stays readable.
const UNLOGGED_PATHS = new Set(["/metrics", "/healthz", "/readyz"]);
 
export const requestLogger: MiddlewareHandler = async (c, next) => {
  const start = performance.now();
  incInFlight();
  try { await next(); }
  finally {
    const durationMs = Math.round((performance.now() - start) * 100) / 100;
    decInFlight();
    recordRequest(c.res.status, durationMs);       // metrics still count these
    if (UNLOGGED_PATHS.has(c.req.path)) return;    // only the log line is skipped
    let level = "info";
    if (c.res.status >= 500) level = "error";
    else if (c.res.status >= 400) level = "warn";
    console.log(JSON.stringify({ level, msg: "request", method: c.req.method,
      path: c.req.path, status: c.res.status, durationMs, requestId: c.get("requestId") }));
  }
};

Before this split, roughly nine in ten log lines were the load balancer and Prometheus talking to themselves — and that noise hid the pattern in the second case study below.

Stage 3 — Collect: push, because the hosts have no inbound path

Both application instances sit on a private subnet with no public IP; egress goes through a gateway, ingress only through the load balancer on port 80. A central Prometheus scraping those hosts would need an inbound route that deliberately doesn't exist. So collection is push: Grafana Alloy runs as a third container next to api and web on each host, tails container logs to Loki, scrapes the API locally over the Docker network, and remote-writes the result.

/metrics is therefore never public — nginx doesn't proxy the path (a request for it on the public domain returns the dashboard's HTML fallback), and it can be additionally gated by METRICS_TOKEN, compared with timingSafeEqual.

discovery.docker "containers" { host = "unix:///var/run/docker.sock" }
 
discovery.relabel "containers" {
  targets = discovery.docker.containers.targets
  // "/license-api" → container="license-api"
  rule { source_labels = ["__meta_docker_container_name"], regex = "/(.*)", target_label = "container" }
}
 
loki.source.docker "logs" {
  host          = "unix:///var/run/docker.sock"
  targets       = discovery.docker.containers.targets
  relabel_rules = discovery.relabel.containers.rules
  forward_to    = [loki.write.monitoring.receiver]
}
 
prometheus.scrape "license_api" {
  job_name        = "license-api"
  scrape_interval = "30s"
  targets         = [{ __address__ = "api:8787" }]   // docker network, never public
  forward_to      = [prometheus.remote_write.monitoring.receiver]
}
 
// Both writers stamp the same identity labels.
external_labels = { host = sys.env("HOST_TAG"), namespace = "license" }

One HOST_TAG per instance is the whole multi-host story: every log line and every metric series carries host="app-1" or "app-2", so "is this one host or both?" is a label selector rather than an investigation.

logs → Loki push · metrics → remote-write · alerts → SMTP

Why not SigNoz, and why not a SaaS

The first cut shipped an OpenTelemetry collector into SigNoz. It went for two reasons. Running a ClickHouse-backed platform to observe two small VMs inverted the resource ratio — the observability stack was the heaviest thing in the deployment. And the traces it existed to collect weren't earning their keep: a handful of endpoints over one database, where a p99 histogram plus per-endpoint log timings answered every question a trace would have.

A hosted vendor was the other option, and for a service whose selling point is that customer documents never leave their infrastructure, shipping operational logs to a third party is an awkward story in a vendor security questionnaire. Self-hosting Alloy + Loki + Prometheus + Grafana keeps the data on the same private network as the service.

Stage 4 — Store and surface: retention, provisioning, gotchas

Loki runs single-node on the filesystem with TSDB indexes, 30-day retention, and the compactor doing the deleting. Prometheus keeps 90 days and scrapes nothing at all — it runs with --web.enable-remote-write-receiver purely as a write target. That asymmetry is intentional: logs are for reading during an incident, metrics are for trend lines and alert evaluation.

Everything Grafana shows is provisioned from files in the repository: both datasources, the twelve-panel overview dashboard, the contact point, the notification policy, and the alert rules. The repo is the source of truth, so UI edits don't survive — worth stating out loud to anyone else touching it.

The or vector(0) in the last rule is the detail I care about most. Without it, an instance whose Alloy stops pushing produces no series rather than a zero, and a naive count(up == 1) < 2 evaluates to no-data instead of firing. Paired with noDataState: Alerting on the sweep rule, the principle is the same both times: absence of telemetry is itself an alertable condition.

Stage 5 — What it caught: three bugs, three different lanes

The point of the build isn't the dashboard screenshot. Within days of turning it on, these three landed — and two were invisible from the application's own point of view.

Operating it: the queries I actually type

Grafana is reachable only over an SSH tunnel (ssh -L 3000:<private-host>:3000 app-1), which is deliberate friction: an ops UI with a password on a public port is a weekly scanning target.

# Is it one host or both?
up{job="license-api"}
 
# Error budget in human terms: 5xx per second, per host
sum by(host) (rate(license_responses_total{class="5xx"}[5m]))
 
# Latency shape, not the average
histogram_quantile(0.95, sum by(le) (rate(license_request_duration_ms_bucket[5m])))
 
# Is anything queuing for a database connection?
license_pg_pool_connections{state="waiting"}

# Errors and warnings across every container on one host
{host="app-2"} |= "level" | json | level =~ "error|warn"
 
# One request end to end, by id
{container="license-api"} | json | requestId = "<uuid>"

The request id is generated per request and included in every log line, which is what makes the last query possible without distributed tracing.

Retrospective

Keep: the hand-rolled registry (it has never been the thing that broke), push-based collection for private hosts, provisioning everything from the repo, and metrics that describe jobs rather than only the process exposing them. The sweep-age gauge is the highest-value series in the set — the only one that can tell me a cron job died.

Still open, in the order I'd do them:

1. An external check on /readyz. Everything today sits inside the perimeter it observes; a DNS or balancer failure is invisible to it. Biggest remaining hole, cheapest to close.
2. Move the monitoring stack off an application host. Documented above as a known trade-off, not an oversight.
3. A path label on the latency histogram. Per-endpoint p95 currently needs Loki timings; a bounded label set (route patterns, not raw paths) would put it where the alerts live.
4. Alert on grant refusals, not just faults. A customer hitting quota is a 402, not a 5xx — invisible to the current rules, and commercially the most interesting event the system produces.

Stack: Grafana Alloy 1.17 · Loki 3.7 · Prometheus 3.13 · Grafana 13.1 · Node 24 · Hono · Postgres 18 · Docker Compose on two small cloud VMs. Metrics registry, log middleware, Alloy config, dashboards and alert rules all live in the application repository and deploy with it.