Penumbra Tech
← All projects
[Case study: Performance engineering]

Diagnostics & load-testing dashboard.

A button-driven k6 test runner exposed as an in-browser dashboard. Server-Sent Events stream latency percentiles, CPU, and memory in real time. Runtime toggles let an operator flip rate limiting or maintenance mode without restarting the backend. The charts below are pulled from real test runs I took during development of this dashboard.

Node.jsExpressSSEk6MySQLPM2SVG charts
[Run 1: API overload]

Graceful degradation

Ramping 50 → 200 → 500 concurrent VUs over four minutes against a real DB-backed endpoint with the rate limiter intentionally off. Throughput plateaus around 944 req/s, latency climbs as the queue grows, but every request still completes. This is the "soft" failure mode: the box gets slow, not broken.

Requests / sec

0.002645297931.1k00:0002:0004:00
req/s
failed/s

Latency (ms)

0ms133ms266ms398ms531ms00:0002:0004:00
mean
p50
p95

Server CPU & memory (%)

0%25%50%75%100%00:0002:0004:00
CPU %
Mem %

Final summary

MEASURED · 240s RUN
Total requests
226,671
Failures
0 (0.00%)
Avg req/s
944.4
p50
294 ms
p95
553 ms
Max
1280 ms
[Run 2: Event-loop block]

Hard failure

Same client load, different endpoint — one that synchronously busy-waits 100 ms per request. Node is single-threaded, so requests queue head-of-line behind the blocked handler. Throughput pins at exactly 1 / 100 ms = ~10 req/s regardless of how many VUs we add; latency climbs linearly with the queue depth and reaches ~12 seconds.

Requests / sec

0.002.85.68.41100:0001:2102:45
req/s
failed/s

Latency (ms)

0ms3259ms6518ms9776ms13035ms00:0001:2102:45
mean
p50
p95

Server CPU & memory (%)

0%25%50%75%100%00:0001:2102:45
CPU %
Mem %

Final summary

MEASURED · 165s RUN
Total requests
1,629
Failures
0 (0.00%)
Avg req/s
9.9
p50
5880 ms
p95
11830 ms
Max
12130 ms

Why this matters

Most stacks have two different failure modes — CPU saturation looks graceful, event-loop blocking looks catastrophic. A dashboard that surfaces both makes it possible to know which one your service is in BEFORE pager fatigue sets in.

What I built into it

Live percentiles streamed over SSE (the wire format is one JSON event per second), a 128-bit per-run bypass token so test traffic can route past maintenance mode, runtime rate-limit and maintenance toggles, copy-log button, automatic teardown if anything goes sideways.

What this engagement looks like

Drop in for a fixed-scope project to instrument an existing service the same way: real load tests run from the GUI, real numbers in CloudWatch / your monitoring of choice, a written post-mortem for each bottleneck so the team doesn't have to rediscover it next quarter.