Release Qualification Gate (Spec 081)
The release qualification gate is a tag-blocking QA harness: on a release tag it runs the existing test suites plus a new server-type matrix and a set of end-to-end invariants, then produces one machine-readable pass/fail verdict. Artifact publication depends on that verdict, so a red gate means no release — mechanically, with no human checklist in the loop.
- Workflow:
.github/workflows/release-qa-gate.yml - Driver:
cmd/release-gate(Go) - Report schema + merger:
internal/gatereport - Fixtures:
cmd/mcpfixture(deterministic MCP server, reused as a Docker image viascripts/gate/build-fixture-image.sh) andtests/oauthserver(mock OAuth 2.1 + PKCE IdP). - Spec:
specs/081-release-qa-gate/spec.md
What the gate runs (T1)
The gate is a reusable workflow (workflow_call) whose jobs each write a JSON
report fragment into a shared directory; a final verdict job merges the
fragments against a hardcoded manifest and exits per the verdict.
| Job | Blocking entries | Timeout | Notes |
|---|---|---|---|
build-candidate | — | 15 min | Frontend build + embed + go build of the candidate mcpproxy, the fixtures, and the release-gate driver; uploaded as one artifact and reused everywhere. |
suite-api-e2e | suite/api-e2e, suite/api-e2e-cleanup-check | 30 min | Runs scripts/test-api-e2e.sh unmodified (FR-003), then a second, real invocation of it under scripts/test-api-e2e-cleanup-check.sh proving the E2E cleanup trap holds (no blanket pkill -f "mcpproxy.*serve"). |
suite-race | suite/unit-race, suite/server-race | 25 min | go test -race ./internal/... + go test -tags server -race ./internal/serveredition/.... |
suite-scan-eval | suite/scan-eval | 10 min | go run ./cmd/scan-eval --gate --min-recall 0.90 --max-fp 0.05 over the detect corpus — runs on every tag regardless of changed paths (FR-015). |
matrix-invariants | matrix/{stdio,http,sse,docker,oauth}, invariant/{activity-request-id,counters,quarantine-flow,upgrade-in-place} | 20 min | Boots the candidate against five local fixture upstreams (connect → list → call → kill/reconnect) and asserts the US2 invariants against the live instance. |
surface-consistency | invariant/surface-consistency (T4) | 20 min | Boots the same matrix, adds quarantined/disabled/unhealthy/live servers and asserts the CLI, MCP and rendered Web UI agree with the core on every server's state — see Surface consistency. |
web-ui-sweep | — (advisory advisory/web-ui-sweep, T2, and advisory/skills-tab-sweep, spec 125) | 35 min | Playwright sweep over the Web UI served by the candidate binary, with a live stdio fixture upstream. continue-on-error: true — see Web UI sweep. |
verdict | (merges all) | 10 min | release-gate report → gate-report.json + $GITHUB_STEP_SUMMARY; its exit code is the gate verdict. |
Every job uploads its fragment with if: always(), so a job that dies before
writing a fragment leaves a missing blocking entry — which the merger scores
as a FAIL (FR-004, fail-closed: no silent skips).
Report fragment schema
Each check writes one Fragment (see internal/gatereport/gatereport.go):
{ "name": "matrix/sse", "status": "pass|fail|flaky|skipped|not-run|advisory-fail",
"reason": "...", "classification": "infrastructure|product",
"duration_ms": 0, "retries": 0, "steps": [...], "details": {...} }
The merger produces gate-report.json: {verdict, blocking_failures, advisory_failures, counts, entries, manifest}. A blocking entry passes only
with status pass or flaky (a flaky is a pass-on-retry, FR-010).
How publication is gated (FR-002)
Both publisher workflows add a qa-gate job that uses: this reusable workflow
and list it in the publish job's needs::
release.yml— thereleasejob (which creates the GitHub Release, and from which every public-facing job cascades) gainsqa-gatein itsneeds. Theqa-gatejob is guarded to the same stable-only condition asrelease, so the expensive gate does not double-run on RC tags here.prerelease.yml— same, guarded to prerelease tag refs only (skipped onnext-branch pushes, matching thereleasejob's tag condition).
Because both the gate job and the publish job share the same trigger condition,
a skipped gate (non-qualifying ref) skips the publish job too — it can never
silently un-gate a release. The invariant is enforced by an audit test,
cmd/release-gate/workflow_audit_test.go,
which parses both publisher workflows and asserts every artifact-publishing
job's transitive needs closure includes the qa-gate job (FR-022 / SC-004).
Statically disabled jobs (if: false…, e.g. the server-edition build-docker)
are excluded — when such a job is enabled it must be re-parented under the gate,
or the audit will fail.
Dry-running the gate before tagging (FR-001a)
Maintainers can qualify a candidate before cutting a tag:
- GitHub UI: Actions → Release QA Gate → Run workflow → optionally set
the
refinput to the branch/SHA to qualify. - CLI:
gh workflow run release-qa-gate.yml -f ref=<branch-or-sha>
A dry run publishes nothing and never counts as qualification for a later tag — the stable/RC tag is always re-qualified on its own commit.
Locally you can run the driver directly (see release-gate --help):
go build -tags nogui -o mcpproxy ./cmd/mcpproxy
go build -o mcpfixture ./cmd/mcpfixture
go build -o oauthserver ./tests/oauthserver/cmd/server
go build -o release-gate ./cmd/release-gate
scripts/gate/build-fixture-image.sh
./release-gate matrix --binary ./mcpproxy --fixture ./mcpfixture \
--oauth-server ./oauthserver --report-dir ./gate-report \
--state-file ./gate-state.json --work-dir ./tmp-gate
./release-gate invariants --state-file ./gate-state.json --report-dir ./gate-report
./release-gate report --report-dir ./gate-report
Web UI sweep (T2) — advisory
The web-ui-sweep job runs the Playwright sweep
(e2e/web-ui-sweep,
see Web UI verification) against the Web UI served by
the candidate binary — embedded frontend, never a dev server — with a live
mcpfixture stdio upstream so the servers/tools screens have real data. It
runs the spec files below (the launchers' playwright test lines are the list, and
TestWebUISweepSpecListIsComplete fails when they, e2e/web-ui-sweep and this
page disagree):
web-ui-sweep.spec.ts: servers list, server detail (+ security tab), tools page and its search, activity log, and settings; fails on uncaught page exceptions.visual-a11y-sweep.spec.ts: WCAG contrast, layouts, accessible names, themes.navigation-consistency.spec.ts: Spec 109 sidebar, header, command palette,+ Addmenu, redirects and filtered deep links.profiles-clients.spec.ts: Spec 108 Profiles and Clients screens, the goal flow, the binding-guard refusal, the Viewing chip, dialogs and layout.profiles-scope.spec.ts: Spec 108 profile scope on Tools, Activity, Sessions and Usage (audit acceptance check 6: the deep links/tools?client=,/usage?profile=and/sessions?client=load filtered, with a removable chip that survives navigation and Back).profiles-unhide.spec.ts: Spec 109-l un-hide of Spec 108 (Profiles in the sidebar and+ Add, the Viewing chip across pages, Clients-row and agent-token-row links,/servers?profile=, a Spec 108 warning surfacing as a needs-attention item, layout and keyboard order).demo-ux-fixes.spec.ts: the layout findings of the live Web UI demo, measured at 1440 and 900 px: the token Profile chip is one line inside its cell, and radio labels sit next to their radios in the create-profile, custom-client and bulk-move dialogs.usertest-web-fixes.spec.ts: the first-run user test findings, at 1440 and 900 px: the header status pill names servers awaiting review, a secret-like Manual-add value is masked with a Show toggle, and profile Try it is readable and labels unsaved edits.skills-tab.spec.ts(spec 125): the Skills tab of Server Detail, run by its own launcherscripts/run-skills-tab-sweep.shas a second step of the same job (it needsenable_skillsand a skills-capable upstream,cmd/skillsfixture, whichbuild-candidatebuilds intogate-candidate). It lists, views and pages skills, approves one, shows a change made while the skill is open as a stale review, then the diff, and blocks and unblocks. It writesskills-sweep-results.json(checked commit, fixture command line and port, each scenario's pass/fail, and the skill records read back after the approve, block and unblock steps) undertmp/web-smoke-artifacts/skills-sweep/, so it is uploaded with theweb-ui-sweep-playwright-reportartifact. It runs with--self-test-first, which first proves (no browser) that no ambientMCPPROXY_*variable (decoys forMCPPROXY_DATA_DIR/MCPPROXY_DATA,MCPPROXY_AUDIT_LOG_PATH,MCPPROXY_TPA_BUNDLE_PATH,MCPPROXY_CERTS_DIR) reaches the throwaway core, that the results writer redacts the API key and local paths, and that aDEBUG=1run never prints the key. The step runs even when the standing sweep failed, throughrelease-gate run-suite --name advisory/skills-tab-sweep --advisory, so its outcome is its own non-blocking manifest entry: a red run is recorded asadvisory-failand listed in the verdict'sadvisory_failures(a run that died before writing its fragment is a missing advisory entry, listed there too), andcontinue-on-errorkeeps the job from blocking the tag. The Playwright config retries a failed test once in CI, but the skills spec is one stateful serial group and turns retries off for itself (a retry would replay against state the first attempt already changed); the results file records the first failed attempt's error.TestEveryRunSuiteNameIsInTheManifestfails when arun-suite --namein the workflow is not a manifest entry, or an advisory entry has no step.
Setup is not duplicated in YAML: the job calls
scripts/run-web-smoke.sh,
the same launcher used by hand (./scripts/run-web-smoke.sh --show-report),
passing MCPPROXY_BINARY_PATH / MCPPROXY_FIXTURE_PATH from the downloaded
gate-candidate artifact.
It is advisory, on purpose while the sweep earns its flake record:
- the job sets
continue-on-error: true, so its failure never becomes the reusable workflow's conclusion — publishers whoseneeds:list the gate still publish; - the driver runs with
--advisory, so a red sweep is recorded asadvisory-failunder the non-blocking manifest entryadvisory/web-ui-sweepand listed inadvisory_failures— reported, never silent; - the Playwright HTML report + server log are uploaded as the
web-ui-sweep-playwright-reportartifact (14 days) for post-mortem.
Do not rename that artifact casually. The gate is a reusable workflow, so its artifacts land in the publisher's own run, and
release.yml/prerelease.ymlbuild their release assets by globbing every*.tar.gz/*.zipout of all run artifacts. A Playwright report stores its traces asplaywright-report/data/<sha>.zip(retained on failure — exactly what an advisory sweep tolerates), so both publishers exclude*/web-ui-sweep-playwright-report/*from that glob by name. Renaming the artifact without updating both publishers would publish unnamed trace zips as public release assets;TestPublishersDoNotShipSweepArtifactsguards this.
Promotion to blocking is a two-line change: set Blocking: true on the
advisory/web-ui-sweep (and/or advisory/skills-tab-sweep) manifest entry and drop continue-on-error from the job
(the FR-016 end state). Do it once the sweep has passed on three consecutive
release tags with no flaky or infrastructure failure — the same criterion T3
uses (FR-021).
Surface consistency (T4) — blocking
invariant/surface-consistency (job surface-consistency, US5 / FR-018)
proves the surfaces do not lie to each other. The job boots the same
five-type server matrix (stdio/http/sse/docker/oauth), then
release-gate consistency adds four servers so every admin state is on
screen — gate-cons-live (healthy), gate-cons-quarantined,
gate-cons-disabled and gate-cons-broken (endpoint refuses → unhealthy) —
and compares every surface against the core's GET /api/v1/servers:
| Surface | Read path | Fields compared |
|---|---|---|
| CLI | mcpproxy upstream list -o json --config <gate config> (daemon client mode) | enabled, quarantined, connected, connection state, health.level, health.admin_state, health.status, tool_count |
| MCP | /mcp upstream_servers operation=list — storage + upstream manager + runtime health, an independent projection | enabled, quarantined, connection state, health.level, health.admin_state, health.status (the row carries no tool count) |
| Web UI | rendered server cards on /ui/servers, read by e2e/web-ui-sweep/surface-snapshot.mjs in one long-lived Chromium page | enabled (⋯ menu Enable/Disable), quarantined (⋯ Review), tool count, status chip colour and text — the expected chip is derived from the core row with ServerCard.vue's rules, because the Web UI never prints health.level as text |
| macOS tray | — | not compared; recorded as not_compared in the fragment (T3) |
Connection state is compared only for enabled, non-quarantined servers (a
disabled server is idle in the StateView and disconnected in the manager,
which no user sees). It runs three phases, each retried until the surfaces
agree or --settle-timeout (60s) runs out; an attempt counts only when the
core did not change while the surfaces were being read:
- settled — the mixed state, Web UI freshly loaded;
- live —
gate-cons-liveis disabled through REST while the page stays open, so the Web UI must follow through its SSE/poll update path; - reload — the same state after a page reload. Live failing while reload passes is reported as a stale live view.
A divergence that survives the retries fails the gate with one line per field:
webui gate-cons-quarantined webui.chip_class: core=badge-secondary webui=badge-success.
The fragment's details carry the last captured rows of every surface per
phase.
Run it by hand next to scripts/test-api-e2e.sh (needs make build for
an embedded Web UI, Node for the reader):
./scripts/test-surface-consistency.sh
Prove it fails: pass --seed-mismatch surface:server:field=value to the
script, or dispatch the gate with the seed_consistency_mismatch input (read
only on workflow_dispatch, never on a tag's workflow_call):
./scripts/test-surface-consistency.sh --seed-mismatch cli:gate-cons-live:health.level=degraded
macOS tray smoke (T3) — advisory
advisory/macos-smoke (job macos-smoke, US4 / FR-019–FR-021) runs on a hosted
macos-latest runner (the toolchain native-tests.yml uses; macos-15's
Swift 6.1 cannot compile the native test suite). It is advisory: continue-on-error: true on the job plus
release-gate run-suite --advisory, so a failure is listed under
advisory_failures and never blocks publication. A missing fragment (runner
never scheduled, job died before upload) is reported the same way.
The job and the by-hand command are the same script, scripts/test-macos-smoke.sh:
swift build -c releasethe tray (native/macos/MCPProxy).swift testthe whole native suite (--skip-unit-testsskips it locally).- Start a core with an isolated
--data-dir/--config/socket under a short scratch root and scratchHOME(never~/.mcpproxy). - Assemble a minimal ad-hoc-signed
.app(the bare SwiftPM binary crashes without a bundle:UNUserNotificationCenter), launch it withMCPPROXY_TRAY_SKIP_CORE=1 MCPPROXY_HOME=… MCPPROXY_SOCKET_PATH=…, and assert from the tray's own log that it reached the core over the Unix socket (connectToCore: core is ready) and read the core's exact version (got version=<core /api/v1/info version>). SIGTERMthe tray: it must exit 0 within 15 s and leave the external core running.
What it cannot do on a hosted runner: grant Accessibility / Screen Recording to
the job, so the mcpproxy-ui-test primitives (status-bar item, menu items,
screenshot) are not used and the tray is not compared in the T4 surface
check yet (still not_compared). No signing or notarization secrets are
involved. Logs (core.log, tray.log, swift build/test output) upload as the
macos-smoke-logs artifact.
Promotion criterion (FR-021): promote to blocking once the job has passed on
3 consecutive release tags with no flaky or infrastructure failure. Promotion is
Blocking: true on the advisory/macos-smoke manifest entry plus dropping
continue-on-error from the job. Adding the Accessibility-driven menu
assertions (TCC grant on the runner) is the follow-up that lets the tray join
the T4 comparison.
Runtime budget (FR-005 / SC-007)
Every job has an explicit timeout-minutes. The blocking portion targets ≤ 30
minutes wall clock; the critical path is build-candidate → matrix-invariants
(the suite jobs run in parallel). If the full non--short race suite empirically
blows its budget on standard runners, fall back to -short in the
suite-race job only, with a loud comment noting the FR-003 deviation — the
heavy property/timing tests still run unguarded in e2e-tests.yml's
stress-tests job, so coverage is preserved.
FR-011 correlation note
The activity-log invariant (invariant/activity-request-id) empirically probes
whether a caller-supplied X-Request-Id lands on the tool-call activity record.
Today internal/server/mcp.go synthesizes its own per-call request ids for both
MCP-native and REST tool calls, so header correlation does not round-trip; the
check falls back to locating each call by a unique argument nonce and then
proves the core-recorded request id resolves through
GET /api/v1/activity?request_id=. The report records this as a limitation
in the fragment details rather than hiding it. Persisting the caller's
X-Request-Id end-to-end (so header correlation is exact) is a middleware change
left out of this stage on purpose.