Applying a merged nginx-proxy fix by hand

Status update (2026-09-04) — this is now pipeline-managed

As of 2026-09-04, scripts/deploy-vps-local.sh no longer permanently excludes landing-page-cms/nginx-proxy/ — its final step syncs the repo’s config onto the live host path (discovered via docker inspect, not assumed), validates with nginx -t inside the container, and reloads only on success. See spec/deployment.md’s “Nginx Config Deploy Pipeline” section for the full mechanism, and the “Cutover to the pipeline-managed path” section near the bottom of this page for the one-time steps needed to adopt it safely on a box whose live config has already drifted from the repo.

This page’s step-by-step manual-apply instructions (sections 1-8 below) are kept as a fallback — for the one-time cutover itself, and for any future case where the pipeline step needs to be bypassed by hand. For an ordinary merged change, the pipeline now applies it automatically on the next deploy; you should not normally need to walk through this by hand.

Why this page originally existed

Before the fix above, scripts/deploy-vps-local.sh — the script the self-hosted-runner pipeline (.github/workflows/deploy-development.yml) runs on every push to development — deliberately and permanently excluded landing-page-cms/nginx-proxy/ from every sync. This was not a bug; it was documented as a safety invariant in the script’s own header and in .github/workflows/README.md:

Never touches nginx/certbot config or the live .env … the RUNNING proxy was created with different mount paths than this compose file declares; recreating it would remap the SSL volumes.

The consequence: a merged, reviewed, CI-green change to any file under landing-page-cms/nginx-proxy/conf.d/ read as shipped in Jira and in git log, but never reached the box. It sat in the repo indefinitely until a human copied it over by hand. See KAN-680 for the incident that surfaced this (KAN-671’s /_next/static/ rate-limit exemption, merged in 5e376db7, never took effect in production).

Two things make this riskier than a normal deploy:

The live config has already diverged from the repo by hand. KAN-680 found hub-https-simple.conf.pre-acme-fix and hub-https.conf.bak sitting next to the live conf.d files on the box — evidence the running config was hand-edited (almost certainly for ACME/certbot path fixes) after whatever deploy last put it there. Never blindly overwrite the live file — diff it first and reconcile, file by file.
This proxy terminates TLS for every hostname on the box (hub, control, docs, api, landing page, CRM, campaigns, workflows), not just EQMS. A bad reload can take down six unrelated services at once.

Scope

This runbook is generic to any file under landing-page-cms/nginx-proxy/. The worked example below is the KAN-671 _next/static exemption in hub-https-simple.conf; substitute the actual file/diff for whatever change you are applying.

Before you start

You need SSH access to the VPS and to already know its host/user (not recorded here — see the ops credential store, not this doc).

Confirm you’re applying the fix that’s actually merged:

git -C /path/to/EQMS log -1 --format='%H %s' -- landing-page-cms/nginx-proxy/conf.d/<file>.conf

Do this at a low-traffic time if at all possible. nginx -s reload is designed to be zero-downtime (it starts new worker processes and lets old ones drain), but you are about to hand-edit six vhosts’ worth of routing — budget for the possibility something is wrong with the diff itself.

1. Back up the live config first

Do this before touching anything, even before you look at the diff.

ssh ops@<vps-host>
sudo mkdir -p /root/innoqualis/landing_page/nginx-proxy/conf.d.backups
stamp=$(date +%Y%m%d-%H%M%S)
sudo cp -a /root/innoqualis/landing_page/nginx-proxy/conf.d \
           /root/innoqualis/landing_page/nginx-proxy/conf.d.backups/conf.d-$stamp
sudo cp -a /root/innoqualis/landing_page/nginx-proxy/nginx.conf \
           /root/innoqualis/landing_page/nginx-proxy/conf.d.backups/nginx.conf-$stamp

This is your rollback source. Confirm it actually copied before proceeding:

ls -la /root/innoqualis/landing_page/nginx-proxy/conf.d.backups/conf.d-$stamp

2. Diff live vs. repo — do not assume the repo copy is authoritative

The live file may already contain hand-fixes (ACME/certbot paths in particular) that the repo copy doesn’t have. Pull the live file down and diff it against the repo before deciding what to change:

# from your local checkout, on the box:
scp ops@<vps-host>:/root/innoqualis/landing_page/nginx-proxy/conf.d/<file>.conf \
    /tmp/live-<file>.conf
diff -u /tmp/live-<file>.conf landing-page-cms/nginx-proxy/conf.d/<file>.conf

If the diff is only the intended fix (e.g. the new location /_next/static/ block from KAN-671) → proceed, apply the repo version.
If the diff shows other hand-made changes on the live side (cert paths, extra headers, a different server_name) → do not overwrite wholesale. Hand-merge: take the live file as the base and apply just the new block/lines from the repo diff. This is the same principle as never reconstructing a file from git show over an uncommitted copy — the live file has state the repo doesn’t.

3. Stage the new file next to the live one (don’t overwrite yet)

scp landing-page-cms/nginx-proxy/conf.d/<file>.conf \
    ops@<vps-host>:/root/innoqualis/landing_page/nginx-proxy/conf.d/<file>.conf.staged

4. Validate before it takes effect

nginx -t only checks whatever is actually named *.conf in conf.d/ (nginx loads the directory wildcard), so validating the .staged file in place requires either a syntax-only check on the file directly, or a swap-and-test inside the container where a failure is still recoverable in the same step:

ssh ops@<vps-host>
cd /root/innoqualis/landing_page/nginx-proxy/conf.d
sudo mv <file>.conf <file>.conf.previous
sudo mv <file>.conf.staged <file>.conf
docker exec innoqualis-nginx-proxy nginx -t

nginx -t reports syntax is ok / test is successful → continue to step 5.
nginx -t fails → immediately restore and stop:
Terminal window
```
sudo mv <file>.conf <file>.conf.rejected
sudo mv <file>.conf.previous <file>.conf
docker exec innoqualis-nginx-proxy nginx -t   # confirm still ok
```
Nginx has not reloaded yet at this point — the currently-running workers are still serving the old config. A failed nginx -t here is a non-event for production; it only means the file you staged is invalid. Go fix the file and start again from step 2.

5. Reload without dropping connections

Only after nginx -t passes on the new file in place:

docker exec innoqualis-nginx-proxy nginx -s reload

-s reload is the zero-downtime primitive: nginx spawns new worker processes with the new config, lets existing workers finish in-flight requests, then retires them. It does not drop established connections the way a container restart would.

6. Verify from outside the box

Do this from your own machine, not from the VPS itself — you want to confirm the public edge is actually serving the new behavior, not just that the container’s local state looks right.

For the KAN-671 case specifically:

curl -sI https://hub.innoqualis.com/_next/static/chunks/webpack-<hash>.js | head -5
# expect: 200, and Cache-Control: public, immutable, max-age=31536000
# NOT a 429, and NOT text/html

For any nginx change generally, at minimum:

curl -sI https://hub.innoqualis.com/            # still 200, still serving the app
curl -sI https://hub.innoqualis.com/api/status  # backend proxy still works
# repeat for every other vhost this proxy terminates (control, docs, api,
# landing page, crm, campaigns, workflows) — a shared nginx.conf change can
# affect all of them even if you only touched one conf.d file

If anything above fails, go straight to rollback — don’t debug live.

7. Roll back if the host stops serving

Rollback is symmetric to step 4, using the backup from step 1 (preferred) or the .previous file staged in step 4 (faster, same effect for a single-file change):

ssh ops@<vps-host>
cd /root/innoqualis/landing_page/nginx-proxy/conf.d
sudo cp /root/innoqualis/landing_page/nginx-proxy/conf.d.backups/conf.d-$stamp/<file>.conf <file>.conf
docker exec innoqualis-nginx-proxy nginx -t && docker exec innoqualis-nginx-proxy nginx -s reload

Then re-verify with the same curl checks from step 6. If nginx -t itself fails on the restored backup (shouldn’t happen — it was live a minute ago) or the container won’t reload, the last resort is a full container restart, which does briefly drop connections but guarantees a clean process state:

docker restart innoqualis-nginx-proxy

Only reach for docker restart after a failed nginx -s reload — a restart is a harder edge than a reload and is exactly the kind of proxy-recreate this deploy pipeline was designed to avoid doing automatically.

8. Close the loop

Record what was applied and when as a comment on the relevant Jira ticket (e.g. KAN-680) — the merge already happened; this is the only record that the deploy happened too.
If the diff step (2) found live-side hand-edits beyond the intended fix, file them as their own gap — see the “general problem” discussion below; don’t let a second undocumented drift start accumulating on top of the one you just found.

Cutover to the pipeline-managed path

This is a one-time procedure for adopting the pipeline-managed sync (scripts/deploy-vps-local.sh’s new step, KAN-680) on a box whose live conf.d has already drifted from the repo by hand. Requires VPS access — do this from the box, not from an autonomous session (see this repo’s escalation rules: HUMAN pre-flight steps / no credentials → can’t act).

Status: the owner has applied KAN-671’s /_next/static/ exemption to the live hub-https-simple.conf by hand (existing runbook flow), inserting it alongside the pre-existing /.well-known/acme-challenge/ location that lives in that file’s HTTPS server block (certbot’s HTTP-01 renewal for hub.innoqualis.com depends on it — the plain :80 server block unconditionally redirects everything, so the challenge is only ever served from the :443 side). The repo copy now carries that same /.well-known/acme-challenge/ block in the same position, so the two should be identical going into step 2 below — if they are not, stop and reconcile before continuing; do not let the sync overwrite a live-only difference you haven’t accounted for.

1. Back up the live `conf.d` (belt-and-suspenders — the pipeline also

backs up automatically on every run, but take an out-of-band copy before the first one)

ssh ops@<vps-host>
docker inspect innoqualis-nginx-proxy --format \
  '{{ range .Mounts }}{{ if eq .Destination "/etc/nginx/conf.d" }}{{ .Source }}{{ end }}{{ end }}'
# note the printed host path — call it $CONFD below
stamp=$(date +%Y%m%d-%H%M%S)
sudo cp -a "$CONFD" "/root/cutover-backup-conf.d-$stamp"

2. Verify the repo copy is identical or intentionally different

Don’t assume — diff every file, file by file:

for f in "$CONFD"/*.conf; do
  name=$(basename "$f")
  repo_f="/root/innoqualis/landing-page-cms/nginx-proxy/conf.d/$name"
  if [ ! -f "$repo_f" ]; then
    echo "=== $name: exists live, NOT in repo checkout — investigate before proceeding ==="
    continue
  fi
  if ! diff -q "$f" "$repo_f" >/dev/null; then
    echo "=== $name: DIFFERS ==="
    diff -u "$f" "$repo_f"
  fi
done

No diffs at all → this is the expected outcome as of KAN-671’s manual apply + this repo’s /.well-known/acme-challenge/ fix (both preserved in hub-https-simple.conf, per the “Status” note above). Proceed to step 3, the first pipeline run will be a no-op.
A diff limited to something you can positively identify as already reconciled (e.g. whitespace/comment-only) → expected, proceed.
Any other diff (hand-fixed ACME paths, extra headers, a cert path that isn’t in the repo, or — the specific failure this cutover exists to catch — a /.well-known/acme-challenge/ location present live but missing from the repo copy) → do not proceed. Port the live-only change into the repo first (same principle as the manual-apply steps above: the live file may hold state the repo doesn’t), get it merged — scripts/check-nginx-acme-challenge.sh will fail CI if the ACME location is missing from any certbot-managed vhost, so this class of gap can’t merge silently again — then restart this cutover from step 1 against the updated repo.

3. First pipeline run

Trigger a normal deploy (push to development, or workflow_dispatch) and watch the “Sync nginx-proxy config” step’s output in the run log. It prints:

the discovered $CONFD (confirm it matches what you found in step 1)
nginx -t output (must say “syntax is ok” / “test is successful”)
either “nginx-proxy reloaded” or a die with the restore-backup path

4. Verify from outside the box

Same checks as “Verify from outside the box” above — confirm every vhost (hub, control, docs, api, landing page, CRM, campaigns, workflows) still responds.

5. Rollback (if the first pipeline run misbehaves)

The pipeline already restores its own pre-sync backup automatically on a failed nginx -t — that path is printed in the die message. If something passes nginx -t but is still wrong behaviorally, restore the step-1 out-of-band backup by hand:

sudo cp -a "/root/cutover-backup-conf.d-$stamp/." "$CONFD/"
docker exec innoqualis-nginx-proxy nginx -t && \
  docker exec innoqualis-nginx-proxy nginx -s reload

Then re-verify with the step 4 checks. If you need to fall back further (stop the pipeline from touching conf.d at all), revert the deploy script change and re-open this doc’s original manual-apply flow — but that reintroduces the GAP-484 drift class, so treat it as a stopgap and re-attempt the cutover once whatever broke is understood.

6. Close the loop

Comment on KAN-680 with the outcome and update docs/gaps.md GAP-484 to RESOLVED once step 4 is clean.

KAN-680 — the incident this runbook was written for.
KAN-671 — the fix that originally could not deploy; the confirmed instance of the GAP-484 class.
scripts/deploy-vps-local.sh — the active deploy script, now including the nginx-proxy config sync step (see the file’s own header comment).
scripts/check-nginx-conf-pipeline-guard.sh / scripts/check-nginx-syntax.sh / scripts/check-nginx-acme-challenge.sh — the CI + pre-commit guardrails that keep this from regressing silently (the last one specifically guards the /.well-known/acme-challenge/ gap the cutover diff found in hub-https-simple.conf).
.github/workflows/README.md — deploy pipeline overview, including the nginx-proxy sync step.
spec/deployment.md — “Nginx Config Deploy Pipeline” and “Nginx Rate Limiting” sections.
docs/gaps.md GAP-484 — the tracked gap entry for this defect class.

Last reviewed

2026-09-04 — pipeline shipped (KAN-680): scripts/deploy-vps-local.sh now syncs, validates, and reloads landing-page-cms/nginx-proxy/ config on every deploy. The sections above (1-8) remain as the manual-apply fallback; the new “Cutover to the pipeline-managed path” section covers one-time adoption on the live box, which is still an owner/human task (VPS access required — out of scope for an autonomous session per this repo’s escalation rules).