Frontend deploy unblock forward-fix + per-service Trivy gate root-cause fix + alert hardening

Goals

• Immediate unblock of Sprint 159 PR #272 (frontend AI feedback parsing hotfix) that had been unreflected in production frontend for 4 hours (Phase A)
• Permanent root-cause fix of the regression — the Update GitOps manifests job guard (needs.trivy-scan.result != 'failure') blocking GitOps update for every service when a single service fails Trivy (Phase B)
• Resolve the alert visibility gap where "git merge success" notifications did not surface the fact that some services were SKIPPED for 4 hours (Phase C, Sprint 159 seed #new4)
• Establish a formal cycle for handling production incidents: external read-only diagnosis → cross-check in this session → immediate unblock → root-cause fix → alert hardening → retrospective

Decisions

• Option B selected (precise 1-line manual PR): For Phase A unblock, chose manual aether-gitops PR over Option A (workflow_dispatch + rebuild_all=true, rebuilds all services) and Option C (trivial commit on main, history pollution). Minimal change scope (1 line in kustomization.yaml), reuses the already-built GHCR image (main-505568a), blast radius limited to frontend
• Forward-fix policy inherited (Sprint 159 pattern): Advance GitOps tag with the exact SHA (main-505568a) instead of revert or trivial commit. The hotfix code is already safe (GHCR build complete + verified), so precise application is the normal flow
• Artifact-based per-service Trivy gate (Phase B root-cause design): To work around the single-value limitation of matrix outputs, export trivy-status/<service>.txt as a per-service artifact (trivy-status-<service>). The deploy job downloads them via pattern: trivy-status-* + merge-multiple: true and performs per-service lookup. The matrix job's own result remains marked failed (preserving commit-level visibility)
• Fail-closed security gate (Critic R1 P1 resolution): Combined with removing the trivy-scan.result != 'failure' guard, a defect could allow services with missing artifacts (infra fail / upload failure) to pass. Now any case where STATUS != "pass" (fail/missing/empty/other) is classified as SKIPPED. Only explicit pass proceeds
• Deploy summary push-outcome branching (Critic R1 P2 resolution): The Phase C Surface deploy summary step's always() condition could output "✅ Deployed services" false-success messages even when GitOps push failed — direct contradiction of the sprint goal (accurate notifications). Branch on steps.push_gitops.outcome: anything other than success → "candidates only (NOT applied)". Reproduces the Sprint 155 pattern (self-contradiction detected after Critic R1 pass)
• GitHub-native only: Phase C alert hardening uses only $GITHUB_STEP_SUMMARY markdown + ::warning:: workflow command. Discord/Slack webhook integration requires secrets/channel decisions → deferred to a separate sprint

Implementation (3 PRs squash merged + 1 aether-gitops PR, origin/main 2ec3747 → b385343)

aether-gitops PR #6 — Phase A frontend deploy unblock

Cross-check results (100% match with external read-only diagnosis):

aether-gitops current state: kustomization.yaml algosu-frontend newTag = main-431356156f615c2e6b215baddaebf307d26c881b (matches the 4313561 from external diagnosis). Other 7 service tags intact.

Change applied: python3 precise substitution — only the algosu-frontend entry advanced to main-505568a229922bf2c77e9e425cfdc846c0eceb70. Zero impact on other services.

Merge: mergeStateStatus: CLEAN, no CI (GitOps data repo), immediate squash merge → aether-gitops main HEAD cb9f9a1 → 3f50eb7 → ArgoCD reconcile auto-sync → frontend pod rollout.

AlgoSu PR #274 — Phase B per-service Trivy gate root-cause fix

Regression scenario reconfirmed: PR #272 CI run 26071766405 → Build Frontend (Next.js) ✅ + Build AI Analysis (FastAPI) ✅ + Trivy Scan — ai-analysis ❌ → Update GitOps manifests SKIPPED (frontend blocked too). After PR #273 (ai-analysis CVE patch) merge, only ai-analysis was updated in GitOps; frontend ran on the pre-hotfix image (main-4313561) for 4 hours.

4-step design:

1. Record per-service results as artifacts in the trivy-scan matrix:

   • Add id: trivy_table to the Trivy scan (table) step
   • New step Record Trivy result for deploy gate — write pass/fail line to trivy-status/<service>.txt (always() + skip == 'false')
   • New step Upload Trivy status artifact — trivy-status-<service> artifact (always, retention 1 day)

2. Remove aggregated trivy-scan.result != 'failure' guard from deploy job:

   diffCopy

   if: |
     github.ref == 'refs/heads/main' && !cancelled() &&
     needs.secret-scan.result == 'success' &&
   - needs.trivy-scan.result != 'failure' &&
     (needs.build-services.result == 'success' || ...)
   

3. Two new deploy job steps:

   • Download Trivy status artifacts — pattern: trivy-status-* + merge-multiple: true flattening
   • Probe Trivy status artifacts (Sprint 157 seed #29) — directory listing + per-service result echo

4. Update image tags logic redesigned:

   • Collect CANDIDATES (build success + detect-changes active service)
   • Per-service lookup of trivy-status/<svc>.txt → fail services moved to SKIPPED_TRIVY
   • Only passing services have their kustomization.yaml tag updated
   • Outputs updated / skipped_trivy exposed (Phase C input)

Critic R1 P1 resolution cycle:

"When a candidate service has no ${SVC}.txt status file, this loop treats it as passing and updates its GitOps tag. ... Treat missing or non-pass status as skipped/failed for the service."

→ Previous logic (if [ -f "$STATUS_FILE" ] && [ "$(cat ...)" = "fail" ]) only classified explicit fails, missing files passed → security gate bypass. Fixed: any case where STATUS != "pass" (fail/missing/empty/other) is SKIPPED. Fail-closed security gate.

STATUS=$(cat "$STATUS_FILE" 2>/dev/null || echo "missing")
if [ "$STATUS" != "pass" ]; then
  SKIPPED_TRIVY="${SKIPPED_TRIVY} ${SVC}"
  echo "  ⚠ algosu-${SVC} SKIPPED (Trivy status: ${STATUS} — ...)"
  continue
fi

Critic R2 Clean ✅: "The change makes the deploy gate require an explicit per-service Trivy pass status and treats missing/empty/failed statuses as skipped, which is consistent with the fail-closed intent. I did not identify a discrete regression introduced by this commit."

AlgoSu PR #275 — Phase C deploy alert hardening

Sprint 159 seed #new4 fulfilled: Two layers of explicit visibility using deploy.outputs.skipped_trivy exposed by Phase B.

1. deploy job outputs: declaration:

   YAMLCopy

   outputs:
     updated: ${{ steps.update_tags.outputs.updated }}
     skipped_trivy: ${{ steps.update_tags.outputs.skipped_trivy }}
   

2. Surface deploy summary step (always()) — markdown to $GITHUB_STEP_SUMMARY:

   • ### ✅ Deployed services section + each service tag
   • ### 🚫 SKIPPED (Trivy fail or status missing) section + remediation guidance
   • When SKIPPED is non-empty: > ⚠️ "git merge success" ≠ "deploy success" blockquote warning

3. notify job Warn on Trivy-skipped services step — if: needs.deploy.outputs.skipped_trivy != '':

   • 3 ::warning:: lines — service list + production image not advanced + recurrence prevention guidance

Critic R1 P2 resolution cycle (sprint-goal self-contradiction detection):

"When git commit or git push fails after update_tags has populated UPDATED, this always() summary still runs and labels those services as deployed even though the GitOps repo was not updated. ... gate the 'Deployed services' section on the commit/push result or phrase these as candidates when the deploy job has failed."

→ Phase C's purpose is "accurate visibility of deploy results" but false-success messages on push fail = direct contradiction with the sprint goal. Fix:

• Add id: push_gitops to the Commit and push to aether-gitops step
• Branch in the summary step on steps.push_gitops.outcome:
  • success → "✅ Deployed services"
  • otherwise → "🚫 GitOps push failed" + "candidates only — production is unchanged" + intended tag with (NOT applied) suffix

Critic R2 Clean ✅: "does not introduce an obvious workflow-breaking issue".

New Patterns

1. Single-sprint 6-phase cycle: external read-only diagnosis → this-session cross-check → immediate hotfix → root-cause fix → alert hardening → ADR — Full cycle from production incident to retrospective within a single sprint. One evolution beyond Sprint 159 (simple hotfix + ADR)

2. Production hotfix option comparison table + single user selection followed by immediate execution — Options A (workflow_dispatch rebuild_all) / B (manual aether-gitops PR) / C (main trivial commit) compared across change-scope/risk/duration/evaluation axes → user selects Option B → plan executed as-is. Direct inheritance of Sprint 156 option-selection pattern

3. Artifact-based per-service gate communication pattern — Workaround for the GitHub Actions matrix output single-value limitation. <key>-<dim> artifact names + pattern + merge-multiple flattening → deploy job performs per-service lookup. Matrix job result remains marked failed (commit-level visibility). Standard workaround pattern for matrix output limitations

4. Fail-closed security gate principle — Critic R1 P1 institutionalization. All cases of STATUS != "pass" (fail/missing/empty/other) blocked. Only explicit pass proceeds. Infra fail / artifact upload failure auto-blocked. Phase B's essential safety principle

5. Critic R1 P2 sprint-goal self-contradiction detection reproduced (Sprint 155/159 pattern reinforced) — Phase C's always() summary false-success messages directly contradict Phase C's essence (accurate notifications). R1 passes → self-contradiction detected just before R2. Second self-contradiction detected in this sprint (Phase B P1 + Phase C P2)

6. Forward-fix policy inheritance — Sprint 159 base image patch pattern reproduced as Phase A. Avoid revert, advance GitOps tag with the exact SHA. Code is already safe (GHCR build complete + verified), so precise application is normal

7. Alert hardening using only GitHub-native — $GITHUB_STEP_SUMMARY + ::warning:: workflow command. No additional secrets / webhooks needed. Discord/Slack integration requires secrets/channel decisions → split into a separate sprint. Incremental hardening policy

8. Critic R1 messages cite precise locations — Critic cites the form line:973-979 / line:1033-1036 with the exact location + regression scenario + fix direction. Architect-side fix cycle wraps up in 1 iteration in this sprint. Effect of Codex cross-review's location-citation precision

9. PR-stage CI green itself acts as the first regression gate — After Phase B/C changes, the PR's CI 27 jobs SUCCESS itself serves as the first regression gate. Empirical validation of the Trivy fail scenario is deferred to a separate sprint

Verification

Branch Discipline ✅ 28 sprints of consecutive compliance

• All 3 PRs (AlgoSu 2 + aether-gitops 1) on new branches + Squash merge
• 0 direct commits to main
• 0 --no-verify uses
• 2 Critic R1/R2 cycles (Phase B/C each)

Changed Files

• .github/workflows/ci.yml — trivy-scan matrix artifact export + deploy job per-service gate + STEP_SUMMARY + notify warning (Phase B/C combined, +131 −15)
• aether-gitops:algosu/overlays/prod/kustomization.yaml — frontend tag advance (Phase A, +1 −1)
• docs/adr/sprints/sprint-159.md — backfill (Sprint 159 KR body)
• docs/adr-en/sprints/sprint-159.md — backfill (Sprint 159 EN body)
• docs/adr/sprints/sprint-160.md — this ADR
• docs/adr-en/sprints/sprint-160.md — this ADR EN