Full zstd Adoption Across 8 Services + Report Metrics Helper Extraction (Seeds #167-1 + #167-3 Recovery)

Goal

• Based on Sprint 167 ai-analysis A/B measurement data, branch-point decision phase: adopt/reject judgment + full rollout or branch removal
• Seed #167-1 recovery: evaluate ≥30% compression saving AND ≤5% time regression AND no CI stability impact conditions
• Seed #167-3 recovery: deduplicate ~30 lines × 3 = 90 lines of "Report build artifact metrics" across 3 build jobs → single helper
• Close seed #167-2 with justification (frontend/blog skip-guard difference is intentional)
• Decision phase of "Security gate (Sprint 165) → Visibility (Sprint 166) → Optimization measurement (Sprint 167) → Full optimization adoption (Sprint 168)" 4-sprint cycle

Decisions

D0. Measurement Data Acquisition Method — User Selection A (workflow_dispatch + rebuild_all)

• After comparing 4 options, user decision: A adopted
  • A (workflow_dispatch + rebuild_all manual one-shot) / B (artifact auto-upload + next-sprint decision) / C (ai-analysis no-op dummy change PR) / D (aggregator + PR comment posting)
• Rationale:
  • Sprint 167 merge run (#26095687152) ai-analysis build had detect-changes paths filter (services/ai-analysis/**) evaluate to false → step skipped → 0 measurement data acquired
  • Option A is the fastest one-shot measurement (single-sprint integration policy priority)
  • Options B/D visibility enhancement deferred to Sprint 169+

D1. Adoption Verdict — All 3 Conditions Passed

Measurement data (run #26097517834, workflow_dispatch + rebuild_all=true, head SHA 500b954):

• docker tarball measurement: gh run download artifact, stat -f %z = 70,197,760 bytes
• zstd tarball measurement: user direct input from GitHub UI Summary (24.3 MB / -63.7%)
• Other service baseline (for regression comparison): auto-calculated from gh api .../jobs step-level started_at/completed_at diff

D2. Full zstd Application — Remove inline ternary + 8 build jobs unconditional generation

• Remove Sprint 167's ${{ matrix.service == 'ai-analysis' && format(...) || '' }} conditional expression
• build-services matrix 6 services + build-frontend + build-blog all add type=oci,compression=zstd,dest=... unconditional outputs line
• Trivy scan job unchanged — uses only docker tarball --input → compatibility regression zero (zstd is measurement/storage only)
• Other matrix branches / paths filter unchanged → detect-changes-based build skip pattern preserved

D3. Seed #167-3 Helper Extraction = scripts/ci/report-build-metrics.sh

• 3 build jobs' "Report build artifact metrics" step duplication ~30 lines × 3 = 90 lines → single helper 75 lines
• Call shortening: run: bash scripts/ci/report-build-metrics.sh <label> <docker_tarball> [zstd_tarball] (1 line)
• ci.yml net reduction: -102 / +18 = -84 net lines (helper 75 lines separately added)
• Argument interface — zstd_tarball optional → compatible with both this sprint adoption (all zstd generated) + future rejection branch
• Body pattern inheritance:
  • awk -v variable passing (Sprint 166 standard)
  • docker buildx du footer (Shared/Private/Reclaimable/Total) explicit exclusion (Sprint 167 R1 P3 fix)
  • || N/A graceful fallback (Sprint 167 policy)
• set -euo pipefail + chmod +x execute permission commit included

D4. Seed #167-2 Justification Close — frontend/blog Skip-Guard Difference = Intentional

• build-frontend / build-blog are single jobs + sufficient with job-level if: needs.detect-changes.outputs.{X} == 'true'
• build-services step-level if: steps.check.outputs.skip == 'false' guard is natural result of matrix fail-fast: false structure — duplicate guard meaningless if added to frontend/blog
• Difference is intentional → close seed via ADR documentation without code change
• Single job vs matrix job skip-guard pattern = justification baseline of this ADR

D5. Build Gateway GHCR Transient Timeout — Deploy Gate Normal Operation Verification

• This sprint force-build run #26097517834 Build Gateway alone failed at docker/login-action@v4 step with Client.Timeout exceeded while awaiting headers
• aether-gitops overlays/prod/kustomization.yaml actual image tag verification:
  • gateway tag: main-3528ad82... (maintained from Sprint 164 dependabot adoption point) — not updated
  • frontend/blog tag: main-500b954... (Sprint 167 merge natural trigger normal update) — unchanged
• "Update GitOps manifests" job commit message simply displays head SHA → main-500b954..., actual manifest delta only for services with Trivy artifact (Sprint 160 deploy gate normal operation)
• Conclusion: ArgoCD sync impact zero. Gateway rerun unnecessary. Transient infrastructure failure (Sprint 169 seed #168-1 — GHCR retry guard follow-up)

D6. aether-gitops Commit Message Ambiguity — Misinterpretation Risk (Sprint 169 Seed #168-2)

• Current commit message: deploy(algosu): update image tags → main-<head_sha>
• Actual changed services list not specified → "all services updated to head SHA" misinterpretation possible
• Follow-up: commit message to specify actual changed service name + new tag (Sprint 169 deferred)

D7. Phase F GHCR retry guard recovered within this sprint (Seed #168-1 → "Zero carryover" Strengthening)

• Right after PR #293 squash merge, main push CI (run #26099776944) Build Submission Service also hit same GHCR /v2/ Client.Timeout — Build Gateway (force-build) + Build Submission (main push) consecutive 2 cases
• Decision: recover Sprint 169 deferred seed #168-1 by extending this sprint scope (Phase F) — "zero carryover" spirit strengthening
• User option A adopted (among 3 options): nick-fields/retry@v3 external action (1.9k stars, 4M+ DL/month trust) — command: wraps shell-level docker login + max_attempts: 3 + retry_wait_seconds: 10 + timeout_minutes: 5
• Option comparison:
  • A (adopted): nick-fields/retry — standard retry action, +1 external dependency (dependabot github-actions filter auto-updates)
  • B: shell-level docker login + custom retry loop (scripts/ci/ghcr-login-with-retry.sh) — zero external dependency but +1 helper
  • C: continue-on-error + verify step + workflow-level rerun — simplest (1 line) but post-fail manual rerun needed (not essential fix)
• docker/login-action replacement — env passes GITHUB_TOKEN/REGISTRY/ACTOR → log mask automatic (best practice). Post-step logout absence is harmless on ephemeral runners

Implementation (2 PRs, 36 sprints continuous branch discipline target)

Branch: feat/sprint-168-zstd-adoption-decision (new from main 500b954)

Phase A — Measurement Data Acquisition (External Trigger, Zero Code Change)

• gh workflow run ci.yml --ref main -f rebuild_all=true forces build of 8 services + frontend + blog
• User inputs zstd MB + saving % directly from GitHub UI Summary ai-analysis box
• This session auto-downloads docker tarball artifact + auto-calculates other service build step durations via gh api

Phase B — Branch-Point Decision (Zero Code Change)

• 3 adoption conditions (saving 30%+ / regression within 5% / stability) auto-evaluated → all passed → adoption decision
• This sprint's primary work → branch to Phase C-Adopt

Phase C-Adopt — Full zstd Application + Helper Extraction (commit 564b5e1)

scripts/ci/report-build-metrics.sh (new, 75 lines):

LABEL="${1:?usage: $0 <label> <docker_tarball> [zstd_tarball]}"
DOCKER_TARBALL="${2:?usage: $0 <label> <docker_tarball> [zstd_tarball]}"
ZSTD_TARBALL="${3:-}"

SIZE_BYTES=$(stat -c %s "$DOCKER_TARBALL")
SIZE_MB=$(awk -v b="$SIZE_BYTES" 'BEGIN {printf "%.1f", b/1024/1024}')

BUILD_DURATION=$((BUILD_END - ${BUILD_START:-$BUILD_END}))
# ... cache size / cache entries ...

if [ -n "$ZSTD_TARBALL" ] && [ -f "$ZSTD_TARBALL" ]; then
  ZSTD_BYTES=$(stat -c %s "$ZSTD_TARBALL")
  SAVE_PCT=$(awk -v d="$SIZE_BYTES" -v z="$ZSTD_BYTES" 'BEGIN {printf "%.1f", (1 - z/d) * 100}')
  echo "- tarball size (oci+zstd): **${ZSTD_MB} MB** (${ZSTD_BYTES} bytes)"
  echo "- compression saving: **-${SAVE_PCT}%** (zstd vs docker)"
fi

.github/workflows/ci.yml build-services change:

outputs: |
  type=image,push=${{ github.ref == 'refs/heads/main' }},name=${{ env.IMAGE_PREFIX }}-${{ matrix.service }}:main-${{ github.sha }}
  type=docker,dest=/tmp/image-${{ matrix.service }}.tar
  type=oci,compression=zstd,dest=/tmp/image-${{ matrix.service }}-zstd.tar
# ...
- name: Report build artifact metrics
  if: steps.check.outputs.skip == 'false'
  run: bash scripts/ci/report-build-metrics.sh "${{ matrix.service }}" "/tmp/image-${{ matrix.service }}.tar" "/tmp/image-${{ matrix.service }}-zstd.tar"

build-frontend / build-blog also same pattern (matrix vs hardcoded only difference).

Phase D — ADR Documentation (commit 3ab5138 + post-R2 fix 0e0e553)

• docs/adr/sprints/sprint-168.md (KR) + docs/adr-en/sprints/sprint-168.md (EN 1:1 mapping)
• docs/adr/README.md count 107→108, range 62167→62168 (lines 18/52/54)

Phase F — GHCR transient timeout retry guard (commit 3604bf3, separate PR)

Branch: feat/sprint-168-ghcr-retry-guard (new from main 6a7f56e, after PR #293 squash merge)

.github/workflows/ci.yml 3 build jobs (build-services matrix + build-frontend + build-blog) all docker/login-action@v4 steps removed → nick-fields/retry@v3 introduced:

- name: Login to GHCR (with retry)
  if: steps.check.outputs.skip == 'false'  # build-services only (frontend/blog has no guard)
  uses: nick-fields/retry@v3
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    REGISTRY: ${{ env.REGISTRY }}
    ACTOR: ${{ github.actor }}
  with:
    timeout_minutes: 5
    max_attempts: 3
    retry_wait_seconds: 10
    command: |
      echo "$GITHUB_TOKEN" | docker login "$REGISTRY" -u "$ACTOR" --password-stdin

ci.yml +39 -12 (3 locations identical pattern, build-services only adds if: steps.check.outputs.skip == 'false' guard). dependabot github-actions filter auto-updates.

Critic Cycle

R1 (codex review --base 500b954, this commit 7c982c9 + 3ab5138):

• Result: P0 0 / P1 0 / P2 1 / P3 0
• P2 detected: scripts/ci/report-build-metrics.sh runs under set -euo pipefail, so docker buildx du nonzero exit terminates the script before the fallback ([ -z "$CACHE_DU" ] && CACHE_DU="N/A") → violates prior inline workflow's graceful fallback promise → buildx cache inspection blip can fail the build job
• forward-fix: dccfccd same-PR same-day — add || true to both pipelines (CACHE_DU + CACHE_ENTRIES) so telemetry absence cannot fail the build job
• CI verification: run #26099215601 — 37 SUCCESS + 9 SKIPPED + 0 FAILURE, mergeStateStatus CLEAN ✅

R2 (codex review --base 500b954, after fix commit dccfccd):

• Result: R1 P2 resolution confirmed ✅, new P2 1 detected
• P2 detected: scripts/ci/report-build-metrics.sh not included in detect-changes paths filter → a future PR changing only the helper passes CI with all build jobs skipped → helper defects never exposed
• forward-fix: <TBD-R2-FIX> same PR — add helper path to all 8 filters (gateway/identity/submission/problem/github-worker/ai-analysis/frontend/blog). Helper changes trigger all build jobs → actual runtime verification
• Self-contradiction detection: this sprint goal "full zstd adoption" ↔ Trivy --input docker tarball usage consistency OK (zstd is measurement/storage only)

Risk / Regression Blocking

Prediction 1: Build Time Regression from Adding zstd outputs

• Sprint 167 measurement: ai-analysis (zstd+docker A/B) 8s vs other services docker only avg 14s → ai-analysis 44.6% faster
• zstd compression operation is part of build step (storage stage). Serialized with main build but additional time < 5% (statistical baseline absence comparing other services avg)
• Other services zstd addition predicted within 5% regression — confirmed by Sprint 169 data validation

Prediction 2: Trivy Scan Regression Zero

• Trivy scan job's --input /tmp/image-${service}.tar matches only docker tarball (zstd is separate file)
• buildx multi-output simultaneous generation → docker tarball always exists → compatibility impact zero

Prediction 3: Helper Script Execution Environment Compatibility

• set -euo pipefail + chmod +x + bash shebang explicit
• stat -c %s GNU coreutils — ubuntu-latest environment guaranteed (macOS stat -f %z mismatch is CI-only step so irrelevant)
• docker buildx du GHA ubuntu-runner standard (Sprint 167 verified)
• GITHUB_STEP_SUMMARY not-set fallback to /dev/stdout (external environment testable)

Prediction 4: aether-gitops Manifest Regression Zero

• Build job fail → Trivy artifact absent → "Update GitOps manifests" job skips tag update for that service (Sprint 160 deploy gate normal)
• This sprint force-build Build Gateway timeout case directly verified — gateway tag main-3528ad82... (Sprint 164) maintained ✅
• zstd outputs addition has zero impact on Build itself stability (multi-output simultaneous generation is buildx standard)

Validation

Local

• python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" PASS
• bash -n scripts/ci/report-build-metrics.sh syntax PASS
• chmod +x applied (executable bit confirmed)
• node scripts/check-adr-en-coverage.mjs --strict 117/117 PASS (Sprint 168 new pair added)
• node scripts/check-doc-refs.mjs 0 broken refs PASS

CI (PR stage)

• 37 SUCCESS + 10 SKIPPED + 1 NEUTRAL + 0 FAILURE, mergeStateStatus CLEAN target
• 8 build jobs (6 services + frontend + blog) all display Summary ### 📦 X build artifact H3 + docker MB + duration + cache size + cache entries + oci+zstd MB + compression saving % (8 services consistent)
• Trivy scan 8 matrix regression zero (--input docker tar matching unchanged)
• aether-gitops update job normal (only changed services tag update)

UAT New (Sprint 168)

• User direct: visual confirmation of consistent oci+zstd MB + compression saving % display across all 8 build job Summaries (Sprint 167 ai-analysis 1 → Sprint 168 full 8 expansion)

Result

Changed files 5 (2 PRs, 6 commits total):

• Modified 1: .github/workflows/ci.yml (Phase C -84 net + R2 P2 fix +8 + Phase F +39 -12, build-services + frontend + blog outputs bulk zstd + helper call shortening + detect-changes filter helper path addition + GHCR retry guard)
• New 1: scripts/ci/report-build-metrics.sh (+75 + R1 P2 fix +3, seed #167-3 helper + graceful fallback)
• New 2: docs/adr/sprints/sprint-168.md (KR) + docs/adr-en/sprints/sprint-168.md (EN 1:1 mapping)
• Modified 1: docs/adr/README.md (lines 18/52/54 — count 107→108, range 62167→62168)

Commits — PR #293 (5 commits squash 6a7f56e):

• 7c982c9 feat(ci): Sprint 168 — zstd full adoption + Report metrics helper (seeds #167-1/#167-3)
• 3ab5138 docs(adr): Sprint 168 ADR (KR + EN) + README update
• dccfccd fix(ci): Sprint 168 R1 P2 — buildx du pipeline graceful fallback restoration
• 82621ce fix(ci): Sprint 168 R2 P2 — add helper path to detect-changes filter
• 0e0e553 docs(adr): Sprint 168 — Critic R1/R2 cycle + commits documentation (KR + EN)

Commits — PR <TBD-PR-PHASE-F> (Phase F, separate PR):

• 3604bf3 feat(ci): Sprint 168 Phase F — GHCR retry guard (seed #168-1)
• <TBD-ADR-PHASE-F> docs(adr): Sprint 168 Phase F — GHCR retry guard ADR update
• Squash merge: <TBD-PHASE-F-MERGE-SHA>

New Patterns

• workflow_dispatch + rebuild_all force-build = Standard Measurement Data Acquisition Pattern — Temporarily bypass detect-changes paths filter's cost-saving trade-off (build skip for unchanged services) in measurement sprints. inputs: rebuild_all override is the safest measurement trigger. All future measurement sprints can leverage the same pattern
• Helper Extraction = 90-Line Duplication → Single-Location Integration + Optional Argument for Adoption/Rejection Both Compatible — [zstd_tarball] optional argument interface compatible with both this sprint's adoption result (all build jobs generate zstd) + future rejection branch (zstd_tarball not passed). Helper value independent of decision outcome
• Artifact Auto-Download + Step Timing API Extraction = Measurement Automation Pattern — Minimize user input dependency. gh run download (docker tarball measurement) + gh api .../jobs step started_at/completed_at diff (each service build duration) → Python auto-calculate regression ratio
• Sprint 160 Deploy Gate Transient Infra Failure Isolation Verification — When Build Gateway GHCR timeout occurred, aether-gitops gateway tag update skip confirmed. Sprint 160 (deploy gate Trivy-based service-scoped blocking) exact operation = 1 service fail does not regress other services' deploy. Infrastructure failure isolation value verified
• set -euo pipefail + ?: Argument Validation = Shell Helper Standard Pattern — LABEL="${1:?usage: ...}" pattern enables immediate fail + clear error on missing argument. Combined with caller simplification (helper call 1 line) = safety + readability dual achievement
• set -euo pipefail and graceful fallback compatibility = || true pipeline isolation (R1 P2 fix pattern) — Within helpers, isolate nice-to-have telemetry (docker buildx du) failures with pipeline || true so they cannot fail the entire build job. fail-fast (argument validation) + graceful fallback (telemetry) coexist within the same script. Sprint 168 R1 P2 → Sprint 169+ standard pattern for shell helper authoring
• Helper Path = Cross-Cutting CI Dependency → Register in All detect-changes filter Image Build Branches Simultaneously (R2 P2 fix pattern) — Helper extraction side-effect: PR changing only helper passes silently with all build jobs skipped. Add helper path to all 8 image build filters → helper changes get actual runtime verification. Cross-cutting helper introduction obligates simultaneous detect-changes update
• Critic R1 + R2 Cumulative 2-Round Detection + All Same-PR forward-fix (Zero Carryover Achievement) — R1 P2 (helper fallback) + R2 P2 (filter registration) both fixed same-PR same-day. This sprint's "no carryover" goal + Sprint 164/167 self-fix policy cumulative strengthening = single-sprint completeness priority policy established
• Phase F = post-merge main push CI failure recovered within this sprint (Zero Carryover Strengthening) — Right after PR #293 squash merge, main push CI Build Submission GHCR timeout occurred → recovered Sprint 169 deferred seed #168-1 (GHCR retry guard) as separate PR (Phase F) within this sprint. True spirit of "zero carryover" = post-sprint-close consequences also handled within this sprint. Sprint scope extension policy established (post-close cross-sprint impact is essentially part of this sprint)
• External retry action introduction = Standard Pattern for Transient Infra Failure Cause Blocking — nick-fields/retry@v3 (1.9k stars, 4M+ DL/month) is de facto standard. shell-level docker login + env-passed token (auto log mask) + max_attempts/retry_wait_seconds/timeout_minutes standard interface. docker/login-action replacement — post-step logout absence harmless in ephemeral runner environment. Same pattern applicable to all external registry logins (GHCR/Docker Hub/GCR)

Deferred Items (Sprint 169+)

Sprint 168 New Deferred Seeds

• Seed #168-1: GHCR transient timeout auto-retry guard → Recovered within this sprint as Phase F ✅ (nick-fields/retry@v3 introduced, applied to all 3 build jobs)
• Seed #168-2: aether-gitops "Update GitOps manifests" commit message specify actual changed services list. Currently displays only head SHA → misinterpretation risk (Sprint 168 D6 analysis)
• Seed #168-3: Accumulate zstd compression measurement data for 7 services + frontend + blog beyond ai-analysis + ADR data documentation (Sprint 168 measurement was ai-analysis only, other services zstd measurement auto-accumulated from Sprint 169 first merge run)
• Seed #168-4: Helper script unit test (tests/ci/report-build-metrics.bats or tests/ci/report_build_metrics_test.sh) — argument validation + zstd branch / not-passed branch / GITHUB_STEP_SUMMARY fallback

Sprint 167 Deferred Continued

• CI Visibility (Sprint 164 seeds #new4/5/7): PR-stage deploy gate simulation / aether-gitops kustomization auto PR template / _parse_group_response raw_text fallback envelope extension
• i18n/lint (Sprint 158 seeds #30/#31): build artifact Korean residue CI step + i18n 3-tier checklist
• plan template (Sprint 157 seeds #24/#18): i18n dual-side mandatory checklist auto / blog post pre-merge cross-check auto
• ADR/blog enhancement (Sprint 157 seeds #26/27/28): README paths filter / build-blog ls out/ / check-adr-links ROOT auto-detection

UAT User Direct (25-Sprint Accumulated)

• Seed #5: Programmers resubmit grading pass confirmation
• Seed #9: English environment + production Grafana CB dashboard ai-analysis visual consistency
• Sprint 160~167 accumulated UAT items all inherited
• Sprint 168 new 1: visual confirmation of consistent oci+zstd MB + compression saving % display across all 8 build job Summaries

Deferred Maintained

• Seed #23: plan template "rebase post-cumulative count fix" checklist

Follow-up (Optional)

• create/edit page.tsx category UI
• Programmers URL auto category inference
• Existing SQL problem data backfill
• coverage-gate skipped allowance removal (Sprint 156 Phase B option B)
• post-merge pre-deploy gate (Sprint 156 Phase B option C)
• prom-client Case B~D check automation
• .claude-tools/ Phase 2 actual deletion (after trigger path verification)
• (adr) layout split (KR + EN override) — Sprint 158 description unification alternative
• Sprint 162 R1 P3: deep relative path .md links uncovered — outside ADR scope
• Sprint 163 addition: H3-only PR table extraction + implementation H2 partial matcher + sprint-87 H3-only carryover

ADR: sprint-168.md (KR) + sprint-168.md EN