Programmers Dataset + Gateway External Integration

Date
ImpactHigh

Key Decisions

Pre-curated JSON bundling over real-time parsing/unofficial API

No official API + Cloudflare JA3 blocking (Sprint 83 precedent) + finite problem pool (373 items)/low update frequency. Operational stability prioritized

Search API also bundling-based

In-memory search endpoint symmetric to Solvedac → Sprint 96 frontend can implement identical UX

Lv.1~5 ↔ BRONZE~DIAMOND 1:1 mapping

0 design token lines changed — reuses existing Difficulty enum and style tokens

DTO `@IsIn` strengthening

From free-string to allowlist. Early mismatch rejection at input boundary. DB retains VARCHAR → Expand-Contract not required

Branch discipline restoration

architect accidentally committed directly to main (adad5cf) → moved to feat/gateway-programmers-dataset branch, main reset (local stage). "No direct push to main" rule enforced

Background

Following the Baekjoon (BOJ) service shutdown, AlgoSu's problem source must migrate to Programmers. The migration scope spans backend, frontend, and submission pipeline — requiring a 3-sprint roadmap (Sprint 95 backend infra → Sprint 96 frontend UX → Sprint 97 submission/docs). This sprint focuses exclusively on backend infrastructure with zero user-visible changes as the guiding principle.

At sprint start, two problems had accumulated in .planning:

  1. No official Programmers API — real-time parsing would require fragile scraping
  2. Problem pool is finite and rarely updated — pre-bundled JSON is more operationally stable

Goals

  • Bundle Programmers problem metadata as a pre-curated JSON file, eliminating external API dependency from the search UX
  • Build /api/external/programmers/* endpoints in Gateway with symmetric structure to the existing BOJ (Solved.ac) endpoints
  • Strengthen DTO sourcePlatform with @IsIn(['BOJ','PROGRAMMERS']) to explicitly declare allowed values
  • Guarantee 0 BOJ path regressions

Work Summary

CommitAgentContent
adad5cfarchitectProgrammersService/Controller skeleton + external.module.ts registration + DTO @IsIn extension
60b7925librarianData bundling decision ADR (docs/adr/topics/sprint-95-programmers-dataset.md)
e460b79postmanPlaywright-based crawler script + initial dataset of 373 problems
18b3932curatorData quality QA report (PROGRAMMERS-QA.md)
2578ae0gatekeeperVerification report + 749 tests PASS + BOJ regression integrity
aff4b47OracleDTO type strengthening follow-up: problem.service.spec.ts sync

Changes

Gateway External Module

  • services/gateway/src/external/programmers.service.ts — JSON loaded at startup, Map<problemId, Info> in-memory cache, fetchProblem/searchProblem — symmetric to SolvedacService interface
  • services/gateway/src/external/programmers.controller.ts — GET /problem/:problemId, GET /search?query=&page=, Swagger External — Programmers
  • services/gateway/src/external/external.module.ts — ProgrammersService/Controller registered (alongside existing Solvedac)
  • Data envelope structure: { version: ISO8601, items: ProgrammersProblemInfo[] } + legacy array backward compatibility (isDataEnvelope() type guard)

Crawler & Dataset

  • services/gateway/scripts/fetch-programmers-problems.ts — Playwright chromium headless, iterates /learn/challenges?levels=N&order=acceptance_desc&page=M. Per-level independent pagination, terminates when 0 new items found, 300–500ms random delay, HTML/URL logging prohibited
  • services/gateway/data/programmers-problems.json — 373 problems collected (Lv.1:95 / Lv.2:132 / Lv.3:95 / Lv.4:31 / Lv.5:20), includes problem 42840 (mock exam), zod runtime validation passed

DTO

  • services/problem/src/problem/dto/create-problem.dto.ts — SOURCE_PLATFORMS = ['BOJ','PROGRAMMERS'] as const extracted, @IsIn(SOURCE_PLATFORMS) applied, type exported
  • DB migration not required (existing source_platform VARCHAR(50) retained)

Test Sync

  • services/problem/src/problem/problem.service.spec.ts L156/L414/L422 — dummy platform literals ('LeetCode','Codeforces') → 'PROGRAMMERS' replacement. sourceUrl retains only @IsUrl validation — minimum change principle applied

Verification

ItemResult
Gateway unit tests (50 suites / 749 tests)✅ PASS
Gateway tsc --noEmit✅ 0 errors
Gateway ESLint (src + scripts)✅ 0 errors
Problem problem.service.spec.ts (35 tests)✅ PASS (initial FAIL → Oracle fix)
BOJ regression (solvedac.{service,controller}.ts diff)✅ 0 lines changed
Data quality (duplicate/missing/encoding/representative problems)✅ 6 PASS / 1 WARN (tags)

Decisions

  • Pre-curated JSON bundling over real-time parsing/unofficial API: No official API + Cloudflare JA3 blocking (Sprint 83 precedent) + finite problem pool (373 items)/low update frequency. Operational stability prioritized
  • Search API also bundling-based: In-memory search endpoint symmetric to Solvedac → Sprint 96 frontend can implement identical UX
  • Lv.15 ↔ BRONZEDIAMOND 1:1 mapping: 0 design token lines changed — reuses existing Difficulty enum and style tokens
  • DTO @IsIn strengthening: From free-string to allowlist. Early mismatch rejection at input boundary. DB retains VARCHAR → Expand-Contract not required
  • Branch discipline restoration: architect accidentally committed directly to main (adad5cf) → moved to feat/gateway-programmers-dataset branch, main reset (local stage). "No direct push to main" rule enforced

Carried Over (Sprint 96–97)

  • tags empty array enrichment: 373 problems have no tags collected. Individual problem detail page breadcrumb crawling for subsequent collection (postman, Sprint 96 or 97)
  • Frontend UX integration: programmersApi, useProgrammersSearch hook, AddProblemModal platform toggle (Sprint 96)
  • GitHub Worker extension: formatPlatform() 'programmers' → 'PROGRAMMERS' case + prg_ filename prefix (Sprint 97)
  • AI feedback prompt: sourcePlatform dynamic injection (Sprint 97)