Skip to content

Content Pipeline Architecture

Status: Local-file pipeline shipped (Phases 0-5, 7). Azure Blob (Phase 6) deferred. Owner: delan Last updated: 2026-05-02 · content-source question settled 2026-08-27

Which card list is authoritative

Settled 2026-08-27, because three sources existed and nothing said which one won — an outside evaluation critiqued the wrong one as a result.

Source Count What it is
data/content/cards.json 584 rows, 476 with effect payloads AUTHORITATIVE. Read by LocalFileContentSource (default path data/content), promoted into the DB by ContentService. All rows isActive: true, status: published; 80 per class, ~36 per House.
ProductionRiftContentCatalog.cs 49 Historical bootstrap seed only. ContentService uses it for a documented "one-shot bootstrap" dump. Not the live pool.
docs/proposals/card-pack-gpt54/ 100 designed Proposal, never implemented. 11 of 12 spot-checked names absent from code.

Consequence for anyone reading the August 2026 evaluation: its claim that the live catalogue is "5 class cards + 1 shared faction card" was read off the bootstrap seed. The actual pool is 80 class cards each. The identity critique in that document still stands as design advice; its inventory does not.

Do not delete ProductionRiftContentCatalog.cs — bootstrap still calls it — but do not treat it as the card list either.

Goal

Replace the hard-coded C# canonical content catalogs (ProductionRiftContentCatalog, ProductionRiftCardUpgrades, ConsumableCatalog, the various *TutorialData classes, WaveCardCatalog, etc.) with externalized JSON files that are:

  • Authored / edited as plain JSON in the repo (or a future admin UI).
  • Versioned via Azure Blob Storage artifacts produced by GitHub Actions.
  • Promoted into each environment's database by an API endpoint that calls existing sp_X_Save stored procedures.
  • Idempotent — every promote is a deterministic upsert keyed on row Id.

Why this shape

Concern Resolution
Drift between code and DB DB is hydrated only by promote; JSON is the truth
Deterministic IDs across envs Id column is part of the JSON, never generated by DB
Auditability Every promote logs (version, user, env, results)
Rollback Re-promote any prior blob version
Test reproducibility Tests load the same JSON files via the library
Designer authoring Edit JSON directly (or admin UI later)
Soft delete Reuse existing ContentStatus.Archived — no new column

Components

┌─────────────────────────────────────────────────────────────────┐
│  GitHub Actions (push to main / tag)                            │
│   1. Bundle data/content/*.json + manifest.json                 │
│   2. Upload to Azure Blob: content/{version}/...                │
│   3. POST /admin/content/promote { "version": "..." }           │
└─────────────────────┬───────────────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────────────┐
│  EchoSpire.API                                                   │
│   POST /admin/content/promote                                    │
│     → IContentSource.FetchAsync(version)                         │
│     → ContentService.ValidateAsync                               │
│     → ContentService.ApplyAsync (calls sp_X_Save / sp_X_Archive) │
│     → record ContentVersion in DB                                │
│   GET  /admin/content/{kind}                                     │
│   GET  /admin/content/diff?version=...                           │
│   GET  /admin/content/current                                    │
└─────────────────────┬───────────────────────────────────────────┘
                      ▼
                  ┌───────┐
                  │  DB   │
                  └───────┘

IContentSource (key abstraction)

Two implementations, selected by config:

  • LocalFileContentSource — reads from ./data/content/ on disk. Used in local dev for fast iteration. No blob round-trip.
  • AzureBlobContentSource — fetches from https://{account}.blob.core.windows.net/content/{version}/manifest.json and the JSON files it lists. Used in dev/staging/prod hosted environments.

Switch via appsettings.json:

{
  "Content": {
    "Source": "LocalFile",         // or "AzureBlob"
    "LocalPath": "./data/content",
    "AzureBlob": {
      "AccountUrl": "https://echospireblob.blob.core.windows.net",
      "Container":  "content"
      // Auth — TBD. See "Deferred decisions" below.
    }
  }
}

ContentService

Single in-process service. Used by:

  • Promote endpoint (production path).
  • Test fixtures (load a fixture JSON via the same service).
  • Dev startup seed (optional convenience: auto-promote LocalFile on boot in dev).

No PowerShell. No CLI. The API endpoint is the loader.

File layout

data/content/
  manifest.json          ← { "version": "v1.4.2", "files": [...] }
  cards.json
  consumables.json
  enemies.json
  artifacts.json
  hero-classes.json
  factions.json
  effect-registry.json
  feature-switches.json
  quest-templates.json
  _well-known-ids.json   ← named GUID constants C# code references

Each file has the shape:

{
  "$sproc": "sp_Card_Save",
  "$archiveSproc": "sp_Card_Archive",
  "rows": [
    { "Id": "...", "Name": "...", ... },
    { "Id": "...", "Name": "...", ... }
  ]
}

The $sproc / $archiveSproc keys make ContentService generic — it doesn't hard-code a switch on filename. Adding a new content type = drop a new JSON file with the right sproc name (sproc + sp_X_Save signature must already exist).

Soft delete

Use the existing ContentStatus.Archived enum value — do not add a DeletedAt column.

  • Row removed from JSON file → ContentService detects "in DB but not in JSON" → calls sp_X_Archive @Id → sets ContentStatusId = Archived.
  • Row re-added → next promote sets ContentStatusId = Published.
  • All Get* queries already filter on ContentStatusId.

Promote endpoint contract

POST /api/v1/admin/content/promote
{
  "version":  "v1.4.2",                  // or "latest" (resolved to current blob)
  "kinds":    ["cards", "consumables"],  // optional filter; default = all
  "dryRun":   true                       // returns diff without applying
}

→ 200 OK
{
  "version":      "v1.4.2",
  "manifestSha":  "abc123...",
  "fetchedFrom":  "https://.../v1.4.2/manifest.json",
  "results": {
    "cards":       { "created": 250, "updated": 12, "archived": 3, "errors": [] },
    "consumables": { "created": 16,  "updated": 0,  "archived": 0, "errors": [] }
  },
  "promotedAt":   "2026-04-29T...",
  "promotedBy":   "delan"
}

Diff endpoint

GET /api/v1/admin/content/diff?version=v1.4.2&kind=cards
→ row-by-row diff (added / changed / removed) between blob snapshot and DB.

Current-version endpoint

GET /api/v1/admin/content/current
→ { "version": "v1.4.1", "promotedAt": "...", "promotedBy": "..." }

Phased plan

Phase Scope Status
0 Fix AssetId range exhaustion (blocks any large seed) ✅ Done
1 ContentService + LocalFileContentSource + consumables.json + endpoints ✅ Done
2 Generic admin React page (browse + diff + promote button) ✅ Done
3 Migrate Cards (largest; includes wave content + upgrades + cross-refs) ✅ Done
4 Soft-delete migration: change sp_GameAsset_Delete to set Archived ✅ Done (V007)
5 Migrate remaining catalogs (factions, hero-classes, enemies, artifacts, biomes, panels, random-events, quest-templates) ✅ Done
6 AzureBlobContentSource + GitHub Actions promote workflow + auth Deferred (use Entra ID + DefaultAzureCredential when it lands)
7 Clustered-index swap — every Guid PK is NONCLUSTERED; clustered keys live on monotonic columns where one exists ✅ Done (V008)

Deferred decisions (revisit when going live)

These are intentionally not solved in the initial implementation. They block production rollout but not local dev iteration.

Azure Blob storage and authentication

  • Container ACL. Public-read (simplest, content isn't secret) vs SAS URLs vs Managed Identity. Will likely start public-read and add auth if/when needed.
  • API → blob auth. If non-public, the API needs either a SAS URL injected at deploy time, a Managed Identity assigned to the App Service, or a service-principal cert (we already have the MyAppKeyVaultAuth cert pattern).
  • GitHub Actions → blob auth. Almost certainly azure/login + a service-principal stored in repo secrets.
  • API → promote-endpoint auth. Bearer token in repo secrets. Already have a Bearer JWT auth scheme; need an admin-scope role for promote.

Versioning scheme

  • Semver tags (v1.4.2) on releases — manual.
  • Auto-generated (20260429-143012-abc1234) — every push.
  • Probably both: every push gets an auto version, tagged releases also write a semver alias.

latest pointer

  • Either GitHub Action overwrites content/latest/manifest.json on each push,
  • Or DB tracks per-env "current version" and the API resolves "latest" via a DB lookup. (Preferred — latest is per-env, not global.)

Lifecycle policies

  • Hot tier: 90 days.
  • Cool tier: 90 days – 1 year.
  • Delete: > 1 year.
  • Adjust based on rollback-window need.

Local-only escape hatches

  • Content:Source = LocalFile — already in the design, used in dev.
  • POST /admin/content/promote { "source": "upload" } (multipart upload) — one-off ad-hoc imports without going through git. Useful for spike work. May or may not ship in v1.

Out of scope

  • A separate ContentDrafts table.
  • A separate EchoSpire.ContentLoader CLI / PowerShell script.
  • A DeletedAt column.
  • Per-row JSON files (e.g. data/content/cards/{id}.json). Single-file-per-kind for now; revisit if diffs get unwieldy.
  • Concurrent multi-user editing safeguards — single-user (delan) for the foreseeable future.
  • Cross-reference resolution via slugs (e.g. "$ref": "card:phase-strike"). Use raw GUIDs + a validation pass that rejects dangling references.