Skip to content

Core 10: Capstone — Proving, Shipping, and Sustaining a FuncPipe System

Module 10 (Conclusion)

Core question:
How do you demonstrate end-to-end mastery of FP-in-Python by shipping a real FuncPipe system with formal laws, measurable guarantees, enforced boundaries, upgrade safety, performance budgets, and a credible scalability roadmap?

This is the payoff.
You ship one complete, audited FuncPipe system (new or deeply refactored) that satisfies every requirement below. The entire governance/CI scaffold may be copied verbatim from the official template repository; graduation assesses correct FP architecture and invariant design, not infra originality.

Motivation Bug: FP knowledge that never ships with verified, versioned, evolvable boundaries remains hobby-grade.

Delta from Core 9: Governance was mostly manual. Core 10 makes it mechanically enforced where reliable, explicitly best-effort elsewhere, and always secondary to architectural separation and review.

Capstone Contract (Normative – All Required for Graduation): - Pure core / effect shell decomposition enforced by import-linter + best-effort AST bans. Architectural separation and code review are the primary defense against impurity; static tools are secondary guardrails only. - Typed public contract surface (events, ports, configs, graphs) using pydantic v2 models, semver (major-only public changes), canonical JSON serialization (strict, cross-language-stable rules below), and normalized JSON schema hashing. - All effect boundaries use explicit protocols (e.g., Protocol[Clock] → datetime). Property tests prove core remains deterministic and equivalent across multiple conforming protocol implementations (pure + representative effectful ones) supplied in the test suite. - Every declared invariant mechanically linked to Hypothesis property tests via invariants.md + governance/check_invariants.py. - Every unsafe extension justified by performance budget, categorized (implementation-only vs semantic), proven with PBT. Semantic-unsafe changes require RFC approval and major version bump of all affected contracts/graphs. - Float fields in contracts forbidden by default: enforced by pydantic runtime validation + @allow_float marker + RFC requirement. - CI fails on: cross-context core imports, detectable observable impurity, law regressions, contract compatibility breaks, modifications to old contract versions, canonical serialization drift, schema hash drift. - Performance budget with CI enforcement using ASV + the single blessed statistical model (Welch’s t-test, rolling 30-run baseline, p<0.01, MAD outlier rejection) defined in budget.yaml. Overrides require RFC. - ParaPipe readiness plan with explicit semantics for partitioning, merging, ordering, delivery, backpressure, deduplication, failure handling, and reconciliation. Any semantic affecting ordering, lossiness, deduplication, delivery guarantees, retries, or merge algebra requires corresponding semantic property tests. - Release process: CHANGELOG.md, MIGRATIONS.md, deprecation enforcement. - Old contract versions immutable. - System-level golden run with canonicalized output snapshot + hash. - Golden files regenerated exclusively via the official regen tool, which is version-tagged in a protected upstream repository. CI verifies golden files were produced by the currently approved tagged tool version (fetched from protected ref; PRs cannot override).

Graduation Artifact: A complete repository that passes all CI checks and serves as a drop-in template for real production pipelines.

1. Graduation Checklist (Enforceable Where Reliable)

Requirement Enforcement
Core/Shell Separation import-linter + ast_bans.py (best-effort) + review
Effect Protocols + Determinism Explicit protocols + Hypothesis tests across multiple implementations
Typed, Versioned Contracts pydantic v2 + normalized JSON schema hash + canonical golden tests
No Unapproved Floats pydantic validation + @allow_float + RFC
Law Suite Hypothesis + check_invariants.py
Impurity Detection ast_bans.py + ruff + import-linter (explicitly best-effort)
Performance Budget CI ASV + blessed statistical model
ParaPipe Readiness Plan PARAPIPE_PLAN.yaml + required semantic property tests
Golden File Trust Protected upstream tool tag + CI verification

2. Deliverables (Reference Repo Layout – Functional Equivalents Explicitly Allowed)

Shared utilities (law strategies, fixtures, contract helpers) belong in top-level funcpipe/ packages with strict OWNERS.

my_funcpipe_system/
  README.md
  funcpipe/
    testing/                # shared strategies, fixtures, protocol impls
    contracts/              # shared canonical utilities, validators
  contexts/
    cleaning/
      OWNERS
      invariants.md
      core.py
      unsafe/
      adapters/
      shell.py
  contracts/
    _canon.py
    v1/
      CleanDocProduced.py
      CleanDocProduced.schema.json.sha256   # normalized export
      CleanDocProduced.golden.json
    v2/
      ...
  laws/
    test_invariants.py
    test_compat.py
    test_parapipe_properties.py
  governance/
    regen_goldens.py        # official tool, tagged in protected upstream
    ast_bans.py
    check_invariants.py
  bench/
    budget.yaml             # blessed statistical model
  docs/
    CHANGELOG.md
    MIGRATIONS.md
    PARAPIPE_PLAN.yaml
  tests/
    test_e2e_smoke.py

3. Required Proofs (Final Specification)

3.1 Observable Purity & Effect Boundaries

  • AST bans are best-effort guardrails (documented limitations in file header).
  • All effects go through typed protocols.
  • Test suite supplies multiple protocol implementations (pure + realistic effectful) and proves identical core results.

3.2 Canonical JSON Serialization – Cross-Language Stable Rules

Designed for Rust/TS/Go/Java interop. Float fields forbidden unless @allow_float + RFC.

  1. Sorted keys (depth-first)
  2. Numeric values → Decimal with exact str(decimal) (float forbidden by default)
  3. NaN / ±Inf → rejected at pydantic validation
  4. datetime → UTC ISO8601 with Z, microseconds if non-zero
  5. bytes → base64
  6. UUID → lowercase string
  7. Include None and empty containers
  8. Lists are ordered sequences (declare ordering invariants if meaningful)
  9. Schema hash = SHA256 of model.model_json_schema() after stripping: title, description, examples, $comment, deprecated, and any pydantic-internal keys. Pydantic version pinned in pyproject.toml lockfile.

3.3 Golden File Integrity

governance/regen_goldens.py is the sole authority and is released as immutable git tags in the protected upstream template repo. CI fetches the approved tool version from the protected ref and fails if any golden file does not match it.

3.4 ParaPipe Readiness Plan Examples

Good: 

partition_key: "doc_id"
merge_algebra: "list_concatenation"
ordering: "stable_per_partition"
delivery: "at_least_once"
backpressure: "bounded_task_queue + drop_newest"  # requires lossiness property test
dedup_key: "chunk_id"
failure_model: "idempotent_replay_from_checkpoint"
reconciliation: "last_writer_wins_per_doc_id"
required_property_tests:
  - merge_associativity
  - merge_identity
  - idempotency_under_replay
  - drop_newest_max_loss_bound

Bad (fails graduation): 

merge_algebra: "concat_i_guess"
ordering: "whatever"
# no property tests for ordering or lossiness → fail

3.5 Performance Budget

Single blessed model in template budget.yaml. No deviations without RFC.