Groundskeeper
Architecture decisions

ADR 0011: Guardrail catalogue source and artifact structure

  • Status: Proposed
  • Date: 2026-10-03
  • Decision owners: architecture and catalogue maintainers
  • Approval also needed from: security/platform for registry trust; privacy/legal for provenance and redistribution; the relevant governance authority for culturally sensitive material
  • Extends: ADR 0004 (policy bundles and immutable snapshots), ADR 0005 (detector extension model), and ADR 0008 (Indigenous Data Governance)

Context

Groundskeeper needs a catalogue that can grow from a small first-party set into many independently maintained packages without making the evaluator resolve a registry, source tree, or dependency graph during a request. The catalogue will contain:

  • generic and jurisdiction-specific PII definitions, including Australian Commonwealth and state/territory variants;
  • credential and secret definitions grouped by provider or issuer;
  • safety, jailbreak, prompt-injection, and other security checks;
  • reusable policy bundles;
  • schemas, small conformance fixtures, large or restricted corpora, source provenance, generated indexes and documentation; and
  • eventually, third-party definitions and detector implementations.

The existing architecture already separates detector observations from policy decisions, policy source from immutable runtime snapshots, and built-in detector code from future gRPC/WASM extensions. This ADR preserves those boundaries. A catalogue definition describes a concept and the evidence needed to recognize it. A detector implements recognition. A policy decides what to do with a finding. None of those identities are interchangeable.

Decision summary

  1. Use independently versioned catalogue packages as the ownership, release, licensing, and distribution boundary. Begin with a few coherent packages; do not create one package per rule or one global monolith.
  2. Organize paths for browsing, never for identity. Domain is the first browsing axis. Geography is used only where meaningful; provider is the equivalent axis for credentials.
  3. Author one YAML file per coherent definition family by default. Split a family when ownership, provenance, licence, schema, or release cadence differs. A file may contain one item when the item is independently complex.
  4. Give every object a permanent, globally namespaced ID of the form <verified-publisher>/<kind>/<name>. Keep Groundskeeper's public dotted finding taxonomy as category; do not use source paths or detector-private labels as public IDs.
  5. Keep existing repository authorities. Catalogue definitions live under catalog/; public policy bundles remain under policies/bundles/; canonical schemas remain under api/schemas/; cross-package and governed evaluation data remain under test/evaldata/. The catalogue links to these locations rather than cloning them.
  6. Compile and validate off-path. Package releases produce canonical JSON, an exact dependency lock, source/licence inventory, generated index, tests and release evidence. Production activation consumes one flattened, immutable snapshot pinned by OCI digest.
  7. Publish packages and deployment snapshots as distinct OCI artifacts. SemVer tags aid discovery. Digests identify dependencies and activations. Signatures, provenance, release evidence, and SBOM/licence inventories are OCI referrers.
  8. Use schema version, package version, item lifecycle, and evidence quality as separate dimensions. Do not add a second independently resolved SemVer to every definition.
  9. Third parties publish from their own verified namespace. Namespace verification, signatures, and provenance prevent identity collisions and establish origin; conformance and trust policy determine whether Groundskeeper will install the content.

What mature ecosystems teach us

No surveyed ecosystem solves the whole problem. Groundskeeper deliberately combines their strongest properties and avoids their weak catalogue boundaries.

EcosystemPattern worth adoptingLimitation not to copy
OpenTelemetry semantic conventionsTyped YAML is authoritative; narrative and registry pages are generated; definitions have stable fully qualified names, stability, area owners, deprecation, and versioned migrations.One centrally released registry is too coarse for unrelated Groundskeeper publishers and licences.
SemgrepLanguage/framework/category paths are discoverable; rule tests are colocated and require both true-positive and true-negative cases.A local rule slug plus registry path is not a sufficiently durable global identity; the hosted registry must not be the only reproducible index.
SigmaA rule has a durable UUID, lifecycle status, dates, references, licence, and typed relationships such as derived, obsolete, merged, and renamed; deprecated rules remain auditable.UUID-only discovery is poor, and arbitrary extension fields without a closed core profile reduce interoperability.
YARA/YARA-XRule files can group related rules; tags, metadata, includes, modules, and compiler namespaces support flexible collections.The language deliberately supplies little catalogue governance: no universal package manifest, lifecycle, provenance, or test layout.
GitleaksProvider-oriented authoring generates one runtime configuration; each rule has positive and negative examples; generated files are not edited by hand.One generated global configuration and shallow override precedence do not provide independent package release or strong provenance.
Microsoft PresidioGeneric, country-specific, NER, and third-party recognizers are separated; entity, language/country, implementation, context, and confidence are distinct; recognizer tests verify offsets and scores.Python classes/names and a loading manifest are not stable cross-implementation catalogue contracts.
OPA bundlesA package has a manifest, owned roots, language version, tests, signatures, and an all-or-nothing activation unit; schema paths are versioned.Runtime path-based data roots and unordered multi-bundle merging are unsafe substitutes for explicit, locked composition.
OCI/package registriesPackage coordinates are namespaced, immutable content is digest-addressed, and signatures/provenance can be attached as referrers. Maven Central demonstrates verified DNS or source-host namespaces.Tags are mutable discovery handles, not deployment identity; a valid signature alone does not establish safety.

These comparisons lead to a T-shaped hierarchy: shallow, predictable package and domain structure for broad discovery, with additional depth only for an axis that changes ownership or meaning. Deeply mirroring every taxonomy component in the filesystem would create empty directories and brittle moves without adding governance.

Source tree

The target repository layout is:

catalog/
  README.md
  namespaces.yaml                    # first-party namespace claims; registry seed
  owners.yaml                        # package areas and required reviewer roles
  packages/
    groundskeeper.dev/
      pii-core/
        package.yaml
        definitions/
          contact-identifiers.yaml
          network-identifiers.yaml
          payment-card.yaml
        sources.yaml
        tests/
          contact-identifiers.cases.yaml
        README.md
      pii-au/
        package.yaml
        definitions/
          national/
            tax-identifiers.yaml
            health-identifiers.yaml
            social-services-identifiers.yaml
            business-identifiers.yaml
          subdivisions/
            au-act/driver-licence.yaml
            au-nsw/driver-licence.yaml
            au-nt/driver-licence.yaml
            au-qld/driver-licence.yaml
            au-sa/driver-licence.yaml
            au-tas/driver-licence.yaml
            au-vic/driver-licence.yaml
            au-wa/driver-licence.yaml
        sources.yaml
        tests/
        README.md
      secrets/
        package.yaml
        definitions/
          aws.yaml
          azure.yaml
          github.yaml
          stripe.yaml
          generic.yaml
        sources.yaml
        tests/
        README.md
      security-prompt/
        package.yaml
        definitions/
          prompt-injection.yaml
          jailbreak.yaml
          tool-abuse.yaml
        sources.yaml
        tests/
        README.md
      safety-core/
        package.yaml
        definitions/
        sources.yaml
        tests/
        README.md
api/
  schemas/
    catalog/
      v1alpha1/
        package.schema.json
        definition-set.schema.json
        source.schema.json
        test-case.schema.json
        index.schema.json
        lock.schema.json
policies/
  bundles/                            # sole source authority for policy bundles
  examples/
test/
  conformance/                        # execution-mode-neutral detector contract tests
  evaldata/
    manifests/                        # corpus metadata, access and digest manifests
    public/                           # redistributable corpus partitions only
docs/
  catalog/                            # generated, committed browsing documentation
  decisions/
dist/                                 # generated, ignored; never hand edited

Why packages, and when to split one

A package is the smallest independently published unit. Everything in one package shares:

  • version and changelog;
  • maintainers and required approval policy;
  • default licence and redistribution posture;
  • dependency lock and compatibility range;
  • OCI artifact and release cadence.

Create a new package when one of those properties materially diverges. Do not split only because another directory level is possible. For example:

  • Start with pii-au, not nine packages for Australia. Split a state package only if an authority, licence, owner, or change cadence becomes independent.
  • Start with secrets, with one family file per provider. Split secrets-aws when provider-specific ownership or release urgency makes the larger package an operational bottleneck.
  • Keep safety and prompt-security packages separate because their evidence, reviewers, corpora, and model dependencies differ even if both inspect text.
  • Never put executable detector binaries in a definition package. A detector package has the separately defined detector manifest and release lifecycle.

This rule gives independent versioning where it has value without recreating the dependency and review overhead of a package per item.

File granularity

The default is one file per coherent family, usually one to roughly ten definitions. Keep variants together when they share the same semantics, validator, sources, maintainers, tests, and release cadence. Examples include the related Australian healthcare identifiers IHI, HPI-I, HPI-O, and HSP-O.

Split definitions when any of the following is true:

  • a source, licence, access classification, or approver differs;
  • an item has substantial independent rationale or test data;
  • reviewers routinely change one subset without understanding the other;
  • merge contention becomes material; or
  • the family would mix unrelated validators merely to reduce file count.

Do not enforce one-file-per-item mechanically. It creates hundreds of tiny files with duplicated provenance and obscures families. Do not use one file per whole jurisdiction or package either; those become merge and ownership hotspots.

Geography, authority, language, and provider

Paths represent the dominant browsing axis, while metadata represents all axes:

  • generic PII is in pii-core;
  • country-specific PII uses ISO 3166-1 alpha-2 in package names and metadata;
  • subdivisions use ISO 3166-2, lower-case in paths (au-nsw) and canonical upper-case in metadata (AU-NSW);
  • national/ means applicable across Australia. Whether the issuer is a Commonwealth, state/territory, private, or other authority is a separate issuingAuthority.level field. This avoids treating “national” and “federal” as competing geographic levels;
  • language and cultural applicability are explicit metadata, not nested under a country unless the definition is actually jurisdiction-bound;
  • credentials use provider/issuer as their browsing axis, not country; and
  • a provider, country, language, entity, implementation, and policy consequence remain independent metadata dimensions.

No definition may infer Indigeneity from a name, language, location, embedding, or correlation. Relevant community material requires the authority, access, withdrawal, and review process specified by the Indigenous Data Governance ADR.

Identity and naming

Coordinates and IDs

The following identities are distinct:

IdentityExamplePurpose
Publisher namespacegroundskeeper.devVerified ownership and collision boundary
Package coordinategroundskeeper.dev/pii-auUnit of source ownership, SemVer release, and OCI publication
Object IDgroundskeeper.dev/definition/privacy.pii.tax_id.au.tfnPermanent audit/reference identity
Public finding categoryprivacy.pii.tax_id.au.tfnStable interoperable taxonomy reported in findings
Detector IDgroundskeeper.dev/detector/pii-regex-checksumImplementation identity from the detector contract
Policy/rule IDgroundskeeper.dev/policy-rule/block-validated-tfnOverride and audit target
Schema API versiongroundskeeper.dev/v1alpha1Authoring document contract
Package version1.4.0Human compatibility/discovery version
OCI digestsha256:…Exact immutable package or snapshot identity

Object IDs use lower-case ASCII and the grammar:

<verified-publisher>/<kind>/<dot-separated-name>

The publisher may be a verified DNS name (example.com) or an approved source-host namespace (github.com/example). Core kinds are definition, detector, validator, policy-bundle, policy-rule, schema, and corpus. Names use lower-case [a-z0-9][a-z0-9_-]* components separated by dots. IDs are immutable, path-independent, and never reused.

The public category remains the existing hierarchical taxonomy. A third-party detector may emit a Groundskeeper category without claiming ownership of it. A new third-party concept uses its fully namespaced object ID and may propose a separate mapping into the public taxonomy. Detector-private labels such as AU_TFN are aliases only.

Human-readable IDs are preferable to UUIDs for discovery, but Sigma demonstrates the value of an identity that survives renames and forks. Groundskeeper obtains the same collision resistance from the verified publisher prefix and immutability. Every compiled item also receives a generated content digest. A fork that changes semantics must mint an ID in the fork's namespace and record derivedFrom; an exact mirror preserves upstream IDs and digest.

Namespaces for third parties

The public registry must verify DNS control with a TXT challenge or source-host account control before granting publish rights, following Maven Central's namespace model. Publication additionally requires an expected signing identity.

  • Losing a domain does not transfer existing IDs automatically.
  • Namespace transfer requires an auditable registry action, old/new owner approval where possible, a notice period, and signer rotation.
  • An abandoned namespace is frozen, not recycled.
  • A package move records old and new coordinates; object IDs remain unchanged.
  • Mirrors cannot become authoritative merely by republishing an artifact.
  • Private registries may allocate internal namespaces but must include a DNS or organization authority that cannot collide when artefacts are exchanged.

The registry index is a discovery service, not the source of truth for artifact contents. Clients verify coordinate, manifest, digest, namespace claim, expected signer, and provenance independently.

Source documents and manifests

Package manifest

Each package has exactly one package.yaml:

apiVersion: groundskeeper.dev/v1alpha1
kind: CataloguePackage
metadata:
  namespace: groundskeeper.dev
  name: pii-au
  version: 0.3.0
  lifecycle: development
  title: Australian identifier definitions
  description: Generic Australian and state/territory identifier evidence
  license: Apache-2.0
  owners:
    - team: pii-maintainers
  requiredReviews:
    - privacy
    - australian-domain
compatibility:
  catalogueSchema: ">=0.1.0 <0.2.0"
  hostProtocol: ">=0.1.0 <0.2.0"
imports:
  - package: groundskeeper.dev/pii-core
    version: 0.2.1
exports:
  roots:
    - groundskeeper.dev/definition/privacy.pii
content:
  definitions:
    - definitions/**/*.yaml
  sources: sources.yaml
  tests:
    - tests/**/*.cases.yaml

The compiler rejects unknown core fields. Vendor metadata is allowed only below extensions.<verified-publisher> so typos in security-relevant fields do not silently become ignored extensions.

Definition family

A family document contains independently addressable definitions and shared defaults; defaults are expanded during compilation and never act as runtime inheritance:

apiVersion: groundskeeper.dev/v1alpha1
kind: DefinitionSet
metadata:
  package: groundskeeper.dev/pii-au
  name: tax-identifiers
  owners:
    - team: pii-maintainers
defaults:
  jurisdictions:
    - country: AU
  languages: [en]
definitions:
  - id: groundskeeper.dev/definition/privacy.pii.tax_id.au.tfn
    category: privacy.pii.tax_id.au.tfn
    title: Australian Tax File Number
    lifecycle: development
    evidenceStatus: controlled
    classification:
      - personal-sensitive
      - government-related
    issuingAuthority:
      level: commonwealth
      name: Australian Taxation Office
    recognition:
      validationLevel: context-required
      validator: groundskeeper.dev/validator/au-tfn
    sources:
      - au-ato-tfn-format
      - au-oaic-tfn-guidance
    tests:
      - tests/tax-identifiers.cases.yaml

Keep legal/privacy classification orthogonal to structural recognition. A valid ABN, BSB, public professional identifier, or infrastructure identifier is not automatically personal-sensitive or subject to redaction.

Source provenance

sources.yaml deduplicates citations within a package. Every source record has:

  • stable package-local source key;
  • publisher, title, document version or publication date;
  • canonical URL and retrieval date;
  • content digest and, where permitted, an archived or vendored snapshot path;
  • authority class (primary, standard, vendor, secondary, or research);
  • claims supported, limitations, and freshness/review date;
  • SPDX licence expression where applicable, copyright holder, attribution, redistribution and commercial-use constraints;
  • access class (public, licensed, controlled, or restricted); and
  • reviewer and approval reference for controlled algorithms or data.

Authoritative facts and heuristic choices are separate fields. A URL's public access does not imply permission to redistribute its content. Restricted algorithms, standards, model weights, and datasets are referenced by digest and approval metadata but are not copied into a public artifact unless their terms allow it.

Imports and composition

Packages are modules, not inheritance trees:

  • Imports are explicit, package-level, exact SemVer references in reviewed source. Floating ranges are not permitted in committed release source.
  • Publication resolves every import to an OCI digest and writes lock.json.
  • Object references cross a package boundary by stable ID, never relative path.
  • The complete import graph must be acyclic. Duplicate IDs, undeclared shadowing, incompatible export roots, ambiguous aliases, and missing references fail the build.
  • Shared defaults may be expanded only inside one source family. There is no cross-package YAML merge, implicit inheritance, or last-writer-wins behavior.
  • extends is permitted only for a schema-defined semantic refinement and must remain visible in compiled provenance. It is not a general merge operator.
  • Definitions can declare requires capabilities or validators. They cannot load executable code.

Policy bundles remain under policies/bundles/. They reference catalogue object IDs and package digests and use ADR 0004's typed override rules. Catalogue packages never embed tenant policy, allowlists, endpoints, credentials, trust roots, or mandatory operator defaults. Provider-level false-positive knowledge that is universally true may be detector evidence; tenant-specific suppression is policy.

Ownership and review boundaries

catalog/owners.yaml is the structured source for generated area documentation and CODEOWNERS sections. Each package has a primary and backup maintainer group, required specialist roles, lifecycle (active, inactive, or needs-maintainer), and escalation contact. Path ownership may narrow a package for a jurisdiction or provider but cannot waive package-level approval.

Required reviews are determined by the changed claims:

ChangeRequired review
Core schema, ID, compiler, package/export ruleCatalogue architecture maintainers
PII definition or classificationDomain maintainer plus privacy; jurisdiction expert for local claims
Controlled algorithm, standard, registry, or datasetDomain maintainer plus legal/licence/data owner
Provider credential patternSecrets maintainer plus provider/source evidence reviewer
Prompt injection, jailbreak, or security ruleSecurity maintainer and threat-model reviewer
Safety definition/model/corpusSafety owner plus risk/governance appropriate to the affected population
Indigenous names, languages, knowledge, or community materialIndigenous-led authority defined by ADR 0008; ordinary CODEOWNERS approval is insufficient
Public policy bundlePolicy owner plus security for weakening/override changes
Namespace or signing identityRegistry security/platform owner

Inactive areas accept fixes, deprecations, and security responses but not feature expansion until ownership is restored, following OpenTelemetry's area model. Authoring, release approval, artifact publication, and production activation are separate roles. A maintainer cannot make an unreviewed artifact trusted merely by publishing it.

Tests and corpora

Colocated package tests

Small, redistributable, deterministic cases live in each package's tests/ directory. Every definition must include inputs that distinguish the intended implementation from plausible wrong ones:

  • true positives and near-miss true negatives;
  • boundary lengths, malformed separators, case and Unicode variants;
  • invalid and valid checksums when public algorithms exist;
  • misleading context and known false positives;
  • expected category, exact Unicode code-point span, validation state, confidence/evidence class, and coverage;
  • synthetic/nonfunctional credential canaries, never live credentials;
  • regression cases for fixed incidents; and
  • declared engine/validator compatibility and performance budget.

This adopts Semgrep's same-basename proximity, Gitleaks' positive/negative rule validation, Presidio's offset/confidence checks, and Sigma's benign-baseline testing. Tests must assert semantic output, not only successful parsing.

Evaluation corpora

Large, shared, generated, licensed, private, or governed corpora do not sit next to source definitions. test/evaldata/manifests/ contains versioned corpus manifests with:

  • globally namespaced corpus ID, version, partitions, and content hashes;
  • source/generation method, seed, transformations, labels, and known limits;
  • licence, consent/authority, permitted purpose and users;
  • redistribution, commercial use, external-judge, cross-border, retention, storage, withdrawal, and deletion constraints;
  • access location or retrieval procedure without embedding credentials;
  • linked definition/package versions and expected metrics; and
  • owner, review date, and incident/change history.

Only redistributable, non-sensitive content belongs in test/evaldata/public/. Restricted material remains in its approved store; CI receives it only in an authorized environment. Release evidence records corpus IDs and hashes, not raw restricted content. Never use real identifiers merely because they are publicly searchable. Checksum-valid synthetic identifiers are labelled as synthetic and not guaranteed unissued.

Generated indexes and documentation

The compiler generates, in one deterministic operation:

  • a repository index of package coordinates, versions, owners, objects, categories, jurisdictions, providers, languages, lifecycle/evidence status, licences, source freshness, dependencies, replacements and documentation;
  • one documentation page per object plus package indexes, support matrices, dependency/replacement graphs, and changelog;
  • expanded canonical JSON with defaults resolved;
  • lock.json with source package version, OCI digest, object IDs/content digests, schema digest, compiler version, and transitive dependencies;
  • licence/notice inventory and source-provenance summary; and
  • release test/coverage report.

docs/catalog/ and the machine-readable public index are committed for browsing and reviewed through generated diffs. CI regenerates and fails on drift. dist/ is build output and is ignored. Generated files carry a “do not edit” header; fixes occur in source, schemas, templates, or generator code. Narrative rationale may be hand-authored in package READMEs, while factual tables are generated from the catalogue as OpenTelemetry does.

Schema evolution and compatibility

There are four independent lifecycle axes:

  1. apiVersion versions the YAML/JSON document schema.
  2. Package SemVer versions the package's exported semantic contract.
  3. lifecycle describes an item: development, experimental, stable, deprecated, or unsupported.
  4. evidenceStatus describes source confidence/access: verified, provisional, controlled, obsolete, or rejected.

Do not give every definition an independently resolved SemVer. The package is the atomic tested unit, and the package digest plus stable object ID exactly identifies an item's semantics. Per-item SemVer would multiply dependency states without making items independently distributable. The generated index includes each item's content digest, introduction/deprecation package versions, and modification history for precise audit.

Before v1, v1alpha1 may change incompatibly and every release must declare that fact. Once v1 is frozen:

  • adding optional fields or new objects is schema/package minor;
  • compatible documentation, source-freshness, and false-positive test fixes are patch when they do not alter output;
  • expanding recognition, confidence, validation state, default enablement, or evidence in a way consumers observe is at least minor and requires decision diff review;
  • removing/renaming a field, category, stable object, or changing its meaning is schema/package major;
  • correcting unsafe behavior may disable an item immediately as unsupported, but does not rewrite an already published artifact; and
  • released schemas and OCI digests are immutable.

Schemas use JSON Schema 2020-12 at immutable versioned URLs. Core objects are closed except for namespaced extension points. CI validates source against the current schema and prior stable releases for forbidden changes. Migrations record renames, aliases, splits, merges, default changes, and replacement IDs, following OpenTelemetry's schema migrations and Sigma's typed relationships.

Deprecation and replacement

Deprecated items are tombstones, not deleted history. Each records:

  • deprecatedIn, reason, planned support end, and migration guidance;
  • typed relations: replacedBy, renamedFrom, derivedFrom, mergedFrom, or similarTo;
  • historical aliases and source/provenance; and
  • whether default bundles still include it.

A stable ID is never reassigned. Default bundles stop selecting a deprecated item after the documented grace period; compatibility bundles may retain it through a major support window. unsupported is for unsafe, invalid, or unmaintainable content and is excluded from new builds. rejected evidence cannot substantiate a validation claim but remains recorded so folklore does not re-enter later.

Compiled artifacts and OCI distribution

Independently published catalogue package

Each package release compiles to a deterministic tar layer:

manifest.json                         # coordinate, version, schema, exports, compatibility
catalog.json                          # expanded definitions, sorted by stable ID
lock.json                             # exact transitive package and schema digests
sources.json                          # distributable provenance and freshness metadata
licenses/
  LICENSES.json                       # SPDX expressions, notices, attribution
schemas/
  definition-set.schema.json          # exact schema needed to inspect this release

Tests and corpora are not loaded by the evaluator. Public test fixtures and release evidence may be a separate evidence artifact that refers to the package digest. Restricted data is never published to OCI merely to make the release self-contained.

Use RFC 8785 JSON Canonicalization Scheme for canonical JSON and hashing rather than a local “sort keys and whitespace” convention. Inputs must satisfy I-JSON; the compiler rejects duplicate keys, non-finite numbers, and values not safely representable by the schema. YAML aliases and merge keys are rejected to keep review and canonicalization unambiguous.

Publish using an OCI image manifest with:

  • artifact type application/vnd.groundskeeper.catalog.package.v1;
  • config type application/vnd.groundskeeper.catalog.package.config.v1+json;
  • package layer type application/vnd.groundskeeper.catalog.package.layer.v1.tar+gzip;
  • OCI source, revision, version, title, description, created and licence annotations; and
  • repository path <registry>/groundskeeper/catalog/<publisher>/<package>.

SemVer tags such as 1.4.0 and optional channels such as stable are discovery handles. Released version tags must be protected, but consumers still resolve and store the manifest digest. Artifact media types should be registered before the format is declared stable.

Deployable runtime snapshot

The control plane resolves packages and policy layers, verifies all inputs, runs tests/replay gates, and emits a second OCI artifact:

manifest.json                         # snapshot format and host compatibility
catalog.json                          # flattened definition closure
policy.json                           # fully composed policy and typed overrides
detectors.json                        # required capabilities and pinned implementations
bindings.json                         # approved deployment bindings/correlation snapshot
lock.json                             # every input coordinate, version and digest
provenance.json                       # source commit/build summary; attestation is a referrer

This snapshot uses artifact type application/vnd.groundskeeper.snapshot.v1. It is complete, immutable, and self-contained for evaluation. The runtime never downloads package imports, resolves SemVer, merges policy, reads catalogue YAML, or queries a registry/database during a request. It verifies the snapshot digest, expected signer identity, provenance subject, schema/host compatibility, and capabilities before atomically swapping the whole snapshot. Failure leaves the last-known-good snapshot active.

Flattening duplicates shared definitions across deployment snapshots, but buys deterministic network-free evaluation, atomic rollback, and simple incident reproduction. Independent source/package versioning is a control-plane concern; it must not create partial runtime activation.

OCI 1.1 subject and the Referrers API associate these with the package or snapshot digest:

  • Sigstore/cosign signature or approved KMS/PKI signature;
  • SLSA provenance whose subject equals the OCI digest and whose materials include source revision, locked packages, schemas, compiler, and relevant generated inputs;
  • SPDX SBOM/licence inventory where executable detector content exists;
  • catalogue source/licence inventory for definition-only packages;
  • release test, corpus-hash, compatibility, and decision-diff attestation; and
  • revocation or verification-summary information.

Verification checks expected signer identity and issuer, builder/workflow, source repository, build type, parameters, provenance subject, and digest. “A valid signature exists” is insufficient. Trust roots and accepted publishers are operator configuration, not tenant policy or package content. Signing establishes origin and integrity, not semantic quality, legal permission, or safety.

Discovery

The public index supports filtering without parsing every package:

  • kind and public category;
  • package, publisher, lifecycle, evidence status and trust tier;
  • country, subdivision, issuing-authority level, language and cultural scope;
  • provider/issuer, modality, stage and detector capability;
  • validation level (shape, context, checksum, reference, registry/service);
  • licence, redistribution/access class and source freshness;
  • owners, last review, compatibility and deprecation/replacement;
  • available policy bundles and corpus/release evidence; and
  • OCI coordinate, SemVer and digest.

The index is generated from signed package manifests and can be mirrored. Search ranking and trust are registry concerns; neither changes object identity. Local install tools can build the same index from a set of OCI digests for air-gapped use.

Publication and activation gates

Publication is all-or-nothing:

  1. Parse YAML safely and validate closed schemas.
  2. Verify IDs, namespace ownership, export roots, source records and licences.
  3. Resolve the acyclic import DAG and produce the exact lock.
  4. Validate references, aliases, lifecycle transitions and compatibility.
  5. Run package cases, conformance tests, benign baselines and authorized corpus replay; compare old/new findings, spans, confidence and policy outcomes.
  6. Generate canonical JSON, docs, index, notices and release evidence; fail on generated drift.
  7. Build the OCI artifact reproducibly, generate provenance, approve, sign and publish without mutating an existing release.

Activation separately verifies trust policy and the complete deployment snapshot, then uses the existing shadow, deterministic canary, atomic promotion and rollback process. Registry availability is not required after an artifact is installed.

Consequences and trade-offs

Benefits

  • Packages allow PII jurisdictions, provider secrets, safety domains, and third parties to release independently without package-per-rule overhead.
  • Stable namespaced IDs keep audits, overrides, aliases, and deprecations valid across source moves.
  • Family files keep shared evidence visible while limiting merge hotspots.
  • Explicit provenance and corpus controls support Australian restricted sources, commercial datasets, and culturally governed material without pretending all public URLs are redistributable.
  • Generated indexes/docs make a distributed package ecosystem discoverable.
  • OCI reuses registry RBAC, replication, content addressing, signatures and referrers while the flattened snapshot keeps the data plane simple.

Costs

  • A compiler, namespace registry, schema compatibility checker, documentation generator and release pipeline are required earlier than with loose YAML.
  • Package boundaries require judgement and occasional package splits.
  • Flattened snapshots duplicate content and are larger than deltas.
  • Closed schemas and mandatory provenance make quick ad hoc contributions slower.
  • DNS/source-host namespace verification needs transfer, abandonment, and signer operations.

These costs are accepted because silent collisions, untraceable validation claims, runtime dependency resolution, and licence-blind redistribution are substantially more expensive to correct after the catalogue becomes public.

Rejected alternatives

One repository-wide catalogue file

Rejected because it creates merge contention, global ownership, one release cadence, and weak provenance boundaries. A generated aggregate remains useful, but it is not source.

One file and package per item

Rejected as the default because hundreds of tiny packages multiply manifests, versions, signatures, dependency locks and review overhead. Use one item per file only when its complexity or governance merits it.

Directory path as stable ID

Rejected because taxonomy cleanup, package splits, and ownership moves would become breaking changes. Paths are navigation only.

Country-first global hierarchy

Rejected because credentials, safety and prompt security are not principally geographic. Domain-first with optional jurisdiction/provider axes is more stable.

Put definitions, policies, schemas and corpora under one catalog/ tree

Rejected because it would duplicate the established authorities under policies/, api/schemas/, and test/, and would blur different ownership, licensing and runtime roles.

Per-item SemVer

Rejected because items are not independently distributed and tested. Package SemVer, immutable item ID, per-item content digest, and OCI digest provide exact auditability with a tractable dependency graph.

UUID-only IDs

Rejected because they are collision-resistant but poor for authoring and discovery. Verified publisher namespaces provide collision resistance while preserving readable identifiers. Optional imported ecosystem UUIDs remain aliases.

Runtime loading of source packages or multiple OCI bundles

Rejected because dependency ordering, partial failure, registry outage, and inconsistent activation would enter the request path. Only the compiled complete snapshot is executable.

Signatures as the trust decision

Rejected because signatures prove possession/identity, not correctness, safety, licensing, or governance approval. Trust policy, provenance verification, conformance and release review are separate gates.

Decisions still requiring review

This ADR recommends defaults but cannot grant the required organizational or legal authority. Reviewers must decide:

  1. Whether the first public schema is groundskeeper.dev/v1alpha1 and when its compatibility promise begins.
  2. Which initial package split is funded. The recommended seed is pii-core, pii-au, secrets, security-prompt, and safety-core.
  3. Whether the permanent ID grammar and core public taxonomy are approved before definitions are published.
  4. Which team operates namespace verification, transfer, abandonment, registry moderation, signing identity, revocation and trust roots.
  5. Which OCI registry and compatibility fallback are supported, and whether the proposed media types will be registered.
  6. Exact CODEOWNERS teams and named privacy, legal, Australian-domain, provider, security, safety and governance approvers.
  7. The controlled-source decisions already tracked for ATO algorithms, DVS, PAF, G-NAF, standards, retention and redistribution.
  8. The Indigenous-led authority and workflow required before relevant catalogue definitions or corpora can be created or released.
  9. Numeric quality, benign-pass, latency and regression gates by risk tier.
  10. Policy DSL/override approval remains under ADR 0004; this ADR deliberately does not settle tenant weakening or failure posture.

Sources

External facts and upstream formats were reviewed on 2026-10-03. Pin upstream versions when implementation begins; links to moving branches document the surveyed projects, not immutable build inputs.

Groundskeeper-specific legal, Australian PII, Indigenous Data Governance, conformance, policy-bundle, detector-extension, and persistence sources remain in the repository research index and their respective design documents.

On this page