Small-Team Multi-Environment Secret Management: SOPS + age + Doc Permissions as Credentials

2026-08-06#secret-management#devops#security#infra#tooling

Background: the problem was process, not encryption

In a multi-service backend project, runtime configuration was once scattered across three places: development config files committed to the repo, hand-maintained plaintext files on the deployment host, and a pile of local envs protected only by .gitignore. Real credentials (model-vendor API keys, JWT secrets, database passwords) had no home that was "reviewed and versioned" — the README taught developers to paste keys into git-tracked files, one git add away from a permanent leak, while the host-side config drifted silently.

The goal was never "stronger encryption" but four process outcomes: one manifest per environment carrying every environment-dependent value; ciphertext committable into the repo; a newcomer able to clone and run; CD decrypting with an environment-level key.

Core model

One env.yaml per environment, encrypted with SOPS + age and committed into the repo as the single source of truth for that environment's values. Shape lives in templates, values live in the manifest, and one render step composes both into each service's startup config — there is no per-environment copy of a full config file. One age key pair per environment; the dev key can never decrypt prod. Private-key distribution creates no new account system: keys go into per-environment-isolated documents on the collaboration platform (a Feishu Base), where document access permission itself is the credential — a newcomer granted the dev document runs one idempotent setup command that fetches the key, decrypts, renders and boots the stack. The CD pipeline's key comes from the pipeline's own environment-level secret store, never from the document platform.

Key decisions

Ciphertext lives in the repo, and the repo is the sole authority. The vault on the document platform is only a mirror + recovery channel; convergence always flows repo → vault, one-way. If someone hand-edits the vault, the sync command pulls it back.
Encrypt per key, not per file. SOPS encrypted_regex only encrypts keys whose names look like credentials ((?i)(key|secret|password|token|dsn|credential)); structure and non-secret values stay readable in PR diffs. This is exactly why git-crypt lost: whole-file binary diffs are unreviewable, and a mis-ordered .gitattributes can leak plaintext into history. Review readability is this scheme's first constraint. But face its weakness squarely: coverage is decided by a key-name heuristic — a secret stored under a name that doesn't look like a credential (database_url, webhook_url and the like) lands in the repo as plaintext, and CI validates SOPS metadata and key parity, which cannot catch it. The real manifest's rule did in fact later get database_url patched in this way. So the key-name regex can only be the last net: a new leaf first goes through the config catalog's sensitivity classification (see item 8), and a guard script additionally rejects plaintext connection-string fields in tracked files. The more thorough design drives coverage from the sensitivity classification, or inverts the default (encrypt everything, explicitly exempt non-secrets) at the cost of some diff readability.
Reuse the collaboration platform's document permissions as the boundary. Three environments use three independent documents rather than one document with three tables, because the platform's permissions inherit downward — distinguishing environments inside a single document relies on advanced permission rules, where one misconfiguration exposes the prod key to everyone. The "independent documents + explicit membership" model is the dumbest one available, which is exactly why it is the hardest to misconfigure.
PR CI never holds a key. The PR stage validates using only the plaintext example manifest (fake values): ciphertext must carry SOPS metadata; ciphertext and example must have key parity; keys must satisfy the cross-environment contract (whatever dev has, staging must have; staging and prod must be isomorphic) — purpose-built to catch "only added to dev" and "forgot prod". Decryption happens only in trusted release jobs, with keys from environment-level secrets, and plaintext wiped when the job exits. Untrusted PR code can never touch a key.
A missing key must not block development. Someone without document access who runs apply automatically falls back to the example manifest + deterministic mocks, and everything still runs — but only dev is allowed this fallback; staging/prod refuse it.
Every credential has exactly one owner. All model-vendor keys exist only under the model gateway service's config subtree; consumer services get only the gateway's base URL and an internal token, and no service may hold its own copy of a vendor key. Rotation then touches exactly one place.
The vault stores only credentials and their companion identities. Initially every config leaf was mirrored into the vault; it turned out that once non-secret values acquire their own write surfaces (local JSON, a config center), the mirror table becomes a drift source. It shrank to: age keys, secret values, and the identities needed to use those secrets (usernames, account IDs). The vault is not a config browser.
Classify first, then pick the channel. A machine-validated config catalog annotates every config leaf: lifecycle (boot-time / hot-reload), sensitivity, owner, rollback method. Secrets and boot-time values go through the encrypted manifest channel; non-secret hot values go through the runtime config channel. Before changing a value, look up its class — don't pick a place by feel.

Practices validated in the flow

Onboarding is one command: install the toolchain, platform login, auto-fetch the key (written to a 0600 local file), decrypt, render, boot the whole stack. No human hands a key to anyone at any point.
Changing a value is one command: silent input of the new value, in-place re-encryption, local config re-render, mirror to the vault, and the change shows up as a PR. The reviewer sees key names and structural changes in the diff; the concrete values are looked up in that environment's vault — reviewing "what changed" is decoupled from keeping "what it changed to" secret.
Deleting a credential row is an explicit operation: routine sync never deletes vault rows; cleanup must first dry-run the rows to be deleted, get human confirmation, then run with an explicit flag. This guards against "sync casually deleting a credential still in use".
Rotation has two layers, and conflating them is the most dangerous misunderstanding: rotating the age key only stops the bleeding, and operationally it must be sops rotate, which regenerates the data key — updatekeys merely re-wraps the old data key with the new age key, so anyone who ever had any historical version can recover that data key and keep decrypting even the new ciphertexts that follow. And even after a rotate, anyone who held the old key can still decrypt all historical commits — the inherent price of ciphertext-in-repo. The real fix is rotating the actual credentials (vendor keys, JWT secrets, database passwords) as well. When a member leaves or a laptop is lost, rotate both layers, and write that into the runbook as a fixed move.
Plaintext in history is not rewritten away: if the guard is bypassed and a plaintext secret lands in a commit, the correct move is to treat that value as burned, rotate the real credential immediately, then fix the manifest — not to rebase the commit away and pretend it never happened.
Guard rails are fully automated: pre-commit and CI run the same checks — every tracked manifest must carry SOPS metadata, an age private key anywhere in the repo is an instant reject, and the example must contain no real-shaped secrets. A post-merge hook re-renders local config after pulls touching the deployment directory; after git pull there is nothing to remember to do.
Commit the encrypted skeleton before prod is even ready: pin the "reviewed shape" into the repo before production infrastructure exists, with values empty or placeholder; the release preflight refuses to ship while placeholders, empty values or mock vendors remain. Shape first, values later, preflight as the backstop.

Alternatives and trade-offs

git-crypt: transparent smudge/clean is convenient, but the diff is unreviewable; and this scheme already has an explicit apply step, so transparency buys nothing while reviewability is a net loss. Rejected.
Cloud KMS / managed Vault: stronger custody and auditing, but operationally heavy for a small team, and clone-and-go onboarding is lost. Kept as an option to re-evaluate during prod hardening, not a day-one choice.
One age key per person (multi-recipient): better revocability — remove the leaver from the recipients list and follow with a sops rotate (for the reason above: updatekeys alone cannot stop a leaver from decrypting subsequent new ciphertexts) with no shared-key redistribution; the cost is maintaining the recipients list and re-encrypting on every membership change. While the team is small, use a shared dev key and write the upgrade path into the runbook — no need to pay this cost on day one.

Transferable judgments

The constraint ordering that decides whether a secret-management scheme survives: how changes get reviewed > how newcomers get keys > cryptographic strength. Per-key encryption secures the first; doc-permissions-as-credentials solves the second; for the third, SOPS + age has long been enough. Outsourcing identity, permissions and audit to the collaboration platform the team already uses — creating no new account system — is why a small-team scheme actually runs. And two of the pieces are team-size-independent, the universal base of any ciphertext-in-repo scheme: "CI split into two trust planes (a keyless validation plane / a keyed release plane)" and "envelope-key rotation ≠ secret rotation".