From five skills to a hardened system

Part two of a series. Part one — "Shipping a Production Stack with AI Coding Agents" — described the five skills. This paper is the full technical story of how those five skills became a hardened system. Part three — "The agentic enterprise" — argues for where it leads. The skills are open-source: clone the public template at github.com/amurthygithub/Sharevalue_claude_skills.

Where this paper picks up

Part one described a working set of slash commands — /work-on, /ship, /agentreview, /promote, /linear — that took the routine parts of shipping code off the human's plate while keeping the human on every irreversible step. It was a set of verbs. It worked at roughly three PRs a day on a one-engineer-plus-agents team.

This paper covers everything that happened after that: the load went up — to a dozen-plus PRs a day, run across multiple parallel agent sessions — and the set of verbs turned into a system. The verbs grew shared primitives. The review grew from one reviewer to four. The pre-push hook grew a rule engine. A coordination layer appeared underneath, because parallel agents collide in ways a single agent never does. And the whole thing got progressively more paranoid: more independent checks between an agent's output and anything irreversible, a sharper threat model for the parts that run unattended.

None of it was planned. Each change was a response to a specific failure mode the previous version produced — which is the same way the original five skills came to exist. We're writing it down because the shape of the evolution turned out to be more transferable than any single skill. The short version, and the thing we'd defend: the human's job never expanded. Everything that changed added machinery below the human, not authority to the agents.

Same warning as part one: this is unglamorous. Mostly bash and markdown. Nothing here is novel in isolation — least-privilege, defense-in-depth, partitioned logs, feature flags. The leverage is in how the pieces compose.

1. The failure modes that drove every change

The honest way to tell this story is through the things that broke, because each one is why a piece exists. Four of them, roughly in the order they bit.

Worktree cross-contamination

We caught an agent merging into the wrong branch. The human had started a session on one ticket; the resulting PR contained two unrelated commits because a sibling agent session — running in the same working tree against a different branch — had switched HEAD underneath it. The shared checkout had no per-session isolation; the second agent's git checkout quietly overwrote the first's branch state. The PR looked clean; the contamination only surfaced because a human happened to read the file list.

The fix was structural: a git worktree per parallel session, made the default. Two agents working two tickets are now physically in two directories and cannot collide on HEAD by accident (§4).

The pre-push gate blew out system memory

The pre-push hook originally ran the full test suite. One day a single push — with review sub-agents spawning behind it, plus an editor and the agent's parent process — killed the whole machine. The cause: test sub-processes spawned by the review sub-agents reparented to init when the parent was killed mid-push, and accumulated across pushes until the OOM killer arrived.

The fix split the gates. Pre-push now runs only the fast lane — lint, format, type-check, ~5 seconds — and never the test suite. Tests run server-side, on an ephemeral machine, where an OOM is someone else's problem. The local gate stays cheap enough to never be the reason a push is slow (§6).

Consensus spoofing

The /ship skill polled for a PR comment whose body began with the consensus-review header and parsed the verdict from it. The hole: anyone with comment access could post a comment with that body. A spoofed "APPROVED" would bypass the merge gate. We caught it internally; the risk is small on a one-person team and not small on a fifty-person one.

The fix: /ship filters candidate comments by author against an explicit trusted set, and the load-bearing approval is a formal review by a bot identity (consumed via branch protection), not a parsed comment (§9). The principle: when an agent is the entity that acts on a verdict, the verdict's authentication is part of the gate.

Reviewers reading the disk instead of the diff

The review sub-agents were told to "review the diff." Several of them hallucinated "production code is missing" findings — they had read the base-branch file on disk and concluded the code in the diff didn't exist. "Review the diff" did not mean to the agent what it meant to us.

The fix is a hard rule in the reviewer's definition: the inline diff the orchestrator passes is the authoritative source; the on-disk file may be the base branch, and assuming otherwise produces confidently-wrong findings. It's the single most common false-positive pattern in agent code review (§9).

The thread connecting all four: as you delegate more authority to agents, the delegation itself becomes part of the threat model. Not "is the agent smart enough" but "what happens when it's confused, the input is adversarial, or two of them race."

2. From skills to a system: the catalog and its primitives

Part one had five skills. The catalog roughly doubled — but the count isn't the point. The point is that somewhere around skill number eight, every skill was carrying its own copy of the same logic: its own danger-zone path regex, its own variant of the review-comment poll loop, its own ad-hoc lock file. When we fixed one skill's scan, the others kept the bug.

The fix was .claude/skills/_lib/ — a small library of playbooks the skills reference instead of copy. A skill is a verb the human invokes; a _lib/ playbook is a primitive the skills compose. When the same logic appears in three skills, it gets extracted; when the primitive needs a fix, every consumer inherits it.

The rule for when to extract is "on the second (or third) consumer, not on speculation." One skill using a primitive keeps it inline — extraction has a cost, and paying it early is its own anti-pattern. A second skill needing the same thing is the cue. This is just library design, but it matters here because skills are also markdown: the primitive lives in a file the skill includes by reference, with no build step, no version, no SDK. The cost of extracting is one file move and a pointer in the consumers. That cheapness is what made the extraction happen at all.

3. Parallel agents = parallel processes

Part one assumed one agent at a time. We now routinely run several in parallel; on heavy days, well into double digits of active worktrees. The frictions that surfaces are unfamiliar in single-agent operation and obvious in retrospect: two agents pushing the same branch; two agents appending to the same audit log and conflicting on every push; one agent's working tree switched out from under it; one agent holding a lock and crashing without releasing it.

Three coordination primitives. None clever; all required.

The sharded log is the smallest change with the biggest payoff: the directory is the log; you read it by listing it. The lock is subtler — the naïve version keys on PID and fails in production, because a skill is not one process but a sequence of short-lived bash invocations. Keying on the worktree path makes it survive that, and makes the failure legible: the lock file says which checkout holds the branch.

The general lesson: when you scale a process from one actor to N, the bottleneck moves from "make each actor faster" to "keep the actors from colliding." The cost of coordination is what you pay for the second 10×. And the lock's deliberate carve-out matters — it gates pushing the same branch (the catastrophic case) but not editing files elsewhere. A lock that prevented all collaboration would be too restrictive.

4. Background-task discipline

"Kick off the review, wait, merge" — the "wait" turned out to be the hardest part to get right. There are three legitimate primitives for a multi-minute wait, and the wrong one wastes resources and hides failures.

The load-bearing rule: the watcher must emit on every terminal state — success, failure, stall, timeout — not just the happy path. We call the anti-pattern "silence is not success." A watcher that only prints when things go well looks healthy when things go badly, and the human walks away assuming a task is still running that died an hour ago. It's a thirty-line convention, and it eliminated the most common form of late-detected failure in the whole workflow.

5. The pre-push rule registry

The most common small failure isn't a bug — it's an agent making the same stylistic mistake you've corrected three times. Beneath a code review, above a linter's defaults. We wanted to add checks for these, and the cost of adding one was awkwardly high: each needed a place in the hook, an opt-out, a way to test it, and — the part we kept getting wrong — a way to roll it out without breaking everyone's push on day one.

The fix is a pluggable registry: one small script per rule in a rules directory; a dispatcher enumerates them and runs each against the diff. It started with two cosmetic rules. It's now eight, and — more importantly — it crossed from style into security and correctness:

Two of the security rules show the registry doing something a linter usually doesn't. secret-echo flags shell that pipes a secret-shaped variable to stdout — it exists because a single 30-day window produced five separate secret-handling incidents; it's least-privilege as a grep. new-deps flags any new top-level dependency, because install scripts run with the full developer shell environment and a compromised transitive dependency can read every token in it; the rule doesn't block the dependency, it forces the addition to be seen.

The load-bearing idea is the lifecycle:

A rule that blocks on day one and false-fires becomes the exact friction it was supposed to remove — you start reaching for --no-verify, and now the gate is worse than nothing. We were burned twice by checks that landed blocking and broke an afternoon of pushes. So every rule ships advisory, soaks while we watch its real false-positive rate, and is promoted only in a deliberate, recorded decision. The rule that touches secrets gets the same WARN-then-backtest treatment as the comment-style nit. This is the "evolve" step — catch the failure mode that just escaped — turned into a standing mechanism.

6. One scan, one script: the danger zone

The danger-zone re-scan is the single most safety-critical primitive in the workflow. Immediately before any auto-merge, a bash check greps the change set against a list of privileged paths — migrations, the env files, the CI config, the skill and agent definitions themselves — without consulting any agent's verdict. It's the defense against a prompt-injected reviewer approving a diff that touches the very files that control how the next agent runs.

It began as a regex copied into each consumer: /ship at commit-time and again at merge-time, /promote, the review skill. Four copies of one regex. Every new protected path meant a four-way lockstep edit, and the review kept catching the times we forgot — the same drift surfaced across three separate PRs.

It collapsed into one shared shell script every consumer sources: one regex, one set of helpers, one place to change it. The detail that makes it trustworthy is that it's fail-closed — if the upstream remote is unreachable, the scan does not return "zero hits," it returns an error and the caller must halt. A gate that silently passes when it can't do its job is worse than no gate, because it reads as safe. The shared script makes that behavior uniform instead of something each copy had to remember.

This is the same two-gate design from part one, now centralized: an advisory scan at commit time (so the human sees the diff with full context) and an independent, bypass-resistant re-scan immediately before merge that does not trust any agent's text. Either gate alone wouldn't be enough; together they're what make us comfortable letting /ship auto-merge on the happy path.

7. The triage council: a review of the review

Once the review became reliable, the bottleneck moved. The reviewers were no longer the weak link — deciding what to do with their findings was. A consensus comment with a dozen findings, three of which matter, still costs a human the time to sort them, and sorting under "let's just clear the review" pressure is where the wrong call gets made.

We made the wrong call enough times to name it. The sharpest version: an agent iterated a second round purely to address nits, and in doing so the reviewer reversed a fix it had asked for in round one. The work went backwards. The lesson got written in the bluntest terms — nits are never merge blockers; the first APPROVED with zero blockers means stop, surface, ship — and then mechanized into /ship itself.

/triage-review is the review of the review. It reads the consensus comment and runs a second council — three agents, three lenses — over the findings:

The aggregation checks the safety-critical verdict first: any lens flagging SECURITY halts for a human before the false-positive check, so a security signal can't be quietly outvoted. A split vote also halts — the council disagreeing is the signal that a human is needed, and the orchestrator does not break the tie.

/triage-review is deliberately read-only: it classifies and prints a report; it cannot edit, push, or update tickets. That's the same split part one drew between authoring and verifying — the thing that decides which findings are real should not also be the thing that applies the fixes. Conflating them produced an orchestrator that was both an author of findings and a judge of which mattered, and the judging was systematically biased toward "do the work."

8. Observability becomes a review lens

We shipped two things that ran "successfully" for weeks while doing nothing useful. The most painful: a scheduled job that exited cleanly every night for ~two months while every database connection inside it silently failed. The task was healthy. The work it claimed to do never happened. There was a monitor on the task's exit, none on its outcome.

So observability became doctrine — and then a review lens. The consensus review went from three reviewers to four, and the bar from 2-of-3 to 3-of-4 APPROVE:

The reason observability is a lens that votes and not just a rule in a document is that exact incident: doctrine is advisory — a tired author under deadline skips it; a lens that fires on every push is not skippable. Its job is mechanical: a new external call with no latency/result instrumentation is a blocker; a new scheduled task missing any of its monitoring layers is a blocker.

The monitoring contract for a scheduled job is three independent layers, because any one alone is insufficient:

Adding the lens forced one boundary: observability and security both touch instrumentation, and instrumentation is exactly where credentials leak (the easy way to "log the request" is to log a URL with an API token in it). The line: observability owns the instrumentation surface; security keeps ownership of credential-scrubbing. A raw URL reaching a telemetry sink unscrubbed is a security blocker, not an observability nit.

9. Hardening the review system itself

/agentreview is the most-touched skill in the workflow, because it's the only one whose verdict another skill acts on automatically. Running it on every push, unattended, under a permission mode that suppresses prompts, means it processes adversarial input — diffs, titles, sub-agent text — without a human watching each step. So we hardened the thing that does the watching, in layers.

The unifying move: assume the injection succeeds and ask what it can then do. Tool starvation, the scrubbed environment, the authenticated verdict, and the independent merge-gate re-scan (§6) are all answers to that question, layered so no single one has to be perfect. We name the residual honestly — the orchestrator keeps a shell, and a shell can read any file the process can — and accept it as a deliberate trade rather than pretend it away.

10. /handoff: continuity across sessions

Context windows end — usually in the middle of a feature, right after the agent has spent its budget discovering which three approaches don't work. A fresh session re-walks all three. /handoff snapshots the state a successor needs into a gitignored sidecar:

The "tried and failed" field pays for the whole skill. An agent's most expensive output is often the knowledge of what doesn't work, and it's exactly the part that evaporates when the conversation does. Writing it to a file the next session reads first turns a sunk cost into an asset.

11. Memory: how a lesson survives the next session

Part one described memory as "save the lesson, promote the stable ones." We've now run that promotion path enough to trace one lesson end-to-end — and it's the clearest illustration of how the system accretes knowledge.

The lesson: nits are never merge blockers. The journey:

Two patterns we noticed. The path is asymmetric: lessons move up (memory → docs → canonical → code), and when one turns out to be wrong we leave the entry in place with a "why this is wrong" note rather than delete it — deleting lets the same wrong lesson land again next month. And most lessons stop early: of dozens of memory entries, only a handful become encoded rules. The rest are too situational or too obvious once written down. That's fine — memory is a hint system, not a policy engine. The value isn't that any lesson is profound; it's that the cost of teaching it falls to near-zero after the first time.

12. What broke in the orchestration layer

The earliest failures (§1) were agents doing the wrong thing. The most recent two were one level up — not in what the agents produced, but in the machinery that runs them.

The auto-merge race. We armed the platform's native auto-merge on a PR, then mid-session asked the agent to fix the review findings first. It didn't matter: auto-merge fires the instant branch protection is satisfied, regardless of anything said in the session. The PR merged out from under several rounds of intended fixes, stranding them on an orphaned branch. The lesson: the merge gate belongs to the skill and the human, not to a platform feature racing them. Native auto-merge gets disarmed before iterating; /ship's own gate is the only thing that decides when a merge happens, because it's the only gate that can see the conversation's intent.

The startup wedge. For a stretch the review orchestrator hung at launch with an empty log. The cause had nothing to do with review: the agent runtime boots every configured background integration at session start, regardless of the agent's tool allowlist, before the agent's own logic runs — and one of those had a cold-start that stalled the whole handshake. The fix was to launch with background integrations explicitly disabled (it doesn't use them), which both fixed the wedge and tightened the surface. The lesson: an autonomous agent's startup is part of its reliability budget; a dependency it never calls can still take it down.

Both share a theme, and it's a healthy one: the agents kept doing the bounded work correctly. The hard problems moved to the orchestration — races between a platform and a session, dependencies loading at the wrong time. That's where the next class of bug lives once the layer below it is stable.

13. The shape of the evolution

Step back from the individual changes and a few properties of the direction stand out. These are the part we'd actually defend.

The human's job never expanded. Every change added machinery below the human, never authority to the agents. The human still holds judgment, scope, and the irreversible steps; still types a present-tense confirmation before anything reaches production. A fourth review lens, a security rule registry, a consolidated danger-zone scan — none moved a decision from the human to the machine. They moved work off the human while leaving every decision in place. This is the property that makes the system safe to run many-agents-wide, and it was a choice.

Every gate was incident-driven and ratcheted in gently. Nothing was designed from first principles. The observability lens exists because a job lied for two months; the secret-echo rule because of five incidents in a month; the shared danger-zone script because four copies drifted three times. And each constraint shipped in its softest form first — advisory before blocking, WARN before fail, backtested before enforced. A gate that arrives blocking and wrong teaches people to bypass gates, which is strictly worse than no gate.

Maturation looks like depth, not breadth. A workflow still finding itself adds verbs. A maturing one adds depth to the verbs it has — more independent checks, a sharper threat model, fewer duplicated primitives — and stops adding verbs. The velocity of new features approaching zero isn't stagnation; it's the system finding its boundaries.

Defense deepened; autonomy didn't. The trend a lot of "AI coding" content implies is more automation over time. Ours went the other way. The agents do the same kind of work they did in part one; what grew is the number of independent checks between their output and anything irreversible. We optimized for being able to trust the output without watching it produced — a function of verification, not cleverness.

The tooling itself is reversible. Rules demote from BLOCK back to WARN when they get noisy; primitives get extracted on the second consumer and deleted when the last one goes; the auto-merge feature got disarmed when it raced us. A workflow that can only accrete becomes its own legacy system. This one is built to be walked back.

14. If you're adopting this

Part one's advice still stands as the foundation: write your danger zone first, force ticket references into commits, pick one verb and live with it for two weeks. This paper's additions are what you reach for once that's working and the bottleneck moves from "shipping" to "not breaking":

1. Shard your audit log the day you run two agents at once. A one-line change that pre-empts a class of merge conflict.
2. Extract a primitive on the second consumer, not the first. Duplication is cheap until it drifts; the danger-zone scan is the one where the third drift is one too many.
3. Build the rule registry when you've corrected the same nit three times — and ship every rule WARN-first. The ratchet is the whole point; skip it and you'll be bypassing your own gates within a month. Let it cross into security only after the style rules have soaked.
4. Add a triage council when sorting your reviewer's findings becomes the bottleneck. That's a sign of success, not a starting move.
5. Make observability a lens that votes once a "successful" job lies to you. A rule in a doc is skippable; a reviewer isn't.
6. Harden the orchestrator before you let review run unattended — tool starvation, a scrubbed environment, authenticated verdicts. Cheap up front, expensive to retrofit after an incident. And treat the orchestration layer as part of the threat model: the agents will be fine; the races and startup dependencies are where the next bug lives.

15. Closing

The honest caveats, same as part one:

• It still doesn't make us ten times faster. We don't have a clean A/B. It lets a very small team run a lot of parallel work against a production system without the safety gates degrading as the concurrency and the stakes go up. The skills are unglamorous; the discipline around them is the product.
• The principles aren't novel. Tool starvation is least-privilege. The independent re-scan is defense-in-depth. Sharded logs are partitioned logs. WARN-before-BLOCK is a feature flag. The contribution is that they compose into a coherent operating discipline.
• AI agents are still bad at the hard parts. Architecture, scope calls, debugging a rare production incident — those stay with a human. The workflow's job is to take the boring 80% off the human's plate so the human spends more time on the 20%.

The skills in this paper — and part one's — are published as a sanitized, MIT-licensed template at github.com/amurthygithub/Sharevalue_claude_skills. It's Markdown and bash; the placeholders are inert until you fill them in.

If part one was the verbs and this paper is the system, part three is the argument for where it goes: that the same primitives — tool starvation, independent gates, sharded audit, triage councils, a WARN→BLOCK policy ratchet — rebuild not just code review but most of the back-office work that humans currently do in queues. That's a bigger and more speculative claim. This paper is the load-bearing part underneath it: an unglamorous system, mostly bash and markdown, that ships our code every day and gets harder to break as it goes.