skill-counterfactual-audit

$npx mdskill add Tibsfox/gsd-skill-creator/skill-counterfactual-audit

Audits a skill's behavioral impact via counterfactual trace comparison.

  • Reveals behavioral changes that pass-rate metrics miss.
  • Depends on agent trace logging and skill loading infrastructure.
  • Compares traces with and without the skill to detect anomalies.
  • Outputs a structured SIP report highlighting key behavioral patterns.

SKILL.md

.github/skills/skill-counterfactual-auditView on GitHub ↗
---
name: skill-counterfactual-audit
description: >
  Audit a skill by running a paired probe — the same task once with the
  skill loaded and once without — segment both traces into goal-directed
  phases, align phases, and emit a SIP report (surface anchoring,
  template copy, excess planning, task recovery, off-task artifact).
  Use whenever a skill is created, modified, or proposed for retirement.
  Pass-rate is BLIND to most skill effects: CTA (arxiv 2605.11946v1)
  shows a single skill can produce 522 measurable behavioural changes
  across 49 tasks while pass-rate moves only +0.3%. Triggers: "audit
  this skill", "is this skill helping", "retire skill", "before shipping
  skill", "behavioural impact of skill X", or any skill review event.
user-invocable: true
version: 1.0.0
format: 2025-10-02
triggers:
  - "audit this skill"
  - "is this skill helping"
  - "behavioural impact of skill"
  - "retire this skill"
  - "before shipping a skill"
updated: 2026-05-16
status: ACTIVE
source: arxiv 2605.11946v1 (Counterfactual Trace Audit / CTA)
---

# Skill Counterfactual Audit

## Why

When you ship a skill, pass-rate is the easy metric — and it's *blind* to almost everything that matters. CTA (arxiv 2605.11946v1) shows a single skill can produce **522 measurable behavioural changes** across 49 tasks while pass-rate moves only **+0.3%**. The interesting effects (surface anchoring on the skill's vocabulary, template copy without adaptation, excess planning loops, recovery from off-task drift, generation of off-task artifacts) live entirely below pass-rate.

The remedy is a paired-trace audit: run the same probe task twice, with and without the skill, and *compare the behaviours directly* rather than the headline metric.

## How

For each skill under audit:

1. **Curate a probe-task bank** (3-5 small tasks per skill domain). Each task should have:
   - A clear goal (a single instruction)
   - An expected category of behaviour
   - A "phase decomposition" (rough segmentation rules)
2. **Run paired** — same prompt, same task, run A *with* skill loaded, run B *without*.
3. **Segment each trace** — phases like {plan, retrieve, edit, verify, fix-loop, summary}.
4. **Align phases** — match B phases to A phases; mark new/dropped/displaced phases.
5. **Classify SIP (Skill Influence Pattern)** — for each aligned phase, label the effect:
   - **surface-anchoring** — A copies vocabulary/structure from the skill itself
   - **template-copy** — A reproduces a skill template literally without adaptation
   - **excess-planning** — A adds extra planning steps absent in B
   - **task-recovery** — A recovers from drift that B falls into
   - **off-task-artifact** — A produces artifacts unrelated to the goal
6. **Emit report** to `.planning/patterns/skill-audits/<skill>-<timestamp>.md` with phase comparison table + SIP distribution + retire/refine/keep recommendation.

## Recommendation rule of thumb

| Dominant SIP | Verdict |
|---|---|
| task-recovery + correct artifacts | **keep** (genuine value) |
| surface-anchoring with no task-recovery | **refine** (skill is leaking style, not capability) |
| template-copy on most tasks | **refine** (lower template specificity) |
| off-task-artifact on >25% of probes | **retire** (skill is causing distraction) |
| excess-planning with no completion delta | **refine** (skill adds ceremony) |

## When to skip

- The skill is a workflow guide that doesn't directly shape output (e.g., `loop`, `schedule`) — audit dominant SIPs would always be "task-recovery" trivially.
- No probe-task bank exists for the domain (build one first; defer audit).
- The skill was just created and has no historical baseline. Run the audit when the skill has been active for ≥3 sessions.

## Integration

- `done-retirement` — required gate before retiring a skill.
- `gsd-cleanup` — sweeps audits on a milestone-close cadence.
- Companion to `simplify`, `code-review` — those audit code; this audits skill behaviour.

## Cross-references

- Rosetta concept #9 (Execution-Grounded Selection) — paired-trace audit is the skill-domain instantiation
- College: `agent-systems / integration-evaluation / agent-paired-trace-audit`
- Related skills: `skill-integration` (manages the skill lifecycle that this audits)

More from Tibsfox/gsd-skill-creator

SkillDescription
3d-interaction-designDesign heuristics for three-dimensional interaction — input modalities (6DoF controllers, gaze, gesture, voice), selection and manipulation techniques (ray-cast, virtual hand, go-go, world-in-miniature), navigation (teleport, continuous, redirected walking), and feedback loops (haptic, audio, visual). Covers Fitts' law in 3D, discoverability vs affordance, safe-zone design, and comfort/simulator sickness mitigation. Use when designing VR/AR interactions, immersive walkthroughs, or any input system where the user's body is the controller.
accessibility-patternsProvides web accessibility best practices for semantic HTML, ARIA, keyboard navigation, color contrast, and screen reader patterns. Use when building UI components, reviewing accessibility, or when user mentions 'a11y', 'accessibility', 'ARIA', 'screen reader', 'keyboard navigation', 'WCAG'.
active-listeningActive listening techniques for effective communication. Covers attending behaviors, paraphrasing, reflective listening, clarifying questions, empathic response, barriers to listening, listening in conflict, and cross-cultural listening. Use when building listening skills, improving understanding in conversation, mediating disputes, or analyzing communication breakdowns.
adversarial-pr-reviewAdversarial spec-compliance PR review — cross-references diffs against approved specs, verifies runtime claims against source, detects competing PRs, audits scope/convention compliance. Use before merging.
agile-methodsAgile and Lean software development methodologies for iterative, adaptive project execution. Covers Scrum (roles, ceremonies, artifacts), Kanban (WIP limits, flow metrics), XP (TDD, pair programming, CI), SAFe overview, Lean Software Development (Poppendieck's 7 principles), user stories (INVEST criteria), estimation via story points and velocity, and the Agile Manifesto's values and principles. Includes Cynefin framework for methodology selection and GSD's relationship to Agile as a structured Lean variant.
algebraic-reasoningSymbolic manipulation, equation solving, and algebraic structures for mathematical reasoning. Covers distributive law, factoring, completing the square, linear through polynomial equation solving, systems of equations (substitution, elimination, Gaussian elimination, matrix methods), algebraic structures (groups, rings, fields), modular arithmetic, polynomial theory, and inequalities. Use when solving equations, simplifying expressions, working with algebraic structures, or performing symbolic manipulation.
algorithmic-awarenessUnderstanding how algorithmic systems shape what users see, know, and do -- from recommendation feeds to search ranking to credit scoring to hiring software. Covers the mechanics of recommendation systems, algorithmic bias and its sources, personalization's effects on information diets, opacity and accountability, AI limitations (hallucination, confident wrongness), and the human-in-the-loop question. Use when a learner needs to think critically about why particular content reached them.
algorithms-data-structuresAlgorithms and data structures from first principles through advanced analysis. Covers sorting (bubble, insertion, selection, merge, quick, heap, radix), searching (linear, binary, BFS, DFS, Dijkstra, A*), fundamental data structures (arrays, linked lists, stacks, queues, hash tables, trees, heaps, graphs, tries), complexity analysis (Big-O, Big-Omega, Big-Theta, amortized), recurrence relations, and algorithm design paradigms (divide-and-conquer, greedy, dynamic programming, backtracking). Use when analyzing, selecting, implementing, or comparing algorithms and data structures.
aminet-browserAminet search and browse: full-text search, category tree navigation, architecture filtering, package detail, and curated collections. Use when searching, browsing, or managing package collections.
aminet-emulatorFS-UAE emulator configuration and launch: hardware profiles, ROM management, WHDLoad integration, config generation, and state snapshots. Use when configuring emulation, managing ROMs, or launching Amiga software.