[safe-output-health] 🏥 Safe Output Health Report - 2026-10-13

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours
• Runs Analyzed: 356 run directories with complete job-level telemetry (out of ~449 downloaded run directories; the initial log download hit its 200-run count limit mid-fetch and returned a partial result, and ~93 downloaded directories were early-exit/partial runs excluded from job-level analysis — consistent with the pattern seen in prior audits)
• Workflows Active: well over 100 distinct workflows triggered in the window (schedule, workflow_dispatch, and event-triggered)
• Safe Output Jobs Executed: 354 (job named safe_outputs with a terminal conclusion; 10 further executions were skipped and are excluded from the denominator below)
• Safe Output Jobs Failed: 3
• Error Clusters Identified: 2

Monitoring cadence: ✅ Healthy — 23.95 hours since the previous successful audit (2026-10-12T04:34:23Z, run 38204331412), well inside the 24h-expected / 48h-alert window. This is a return to healthy cadence after the ~144h gap flagged critical in the 2026-10-12 report.

Safe Output Job Statistics

Job Type	Total Executions	Failures	Success Rate
safe_outputs (all types, aggregate)	354	3	99.15%
— create_issue (failed items)	—	2	n/a (both within otherwise-successful jobs)
— create_pull_request (failed items)	—	1	n/a

Note: this repository's compiled workflows run all configured safe-output types (create_issue, create_discussion, add_comment, create_pull_request, add_labels, etc.) through a single consolidated safe_outputs job (Process Safe Outputs step), rather than one job per output type, so per-type breakdown reflects which output type was being processed when a job failed, not separate job executions.

Error Clusters

Cluster 1: Stale lock file / provenance validation fails closed (GH_AW_JOB_CHECK_RUN_ID)

• Count: 2 occurrences
• Affected Jobs: safe_outputs → Process Safe Outputs step, create_issue output type
• Affected Workflows: Smoke Copilot Routed Claude (run 38341741138), Smoke Copilot SDK Routed Claude (run 38341743666) — both workflow_dispatch smoke tests, ~03:26 UTC on 2026-10-13
• Sample Error:

  1 safe output(s) failed:
    - create_issue: Missing or invalid safe-output execution provenance: GH_AW_JOB_CHECK_RUN_ID
  

• Root Cause: ADR-68077 ("Always-on execution provenance marker for safe outputs") landed on 2026-10-12, making every safe-output publish fail closed if GH_AW_JOB_CHECK_RUN_ID is missing or invalid, by design ("Missing or malformed runtime metadata fails closed, preventing publication of unattributed artifacts"). The ADR itself warns: "Existing compiled workflows must be recompiled to supply the publishing check-run context; stale .lock.yml files will fail closed at publish time."
  • smoke-copilot-routed-claude.lock.yml is confirmed stale: it has no GH_AW_JOB_CHECK_RUN_ID env var at all in its Process Safe Outputs step — a straightforward recompile fixes this one.
  • smoke-copilot-sdk-routed-claude.lock.yml is not stale — it already has GH_AW_JOB_CHECK_RUN_ID: ${{ job.check_run_id }} wired in — yet it still hit the identical failure. That means the GitHub Actions job.check_run_id context itself resolved empty/invalid at runtime for this job. Both affected workflows share an unusual job shape: engine.bare: true with model-routing and sandbox.agent.id: awf. This suggests the Checks-API check run that resolvePublishingJobId depends on may not be available/resolvable yet for this specific engine/sandbox configuration — a second, deeper issue beyond "just recompile."
• Impact: Both are low-traffic internal smoke-test workflows (private: true, no user-facing effect), but the underlying provenance-resolution gap could affect any other workflow using the same bare + model-routing + custom sandbox-agent-id combination, and any other not-yet-recompiled .lock.yml using similar patterns.

Cluster 2: create_pull_request patch size exceeds configured limit

• Count: 1 occurrence (first-time, not yet recurring)
• Affected Jobs: safe_outputs → Process Safe Outputs step, create_pull_request output type
• Affected Workflows: Linter Miner (run 38284938021, 2026-10-12 ~17:48 UTC)
• Sample Error:

  1 safe output(s) failed:
    - create_pull_request: Patch size (11889 KB) exceeds maximum allowed size (4096 KB)
  

• Root Cause: Linter Miner's agent generated a diff (new linters under pkg/linters) totaling ~11.9 MB, roughly 3x the handler's 4 MB patch-size cap, so the whole output was rejected rather than partially applied.
• Impact: Workflow-specific; no PR was published for that run. Single occurrence so far — flagged for recurrence-watching, not yet a confirmed systemic issue.

Root Cause Analysis

API-Related Issues

None found. mcp_server_health was clean (0 errors, 1 healthy server) for every failing run inspected, and no rate-limit, authentication, or network-layer errors were present in any of the 3 failures.

Data Validation Issues

Both clusters above are validation/configuration failures, not transient infra flakiness:

• Cluster 1 is a fail-closed design behavior (intentional, by ADR-68077) tripped by stale compiled output and/or an unresolved runtime context value.
• Cluster 2 is a hard size-limit check behaving as designed, triggered by an oversized agent-generated diff.

Permission Issues

None found this cycle. The previously-tracked approve_workflow_run / fork-PR permission-gap pattern (PR Sous Chef) did not recur.

Other Issues

None found. missing_tools/missing_data/mcp_failures fields were null across every sampled run_summary.json.

Recommendations

Critical Issues (Immediate Action Required)

None. Both failure clusters affect low-traffic internal smoke-test / maintenance workflows, not user-facing publishing paths, and no systemic or security-relevant failure was found.

Bug Fixes Required

1. Recompile smoke-copilot-routed-claude.md

   • File/Location: .github/workflows/smoke-copilot-routed-claude.md → .github/workflows/smoke-copilot-routed-claude.lock.yml
   • Problem: The compiled lock file predates ADR-68077 and has no GH_AW_JOB_CHECK_RUN_ID env var in its Process Safe Outputs step, so it fails closed on every create_issue publish.
   • Fix: Run gh aw compile (or make recompile) to regenerate the lock file with the new provenance wiring.
   • Affected Jobs: safe_outputs / create_issue

2. Investigate job.check_run_id resolution for bare + model-routing + custom sandbox-agent-id jobs

   • File/Location: actions/setup/js/execution_provenance.cjs and resolvePublishingJobId (referenced from pkg/workflow/compiler_safe_outputs_steps.go, safe_jobs.go)
   • Problem: smoke-copilot-sdk-routed-claude.lock.yml already supplies GH_AW_JOB_CHECK_RUN_ID: ${{ job.check_run_id }}, yet the value was missing/invalid at runtime, meaning the GitHub-provided job.check_run_id context (or its downstream resolution against the recorded run ID/attempt) did not resolve correctly for this job shape.
   • Fix: Reproduce with a workflow using engine.bare: true + model-routing + sandbox.agent.id: awf, and confirm whether the Checks API check-run is created/available at the point resolvePublishingJobId queries it for this configuration. This is a provenance-feature regression risk for any workflow sharing this job shape, not just these two smoke tests.

Configuration Changes

1. Linter Miner patch-size limit
   • Current: Default/configured create-pull-request max patch size of 4096 KB.
   • Recommended: Either raise the limit for Linter Miner specifically (if large generated-linter diffs are expected/legitimate) or scope the prompt to generate fewer linters per run so diffs stay under the cap.
   • Reason: A single ~11.9 MB diff exceeded the cap by ~3x; this is a workflow-authoring/task-scoping decision, not a platform bug.

Process Improvements

1. Lock-file staleness auditing after provenance-affecting ADRs
   • Current State: ADR-68077 landed and immediately produced two failures in workflows whose .lock.yml files were not recompiled (one fully stale, one partially).
   • Proposed: When a safe-outputs-publishing-affecting change lands (provenance, schema, or handler-manager changes), run a repo-wide check for .lock.yml files missing the newly-required env var/field before merging, rather than discovering staleness via this next-day audit.
   • Benefits: Catches "needs recompile" regressions same-day instead of after at least one failed smoke-test run.

Work Item Plans

Work Item 1: Recompile stale smoke-test lock file

• Type: Bug Fix
• Priority: Medium (internal smoke test only, no user-facing impact, but blocks that smoke test's own signal)
• Description: smoke-copilot-routed-claude.lock.yml is missing the GH_AW_JOB_CHECK_RUN_ID env var that ADR-68077 requires, causing every create_issue safe-output publish in that workflow to fail closed.
• Acceptance Criteria:
  • gh aw compile run against smoke-copilot-routed-claude.md, diff reviewed and committed
  • Next scheduled/dispatched run of Smoke Copilot Routed Claude successfully creates its issue
• Technical Approach: make recompile (or targeted gh aw compile .github/workflows/smoke-copilot-routed-claude.md), verify the new lock file includes GH_AW_JOB_CHECK_RUN_ID: ${{ job.check_run_id }} in the Process Safe Outputs step.
• Estimated Effort: Small
• Dependencies: None

Work Item 2: Root-cause job.check_run_id empty-at-runtime for bare/model-routing/custom-sandbox jobs

• Type: Investigation
• Priority: Medium-High (recompiling alone will not fix this one — the env var is already present and still failed)
• Description: Determine why GitHub Actions' job.check_run_id context (or resolvePublishingJobId's resolution of it) returned empty/invalid for smoke-copilot-sdk-routed-claude's safe_outputs job, despite the lock file already being up to date.
• Acceptance Criteria:
  • Reproduce the failure in isolation (or confirm it is timing/engine-shape dependent)
  • Identify whether the Checks API check run is created late, under a different name, or not at all for engine.bare + model-routing + sandbox.agent.id jobs
  • Ship a fix or a documented constraint/workaround
• Technical Approach: Instrument execution_provenance.cjs / resolvePublishingJobId with additional diagnostic logging for this job shape, or trace through the Checks API pagination logic described in ADR-68077 against a reproduction run.
• Estimated Effort: Medium
• Dependencies: Work Item 1 (recompile) should land first so this investigation isn't confounded by the simpler staleness case.

Work Item 3: Watch Linter Miner patch-size failure for recurrence

• Type: Investigation (low priority, single occurrence)
• Priority: Low
• Description: Confirm whether Linter Miner's ~11.9 MB diff was a one-off (e.g., it accumulated a backlog of linter ideas in one run) or a recurring pattern as the workflow continues to run daily.
• Acceptance Criteria:
  • Check next 2-3 Linter Miner runs for the same patch-size error
  • If recurring, decide between raising the limit or scoping the prompt
• Technical Approach: Tracked in cache memory (error-patterns.json → create-pull-request-patch-size-exceeds-limit) for next audit to check.
• Estimated Effort: Small
• Dependencies: None

Historical Context

Compared against the 2026-10-12 audit (0 failures across 178 executions, 100% success — the first fully clean audit since 2026-09-11):

Trends

• Error rate trend: Slightly up from 0% to ~0.85% (3/354), but both clusters are brand-new signatures, not recurrences of any previously-tracked issue.
• Most common recurring issue: None recurred today. All 7 previously-tracked recurring signatures (PR Sous Chef approve_workflow_run, Design Decision Gate allowed-files decline, Smoke Issues jira/linear credentials, Smoke Project update_project, Auto-Triage Issues add_labels, push_ledger_changes validation bugs, ledger_append invalid-fields) were absent this cycle.
• Improvement since last audit: Cadence itself improved — back to a healthy ~24h gap after the ~144h gap flagged critical on 2026-10-12. The two new failures are a direct, traceable side effect of a same-week platform change (ADR-68077) landing without a full recompile sweep, rather than a regression in existing functionality.

Metrics and KPIs

• Overall Safe Output Success Rate: 99.15% (351/354 terminal safe_outputs job executions succeeded; 10 further executions were skipped and excluded from this ratio)
• Most Reliable Job Type: All non-create_issue/create_pull_request output types observed (e.g. create_discussion, add_comment, add_labels) had 0 failures this cycle
• Most Problematic Job Type: create_issue (2 failures, both from the same new provenance-validation cluster) and create_pull_request (1 failure, patch-size limit)
• Average Time to Failure: Process Safe Outputs step failed within 1-24 seconds of starting in all 3 cases (fast-failing validation, not a timeout)

Next Steps

• Recompile smoke-copilot-routed-claude.md to restore its GH_AW_JOB_CHECK_RUN_ID wiring (Work Item 1)
• Investigate empty job.check_run_id at runtime for engine.bare + model-routing + custom-sandbox-agent-id jobs (Work Item 2)
• Watch Linter Miner for a recurring patch-size failure over the next few scheduled runs (Work Item 3)
• Next audit: re-check whether any other .lock.yml files are stale relative to ADR-68077's provenance requirement

Generated by 🔒 Safe Output Health Monitor · claude · agent · 625.5 AIC · ⊞ 6.7K · ◷

• expires on Oct 13, 2026, 8:59 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #68596.