[prompt-clustering] Prompt Clustering Analysis - 2026-10-14

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-14 to 2026-10-14)
Total Tasks Analyzed: 588
Clusters Identified: 9
Overall Success Rate: 84.7% (498 merged / 588 total)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 4 — safe-outputs validation, test coverage & general correctness (160 tasks, 27.2% of all PRs), merge rate 79.4%. This is the catch-all "general correctness" bucket (strict-mode validation, engine guard hits, attribution/provenance fixes) and it's the single largest cluster, so its below-average merge rate pulls down the overall figure.
• Second Most Common: Cluster 6 — workflow/CI job hardening & audit reporting (113 tasks, 19.2%), merge rate 83.2%.
• Highest Success Rate: Cluster 8 — memory/ledger validation & state push (93.3% merged, 30 tasks). Tightly-scoped schema/state-repair fixes continue to converge fastest, consistent with prior runs.
• Second Highest: Cluster 2 — firewall/network domain handling (91.3% merged, 46 tasks). As in previous runs, narrowly-scoped, testable domain/firewall-pin changes merge reliably.
• Most Complex Tasks: Cluster 5 (gh-aw CLI version pins & dependency bumps) at 111.8 avg files changed per PR — pin bumps regenerate many lockfiles/manifests at once, by far the largest footprint this run, ahead of Cluster 2 (67.9 avg files changed).
• Lowest Success Rate: Cluster 4 — safe-outputs validation & general correctness (79.4% merged, 160 tasks). 5.3 points below the overall rate — the largest gap this run, but still short of the outlier bar.
• Highest Review Engagement: Cluster 7 — PR Sous Chef & agent run fixes (14.0 avg comments, 6.7 avg reviews) — heaviest reviewer engagement of any cluster despite a solidly above-average 88.6% merge rate.
• Outliers: None this run. No cluster met the outlier threshold (≥15 PRs and ≥10-point merge-rate gap below the 84.7% overall rate). Cluster 4 is the closest to the bar at a 5.32-point gap — see the note below.
• Data coverage: Full PR detail (comments, reviews, file counts, review decision) was available for all 588 PRs analyzed this run (100% coverage) via the cached pr-full-data store, which now holds 665 PRs total (a superset spanning beyond the current 30-day window). Workflow run-log metrics (turns/duration/cost via agenticworkflows logs) were not joined this run — these PRs are created by the GitHub-native Copilot coding agent rather than gh-aw-orchestrated workflow runs, so there is no corresponding aw_info.json turn-count data to match against; findings rely on PR metadata, comments, and reviews only.

Cluster Analysis

Cluster 4: Safe-outputs validation, test coverage & general correctness

• Size: 160 tasks (27.2% of total)
• Success Rate: 79.4%
• Avg Files Changed: 25.0 (comments: 4.6, reviews: 3.4)
• Top Keywords: issue, output, safe, test, coverage, failure, mcp, add
• Characteristics: The largest and most heterogeneous cluster this run — general correctness and safe-output hardening work (strict-mode pull_request_target checks, engine guard-hit normalization, attribution/provenance fixes, MCP coverage). Its size and breadth make it behave like a baseline bucket rather than a single theme, and it has the lowest engagement of the top clusters (lowest avg files changed among the top 5 by size), suggesting many of its PRs are either very small fixes or get abandoned early.
• Example PRs: Reject unsafe pull_request_target role configurations in strict mode #68554, Add engine max-context-rebuild support and normalize Codex guard hits #68504 (closed), Fix WIP attribution and specify persistent run provenance #68501, Document Codex context-rebuild guard tuning and identify guard cutoffs #68460, Exclude trusted base merges from unsigned PR bundle size checks #68459

---

Cluster 6: Workflow/CI job hardening & audit reporting

• Size: 113 tasks (19.2% of total)
• Success Rate: 83.2%
• Avg Files Changed: 62.7 (comments: 5.6, reviews: 2.9)
• Top Keywords: job, workflow, actions, repository, usage, artifact, github, daily
• Characteristics: CI/job-scoping and audit-report fixes (OTLP credential scoping, generator quoting hardening, unified session audit reports). Second-largest average diff footprint with below-average review depth relative to size — mechanical but broad changes.
• Example PRs: Scope OTLP credentials to environment-protected telemetry jobs #68555, Harden quoting in workflow generators #68430, Include AWF steering events in unified session audit reports #68428, Keep safe-output PR/issue/commit links when the conclusion job rewrites the activation comment #68413, Clarify daily AIC accounting and exclude confirmed unstarted AI jobs #68079

---

Cluster 7: PR Sous Chef & agent run fixes

• Size: 70 tasks (11.9% of total)
• Success Rate: 88.6%
• Avg Files Changed: 43.3 (comments: 14.0, reviews: 6.7)
• Top Keywords: sous chef, chef, pr sous, run, em, sub, aic
• Characteristics: Highest comment/review activity of any cluster by a wide margin, yet still converges above the overall rate — reviewers engage heavily on PR Sous Chef, lethal-trifecta copy, and agent-run provenance fixes, but the back-and-forth resolves rather than stalling.
• Example PRs: Update lethal trifecta landing page copy #65115, Prevent base config restore from clobbering an external root checkout #65069, Fail add-comment when the triggering target cannot be resolved #65068, Handle transient stdin reads in the safe-output transport #65067, Clarify PR branch target errors in scheduled runs #65066

---

Cluster 0: Model routing (Copilot/Claude/Codex engines)

• Size: 62 tasks (10.5% of total)
• Success Rate: 87.1%
• Avg Files Changed: 50.1 (comments: 6.7, reviews: 4.0)
• Top Keywords: model, routing, copilot, models, routed, endpoint, claude, awf
• Characteristics: Model-identifier/endpoint routing fixes across engines (routed-selection persistence, allowed/blocked model policy). Above-average merge rate with moderate review depth.
• Example PRs: Recompile smoke-copilot-routed-steering lock file #68540, Model routing: keep the routed selection when a proxy failure follows it #68498, Model routing: make models.allowed/models.blocked the only request policy, and allowed-models optional router candidates #68495, Fix guidance contracts after model checklist extraction #68462, Ignore Codex custom-slug fallback-metadata warnings in model_not_supported_error detection #68427

---

Cluster 5: gh-aw CLI version pins & dependency bumps

• Size: 53 tasks (9.0% of total)
• Success Rate: 86.8%
• Avg Files Changed: 111.8 (comments: 5.7, reviews: 2.6)
• Top Keywords: aw, gh, gh aw, version, v0, pins, cli, pin
• Characteristics: Firewall/MCP-gateway/CLI pin bumps and dependency consolidation. By far the largest average diff footprint this run (pin bumps regenerate many lockfiles/manifests at once), yet still a solidly above-average merge rate with the lowest review engagement — mechanical, low-friction changes.
• Example PRs: Bump gh-aw-firewall and gh-aw-mcpg dependencies #68598, Consolidate GHES action pin lookup #68422, Update Copilot CLI and Yamllint pins #68415, Upgrade charmbracelet ANSI module to v0.11.9 #68412, Add guided-error codemod for post-steps accessing safe-outputs file directly #68409

---

Cluster 2: Firewall/network domain handling

• Size: 46 tasks (7.8% of total)
• Success Rate: 91.3%
• Avg Files Changed: 67.9 (comments: 12.7, reviews: 5.9)
• Top Keywords: blocked, firewall, network, domain, details, summary
• Characteristics: Firewall pin bumps, blocked-domain handling, and unified agent session/AIC accounting normalization. Second-highest merge rate with above-average review engagement — consistent with prior runs where this theme converges reliably despite active review discussion.
• Example PRs: Deduplicate memory allowed-extension parsing #67543, Bump gh-aw-firewall to v0.28.50 #67035, Normalize unified agent session metadata and AIC accounting #65344, Claude: default to strict MCP configuration #64970, Report MCP payload sizes in audit and aggregate analysis #63687

---

Cluster 8: Memory/ledger validation & state push

• Size: 30 tasks (5.1% of total)
• Success Rate: 93.3%
• Avg Files Changed: 34.7 (comments: 8.6, reviews: 4.6)
• Top Keywords: ledger, memory, repo, validation, push, job, workflow, safe
• Characteristics: Highest merge rate this run — safe-output server startup repair, post-agent state-push retry fixes, and declarative memory schema validation. Narrowly-scoped, well-tested fixes that converge fast despite moderate review engagement.
• Example PRs: Restore safe-output server startup after memory schema change #67506, Repair post-agent state push retries and expose git errors #67474, Add declarative JSON schema validation for memory #67439, Make invalid repo-memory baselines repairable during the agent turn #67400, Increase arXiv repo-memory patch limit #66562

---

Cluster 3: Work-queue protocol & issue dispatch

• Size: 29 tasks (4.9% of total)
• Success Rate: 82.8%
• Avg Files Changed: 32.9 (comments: 8.5, reviews: 5.7)
• Top Keywords: work, queue, work queue, issue, dispatch, git, policy
• Characteristics: Work-queue protocol updates, multi-claim support, and experimental long-running agentic work across runs. Below-average merge rate with above-average review depth — newer/experimental surface area needing more iteration.
• Example PRs: Updating work queue protocol and related components #68582, Show work titles and descriptions in work-queue issues #68421, [Experimental] Long-running agentic work across multiple runs #68315, Cancel queued work when a related Issue closes #68133, Make work queue Issues easier to scan #68084

---

Cluster 1: Threat detection & engine execution guards

• Size: 25 tasks (4.3% of total)
• Success Rate: 84.0%
• Avg Files Changed: 36.7 (comments: 6.0, reviews: 4.4)
• Top Keywords: detection, engine, threat, threat detection, awf, copilot, agent
• Characteristics: Agentic-engine execution-boundary detection, OIDC support, and manual-cleanup-call detection. Merge rate tracks the overall average closely.
• Example PRs: Detect agentic engine execution outside the agent job #68579, Support Azure Entra OIDC in native Codex #68457, Detect parenthesized manual cleanup calls #68419, Improve frontmatter diagnostics for schedule and engine typos #68411, Require opt-in for Copilot telemetry during threat detection #67989

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Top Keywords
4	Safe-outputs validation & general correctness	160	79.4%	25.0	issue, output, safe, test, coverage
6	Workflow/CI job hardening & audit reporting	113	83.2%	62.7	job, workflow, actions, repository
7	PR Sous Chef & agent run fixes	70	88.6%	43.3	sous chef, chef, pr sous, run
0	Model routing (Copilot/Claude/Codex)	62	87.1%	50.1	model, routing, copilot, endpoint
5	gh-aw CLI version pins & dependency bumps	53	86.8%	111.8	aw, gh aw, version, pins, cli
2	Firewall/network domain handling	46	91.3%	67.9	blocked, firewall, network, domain
8	Memory/ledger validation & state push	30	93.3%	34.7	ledger, memory, repo, validation
3	Work-queue protocol & issue dispatch	29	82.8%	32.9	work, queue, dispatch, policy
1	Threat detection & engine execution guards	25	84.0%	36.7	detection, threat, engine, awf

Nearest-Miss Cluster Note (No Outlier This Run)

• Cluster: 4 — safe-outputs validation, test coverage & general correctness
• Merge Rate Gap: 79.4% vs 84.7% overall (gap: 5.32 points) — below the 10-point outlier threshold, so no formal root-cause investigation was triggered.
• Light-touch review: Of the 33 non-merged PRs in this cluster, 16 (48%) closed with zero comments and zero reviews — consistent with abandoned/superseded or experimental [WIP] tasks (e.g. "[WIP] Close designer mapping and reference-routing gaps", "[WIP] Expand Windows CLI integration coverage across scenario matrix") rather than review friction. The remaining 17 show active iteration (e.g. Add engine max-context-rebuild support and normalize Codex guard hits #68504 "Add engine max-context-rebuild support and normalize Codex guard hits" — 11 comments/6 reviews, explicit CI-failure and merge-conflict follow-up requests; Harden docker-sbx install: remove curl|sudo sh root install #62991 "Harden docker-sbx install" — 12 comments/10 reviews).
• Takeaway: The size and breadth of this catch-all cluster, not a single systemic blocker, is driving its below-average rate — a mix of experimental/WIP scope mismatches and a smaller tail of CI/review iteration.

Key Findings

1. No outlier cluster for the second run in a row — cluster merge rates this period range from 79.4% to 93.3%, a narrower band than earlier runs, suggesting task routing/sizing across themes has stabilized somewhat.
2. Narrowly-scoped, testable work keeps winning — memory/ledger validation (93.3%) and firewall/network domain handling (91.3%) are the top two clusters again, continuing the pattern from prior reports that tightly-bounded, verifiable changes converge fastest.
3. Size correlates with risk — the three largest clusters (safe-outputs/correctness, workflow/CI hardening, PR Sous Chef) sit at or below the overall merge rate except Sous Chef, which bucks the trend despite the heaviest review load; broad catch-all work is where the agent most often stalls.
4. Diff size does not predict failure — the largest-footprint cluster (CLI version pins, 111.8 avg files) merges at 86.8%, well above the lowest-merging cluster's 25.0 avg files — mechanical, repetitive diffs are low-risk even at scale, while smaller, exploratory diffs in Cluster 4 are the higher-risk category.

Recommendations

1. Split the safe-outputs/correctness catch-all — Cluster 4's breadth (strict-mode validation, MCP coverage, attribution, test coverage) makes it hard to target improvements; consider tagging or sub-labeling these PRs by sub-theme to track whether specific slices (e.g. MCP coverage vs. provenance fixes) are systematically weaker.
2. Reduce [WIP]/experimental PR noise — nearly half of Cluster 4's closures had zero review engagement, suggesting many exploratory tasks are opened and abandoned without ever reaching a reviewer; consider closing or re-scoping these earlier (e.g. via an explicit "experimental" label workflow) rather than letting them accumulate as open PRs.
3. Keep leaning into narrow, testable scopes — the two highest-merging clusters (memory/ledger, firewall/network) both reflect small, verifiable diffs; prompt templates for future tasks in lower-merging clusters could borrow this pattern (explicit acceptance tests, narrow blast radius).
4. Monitor Cluster 7's review load — PR Sous Chef tasks draw the heaviest reviewer engagement (14.0 comments/6.7 reviews avg) despite merging well; if this trend grows, consider whether some of that review burden can be shifted left (e.g. pre-submission linting/checks) to reduce reviewer cost without hurting the merge rate.

---

Generated by Prompt Clustering Analysis (Run: 38544993618)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 170.5 AIC · ⊞ 6.9K · ◷

• expires on Oct 15, 2026, 2:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #69010.