Proposal: AI-assisted research to help close stale/resolved issues in the backlog

[theaminulai]

Proposal: AI-assisted research to help close stale/resolved issues in the backlog

The problem

As of today, WordPress/gutenberg has 4,445 open issues. The stale-issue-gardening.yml workflow already closes issues that sit in [Status] Needs More Info or Needs Testing with no follow-up, and the triage docs lay out a clear manual process. But inactivity is a weak signal for relevance: plenty of old reports are already fixed, duplicates, or "working as intended," yet nobody has gone back to confirm that, so they sit open indefinitely and make it harder to spot the regressions that actually matter right now. Closing them requires real investigation — reading the thread, checking whether a linked PR shipped, and trying the repro against trunk — which doesn't scale to thousands of issues with a volunteer triage team.

Next.js hit the same problem (backlog peaked at 3,109, later 2,244) and wrote up how they closed 1,462 issues in about three weeks using an agent-assisted triage pipeline: "How we closed 1,500 GitHub issues" (Vercel, Sept 2026). I think a scaled-down version of that approach fits well within Gutenberg's existing triage workflow, without requiring new infra spend or bypassing the triage team's role.

What Next.js did (summary)

• A read-only research agent (closability) investigates one issue at a time in a sandbox with the repo checked out: reads the thread, searches related issues/PRs/commits/releases, and tries to reproduce the bug on the reported version, latest stable, and canary.
• It outputs structured evidence and a conservative confidence score + reason (fixed / duplicate / expected behavior / no longer reproducible / unsupported), never acts directly (no commenting, closing, or pushing code), and is instructed to ignore any instructions embedded in issue text (prompt-injection defense).
• High-confidence results go into a "Close Queue" that maintainers review and act on — the agent informs the decision, a human still makes it.
• A GitHub Action lets anyone reopen a closed issue within a grace window, which both catches bad calls and builds confidence in the process (most closures were never reopened).
• Only after validating accuracy over time did they let two agents (proposer + skeptic) auto-close the clearest cases (score ≥ 80, no contradicting evidence found), still with the reopen safety net, capped at ~25/week.

Proposed pilot for Gutenberg

1. Research-only agent, no write access. Build a workflow (manually triggered or scheduled, e.g. via GitHub Actions + an LLM API, similar in spirit to closability) that takes a batch of old, low-activity open issues, checks whether the described bug still reproduces against current trunk, and searches merged PRs/CHANGELOG entries for an existing fix. It only ever posts a comment with its evidence and a suggested reason (fixed / duplicate / can't reproduce / needs more info) — it does not close issues or touch labels, since that requires triage-team permissions per the existing policy.
2. Human review stays the closing step. Triage team members review agent-drafted findings the same way they'd review a fellow triager's comment, and close/label using the existing process in triage.md. This adds a research assistant to the existing workflow rather than replacing any part of it.
3. Start narrow. Pilot against a bounded, low-risk slice first — e.g. issues with Needs Testing or no activity in 12+ months and zero comments — rather than the whole backlog, so false positives are cheap to catch.
4. Reopen safety net from day one. Reuse the existing stale-bot pattern to add a "reopen within N days, no questions asked" comment on any issue closed off the back of an agent finding, mirroring Next.js's approach, so bad calls are self-correcting.
5. Only consider auto-close much later, if ever, and only for extremely high-confidence, dual-checked cases, with a hard weekly cap — and that decision should rest with core maintainers, not something an outside contributor's tooling does unilaterally.

Open questions for the core team

• Who owns/hosts this (a #core-editor-sponsored GitHub Action + API key vs. something run by individual triage-team volunteers)? Next.js's version runs on Vercel's own agent infra; Gutenberg doesn't need anything that heavy for a comment-only assistant.
• What's an acceptable false-positive rate before a finding gets posted, and how conservative should confidence thresholds be?
• Should this be scoped to bug reports only, or also help with duplicate-detection on enhancement/feature requests?
• Is there appetite to pilot this on a small label/filter first, and who from the triage team would be willing to review the pilot's output?

CC: @youknowriad @Mamaduka @t-hamano

References

• Marcos Hernanz, "How we closed 1,500 GitHub issues in one month", Next.js blog, Sept 4 2026.
• Vercel, Introducing eve (the agent framework Next.js built closability on).
• Gutenberg Triage docs.
• Gutenberg stale-issue-gardening.yml workflow (existing inactivity-based automation this proposal builds on).
• Gutenberg Repository management docs (labeling conventions referenced above).
• WordPress.org, How to do triage on GitHub.

[youknowriad]

I've been doing some informal form of this already (with some other folks already). Here's the graph of issues of Gutenberg. In the last month, we closed around 1500 issues.

So I guess what I'm saying is that we should just continue doing this as the results prove themselves.

What we've found though is that we can't fully trust the AI results, the AI gives you an assessment (95% of the work) and there needs to be a human to quick verify it.

[karmatosed]

Thanks @theaminulai for posting this, and for building on a lot of thinking @youknowriad, myself and others have done over the years about triage before AI, and now with a human in the loop. As you can see there is work and conversations already going on, so I would first encourage you to look at those and then everyone can be part of the same solution together.

One point I'd flag whatever path is taken: the human ideally isn't the last step in the loop. When review happens only at the end, you get an assessment that looks right, passes a quick check, and quietly erodes trust as the misses compound.

I know the pull towards clearing fast. I want that too and I actually want more tooling in our systems but done in a way that stays, doesn't get added ignored as a report or trust is broken. It's about balance, and @youknowriad's numbers show we can get there collaborating with these tools. This is not an argument against AI. It is a good assist for triage, and a real collaborator. The opportunity is to move human insight upstream, into what the agent knows before it starts and into pairing while it works, rather than only into checking what it produced.

Worth naming that surfacing is one skill. Catching and reviewing AI output is a different one, and most of us are still refining it. A loop designed as if only the first skill matters puts all the weight on the one we have least practice with.

This gets sharper when triage moves past bugs into enhancements and requests. A bug either reproduces or it does not. An enhancement carries context about direction, prior decisions, and what the project has already declined. Matching against things previously closed as "won't do" is a simple start, but it is not the common case. That is where the collaboration and the feedback loop need to be dialled up throughout, not just at the end.

[theaminulai]

Thanks both. Good to know this is already running informally, not starting from zero.

@youknowriad makes sense to just keep supporting what's already working rather than propose a parallel track.

@karmatosed fair correction: feeding context in upstream and pairing during the investigation makes more sense than only gatekeeping the result. Agreed on bugs vs. enhancements , too reproduce/doesn't-reproduce is tractable, "conflicts with a prior won't-do" isn't.

Where should I look to see what's already running, and is there anything concrete I can help with?

[youknowriad]

As I said there's nothing formal running at this point. But myself for instance, I run an AI assisted triage almost daily. I know others do that as well. Please continue doing the same, maybe one way to help is to share a skill/prompts that you found useful in this context.

[t-hamano]

I'm sure that @youknowriad has been closing a huge number of issues recently, so I figured you were probably using AI in some way. I also often use AI for triaging. Basically, I just provide a list of open issues and ask the AI to test them locally and let me know which ones are no longer reproducible. Based on the results, I carefully verify them myself and only close the ones where the AI's findings were accurate.

However, this naturally affects the usage limits of my personal AI subscription, so I might not be able to recommend this exact approach to all contributors.

[aduth]

However, this naturally affects the usage limits of my personal AI subscription, so I might not be able to recommend this exact approach to all contributors.

New "System One" models like Jev and OpenAI's recently(today)-announced Decisions API could end up being helpful tools for low-cost, first-pass triage, even to the extent of integrating some automation into the project's CI. It may not be able to support a full end-to-end workflow of reproducing bugs in a real, local environment. But, toward the broader point of using AI assistance and the costs associated with it, it feels like a promising direction for helping automate some level of structured triage in an efficient way.