TriageLens
Which issues in this backlog will never get fixed? Generated 2026-10-02 14:58 UTC · fully offline, no network calls at inference.
Run
| Repository | astral-sh/ruff |
| Corpus retrieved | 2026-10-02T11:46:20 UTC |
| Issues fetched (PRs excluded) | 6638 closed of 28154 raw items |
| Base rate of “closed without a fix” | 13.5% |
| Training period | 2022-08-24 → 2025-03-21 |
| Held-out period | 2025-03-22 → 2026-10-01 |
| Backlog scored | 1716 open issues |
| Backend | logistic, fitted in 49.8s |
Finding 1 — the loudest-first heuristic is worthless. Ranking by comments and reactions scores AUC 0.5215 against a held-out future period. A constant guess scores 0.5000. At precision@100 the heuristic returns 0.190 — worse than picking at random (0.240). Median comments is 2 on issues that never shipped and 2 on issues that did.
Finding 2 — but an open tabular foundation model does learn something. TabPFN v2 on triage-time features only reaches AUC 0.8202 and precision@100 0.650 against a 0.195 base rate: the 100 issues it flags contain 65 genuine duds where guessing gives 19.
Finding 3 — the signal is thinness, not attention.
The strongest features are lbl_bug, log_author_prior_issues,
log_n_labels — four of the top five run backwards. What
predicts an issue dying is looking under-specified: a new reporter, few labels,
phrased as a question.
Finding 4 — the leaks made it worse. Adding resolution-time labels and comment counts dropped AUC from 0.8202 to 0.8095. Every tutorial that says “just add engagement features” would have made this tool worse while reporting a nicer table.
Measured, on a held-out future period
| Model | AUC | Precision@100 | Recall |
|---|---|---|---|
| always-alive (trivial) | 0.5000 | 0.240 | 0.000 |
| single best feature: n_labels | 0.7398 | 0.610 | 0.778 |
| logistic, all triage-time features | 0.7869 | 0.560 | 0.778 |
| FOLK BELIEF: loudest-first (leaky features) leaky feature | 0.5208 | 0.190 | 0.487 |
| logistic, all triage-time (RANDOM split) | 0.7870 | 0.430 | 0.756 |
| TabPFN v2, triage-time features only | 0.8202 | 0.650 | 0.838 |
| TabPFN + resolution-time labels & comments (LEAK) leaky feature | 0.8095 | 0.580 | 0.855 |
Backlog, ranked by P(closed without a fix)
| # | Issue | Score |
|---|---|---|
| 1 |
#14638 Allow (formatter?) config to never introduce implicitly concatenated strings or just accept that a check run will complain about it
no strong triage signal
|
88%
|
| 2 |
#4368 Add config to disable S101 (assert detected) and a few others in test folders
no strong triage signal
|
81%
|
| 3 |
#20362 `RUF022` does not follow `lint.isort.classes` option
no strong triage signal
|
80%
|
| 4 |
#7568 Allow disabling sub-config required-versions
no strong triage signal
|
78%
|
| 5 |
#29073 B014 does not detect redundant built-in exception subclasses
no triage labels yet
|
78%
|
| 6 |
#20280 Relax DOC rules for private/inner methods
no strong triage signal
|
77%
|
| 7 |
#6139 Ignore specific lines in every file
no strong triage signal
|
77%
|
| 8 |
#29071 B008: False positive for frozen dataclass defaults with immutable fields
no triage labels yet
|
77%
|
| 9 |
#24955 [pydocstyle] allow configuring indentation level for Google style
no strong triage signal
|
77%
|
| 10 |
#7401 pydocstyle: Ignore matching functions
no strong triage signal
|
76%
|
| 11 |
#14365 Add lint.flake8-bandit setting to ignore specific argument names for S106 and S107
no strong triage signal
|
76%
|
| 12 |
#14240 Request: fix Autofix for multiple-with-statements/SIM117
no triage labels yet
|
75%
|
| 13 |
#22791 Adding a new setting for rule E501 (`ignore-all-comments`)
no strong triage signal
|
75%
|
| 14 |
#27926 Add rule checking for `if __name__ == "__main__":` clause in test files
no strong triage signal
|
75%
|
| 15 |
#27925 Add rule to keep rule names in configuration sorted
no strong triage signal
|
74%
|
| 16 |
#29030 [flake8-pytest-style] Flag tests whose only assertion is inside an `if`/`for`
no strong triage signal
|
74%
|
| 17 |
#27603 Detect dataclasses that silently override an inherited __hash__
no strong triage signal
|
73%
|
| 18 |
#5527 Add opposite of `extend` field or add it as cli option
no strong triage signal
|
73%
|
| 19 |
#27448 Add option to suppress `PLR6301` for custom instance method decorators
no strong triage signal
|
73%
|
| 20 |
#12743 Add support for nosec comments for bandit rules
no strong triage signal
|
73%
|
| 21 |
#18976 Possible to automatically ignore UP046 in cases of manual variance setting
no strong triage signal
|
73%
|
| 22 |
#6258 Feature request: Use `pathlib` objects
no strong triage signal
|
73%
|
| 23 |
#27556 Add support for PyMdownx Snippets when formatting Markdown
no strong triage signal
|
73%
|
| 24 |
#19559 Rule suggestion: always import from definition
no strong triage signal
|
73%
|
| 25 |
#9870 Different length sort behavior against isort
no strong triage signal
|
73%
|
| 26 |
#23705 BLE001 (blind-except) add option to flag even if logger.exception / exc_info is used
no strong triage signal
|
72%
|
| 27 |
#9019 Attempting to use `ruff check` with includes, but without specifying a directory, picks up nested projects
no triage labels yet
|
72%
|
| 28 |
#28340 formatter ignores `fmt: off` pragma inside match expression
no strong triage signal
|
72%
|
| 29 |
#10079 Allow banning of certain builtins in "banned-api"
no triage labels yet
|
72%
|
| 30 |
#18492 Proposal: C422 Unnecessary filter() usage
no strong triage signal
|
72%
|
| 31 |
#19685 Allow for certain global names in F821
no strong triage signal
|
72%
|
| 32 |
#15801 pyproject.toml fix-only=True cannot be toggled per directory by using multiple pyproject.toml files and the extend setting.
no strong triage signal
|
72%
|
| 33 |
#25420 Should `lint.flake8-tidy-imports.ban-lazy` automatically count as exclusions for `lint.flake8-tidy-imports.require-lazy.include="all"`
no strong triage signal
|
72%
|
| 34 |
#21133 Detect variables used after context-manager exit
no strong triage signal
|
71%
|
| 35 |
#13330 Allow character ranges or locales in `allowed-confusables` for RUF001, RUF002, & RUF003
no triage labels yet
|
71%
|
| 36 |
#23703 New rule: raw-string-in-assert (like EM101)
no strong triage signal
|
71%
|
| 37 |
#15056 New rule: Unnecessary unicode escape character
no strong triage signal
|
71%
|
| 38 |
#8832 ruff should respect targeting Python 3.6 for UP036
no strong triage signal
|
71%
|
| 39 |
#5860 Feature request: docstring-min-length (pylint)
very short body (<200 chars)
|
70%
|
| 40 |
#21690 Detect `if-elif` with same condition causing unreachable branch
no strong triage signal
|
70%
|
The rule that costs nothing
| Body length | n | Dead rate | |
|---|---|---|---|
| 0–0 | 155 | 0.0323 | |
| 1–200 | 608 | 0.1135 | |
| 201–500 | 1552 | 0.1488 | |
| 501–1000 | 2092 | 0.1439 | |
| 1001–2000 | 1541 | 0.1324 | |
| 2001–4000 | 690 | 0.1275 |
An issue with an empty body is 4.6× less likely to ever ship than a well-formed one (3.2% vs 14.9%). No model required. For the most common triage action — ask for more information, or close it — this one rule may be all you need.
What to do with this
- Stop sorting by comments. It is worse than random at the top of the list.
- Use the ranking to find dead weight, not to pick what to build. This model is good at the question it was asked and no good at the inverse.
- Handle empty-body issues first, with a template reply, before reading any of the rest.
- Ignore the engagement metrics. They were measured and they hurt.