Repository navigation
Update the RustPython Ruff fork to 0.17.2 - #8
Draft
youknowdot wants to merge 496 commits into
Draft
youknowdot wants to merge 496 commits into
youknowdot wants to merge 496 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Co-authored-by: typeshedbot <> Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
) ## Summary ty finds existing imports for fixes, completions, and inlay hints by inspecting the module's immediate statements. The importer previously visited nested statement bodies and discarded every import it found there. On the `dataset.py` benchmark fixture with a diagnostic that requests an import fix, the query executed 2,005 instructions, down from 112,090 on main (−98.21%). The full `ty check` used 0.059% fewer instructions across three paired runs and produced the same diagnostic. ## Test Plan The auto-import tests and the module import mdtest passed.
## Summary
Support generic `TypedDict`s in function calls, for example:
```py
class Item[T](TypedDict):
value: T
def get_value[T](item: Item[T]) -> T:
return item["value"]
def _(item: Item[int]):
reveal_type(get_value(item)) # int, previously Unknown
```
closes astral-sh/ty#4884.
## Test strategy
New Markdown test
…stral-sh#29883) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
…tion (astral-sh#29885) ## Summary Stop scanning a package initializer once ty recognizes a legacy namespace declaration. The resolver examines only module-level statements and accepts the existing `pkgutil.extend_path` and `pkg_resources.declare_namespace` forms. On the large parser fixture, the declaration check used the following instruction counts. These cover the top-level scan and pattern matching, excluding the calls to obtain and load the parsed module. | Declaration in `__init__.py` | Top-level statements | Visitor (before) | Iterator (after) | Reduction | | --- | ---: | ---: | ---: | ---: | | First statement | 113 | 2,595 | 454 | 2,141 (82.5%) | | Absent | 112 | 3,189 | 1,290 | 1,899 (59.5%) | All three paired runs produced the same counts. The full `ty check` used 0.004% fewer instructions with the declaration first. ## Test Plan The legacy namespace and partial stub package import mdtests passed.
## Summary Today, if a CI job exports `FORCE_COLOR=1`, running `NO_COLOR=1 ruff check .` still prints colored diagnostics. The same happens with `ty check`. Giving a non-empty `NO_COLOR` priority lets users request plain output without having to unset a variable inherited from their environment. This matches how uv and the Python interpreter resolve that conflict. Users who want color for one command can still pass `--color=always`. [uv’s resolver](https://github.com/astral-sh/uv/blob/39a02a0d9085f14dc3d5aa8655ac010a1fd377f8/crates/uv-cli/src/settings.rs#L172-L197) · [Python’s color precedence](https://docs.python.org/3/using/cmdline.html#controlling-color) Previously, the shared resolver turned `FORCE_COLOR` into an override before `colored` could honor `NO_COLOR`. The resolver now handles the color environment for both tools, and `colored` styles diagnostics on stdout according to that decision. The first matching rule wins. 1. Use an explicit `--color=always` or `--color=never`. 2. A non-empty `NO_COLOR` disables color. 3. A non-empty `FORCE_COLOR` or `CLICOLOR_FORCE` enables color, including in a pipe. 4. If stdout is a pipe or file, disable color. 5. On a terminal, enable color if `CLICOLOR` is non-empty or `TERM` is anything other than `dumb` (including unset). All four color variables treat `0` as non-empty and an empty string as unset. In particular, `FORCE_COLOR=""` previously disabled color. It now lets the remaining settings and terminal detection decide. ## Comparison with the upstream conventions - The [`NO_COLOR` convention](https://no-color.org/) specifies that a non-empty value disables color by default and that command-line options can override it. It gives no rule for a conflict with `FORCE_COLOR`. Its FAQ also limits the convention to color, while Ruff and ty suppress other diagnostic styling, such as bold text, when color is disabled. - The [`FORCE_COLOR` convention](https://force-color.org/) specifies that any non-empty value enables color, even in a pipe. Its example applies `FORCE_COLOR` last, so the example produces colored output when both variables are set. Here we follow uv and CPython in giving `NO_COLOR` priority. - The [CLICOLORS specification](https://bixense.com/clicolors/) gives `NO_COLOR` priority over `CLICOLOR_FORCE` and makes `CLICOLOR` a terminal-only request. We follow those rules, add `FORCE_COLOR` as an alias for `CLICOLOR_FORCE`, and use `TERM=dumb` for the default the spec leaves to applications. ## Comparison with uv When `--color` is omitted, uv also checks non-empty `NO_COLOR` before either force variable. In Ruff and ty, passing `--color=auto` has the same effect as omitting it. In uv, passing it delegates the decision to `anstream`, which recognizes `CLICOLOR_FORCE` but ignores `FORCE_COLOR`. The terminal heuristics also differ: uv delegates to anstream, which handles `TERM=dumb` differently depending on which platform uv is being run on, and also honours the `CI` variable. I'd prefer not to have platform-dependent behaviour or to have behaviour automatically be different in CI; that feels too confusing/magical to me. [uv’s resolver](https://github.com/astral-sh/uv/blob/39a02a0d9085f14dc3d5aa8655ac010a1fd377f8/crates/uv-cli/src/settings.rs#L172-L197) · [`anstream` detection](https://docs.rs/anstream/1.0.0/src/anstream/auto.rs.html#198-219) · [`anstyle-query` environment checks](https://docs.rs/anstyle-query/1.1.5/src/anstyle_query/lib.rs.html#23-82) Each example assumes other color settings are unset. | Case | Ruff and ty with this change | uv | | --- | --- | --- | | `FORCE_COLOR=1` with `--color=auto`, output piped | Colored, as when the flag is omitted | Plain | | `CLICOLOR=0 TERM=xterm`, on a terminal | Colored | Plain | | `CLICOLOR="" TERM=dumb`, on a terminal | Plain | Colored | | `CI=1 TERM=dumb`, on a terminal | Plain | Colored | | `TERM` unset, on a terminal | Colored on all platforms | Plain on non-Windows platforms, colored on Windows | An empty `TERM` allows color on a terminal in both implementations.
…sh#29899) On `main`, `ruff check --color=always example.py >report.txt` saves colored diagnostics, but replacing `>report.txt` with `--output-file report.txt` strips the color. This PR makes both ways of saving the report honor `--color=always`. Automatic color also goes wrong with `ruff check --fix -`. Ruff writes the fixed source to stdout and diagnostics to stderr, yet it checks whether stdout is a terminal to decide whether to color the diagnostics. Redirecting only stderr can put ANSI escapes in the saved report, while redirecting only stdout can leave the diagnostics plain on the terminal. Ruff now applies the color policy from astral-sh#29898 to the destination that receives the diagnostics. When the flags and environment leave color up to terminal detection, Ruff checks that destination. With `--diff -`, for example, a patch printed to the terminal on stdout can keep its color while the fix summary redirected from stderr stays plain. ## Test plan This video compares the base of this PR (astral-sh#29898 at `baa3088701`) with the fix (`779a511358`). Each command uses the same one-line input, `print(unknown)`. https://github.com/user-attachments/assets/d55b0fb3-60aa-4ac8-806c-f51c394890f5 <details> <summary>Manual split-stream reproduction and results</summary> Run these commands from a terminal in a scratch directory, first with `ruff` from the base commit (`baa3088701`) on `PATH`, then with the build from this PR. The `tr` pipelines count escape bytes in the redirected files. ```sh unset NO_COLOR FORCE_COLOR CLICOLOR CLICOLOR_FORCE export TERM=xterm-256color FORCE_HYPERLINK=0 check_stdin() { ruff check --isolated --no-cache --color=auto --output-format=concise "$@" - } printf 'print(unknown)\n' | check_stdin --fix --select F821 2>err LC_ALL=C tr -cd '\033' <err | wc -c printf 'print(unknown)\n' | check_stdin --fix --select F821 >out cat out ``` On the base commit, `err` contained 11 escape bytes. With this change it contains zero. Reversing the redirection (`>out`) changed the F821 diagnostic on the terminal from plain to colored. On both revisions, `out` contained the unchanged source, `print(unknown)`. With `--diff`, stdout carries the patch and stderr carries the fix summary. ```sh printf 'import os\nprint(unknown)\n' | check_stdin --diff --config 'show-fixes = true' 2>summary LC_ALL=C tr -cd '\033' <summary | wc -c printf 'import os\nprint(unknown)\n' | check_stdin --diff --config 'show-fixes = true' >patch LC_ALL=C tr -cd '\033' <patch | wc -c ``` The removed import was red on the terminal on both revisions, while the escape count in `summary` dropped from 10 to zero. With `>patch`, the patch contained zero escapes on both revisions, and the fix summary on the terminal changed from plain to colored. </details>
## Summary Prevent exponential growth in equality narrowing with large union of intersections. fixes astral-sh/ty#4867 ## Test Strategy Regression test
) ## Summary Eleven lint checks collect `return` statements from the current function. Their visitor now skips expressions as well as nested functions and classes. The annotation fixer keeps its existing visitor because it also needs to detect `yield` and `yield from` before suggesting a return type. For PLR0911 alone, 196 calls with all stable rules enabled, this cuts instructions from 323,004 to 64,650 (−79.98%) ## Test Plan The existing Ruff linter unit suite passes with no snapshot changes.
…ral-sh#29840) ## Summary ty can panic on the call to `extend` below. The fuzzer started hitting this in astral-sh#29747, but this example also panics on its parent. ```python from typing import Annotated type Alias = Annotated[int, list(iter(aliases))] aliases = [Alias] aliases.extend([Alias]) ``` When ty checks whether the return type of `Iterator.__next__` refers back to the protocol, it encounters the runtime `Alias` object. Expanding its value checks the iterator in the `Annotated` metadata and re-enters the same protocol check. We now visit an alias object separately from the type it denotes. The protocol check can leave `Alias.__value__` unevaluated here and still expand aliases used in annotations. Visitors that look for type variables can read the `T` in a specialized alias object like `Box[T]` without evaluating its value. I reran all 20,001 seeds (`0` through `20000`) against the nightly baseline, `2469f71c13`. The fuzzer found no new panics or timeouts. Seed `9098` produced the same seven diagnostics as the baseline. [Original nightly run](https://github.com/astral-sh/ty/actions/runs/40978355766) Closes astral-sh/ty#4872.
A generic type alias defined in a class could report a class-local name
as undefined or silently resolve it to a module-level name. For example,
in:
```py
class Outer:
class Inner:
type Pair[T] = tuple[T, Inner]
```
ty skipped the class scope that defined `Inner` because the alias's
type-parameter scope sat between its value and the class.
Name lookup now sees the enclosing class through consecutive annotation
scopes. The semantic index and type inference use the same visibility
rule.
Fixes astral-sh/ty#4886.
…l-sh#29897) ## Summary We now prepare a static class’s constructor bindings once and match each call’s arguments against its own copy. We also defer walking the caller’s enclosing scopes until a generic call might refer to one of their type variables. Compared with `main`, a cold `ty check` used 2.9% fewer instructions on PyTorch and 2.1% fewer on SQLAlchemy. --------- Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary
In the original local debug runs at depth 18, `ty check` went from 7.31
s to under 0.1 s for `<` and from 11.10 s to under 0.1 s for `==`.
Both operators check whether corresponding tuple elements are equal. If
the first pair could be equal, ty also checks the second pair. In this
example both positions have the same type, so ty repeats the same check
and each extra level doubles the work.
```python
type T0 = tuple[int]
type T1 = tuple[T0, T0]
type T2 = tuple[T1, T1]
def compare(a: T2, b: T2) -> bool:
return a < b # Before this change, ty checks T1 against T1 twice.
```
We now reuse completed equality checks. We skip caching results that may
depend on a recursive fallback, since a pair that looks ambiguous during
recursion can be resolved on a later visit.
Comparing large unions can visit many distinct pairs with little reuse,
so we clear the completed entries when the cache would exceed 1,024
results. In a local debug run comparing single-element tuples with
disjoint 1,024-member literal unions, peak memory was about 46 MiB with
the cache and 47 MiB on `main`.
Closes astral-sh/ty#4883.
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary When we build a union, we compares each new member with earlier ones to see which are redundant. For example, adding `int` can eliminate an earlier `bool`. On discord.py, 85% of the "wide" unions built at the three affected sites had the same members, in the same order, as a union we'd already built. We now cache the normalized union when we know the input has at least eight members. We reuse the result when the ordered members, program, alias-expansion policy, and recursion flag match. If the cache lookup encounters a cycle, we use the existing union builder directly. (A cold check of discord.py used 3.17% fewer instructions locally.) --------- Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
…tral-sh#29942) A contrived function with 26 consecutive `data += data` statements makes ty 0.0.96 abort under a 512 MiB address-space limit. Each statement doubles the bytes literal ty builds during type inference, even though nothing calls the function. Check the combined length before concatenating two bytes literals. ty keeps the exact value for results up to 4,096 bytes and infers `bytes` for larger results. The same file now passes under the same limit.
…tral-sh#29785) ## Summary This PR adds support for `ForeignKey` and `OneToOneField` used as primary keys. The `.pk` attribute now has the value type of the related model's primary key, rather than the related model's type. An explicit `to_field` selects that field's value type instead. Example: ```py from django.db import models class Book(models.Model): id = models.UUIDField(primary_key=True) class Ebook(models.Model): book = models.OneToOneField(Book, on_delete=models.CASCADE, primary_key=True) def _(ebook: Ebook): reveal_type(ebook.book) # Book reveal_type(ebook.book_id) # UUID reveal_type(ebook.pk) # UUID (previously Book) ``` closes astral-sh/ty#4847.
Report `invalid-glob` as a warning when ty can continue type checking. This includes invalid module globs in `allowed-unresolved-imports` or `replace-imports-with-any` within an analysis override. When an invalid glob prevents type checking, ty reports `error[invalid-glob]`. A script’s top-level analysis settings are one example. --------- Co-authored-by: Zsolt Dollenstein <zsol@openai.com>
…h#29959) ## Summary A cold `ty check` of NumPy went from 1.15 s on `main` to 1.00 s with this change (medians across 48 paired release runs on Linux with a four-CPU quota). The two independent batches took 11.7% and 9.8% less wall time, respectively. Both builds checked the same 205 files at NumPy revision `f9f1772` with Python 3.12.15 and emitted identical diagnostics. Whole-command instructions fell from 14.91 billion to 14.26 billion. When an overloaded method is incompatible with a base method, ty checks the callable types a second time to explain the error. The regular overload search turns off context collection while trying alternative source signatures. On NumPy, the explanation pass revisited more than 11,000 signature pairs even though most of those comparisons could never add a note. We now return an empty explanation directly when the source has multiple overloads and none of the target signatures can use unary overload aggregation. We still run the full validity check. When the overloads cover a union parameter, aggregation can explain which part of the combined return type is incompatible. For example, the new test keeps the note that the `bytes` member of `bytes | str` is not assignable to `str`. --------- Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
…stral-sh#30220) ## Summary Closes astral-sh/ty#4961. When a parameter accepts either `T` or a generic type containing `T`, ty can infer extra layers of that generic type when it revisits an argument expression whose type is context-sensitive. We do fixpoint iteration on generic calls to allow contextual inference to propagate types from sibling arguments, and this nested growth can prevent that iteration from settling. An example: ```python from typing import reveal_type def as_list[T](value: T | list[T]) -> list[T]: if isinstance(value, list): return value return [value] def example(n: int, xs: list[int]) -> None: # The type of `n` is always `int`. reveal_type(as_list(n)) # list[int] (before and after) # This case also already converges quickly because the type of `xs` is always `list[int]`. # We might prefer to infer a type of `list[int]` in this case, but that would be a separate change: reveal_type(as_list(xs)) # list[list[int] | int] (before and after) # Both of these cases are context sensitive and we build repeatedly deeper nested types before this PR: reveal_type(as_list([n])) # list[list[int] | int] with this change reveal_type(as_list(as_list(n))) # list[list[int] | int] with this change ``` The arguments `xs`, `[n]`, and `as_list(n)` all start out with type `list[int]`. Each can match the `T` arm with `T = list[int]` or the `list[T]` arm with `T = int`. We then revisit the same argument expression using the specialized parameter type as type context. This is how one argument can help determine the type of another in a generic call; this is particularly useful for e.g. typing a lambda in a call to `map`, where the type of data passed to `map` can help us understand the type of the untyped lambda. On main, ty combines the two bindings into `int | list[int]`, so the type context from the `list[T]` arm becomes `list[int | list[int]]`. For the context-sensitive argument expressions (literal `[n]` or the inner `as_list(n)`), that context can widen the inferred element type to `int | list[int]`, which then feeds back into the outer call and can add yet another list layer on each iteration. With this PR, the hint from the `list[T]` arm is `list[int]`, so all three argument expressions keep that type. Both bindings still contribute to the result of the outer call, `list[list[int] | int]`. The same feedback loop made the nested `factory.Maybe` calls in astral-sh/ty#4961 slow to check. I don't love this fix -- it's too narrowly targeted to unions and it can miss some cases. But it seems like more general approaches will be quite large/invasive, so I think this is the best fix for now to solve the known problem case. ## Comparison with astral-sh#30195 astral-sh#30195 detects when inference paths disagree on a type variable. If the resulting parameter type already covers the previous argument type, it reuses that whole argument type as context. This can discard information from a sibling argument, including the parameter type needed to check a lambda body. This change computes a separate specialization for the nested union members. For `T | list[T]`, ty tries matching the list arguments to `list[T]` while retaining the constraints from the other arguments and the expected return. It uses that hint only if one specialization works across the call and preserves an already-compatible return. The nested member must have a solution for its type variables, and the argument’s previous type must be redundant with that specialized member. For example, `list[int] | Unknown` is assignable to `Iterable[int]`, but using that as the next hint can invalidate a nested call. The bare `T` arm keeps its ordinary specialization. The benchmark covers the issue's nested constructor call with no expected type for the outer expression, including a variant where a type alias hides the union members. ## Test plan <details> <summary>Dependency-free constructor reproducer</summary> ```python class Declaration[K, V]: def __init__(self, key: K | None = None, *choices: "V | Declaration[K, V]"): ... class Attribute[V](Declaration[str, V]): ... class SubFactory[V](Declaration[str, V]): ... Declaration("name", Declaration("name", Attribute(), SubFactory()), None) ``` </details> I checked this constructor example three times with each debug binary. On `4c51cc646d`, all three runs hit a 10-second timeout. On this branch, all three checks completed without diagnostics, with a median of 49 ms. With the `types-factory-boy` 0.4.1 stubs, the reporter's full example finished in 231–255 ms across three Linux ARM64 debug runs. The baseline hit a 35-second timeout. The only diagnostics were the two `Meta` override errors also present with the nested expression removed.
…aliases (astral-sh#30149) ty 0.0.99 overflows its stack on the call to `map_leaves` inside `visit` below. It should report a return-type error instead. If `V` is `int`, for example, `visit` should return an `int` but `map_leaves` returns a `Tree[int]`, which expands to `int | tuple[Tree[int]]` -- the tuple part is not compatible with the required return type. ```python from collections.abc import Callable type Tree[T] = T | tuple[Tree[T], ...] def map_leaves[T, U](fn: Callable[[T], U], value: T) -> Tree[U]: raise NotImplementedError def visit[V](value: V) -> V: return map_leaves(lambda leaf: leaf, value) ``` ## Why ty specializes alias arguments Suppose ty knows `U = int` and encounters `Tree[U]`. Expanding the definition would lead straight back to `Tree[U]` inside the tuple. Before astral-sh#24773, expanding that infinite recursion would just fall back to `Tree[Any]` at the recursion point and lose type information. After astral-sh#24773, we can re-specialize `Tree[U]` as `Tree[int]` directly without needing to expand the alias. astral-sh#24773 added the `as_specialization` method because substituting into an alias's stored arguments requires a mapping with one entry per type parameter. Given a partial mapping, the helper kept the original type variable. But there are two cases where it did that, and only one of them is correct: **Missing slot.** Suppose we know only `T = int` for `map_leaves`. Its callback type becomes `Callable[[int], U]` and its return type stays `Tree[U]`. Keeping `U` lets ty specialize the return type to `Tree[str]` if it later learns that the callback returns `str`. This is correct. **Skipped slot.** Since astral-sh#26680, ty uses both the arguments and the expected return type to infer a call. In the reported example, that can give `U` a binding that mentions itself. ```text U = V | tuple[Tree[U], ...] ``` While resolving this binding, ty marks `U` as skipped. If `as_specialization` keeps the inner `U` (as it does on main), substituting this binding into another variable's solution copies the same `U` one tuple level deeper on every pass. We now call the existing `ApplySpecialization::get`, which already distinguishes missing vs skipped, and supplies `Never` (instead of the original type variable) for a skipped typevar. In the recursive "skipped" case, `Never` works because the recursive occurrence of the typevar doesn't add new evidence about the leaf type. This use of `Never` pre-dates this PR. Closes astral-sh/ty#4972. ## Test plan Bisected the reporter's Python 3.12 example to astral-sh#26680. Its parent accepted the example in 0.04 seconds. The merge commit overflowed its stack after 22 seconds. With this fix, the example passes with the cast and reports `invalid-explicit-return` when the cast is removed.
) ## Summary ty can panic when two recursive `NamedTuple` fields share a wide union. For example, both fields in the regression test simplify to `int | Leaf`, but `Leaf` itself refers back to the two named tuples. Checking whether we can drop `Leaf` takes us back to the union we're still building. Its cached result can alternate between `int` and `int | Leaf` until ty panics. We now build the union directly in the caller when the cache lookup encounters a cycle. Both fields resolve to `int | Leaf`.
…ral-sh#30242) Unary `+`, `-`, and `~` could report an invalid return type when a method returned `Self`. For example, given `x: T` with `T` bounded by `Base`, ty inferred `Base` for `-x` even though `x.__neg__()` returned `T`. Use the upper bound for literal folding and the original operand as the receiver when binding the unary method. For a union bound, each method binds to the part of the operand that belongs to its union member. This also keeps the diagnostic that identifies a member missing the operator. For an intersection bound, ty checks the call on each component and combines the return types of the matching calls. Fixes astral-sh/ty#4983 ## Ecosystem report At `b79065f153a6c39d5c4ea2c53f6b11f0ee553a9b`, ty removes three false positives in Ibis, PySpark, and SymPy. In SymPy’s matrix subtraction, preserving the operand type also exposes an existing decorated-method inference issue (astral-sh/ty#2520). The unsupported addition and its return warning move from reflected subtraction to forward subtraction, replacing an incorrect return-type error there. [Full ecosystem report](astral-sh#30242 (comment)) ## Test plan Compared a debug build to the base revision on a file with 300 `with x` statements and a 48-class union bound. Three runs of each build used 0.42–0.43 seconds of user CPU. This checks that other implicit method calls on union-bounded type variables still benefit from cached lookup.
## Summary When Ruff removes the sole named import from a statement, the fix deletes the entire statement. Check the AST for that case before parsing the statement again with LibCST. On the Pypinyin default-rules benchmark, the first lint call fell from 4.75M to 3.25M Callgrind instructions (31.6%), and repeated lint calls fell from 248 µs to 229 µs (7.6%). ## Test Plan Ran the linter unit tests, including the import-fix snapshots.
…l-sh#30346) ## Summary UP009 compiles a regular expression the first time it checks a leading comment for an encoding declaration. Scanning for the PEP 263 encoding name directly reduced the first lint call on the Pypinyin default-rules benchmark from 4.75M to 2.64M Callgrind instructions (44.4%) and from 1,164 µs to 867 µs. Repeated calls measured 249 µs with either implementation. ## Test Plan Added coverage for multiple declaration markers and invalid encoding names, and ran the linter unit tests.
…stral-sh#30353) ## Summary PYI033 now reads the `# type:` prefix and checks for `ignore` directly, replacing two regular expressions. On repeated lints of the NumPy globals benchmark with preview rules, this reduced Callgrind instructions from 960,493 to 956,855 (0.38%). ## Test Plan Ran the linter test suite and checked comment recognition against the previous implementation.
…al-sh#30352) ## Summary A single assignment target such as `x = 1` cannot contain a redeclared name. For unpacking and chained assignments, track borrowed names and consult the configured dummy-variable pattern only when a name repeats. On repeated lints of the NumPy globals benchmark with default rules, this reduced Callgrind instructions from 320,256 to 313,889 (2.0%). ## Test Plan Added coverage for repeated and dummy targets and ran the linter test suite.
…stral-sh#30351) ## Summary E501 and W505 skip measuring display width when a line contains no tabs and its UTF-8 byte length fits the column limit. Lines containing tabs use the existing width calculation. On an uncached E501 check of 110 Home Assistant files, this reduced whole-process Callgrind instructions from 342.4M to 313.6M (8.4%, median of three runs). ## Test Plan Ran the linter test suite. All 116 Home Assistant diagnostics matched, and E501 and W505 produced identical diagnostics for tab widths of 1, 2, 4, and 8 on lines with tabs, wide characters, and combining characters.
) ## Summary Previously, Ruff parsed the leading words of each docstring line as a possible section name. Both `Returns:` and `Returns: the new value.` matched `Returns`, so Ruff extracted the name again and checked the remaining text to reject the second line. Ruff now skips the indentation already measured by `leading_space`, trims trailing whitespace, and strips one optional trailing colon. It passes the rest of the line to the existing section-name parser. `Returns:` yields `Returns`. The inline description in `Returns: the new value.` makes the match fail immediately. On a match, `suspected_as_section` returns both the section kind and the borrowed name so the caller can record its source range. Each measurement counts instructions for one uncached lint of the NumPy `globals.py` benchmark after initialization. | Rules | Before | After | Reduction | | --- | ---: | ---: | ---: | | All stable | 819,973 | 767,658 | 6.4% | | All with preview | 951,580 | 899,315 | 5.5% | ## Test Plan Ran the linter and Python AST tests and compared diagnostics and edits across the benchmark files and docstring conventions.
…astral-sh#30348) ## Summary Use Ruff's naming convention for new ty rules so their names read naturally in suppression comments. Explain how to use Ruff's categories to choose a default severity, with examples from existing ty rules. ## Test Plan Checked the example rules against their declared defaults and ran the documentation checks.
## Summary Ruff scans the enabled rules separately to decide whether to run each lint pass. For example, the import pass walks the file’s enabled rules looking for either I001 or I002. This change builds one rule set per lint source at compile time, so Ruff can select the import pass by intersecting its two rules with the file’s enabled rules after per-file ignores. Callgrind measured the following instruction counts against `5b21d854bc`. | Workload | Before | After | Reduction | | --- | ---: | ---: | ---: | | NumPy `globals.py`, CodSpeed default rules, per uncached lint after initialization | 306,801 | 283,188 | 7.70% | | Home Assistant `homeassistant/components`, uncached CLI run over 9,223 files | 16,381,508,852 | 16,130,855,302 | 1.53% | ## Test Plan The linter tests passed, and diagnostics and suggested edits matched across the benchmark fixtures, per-file ignore cases, Home Assistant, and Airflow.
…h#29446) <!-- Thank you for contributing to Ruff/ty! To help us out with reviewing, please consider the following: - Does this pull request include a summary of the change? (See below.) - Does this pull request include a descriptive title? (Please prefix with `[ty]` for ty pull requests.) - Does this pull request include references to any relevant issues? - Does this PR follow our AI policy (https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)? --> - fixes: astral-sh#29445 ## Summary - `TID251` detects uses of APIs banned by project configuration, including access through import aliases. For example, a project can ban os.path.join with this configuration: For example, ``` toml [lint] extend-select = ["TID251"] [lint.flake8-tidy-imports.banned-api] "os.path.join".msg = "Use pathlib.Path instead." ``` - This example configures only one banned API. For each resolved attribute access, Ruff currently searches the banned API entries until it finds a match or reaches the end. Repeating this search becomes costly when the banned list is large. - This change looks up the resolved name directly in the existing hash map. It retains the qualified-name comparison and preserves the distinction between implicit builtin access, such as dict.fromkeys, and explicit access through builtins.dict.fromkeys. - This is intended to preserve existing matching behavior and diagnostic messages. ## Test Plan - All 29 related tests pass before and after the change. - Added 12 regression cases covering aliases, exact matching, implicit and explicit builtins, relative imports, shadowed imports, Unicode identifiers, and long qualified names. - Verified identical stdout, stderr, and exit status across the benchmark workloads. - Formatting and applicable pre-commit checks pass. Local benchmark results, using the median of 11 runs per binary in randomized order: ### Benchmark result Workload | Before | After | Time reduction -- | -- | -- | -- Python stdlib, default rules | 604.6 ms | 609.4 ms | −0.8% Python stdlib, default rules + TID251, 100 banned entries | 639.6 ms | 606.8 ms | 5.1% Python stdlib, default rules + TID251, 1,000 banned entries | 1,022.3 ms | 633.9 ms | 38.0% Synthetic input, TID251, 1,000 banned entries | 282.9 ms | 19.2 ms | 93.2% --------- Co-authored-by: Micha Reiser <micha@reiser.io>
…al-sh#30362) ## Summary When a rule asks whether `typing.Annotated` is `Literal`, Ruff tries each configured typing module. For Airflow, it builds `airflow.typing_compat.Literal` even though `Annotated` and `Literal` already differ. Check the member first. When it matches, compare the module segments directly. | Stock Airflow workload | Baseline instructions | Patched instructions | Reduction | | --- | ---: | ---: | ---: | | CLI API datamodels, recurring uncached lint | 42,441,754 | 37,060,098 | 12.68% | | SDK API datamodels, recurring uncached lint | 14,333,215 | 13,086,030 | 8.70% | | Full checkout, whole uncached command | 13,984,390,679 | 13,833,412,753 | 1.08% | ## Test Plan The linter suite passed, and both versions produced byte-identical diagnostics and edits on Airflow.
## Summary Ruff’s suppression-comment scanners call `ruff_python_trivia::Cursor::first` and `bump` for each character. Add `#[inline]` to the public cursor methods and the forward and backward predicate loops so callers can inline these operations. Callgrind instruction counts for uncached Ruff lint on x86-64 with the profiling build (without fat LTO): | Workload | Main | With this change | Reduction | | --- | ---: | ---: | ---: | | JAX docs | 871,822,822 | 747,817,978 | 14.22% | | JAX shard_map.py | 210,767,215 | 98,080,629 | 53.46% | | Jinja | 204,158,352 | 201,518,289 | 1.29% | ## Test Plan The trivia unit tests passed, and the benchmark runs produced identical diagnostics.
…stral-sh#30369) ## Summary When Ruff checks whether `np.ndarray` denotes `typing.Literal`, it can reject the spelling `ndarray` before resolving `np` and constructing a qualified name. For a bare name spelled differently from the target, return early if its recorded binding is unaliased. An alias such as `L` in `from typing import Literal as L`, or a name without a recorded binding, goes through normal resolution. | Uncached lint workload | Main instructions | With this change | Reduction | | --- | ---: | ---: | ---: | | SciPy-stubs `linalg`, whole command | 443,695,416 | 419,378,831 | 5.48% | | Full SciPy-stubs, whole command | 4,207,292,766 | 4,030,975,285 | 4.19% | | Airflow CLI API datamodels, recurring lint | 36,669,082 | 34,882,858 | 4.87% | | Full Airflow, whole command | 12,622,939,750 | 12,573,708,458 | 0.39% | ## Test Plan The linter and semantic-model tests passed, and diagnostics and fixes matched main across the linter fixtures, Home Assistant, and import-alias cases.
## Summary When Ruff reads a notebook from disk, it passes `&mut BufReader<File>` to `serde_json`. Serde calls `.bytes()` on that reference, which uses Rust’s generic one-byte read path. Passing the reader by value lets the byte iterator consume bytes directly from the existing buffer and refill it as needed. The counts measure whole-command instructions under Callgrind for an uncached `ruff check` with one worker. | Workload | Main | With this change | Reduction | | --- | ---: | ---: | ---: | | [JAX docs](https://github.com/jax-ml/jax/tree/55029b8f7e65adb79764103871ec0357fa440227/docs) | 747,698,659 | 486,331,560 | 34.96% | | [Full ipywidgets](https://github.com/jupyter-widgets/ipywidgets/tree/8ada3542abe85fdc52e2bfeeecc07f8db351cb9c) | 275,380,491 | 255,782,455 | 7.12% | ## Test Plan The notebook tests and Clippy passed, and diagnostics and fixed notebook bytes matched main.
## Summary Each matching per-file ignore already contains a `RuleSet`. Ruff iterates those rules to build a new set, then iterates that set to disable each rule. Union the matching sets and subtract the result from the enabled and fixable rules. This helps configurations like attrs, which ignores `ALL` for its source stubs. Callgrind counts for an uncached `ruff check` on [attrs](https://github.com/python-attrs/attrs/tree/9d9f94a043de7fc4012459853a4ed597a5577855). | Uncached workload | Main instructions | With this change | Reduction | | --- | ---: | ---: | ---: | | All attrs source stubs | 26,453,751 | 25,103,368 | 5.10% | | Full attrs | 259,301,442 | 258,219,987 | 0.42% | ## Test Plan The linter and per-file-ignore CLI tests passed, and diagnostics and fixes matched main.
Summary -- This PR prepares the linter to emit diagnostics without files by moving the last output format from the linter into `ruff_db` to take full advantage of the `FileResolver` and `UnifiedFile` types, as well as the `DiagnosticStylesheet`. The first commit is mostly a move, with the minimal updates required to get things compiling. There's one small behavior change here in that we sort by the relative file path instead of the full path. This can apparently matter in cases where you invoke Ruff with a mix of paths within the working directory and from outside of it, such as: ```sh ruff check . /absolute/path/to/file.py ../another/file.py ``` but I think that's both pretty niche and not really breaking as it just changes the output order. The second commit cleans up the remaining `expect` calls so that diagnostics without files or without ranges can actually be handled. Diagnostics without files sort first and are rendered under a `<no filename>` heading, as you can see in the new tests. The third commit cleans up some now-unused code and a TODO in the junit code. I initially intended to re-unify the grouping code in junit with the grouped output format, but `grouped` needs a relative file path while `junit` needs the absolute path, and it's really not that much code anyway. Test Plan -- Existing tests ported to `ruff_db` and a new test for a missing file
## Summary On main, parsing `b"\x41"` calls `parse_unicode_literal(2)`. The x86-64 profiling build uses one function for widths 2, 4, and 8, so the loop bound is a runtime argument. For each digit (`4`, then `1`), the loop saves its source position and calls `next_char()` to decode and consume a UTF-8 character before checking whether it is a hex digit. This PR calls `parse_unicode_literal::<2>()` for `\x`, which lets the compiler specialize the loop for two digits. This is the largest measured improvement. The new loop also peeks at the next byte and advances by one when it sees a valid ASCII hex digit. Only if that check fails does it save the source position and call `next_char()`. For an invalid escape such as `"\x4é"`, both versions consume `é` and report its two-byte range. Callgrind instruction counts for a whole uncached `ruff check` on [JAX](https://github.com/jax-ml/jax/tree/55029b8f7e65adb79764103871ec0357fa440227/jax/_src/internal_test_util/export_back_compat_test_data). On the compatibility fixtures, changing only the width to `const N` saves 8.36 million of the 13.19 million instructions saved by the full change. | Workload | Main | `const N` only | `const N` + deferred error position | Full change | Total reduction | | --- | ---: | ---: | ---: | ---: | ---: | | JAX export compatibility fixtures | 131,434,592 | 123,077,887 | 120,580,490 | 118,244,820 | 10.04% | | JAX export implementation, compatibility fixtures, and export tests | 226,412,243 | 218,062,626 | 215,565,059 | 213,219,152 | 5.83% | ## Test Plan The parser tests passed, and diagnostics matched main on valid, malformed, and non-ASCII escapes.
## Summary PYI034 checks whether a class is a metaclass before it knows whether a method should return `Self`. That check can scan the entire class body for `__new__`. Move it to the point where PYI034 would emit the diagnostic. This avoids 2,371 metaclass checks in `pandas-stubs/core`. Callgrind counts for an uncached `ruff check` on [pandas-stubs](https://github.com/pandas-dev/pandas-stubs/tree/72db8965d09e0cbfc00a5f20603ef4c1559b77d8/pandas-stubs/core). | Uncached workload | Main instructions | With this change | Reduction | | --- | ---: | ---: | ---: | | pandas-stubs `core` | 334,791,285 | 317,214,869 | 5.25% | | Full pandas-stubs | 1,641,045,567 | 1,626,464,319 | 0.89% | ## Test Plan The flake8-pyi tests passed, and diagnostics and fixes matched main across Ruff’s linter fixtures.
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com> Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com> Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
…tral-sh#30376) ## Summary The parser preserves an unknown escape like `\q` in the decoded literal, including its backslash. If a successfully decoded string or bytes literal has no backslash, W605 can skip scanning its source. This avoids walking the full spelling of escaped binary data a second time. Callgrind counts for an uncached `ruff check` on [JAX](https://github.com/jax-ml/jax/tree/55029b8f7e65adb79764103871ec0357fa440227/jax/_src/internal_test_util/export_back_compat_test_data). | Uncached workload | Main instructions | With this change | Reduction | | --- | ---: | ---: | ---: | | JAX export compatibility fixtures | 131,472,981 | 123,959,397 | 5.71% | | JAX export implementation, compatibility fixtures, and export tests | 226,452,648 | 218,584,652 | 3.47% | ## Test Plan The linter tests passed, and diagnostics and fixes matched main on Ruff’s fixtures and Home Assistant.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update the RustPython fork from Ruff 0.17.1 to 0.17.2, using upstream commit 30516b4.
Ruff 0.17.2 makes parser stack growth opt-in. Downstream consumers that need the previous behavior must enable the parser's
stackerfeature explicitly.This is the baseline upgrade only. The remaining operand and Python 3.15 diagnostic changes in #7, and diagnostic-provenance additions, are not included.
Validation: parser library check and strict parser-only Clippy passed with
stackerenabled and the lockfile unchanged. The check reports the existing AST dependency warnings for disabledcache/salsaconfiguration; the parser passes-D warnings. Runtime and workspace-wide testing have not been performed.AI assistance: OpenAI Codex (model version unavailable) prepared the release integration, conflict resolutions and validation.