Skip to content

Prefer item lookup when converting mappings to objects - #1266

Open
MbappeWU wants to merge 2 commits into
msgspec:mainfrom
MbappeWU:fix/mapping-method-named-fields
Open

MbappeWU wants to merge 2 commits into
msgspec:mainfrom
MbappeWU:fix/mapping-method-named-fields

Conversation

@MbappeWU

@MbappeWU MbappeWU commented Oct 13, 2026 •

Copy link
Copy Markdown

Fixes #1246

What changed

msgspec.convert reads fields from a non-dict mapping with __getitem__ first when from_attributes is false, and falls back to the attribute if the item lookup fails. Struct, dataclass, attrs, and tagged struct union targets share that lookup. A field named keys, items, values, or get receives the value stored under that key, and objects that expose fields only as attributes (such as SQLAlchemy Row) still convert as on main.

from_attributes=True still checks the attribute first, then the item.

Tests

tests/unit/test_convert.py covers those four targets. With from_attributes=True, conversion still raises ValidationError because the field value is a method. test_getitem_raises_falls_back_to_attributes covers an object whose __getitem__ raises while its fields are attributes, and the (GetAttrOrItemObj, False, False) case stays in the shared parametrize table.

uv run --only-dev just rebuild=1 test-all reported 6568 passed, 116 skipped for the unit suite and 7 passed for doctests. uv run --only-dev just check passed.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-13T14:21:20.961020Z 3850421 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Non-dict mappings were converted with getattr before __getitem__, so
fields named keys, items, values, or get received bound methods.
from_attributes=True still prefers attributes.

Fixes msgspec#1246.
@MbappeWU
MbappeWU force-pushed the fix/mapping-method-named-fields branch from 3850421 to 272b924 Compare October 16, 2026 01:14

@cig13zs cig13zs left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this (272b924) on Python 3.11 against main 75e1341. On main, #1246 reproduces for UserDict and sqlite3.Row (a keys column gives "Expected int, got builtin_function_or_method"). With this branch those convert correctly, and so do attrs targets, tagged struct unions, and fields named pop, setdefault, update, clear, copy, fromkeys. pytest tests/unit passes (6347 passed, 324 skipped).

One regression though: anything that has __getitem__ but exposes fields only as attributes no longer converts without from_attributes=True. SQLAlchemy 2.x Row is the common case, since row["id"] raises and the columns are attributes:

import sqlalchemy as sa, msgspec

class U(msgspec.Struct):
    id: int
    name: str

with sa.create_engine("sqlite://").connect() as c:
    row = c.execute(sa.text("select 1 as id, 'x' as name")).first()
    msgspec.convert(row, U)

main: U(id=1, name='x'). This branch: ValidationError: Object missing required field `id` (SQLAlchemy 2.1.4). Same for a plain class with __getitem__ plus attributes. The PR also drops the (GetAttrOrItemObj, False, False) row from the test table in test_convert.py, which is the test that covered this case.

Trying the item first and falling back to the attribute keeps both working:

static PyObject *
getitem_then_getattr(PyObject *obj, PyObject *key) {
    PyObject *out = PyObject_GetItem(obj, key);
    if (out == NULL) {
        PyErr_Clear();
        out = PyObject_GetAttr(obj, key);
    }
    return out;
}

with getter = getitem_then_getattr; in the from_attributes=False branch. With that, the unit suite still passes (6347 passed), including your new tests, the #1246 cases above convert, and the SQLAlchemy row converts again. The one thing it doesn't cover is a mapping that lacks a keys key while the target has an optional keys field: that still picks up the bound method, but that is what main does today. Maybe worth a test with an attribute-only __getitem__ object either way.

Objects with __getitem__ that expose fields only as attributes (for
example SQLAlchemy Row) converted on main but failed after the
previous commit. Try the item first, then the attribute.

Co-authored-by: Cursor <cursoragent@cursor.com>
@MbappeWU

Copy link
Copy Markdown
Author

Thanks for testing this so thoroughly, @cig13zs. You're right, dropping the attribute fallback broke objects like SQLAlchemy's Row, and removing that test row hid it.

Pushed 13022db:

  • from_attributes=False now uses your getitem_then_getattr: item first, then the attribute.
  • Restored the (GetAttrOrItemObj, False, False) row.
  • Added test_getitem_raises_falls_back_to_attributes, with a Row-like object whose __getitem__ raises and whose fields are attributes.

just rebuild=1 test-all: 6568 passed, 116 skipped; doctests 7 passed. just check passes.

Agreed on the remaining case (a mapping without a keys item and an optional keys field still picks up the bound method). That matches main, so I've left it out of this PR.

@cig13zs

cig13zs commented Oct 20, 2026

Copy link
Copy Markdown

Rechecked 13022db on Python 3.11 with SQLAlchemy 2.1.5. The Row case converts again (U(id=1, name='x')), UserDict and sqlite3.Row with keys and get columns still convert correctly, and from_attributes=True still picks the attribute. pytest tests/unit passes for me (6527 passed, 157 skipped). LGTM, and fine to leave the optional keys case for later since main does the same.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

convert: mapping fields named like mapping methods (keys, get, ...) resolve to the bound method

2 participants