Use non-BOM encodings - #2370

* Use non-BOM encodings

The documentation of the used `PyUnicode_DecodeUTF16` states that not passing `*byteorder` or passing a 0 results in the first two bytes, if
they are the BOM (U+FEFF, zero-width no-break space), to be interpreted and skipped, which is incorrect when we convert a known "non BOM" string, which all strings from C# are.

(cherry picked from commit 195cde6)

* Initial 3.14 commit

(cherry picked from commit caac33d)

* Apply alignment fix

(cherry picked from commit e10d333)

* Disable problematic GC tests

(cherry picked from commit e976558)

* Set ht_token to NULL in Python 3.14

(cherry picked from commit 65af098)

* Workaround for blocked PyObject_GenericSetAttr in metatypes

Python 3.14 introduced a new assertion that prevents us from using
PyObject_GenericSetAttr directly in our meta type. To work around
this, we manipulate the type dict directly.

This workaround is a simplified variant of Cython's workaround from
cython/cython#6325.

The relevant Python change is in
python/cpython#118454

(cherry picked from commit 08550d0)

* Use PyThreadState_GetUnchecked on Python 3.13

(cherry picked from commit f3face0)

* Remove deprecated function call

(cherry picked from commit 8dfe408)

* Assign True instead of None to __clear_reentry_guard__

Not at all sure why this helps, but when assigning `None` instead, the
object is gone at the time of garbage collection.

(cherry picked from commit 8e0333d)

* Move tp_clear workaround to .NET

In Python 3.14, the objects __dict__ seems to already be half
deconstructed, leading to crashes during garbage collection.

Since gc in Python is single-threaded (I think :)), it should
be fine to have a single static for this. If that is not true,
we can always use a thread-local instead.

(cherry picked from commit 908e13b)

* Use non-BOM encodings (pythonnet#2370)

* Use non-BOM encodings

The documentation of the used `PyUnicode_DecodeUTF16` states that not passing `*byteorder` or passing a 0 results in the first two bytes, if
they are the BOM (U+FEFF, zero-width no-break space), to be interpreted and skipped, which is incorrect when we convert a known "non BOM" string, which all strings from C# are.

(cherry picked from commit 195cde6)

* Preserve SyntaxError source line in message on Python 3.12+

Python 3.12 eagerly normalizes the error indicator, so PyErr_Fetch now
hands us the SyntaxError instance (whose str() omits the offending source
line) instead of the raw args tuple (whose str() included it). Callers that
surface PythonException.Message for compile diagnostics therefore lost the
offending source text on 3.12+.

GetMessage now re-appends the SyntaxError 'text' attribute when present.
This is a no-op on <=3.11 (there the fetched value is a tuple without the
SyntaxError attributes) and only affects SyntaxError messages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Make embed tests compatible with Python 3.12+ behavior changes

Three CPython behavior changes surfaced as failures/crashes once the
overload-resolution crash was fixed, all on 3.12+:

- ClassManagerTests.BindsCorrectOverloadForClassName crashed the host with
  "Python memory allocator called without holding the GIL". TestClass2's
  Get(PyObject o) re-enters Python via ToPython() while MethodBinder has
  released the GIL (allow_threads) around the managed call. A managed
  callback that re-enters Python must re-acquire the GIL; tolerated on
  <=3.11, fatal on 3.12+. Wrap the body in using (Py.GIL()).

- TestGetsPythonCodeInfoInStackTrace[ForNestedInterop]: 3.12+ adds caret
  indicator lines (e.g. "~~~^^^") under source lines in tracebacks, shifting
  the positional assertions. Drop caret-only lines before asserting
  (no-op on <=3.11).

- Codecs.ExceptionDecodedNoInstance: 3.12 eagerly normalizes exceptions, so
  the error indicator always carries an instance ("value"); the instanceless
  scenario this decoder targets can no longer be produced. Guard the test to
  <3.12.

Verified: full embed suite green on 3.11 (910/910) with these changes; the
three previously-failing tests pass on 3.14.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Add Python 3.12 / 3.13 ABI offsets and CI jobs

The fork resolves PyTypeObject field offsets from the hardcoded
TypeOffset{major}{minor} tables (it does not run geninterop at build),
so a missing table makes ABI.Initialize throw "Python ABI v... is not
supported" and every test on that version fails at PythonEngine init.
Only 3.6-3.11 and 3.14 tables were present.

Vendor the 3.12 and 3.13 tables from pythonnet/pythonnet upstream
(byte-identical to upstream master; same source as the already-present
TypeOffset314) and add 3.12 + 3.13 to the CI matrix.

Local verification (uv standalone CPython 3.13, this branch's fixes):
embed suite ABI-initializes correctly and runs 847 passed / 0 failed
(parity with 3.14). 3.12 table is vendored from the same authoritative
source; CI exercises it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Support Python 3.13/3.14 re-init in tests; drop obsolete TestDomainReload

The embed-test suite re-initializes the interpreter per fixture. On CPython
3.13/3.14 the suite aborted with "Failed to import encodings module" - not a
filesystem problem (strace shows the file opens fine) but interpreter import
state corrupted across re-initialization.

Root cause isolated to a single test: TestPythonEngineProperties.SetPythonPath.
It uses PythonEngine.PythonPath, which pins a fixed module search path via the
deprecated Py_SetPath. CPython 3.13+ keeps that path config in _PyRuntime across
Py_Finalize and offers no way to reset it back to auto-computation without the
PyConfig API, so once this test runs every later re-initialization in the same
process is forced onto the pinned path and eventually cannot bootstrap encodings.
All other fixtures - including the normal Initialize/Shutdown cycles in
pyinitialize and TestFinalizer - run fine in a single process.

Run only SetPythonPath in its own test process so it cannot pollute the rest of
the suite. Verified locally: full embed suite green on 3.11, 3.13 and 3.14
(main run 908 / SetPythonPath 1, 0 failures); no regression.

Also delete TestDomainReload: AppDomain reload is not supported on modern .NET
(single-domain), so those tests (MarshalByRefObject / AppDomain.CreateDomain)
are obsolete.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Benedikt Reinartz <filmor@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>