Switch `configobj` from bundled dependency to PyPI package by Andrej730 · Pull Request #20172 · astropy/astropy

Andrej730 · 2026-07-30T19:10:30Z

Description

Hi, noticed Python 2 workaround still present on the repo and wanted to help to get rid of them.

astropy/astropy/extern/configobj/validate.py

Lines 167 to 185 in 55da6b4

    
           from pprint import pprint 
        
           #TODO - #21 - six is part of the repo now, but we didn't switch over to it here 
        
           # this could be replaced if six is used for compatibility, or there are no 
        
           # more assertions about items being a string 
        
           if sys.version_info < (3,): 
        
               string_type = basestring 
        
           else: 
        
               string_type = str 
        
               # so tests that care about unicode on 2.x can specify unicode, and the same 
        
               # tests when run on 3.x won't complain about a undefined name "unicode" 
        
               # since all strings are unicode on 3.x we just want to pass it through 
        
               # unchanged 
        
               unicode = lambda x: x 
        
               # in python 3, all ints are equivalent to python 2 longs, and they'll 
        
               # never show "L" in the repr 
        
               long = int 
        
           _list_arg = re.compile(r'''

Checked the code upstream and it's already resolved there.

https://github.com/DiffSK/configobj/blob/b7707c94c0317b8f89b704a01525a68eb6c72521/src/configobj/validate.py#L164-L169

import re
import sys
from pprint import pprint

_list_arg = re.compile(r'''

So submitting in this PR the latest package for configobj (https://pypi.org/project/configobj/5.0.9/, nvm typo in branch name)) without any Python 2 symbols involved. configobj restructured code a little bit, so in the future the won't global validate module available so some files were moved around / renamed.

I've made two changes that deviate from the actual published package:

I've dropped the whitespaces that were cleaned up on this repo in ecaa91c just to keep diff minimal and more readable. This whitespace undo in a separate commit 1f8820d. Maybe makes sense to undo this commit to keep diff with upstream absolutely minimal, but upstream is not very active, so it's up to you - what do you prefer?
I've replaced from configobj.validate import VdtMissingValue with relative paths from ..validate import VdtMissingValue, similar patch already exists on this repo (e5ce1e3) and without it from astropy.extern.configobj import configobj, validate was failing with ModuleNotFoundError: No module named 'configobj' since it was assuming configobj to be available as a package globally. I've added ASTROPY PATCH comments to make to make these patches nature easily recognizable.

By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

github-actions · 2026-07-30T19:10:43Z

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

Do the proposed changes actually accomplish desired goals?
Do the proposed changes follow the Astropy coding guidelines?
Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

pllim

Sorry. We do accept patches to extern. Packages in that directory are bundled as-is.

Andrej730 · 2026-07-30T19:16:28Z

Sorry. We do accept patches to extern. Packages in that directory are bundled as-is.

@pllim But I just bumped bundled version of configobj, code comes completely from upstream. It's not acceptable too?

pllim · 2026-07-30T20:08:31Z

OK looks like at some point things got moved around and upgrade became non-trivial. Looks like you attempted to account for that, so I will re-open this for review but I cannot promise when someone can get to it. Thanks.

pllim · 2026-07-30T20:09:55Z


 from codecs import BOM_UTF8, BOM_UTF16, BOM_UTF16_BE, BOM_UTF16_LE

+from ._version import __version__


Why is this needed though. It is not used.

It's just a convention to quickly retrieve package version. It's not really needed, just kept everything exactly as it is upstream, so the next person bumping dependency wouldn't need to think about whether to keep this diff or not.

>>> import configobj >>> configobj.__version__ '5.0.9'

pllim

I wonder if it would be easier on us in this repo if we undo the renaming stuff from upstream and keep the original filenames here. Then add a note about that for the next upgrade in the future.

pllim · 2026-07-30T20:12:10Z

Would need a change log under "other" as well.

Andrej730 · 2026-07-30T20:38:13Z

I wonder if it would be easier on us in this repo if we undo the renaming stuff from upstream and keep the original filenames here. Then add a note about that for the next upgrade in the future.

From the current packaging mechanism it seems to bundle everything it can find in astropy folder, so there's no concern here at least.

astropy/pyproject.toml

Lines 213 to 216 in c586ab9

    
           [tool.setuptools.packages.find] 
        
           include = ["astropy*"] 
        
           exclude = ["astropy._dev*"] 
        
           namespaces = true

But it's really up to you, whatever seems more convenient from your perspective.

From my point of view, ideally, is to keep differences with upstream minimal, so bumping bundled dependency can be -

removing astropy/extern/configobj contents completely
pasting bumped ones
adding empty __init__.py to keep astropy.extern.configobj recognizable as a package
updating absolute imports to relative ones
quickly look through diff after bump if anything is concerning appeared
the bumping process documented in README.md in astropy/extern/configobj or astropy/extern

So any difference from upstream requires the next person maintaining this to reason it through - whether moving files is okay or it should be undone, they will also need to clean up whitespace changes each time, etc.

But this dependency had 70 LOC updated in 12 years, so any solution will work 😄

Would need a change log under "other" as well.

Added it.

pllim · 2026-07-30T20:56:12Z

@astrofrog or @neutrinoceros , any preference on how we want to bundle this going forward?

Now that I dig more, we seem to have patched some important stuff over the years without renaming files. 🤷‍♀️

I will wait to run CI until we figure out the file organization.

Thanks!

neutrinoceros

From my point of view, ideally, is to keep differences with upstream minimal

strong +1
if we can, we shouldn't rename anything. Though I reckon I don't know the history here, does anyone know what we did so far and why ?

neutrinoceros · 2026-07-31T10:38:24Z

Not sure porting this file as is makes sense.

import validate will not work

we don't need a backward compatible layer since this module is only meant for internal use

But then removing this shim means we also need to skip rename and keep validate.py as astropy/extern/configobj/validate.py, so it's an extra thing to remember when bumping this dependency in the future. Any harm in keeping the shim as-is?

You're right, better keep everything close to source

astrofrog · 2026-08-02T06:57:35Z

Can anyone remember why we are even bundling this in the first place? Does this date back to when we could only have numpy as a dependency? Could we just not bundle it anymore?

neutrinoceros · 2026-08-02T07:11:56Z

Good point. I'd be in favour of just depending on the original third party package and found no argument against it: it has (pure Python) wheels on PyPI and is available through conda-forge too.

astrofrog · 2026-08-02T08:07:06Z

Can someone check the commit history and maybe issue/PR history for why it was bundled, to double check I am not missing something?

Andrej730 · 2026-08-02T17:50:48Z

Ha, it's been a while since 6560e0c

Don't know the reasoning, but there is this sentence in the code guide:

astropy/docs/development/codeguide.rst

Lines 20 to 23 in 02f3328

    
           * The core package should be importable with no 
        
             dependencies other than components already in the Astropy core, the 
        
             `Python Standard Library <https://docs.python.org/3/library/index.html>`_, 
        
             and |NumPy| |minimum_numpy_version| or later.

But it seems to be stale now, since repo has pyyaml, pyerfa and packaging. So, time to move on from it?

astropy/pyproject.toml

Lines 45 to 51 in 02f3328

    
           dependencies = [ 
        
               "astropy-iers-data>=0.2026.7.27.0.56.29", 
        
               "numpy>=2.0", 
        
               "packaging>=25.0", 
        
               "pyerfa>=2.0.1.3",  # for >=2.0.1.7, adjust structured_units.rst and doctest-requires 
        
               "PyYAML>=6.0.0", 
        
           ]

A heads up - another bundled dependency, ply, is now archived (see https://github.com/dabeaz/ply), so it will make sense to keep it bundled (as author also advises) to allow unlimited patching per astropy needs.

astrofrog · 2026-08-02T19:03:34Z

Thanks! If configobj is a well maintained dependency, I don't see any reason to not just have it as a regular dependency.

pllim · 2026-08-03T18:58:43Z

I have an old note somewhere that said it was unmaintained. Or maybe I was just confused... or someone took it up?

pllim · 2026-08-03T19:07:40Z

@eteq why did you bundle configobj 15 years ago in #73 , do you remember? 😆

neutrinoceros · 2026-08-03T19:49:56Z

 from warnings import warn

+import configobj
+import configobj as validate


I'd rather we just use a single alias for every namespace.

ahh, that was a typo, sorry, addressed in 6b03ca3

neutrinoceros · 2026-08-03T19:51:45Z

+            "pyerfa>=2.0.1.3",
+            "configobj>=5.0.9"]


I don't understand why this would be needed. Could you show me where is configobj used at build time ?

Sure, it's needed because of install_stubs in setup.py, which triggers astropy import and then astropy.config.configuration import.

astropy/setup.py

Lines 19 to 27 in 345a338

def install_stubs(build_lib, output_dir):

sys.path.insert(0, build_lib)

try:

from astropy.units.typing_utils import create_stubs # noqa: PLC0415

create_stubs(Path(output_dir))

finally:

# Undo the path modification.

sys.path.pop(0)

Here's the traceback when trying to uv pip install -e . without configobj as a build dependency (when running uv build traceback is similar)

File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/build_meta.py", line 410, in _build_with_temp_dir self.run_setup() ~~~~~~~~~~~~~~^^ File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/build_meta.py", line 317, in run_setup exec(code, locals()) ~~~~^^^^^^^^^^^^^^^^ File "<string>", line 52, in <module> File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/__init__.py", line 117, in setup return distutils.core.setup(**attrs) # type: ignore[return-value] ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/_distutils/core.py", line 186, in setup return run_commands(dist) File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/_distutils/core.py", line 202, in run_commands dist.run_commands() ~~~~~~~~~~~~~~~~~^^ File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/_distutils/dist.py", line 1001, in run_commands self.run_command(cmd) ~~~~~~~~~~~~~~~~^^^^^ File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/dist.py", line 1107, in run_command super().run_command(command) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^ File "~/.cache/uv/builds-v0/.tmpnRSaPV/lib/python3.14/site-packages/setuptools/_distutils/dist.py", line 1020, in run_command cmd_obj.run() ~~~~~~~~~~~^^ File "<string>", line 43, in run File "<string>", line 22, in install_stubs File "~/astropy/astropy/__init__.py", line 64, in <module> from . import config as _config File "~/astropy/astropy/__init__.py", line 55, in __getattr__ return import_module("astropy." + attr) File "~/.local/share/uv/python/cpython-3.14.4-linux-x86_64-gnu/lib/python3.14/importlib/__init__.py", line 88, in import_module return _bootstrap._gcd_import(name[level:], package, level) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "~/astropy/astropy/config/__init__.py", line 8, in <module> from .configuration import * File "~/astropy/astropy/config/configuration.py", line 31, in <module> import configobj.validate ModuleNotFoundError: No module named 'configobj'

Makes sense. I forgot these stubs were so involved to generate... well, that's more dependencies to trim for me, but clearly out of scope for this PR. Thanks !

In 84b08f9 found that one of the CIs is building astropy manually, so it will need configobj too to succeed.

neutrinoceros · 2026-08-03T21:24:15Z

 from warnings import warn

+import configobj
+import configobj.validate as validate


Non-aliasing as imports tend to be used, by convention, to indicate an intentional re-export, which I don't think we want here

Suggested change

import configobj.validate as validate

from configobj import validate

Addressed in c6f44cd

On Ubuntu 24.04 configobj is available as 5.0.8, we have 5.0.9 pinned. There's no critical difference between 5.0.8 and 5.0.9, but 5.0.9 is already almost 2 years old, so it's probably the better one to pin.

…sult

Andrej730 · 2026-08-04T06:10:49Z

Ah, tried to cherry-pick lowest-resolved-tree.txt changes to resolve merge conflicts, but github still detects one. I guess it's okay, since it's one-click resolvable.

neutrinoceros · 2026-08-04T06:38:44Z

The way to fix it is to checkout the file from main then run the update script (uv run scripts/check-lowest-resolved-tree.py --overwrite)

Andrej730 · 2026-08-06T21:35:50Z

Should be complete now, remaining CI failures are unrelated.

neutrinoceros · 2026-08-07T14:17:18Z

            "extension-helpers>=1.4,<2",
-            "pyerfa>=2.0.1.3"]
+            "pyerfa>=2.0.1.3",
+            "configobj>=5.0.9"]


Nit: this makes it so that next time we need to add something there, we won't loose the ability to use git blame for this one

Suggested change

"configobj>=5.0.9"]

"configobj>=5.0.9",

]

Applied in a0428c8

Co-authored-by: Clément Robert <cr52@protonmail.com>

pllim · 2026-08-10T16:43:01Z

Are we sure we don't want to wait till v9.0 for such things? This has real downstream impact:

use configobj instead of astropy.extern.configobj spacetelescope/stpipe#358

neutrinoceros · 2026-08-10T18:49:33Z

AFAIC I think we better bite the bullet in the next feature release. We could easily add a backwards compatibility layer in the form of a (deprecated) re-export if necessary

github-actions Bot added config external PRs and issues related to external packages vendored with Astropy (astropy.extern) labels Jul 30, 2026

pllim added this to the v8.1.0 milestone Jul 30, 2026

pllim requested changes Jul 30, 2026

View reviewed changes

pllim closed this Jul 30, 2026

This comment was marked as outdated.

Sign in to view

Andrej730 changed the title ~~Configobj 5.0.8 update (drop Python 2 workarounds)~~ Configobj 5.0.9 update (drop Python 2 workarounds) Jul 30, 2026

pllim reopened this Jul 30, 2026

pllim reviewed Jul 30, 2026

View reviewed changes

Andrej730 added 3 commits July 31, 2026 01:27

configobj 5.0.9

baa1483

Undo whitespace changes

66905d3

Patch imports to be relative

745b7a6

Andrej730 force-pushed the configobj-5.0.8-update branch from 7395348 to 745b7a6 Compare July 30, 2026 20:28

Add changelog for configobj 5.0.9 update

b8a1ca8

neutrinoceros reviewed Jul 31, 2026

View reviewed changes

Andrej730 changed the title ~~Switch configobj from bundled dependency to PyPI package~~ Switch configobj from bundled dependency to PyPI package Aug 3, 2026

neutrinoceros reviewed Aug 3, 2026

View reviewed changes

Andrej730 added 2 commits August 4, 2026 02:06

Fix typo in configobj.validate import

6b03ca3

Add configobj to CI that buildsastropy manually

84b08f9

neutrinoceros added Extra CI Run cron CI as part of PR dependencies Pull requests that update a dependency file build Build all wheels Run all the wheel builds rather than just a selection labels Aug 3, 2026

neutrinoceros reviewed Aug 3, 2026

View reviewed changes

Andrej730 and others added 6 commits August 4, 2026 10:51

Don't use aliasing during import

c6f44cd

lowest-resolved-tree: add configobj

307ec77

ci_cron_monthly: add configobj needed during installation

f8a6f93

On Ubuntu 24.04 configobj is available as 5.0.8, we have 5.0.9 pinned. There's no critical difference between 5.0.8 and 5.0.9, but 5.0.9 is already almost 2 years old, so it's probably the better one to pin.

TST: Add back pytest-filter-subpackage

44bc1dd

MNT: upgrade check-lowest-resolved-tree.py's pinned uv version and re…

6be7a7b

…sult

Update minimum required version of astropy-iers-data

018232e

Andrej730 added 2 commits August 4, 2026 11:41

Merge branch 'main' into configobj-5.0.8-update

aa78941

lowest-resolved-tree: update after pulling changes from main

6f24fdc

neutrinoceros reviewed Aug 7, 2026

View reviewed changes

Comment thread docs/changes/20172.other.rst Outdated

neutrinoceros reviewed Aug 7, 2026

View reviewed changes

Andrej730 and others added 2 commits August 7, 2026 21:43

Update docs/changes/20172.other.rst

abb8942

Co-authored-by: Clément Robert <cr52@protonmail.com>

pyproject: make configobj dependency more diffable

a0428c8

neutrinoceros approved these changes Aug 7, 2026

View reviewed changes

braingram mentioned this pull request Aug 10, 2026

Stop using astropy configobj spacetelescope/stpipe#357

Closed

pllim requested a review from astrofrog August 10, 2026 16:43

	from pprint import pprint

	#TODO - #21 - six is part of the repo now, but we didn't switch over to it here
	# this could be replaced if six is used for compatibility, or there are no
	# more assertions about items being a string
	if sys.version_info < (3,):
	string_type = basestring
	else:
	string_type = str
	# so tests that care about unicode on 2.x can specify unicode, and the same
	# tests when run on 3.x won't complain about a undefined name "unicode"
	# since all strings are unicode on 3.x we just want to pass it through
	# unchanged
	unicode = lambda x: x
	# in python 3, all ints are equivalent to python 2 longs, and they'll
	# never show "L" in the repr
	long = int

	_list_arg = re.compile(r'''


		from codecs import BOM_UTF8, BOM_UTF16, BOM_UTF16_BE, BOM_UTF16_LE

		from ._version import __version__

	def install_stubs(build_lib, output_dir):
	sys.path.insert(0, build_lib)
	try:
	from astropy.units.typing_utils import create_stubs # noqa: PLC0415

	create_stubs(Path(output_dir))
	finally:
	# Undo the path modification.
	sys.path.pop(0)

	import configobj.validate as validate
	from configobj import validate

Uh oh!

Uh oh!

Conversation

Andrej730 commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

Andrej730 commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

pllim commented Jul 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

pllim commented Jul 30, 2026

Uh oh!

Andrej730 commented Jul 30, 2026

Uh oh!

pllim commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

astrofrog commented Aug 2, 2026

Uh oh!

neutrinoceros commented Aug 2, 2026

Uh oh!

astrofrog commented Aug 2, 2026

Uh oh!

Andrej730 commented Aug 2, 2026

Uh oh!

astrofrog commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pllim commented Aug 3, 2026

Uh oh!

pllim commented Aug 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andrej730 Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andrej730 commented Aug 4, 2026

Uh oh!

neutrinoceros commented Aug 4, 2026

Uh oh!

Andrej730 commented Aug 6, 2026

Andrej730 commented Jul 30, 2026 •

edited

Loading

Andrej730 commented Jul 30, 2026 •

edited

Loading

pllim commented Jul 30, 2026 •

edited

Loading

astrofrog commented Aug 2, 2026 •

edited

Loading

Andrej730 Aug 3, 2026 •

edited

Loading