Repository navigation
Closed
Labels
3.11only security fixesonly security fixesOS-macbuildThe build process and cross-buildThe build process and cross-buildtype-featureA feature request or enhancementA feature request or enhancement
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-11-07.16:37:24.260>
labels = ['OS-mac', 'type-feature', 'build', '3.11']
title = 'Cleanup and simplify setup.py'
updated_at = <Date 2021-11-09.08:56:25.176>
user = 'https://github.com/tiran'bugs.python.org fields:
activity = <Date 2021-11-09.08:56:25.176>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build', 'macOS']
creation = <Date 2021-11-07.16:37:24.260>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 45743
keywords = ['patch', 'patch']
message_count = 10.0
messages = ['405908', '405912', '405914', '405915', '405942', '405943', '405944', '405998', '406002', '406006']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'ronaldoussoren', 'christian.heimes', 'ned.deily', 'erlendaasland']
pr_nums = ['29456', '29456', '29457', '29464']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45743'
versions = ['Python 3.11']Metadata
Metadata
Assignees
Labels
3.11only security fixesonly security fixesOS-macbuildThe build process and cross-buildThe build process and cross-buildtype-featureA feature request or enhancementA feature request or enhancement
Activity
tiran commentedon Nov 7, 2021
Motivated by deprecation of distutils, I like to move more logic and checks from setup.py into configure.ac. Eventually I like to get rid of setup.py. The file contains a bunch of complicated checks and macOS-specific adjustments that I cannot verify on Linux.
socketmodule setup defines __APPLE_USE_RFC_3542 https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1229 Can we move the define into the __APPLE__ block of socketmodule.c?
-Wl,-search_paths_first linker arg, e.g. https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1616 Would it be safe to make the option default for all core extensions on $ac_sys_system = Darwin? We could add it to PY_CORE_LDFLAGS or add a new Makefile variable for core extensions.
detect_dbm_gdbm has about 200 lines of complicated code to detect old to ancient versions of Berkeley DB (libdb), https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1233 . Can I remove support for libdb-3 / libdb-4 and only support libdb-5 in standard locations? libdb-5.3 has been around for over a decade.
Note: libdb, gdbm, ndbm, and gdbm-compat don't provide pkg-config .pc files. We cannot use pkg-config to detect them.
sqlite's setup https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1531 does extra work to search for header and library files in non-standard locations. Can we replace the code with pkg-config checks in configure.ac with a fallback to AC_CHECK_LIB() and AC_CHECK_HEADERS()?
zlib's setup code https://github.com/python/cpython/blob/v3.11.0a2/setup.py#L1661 has a check for a CVE from 2002 (!). I think we can safely assume that everybody has upgraded to a fixed version. The check is mixed with macOS specific code.
ned-deily commentedon Nov 7, 2021
It seems to me that some (most?) of the macOS-specific workarounds in setup.py were added to make it easy to build with the system-provided copies of the third-party libraries, like zlib and sqlite3 and openssl. At least one of the workarounds, search_paths_first, became the default ld behavior since around macOS 10.6 (Xcode 4). So when building on more recent macOS systems, either the workarounds aren't necessary any more or the third-party library is no longer provided (like openssl) or is otherwise too old (like Tk) so that you can no longer rely on building with just using system-supplied libraries to have a useful Python on macOS. And, if you are building on very old systems (because of hardware requirements or whatever), in most cases, the versions of the system-supplied libraries are so old that you shouldn't be relying on them anyway. So I think that, for 3.11, we could take a stand and say that we no longer support building with specific system-supplied third-party libraries on macOS systems older than, say, macOS 10.9; IOW, you will need to supply your own copies of those libs on older systems. That's essentially the approach the MacPorts project has taken for years; they still support providing current Pythons on old versions of macOS but they also supply current version of the third-party libs, too.
And Christian's work here for 3.11 to move away from setup.py and use pkg-config to remove the guesswork on header and lib locations should make life easier for everyone. There is a small issue here in that macOS does not supply a built-in pkg-config but there are implementations available, if you are unable or unwilling to use open source distributors like MacPorts or Homebrew, including an actively-maintained permissive-licensed version with no build dependencies that seems to work on even macOS 10.6 (I didn't try anything older than that!): https://github.com/pkgconf/pkgconf.
Ronald, what do you think? If this sounds reasonable, we could draw up a list of libs and document it somewhere in the 3.11 code.
tiran commentedon Nov 7, 2021
Thanks for the detailed explanation, Ned. Much appreciated! For non-Mac users: macOS 10.6 was released in 2009 and 10.9 in 2013. IMHO it is reasonable to ask people to provide their own copies of libraries on older system. We also require OpenSSL 1.1.1, which is not available on older systems like CentOS 7 (released 2014).
Regarding pkg-config, I like to tidy up setup.py first to get a better understanding what kind of builds and systems I need to support. Thanks for pointing out https://github.com/pkgconf/pkgconf! I keep it in mind.
tiran commentedon Nov 7, 2021
For reference: https://opensource.apple.com/source/ld64/ld64-242/doc/man/man1/ld.1.auto.html
According to Wikipedia, Xcode4 was released in 2011.
tiran commentedon Nov 8, 2021
New changeset 24af9a4 by Christian Heimes in branch 'main':
bpo-45743: Move __APPLE_USE_RFC_3542 into socketmodule.c (GH-29456)
24af9a4
ronaldoussoren commentedon Nov 8, 2021
__APPLE_USE_RFC_3542 should have been in socketmodule.c from the start, not sure why it was added in setup.py.
as you and Ned noticed -search_paths_first has been the default for a long while, we can just drop it and anyone building on ancient systems can add the flag to CFLAGs/LDFLAGS as needed. IIRC we added the flag to be able to build with local copies of libraries also shipped in the OS, in particular when using static libraries for those local copies.
I don't know about detect_dbm_gdbm, and don't use those libraries myself. It would be nice to be able to open system files created using dbopen etc, but on the other hand there is at least one bug report complaining about data corruption when using one o the dbm extensions linked with the system version of the library.
4/5) Fine by me, although I'm slightly worried about using pkg-config because the system doesn't ship that tool.
Something you don't mention is the logic dealing with SDK roots. I haven't checked yet if similar logic would be necessary in configure. With some luck it isn't, but that depends on what's supported by autoconf and the particular probes we want to use.
@ned: Not being able to use system versions of libraries is a bit annoying, but that's something you already have to do due to openssl. Maybe we should also try to clean up and refactor the build-installer.py script to do have a way to build/install those 3th-party dependencies.
erlend-aasland commentedon Nov 8, 2021
FTR, it was added by me in bpo-35569, #63725. I moved the check to setup.py because Ned made me do it :P #19526 (comment)
6 remaining items
tiran commentedon Jul 23, 2022
You should not close tickets that affects supported versions of Python without seeking consent from the ticket author first, especially when the ticket author is a triager or core dev. 3.11 is still affected.
kumaraditya303 commentedon Jul 23, 2022
Okay, but I did say that "feel free to reopen if required." and there was no activity in the last 6 months or so and setup.py is now removed.
tiran commentedon Jul 23, 2022
It's not removed from 3.11.
kumaraditya303 commentedon Jul 23, 2022
This issue is marked as
type-featureand we don't usually backport enhancements after beta, only bug fixes. Do you consider this as a bug? If yes would you mind labelling it as a bug?erlend-aasland commentedon Jul 25, 2022
With beta 5 around the corner and the release candidates appearing in near future, I believe it would be wise to just leave setup.py as it is for 3.11. What do you say, Christian?
tiran commentedon Jul 25, 2022
There is still some documentation work and checks to do for 3.11.
CAM-Gerlach commentedon Nov 3, 2022
@tiran Do you still intend to do additional work on this for 3.11, given
setup.pyis completely gone in 3.12?arhadthedev commentedon Jun 19, 2023
@erlend-aasland For issue archeology of the future, could you link to PRs that resolve these points or explain why they can be abandoned, please?
erlend-aasland commentedon Jun 19, 2023
@arhadthedev: see gh-90005
erlend-aasland commentedon Jun 19, 2023
@arhadthedev: for sqlite3: gh-89932