Repository navigation
"files" is wrong when installed without "wheel" #115
Description
In GitLab by @blueyed on Mar 17, 2020, 12:32
When "wheel" is not installed/used, installing a package will not contain "RECORD", and it falls back to reading "SOURCES.txt" (https://gitlab.com/python-devs/importlib_metadata/blob/3150ed4da9e1267d0787c6f4c1f8258a26a1dd93/importlib_metadata/__init__.py#L270), which then might result in paths not being locate()able, when a src-based setup is used:
setup(
...
packages=find_packages('src'),
package_dir={'': 'src'},
)
This is similar to https://gitlab.com/python-devs/importlib_metadata/-/issues/112, but in this case here it could use installed-files.txt from the egg-info.
Activity
jaraco commentedon Oct 22, 2020
In GitLab by @blueyed on Mar 17, 2020, 12:34
mentioned in commit blueyed/importlib_metadata@5d73789b0619d5ac53dfb2966902c13b24d31e2b
jaraco commentedon Oct 22, 2020
In GitLab by @blueyed on Mar 17, 2020, 12:34
mentioned in merge request !114
jaraco commentedon Oct 22, 2020
In GitLab by @jaraco on Mar 25, 2020, 15:55
Before adding support for
installed-files.txt, I'd like to know what command generates that file. It looks like it's generated in the legacy install.I'm leaning toward saying that importlib_metadata shouldn't support this use-case, that if someone wants to have metadata support, they should use the non-legacy features of the packaging ecosystem.
What is the impact of this issue? Are there real-world use-cases that experience issues as a result of this non-support?
jaraco commentedon Oct 22, 2020
In GitLab by @jaraco on Jun 5, 2020, 15:37
Bump.
jaraco commentedon Oct 22, 2020
In GitLab by @jaraco on Sep 23, 2020, 13:41
Happy to revisit when there's additional information available, but for now, I'm declaring "won't fix" without prejudice.
jherland commentedon Mar 9, 2023
I've recently run into this issue with the Qiskit (meta)-package:
qiskitpackage namespace).SOURCES.txtwhich is incorrect from a runtime POV: it references 3.pyfiles, asetup.pyand two files undertest/, none of which are actually installed.top_level.txtwhich is empty (contains a single newline). This is arguably correct given that it does not directly install any importable packages/modules.IMHO, there are two issues with how
importlib_metadatahandles this package:SOURCES.txtdoes not reflect the installed files,importlib_metadata.files("qiskit")ends up returning paths that don't actually exist on disk. According to setuptools docs,SOURCES.txtshould not be relied upon at runtime (although I understand that it is likely used as a last resort in this case).packages_distributions()returns nonsense for this package:_top_level_declared()does not differentiate between an emptytop_level.txtand a missingtop_level.txt, which triggers a fallback to_top_level_inferred(). This fallback ultimately ends up looking for*.pypaths inSOURCES.txt, which finds the three non-existing paths mentioned above. The end result is thatpackages_distributions()associates the import namessetupandtestwith this package, neither of which exist, or make any sense.Some possible ways to fix this:
installed-files.txtbefore falling back toSOURCES.txt. This file is written bypipon installation, and is a more accurate source thanSOURCES.txt. (Ifinstalled-files.txtdoes not exist for any reason, then fall back toSOURCES.txtas last resort.)SOURCES.txt, then at least make sure that only the paths inSOURCES.txtthat actually remain after installation are returned.top_level.txtdifferent from a missingtop_level.txt. In the above case the emptytop_level.txtcorrectly communicates that this packages does not expose any importable names. Having said that, there might be good reasons for the current behavior.jaraco commentedon Mar 9, 2023
I believe the issue may be rooted in how qiskit is installed.
The preferred way to install Python packages is as a wheel using dist-info metadata. As eggs are deprecated and superseded by wheels, support for egg-info is incomplete and best-effort.
Consider this example where I've installed qiskit from the source archive as a wheel and am able to avoid the undesirable behavior:
The key here was to pass
--use-pep517, which caused the source archive to build as a wheel and produce dist-info metadata. Other parameters topip install(via pip run) are there to force the source dist install and bypass other warnings.This approach utilizes the preferred code path for metadata and avoids the undesirable behavior.
Honestly, if it weren't for the fact that editable installs still rely heavily on egg-info metadata, I'd deprecate that format outright.
When I wrote
files()to resolve fromSOURCES.txt, I did that knowing it was an imperfect approximation for editable installs. I was unaware thatinstalled-fileswas also a thing :(.I'm not confident that
installed-files.txtis properly supported either. I see now there was some discussion some years ago in pypa/setuptools#371 aboutinstalled-files.txt. Ideally, I'd like to rally around supported standards and not add support for inferred interfaces.Does the
--use-pep517method of installation provide a suitable approach for your use-case?jherland commentedon Mar 9, 2023
Yes, installing it with dist-info metadata (either using
--use-pep517or by installing with modern pip in a venv that already haswheelavailable) does solve the issue (*.dist-info/RECORDis obviously more accurate than*.egg-info/SOURCES.txt).I 100% agree on using modern packaging practices. However, in this case, I'm not in the position to control the package installation process: I'm working on a tool to find undeclared and unused dependencies in the user's project, and as such I'm using
importlib_metadatainside the user's existing venv, and I do not control which packages are installed, or how they were installed.installed-files.txtseems to have been written by pip since version 0.3 (Jan 2009), and even if it has not been adopted by the rest of the packaging ecosystem, I think the current prevalence of pip makesinstalled-files.txtuseful as a better source thanSOURCES.txtwhen it exists. When it does not exist (e.g. it seems to not be created in the case of editable installs), falling back toSOURCES.txtis still available as a final - albeit imperfect - resort).Do you foresee problems with any of the three fixes I outlined above? In any case, I'm happy to whip up a PR.
jaraco commentedon Mar 9, 2023
All good points. Yes, I think I agree with the recommendation, and I'm delighted to hear you'd like to work on an implementation. Crucial will be to have tests that capture the relevant factors and expectations (including which scenarios apply to which logic branches). I'm slightly tempted to deprecate the use of SOURCES.txt altogether, but that probably should happen separately. Feel free to take a stab at it and let me know if you have any questions. I can be reached in Gitter (except I haven't enabled this project) and Discord.
FFY00 commentedon Mar 10, 2023
I recently touched this code and also think looking at
installed-files.txtwhen available is a good approach 👍Add tests for egg-info package with no installed modules
Distribution.files: Prefer *.egg-info/installed-files.txt to SOURCES.txt
Add tests for egg-info package with .files from inaccurate SOURCES.txt
Distribution.files: Only return files that actually exist
Add tests for egg-info package with no installed modules
Distribution.files: Prefer *.egg-info/installed-files.txt to SOURCES.txt
Add tests for egg-info package with .files from inaccurate SOURCES.txt
Distribution.files: Only return files that actually exist