Repository navigation
refuses to pip install openml #727
Description
Description
Installing openml like the guide says using pip install openml fails deliberatly, and says pip install -e . (i.e. on a local checkout) should be used:
ERROR: Please install this package with pip: `pip install -e .` Installation requires pip>=10.0.
Apparently this was added to setup.py to prevent setuptools building numpy: https://github.com/openml/openml-python/blob/develop/setup.py#L9..L14
But since pip is able to pull binary numpy from wheels, I suspect this test should not fire if setup.py is called through pip.
Versions
pip 19.1.1
openml on pypi is currently 0.9.0
Metadata
Metadata
Assignees
Labels
No labels
Activity
mfeurer commentedon Jul 9, 2019
Thanks for reporting this. We'll have a look into this.
PGijsbers commentedon Jul 12, 2019
I can not reproduce the issue locally. Does the full output of the
pip install openmlcommand provide more information (and if so, could you provide the full output)?azrdev commentedon Jul 13, 2019
@PGijsbers full stdout/stderr below. As I wrote, it prints that warning from
setup.py.PGijsbers commentedon Jul 15, 2019
Thanks @azrdev. Of course I did not doubt you were speaking the truth, the reason I asked for the output is that we can see if there is other identifying information which allows us to spot if
setup.pyhas been called frompipor not. Sorry if you had the wrong impression.In this case, we see
setup.pygets called with quite a few additional arguments if it is called frompip, in particular a--recordoption to a temporary file with 'pip' in its name. @mfeurer what about specifically checking for this condition to detect usage ofpip?mfeurer commentedon Jul 15, 2019
Hey, could it be that a missing wheel is the reason why
python setup.py installinstalls packages by compiling the source? I just created an empty conda environment and install openml withpython setup.py installand it took ~1 minute and didn't compile anything.The log:
If you guys agree, I suggest to replace the check to see if the wheel package is installed, and if not, refuse installation.
PGijsbers commentedon Jul 17, 2019
Sorry I'm confused, why did
not error out? Did you change to setup script to ignore it to see if it would fail(/build numpy)?
But I guess it makes sense that we could check for
wheel.mfeurer commentedon Jul 17, 2019
I commented it out to see what happens. Sorry for not mentioning that.
PGijsbers commentedon Jul 17, 2019
Ok, cool. Then it seems like it makes the most sense to just check for the
wheelpackage indeed. I would be fine with that solution.pganssle commentedon Jul 21, 2019
I think you may want to drop this requirement entirely, or replace it with a warning of some sort, plus strong documentation to not invoke
setup.py install. We over atsetuptoolsare trying to deprecatesetup.py installand other direct invocations ofsetup.py, but it's been difficult for reasons like this - things that are doing the right thing but are implemented in terms ofsetup.py install(which is fine).I was originally thinking that adding a
pyproject.tomlwith abuild-systemsection (for PEP 517 support) might solve your problem somewhat, but I think it won't, particularly if you are using editable installs (I still think it's a good idea, just I can't think of a way to salvage the "error out when someone invokedsetup.py installdirectly" behavior).mfeurer commentedon Jul 22, 2019
Thanks for your comments @pganssle
The original reason for prohibiting the invocation of
setup.py installwas to make sure that numpy, scipy and scikit-learn aren't compiled. This is most likely the product of a non-conda environment. I'm afraid that a warning won't do because an installation withsetup.py installproduces a long log in which the warning won't be easily visible.Potentially we should ask for confirmation by making the user type a
yes?In any case, we should add a link to the docs to the error message.
pganssle commentedon Jul 22, 2019
@mfeurer The problem is that I don't think it's possible to do what you want to do, because there's no clear way to distinguish between direct and indirect invocations of
setup.py install. Additionally, evenpip install openmlwill end up buildingnumpyfrom source if it's not already installed and no wheel is available for the platform the person is trying to build upon.The problem is that as long as you're not installing from a wheel (or installing in develop mode),
setup.py installwill be invoked.See:
This will always fail. Using
pip install -eis just sort of accidentally getting around the problem because it's invokingsetup.py develop, which is probably more problematic thansetup.py install!I think you should probably just give up on trying to stop your users from doing the wrong thing by accident, because you are likely to cause problems for people actually doing the right thing in the first place.
This will almost certainly fail, because
pip installmay eat any outputs tosys.stdout, and will probably hard-fail on any inputs.pip installis not usually an interactive command.Like I said, what you are trying to do here is admirable, but if it were possible, we would have already done it in
setuptools. The best thing you can do is to document that this is installed withpip installand notsetup.py install, and that in some situations installation withcondawill provide a smoother user experience.mfeurer commentedon Jul 22, 2019
Thank you for your help and explanations @pganssle. What do you think of the solution of scikit-learn which checks itself that the libraries in question are installed?
Anyway, I guess we'll better remove these lines again as they appear to cause more trouble than good.
pganssle commentedon Jul 22, 2019
@mfeurer I actually recommend removing as much logic as possible from your
setup.py. Unless you need those libraries installed at build time (in which case they should go inpyproject.toml), I would recommend against checking for them in yoursetup.py.Generally speaking, you should think of your
setup.pyas a script that takes your project and turns it into awheelto be installed, not as something that is to be executed in the target environment (assuming you can avoid it). This is useful because ideally your package (which in this case seems to be pure Python, so is very amenable to this) should be able to be "cross-compiled", which is to say that you can generate a universal (not python version or environment specific) wheel and publish that. In that casesetup.pyis only ever executed on the build machine.For a good overview of current python packaging, see Bernat Gabor's series of 3 articles on The State of Python Packaging.