Issues instantiating/running models from flows #718

Activity

1. PGijsbers commented on Jun 25, 2019

   

   PGijsbers

   on Jun 25, 2019

   Collaborator

   More actions

   Sorry for the delay.
   To reproduce the issue, please let us know which flow and/or setup you are working with.

   That said, running a flow on a task is easiest done in the following way (e.g. task 3 and flow 7684):

   import openml
   
   task = openml.tasks.get_task(3)
   flow = openml.flows.get_flow(7684, reinstantiate=True)  # Necessary otherwise no model is made.
   
   run = openml.runs.run_flow_on_task(flow, task)

   Note the one line in the middle, where the model is reinstantiated after downloading the flow. I think it's fair to say that we should check for this automatically in the run_flow_on_task method.
2. hp2500 commented on Jun 25, 2019

   

   hp2500

   on Jun 25, 2019

   Author

   More actions

   Hi, thanks a lot for your reply! I have tried your code, but it seems to require an older sklearn version.

   

   I am using sklearn version 0.21.2, because I am interested in some functions that were added only recently. Do you think there is a workaround?

   P.S. I originally used flow #6970, but I don't think it matters because I am getting the same errors for different flow ids.

   @amueller
3. hp2500 commented on Jun 25, 2019

   

   hp2500

   on Jun 25, 2019

   Last edited by hp2500

   Author

   More actions

   But maybe, the required sklearn version depends on which version was used in the original flow... When I try a different flow, the error mentions a different sklearn version.

   

   @amueller
4. PGijsbers commented on Jun 25, 2019

   

   PGijsbers

   on Jun 25, 2019

   Last edited by PGijsbers

   Collaborator

   More actions

   Yes, that is the case. The flows require a specific scikit-learn version to ensure reproducibility. I don't believe there currently is a way to automatically create a new flow for your scikit-learn version based on a flow-id and your current environment (or find one on the server that matches). I can understand this would be a desirable feature, which for pure-scikit-learn based flows should be feasible.

   A little disclaimer, I was not involved at the time this flow (6970) was created or the related study (study14) was done. With that said, it looks like the flow you are trying to deserialize (6970) in particular also relies on non-scikit-learn components (openmlstudy14.preprocessing.ConditionalImputer). I don't currently know the status of reproducing that scenario, perhaps @mfeurer knows. I image you need to make sure the original source code can be found (https://github.com/openml/Study-14), which @janvanrijn should be more familiar with.

   Sorry to not be of more use.
5. amueller commented on Jun 27, 2019

   

   amueller

   on Jun 27, 2019

   Contributor

   More actions

   @PGijsbers Thanks for all the replies! It would be great to do the instantiation automatically for run_flow_on_task. I couldn't find that in the documentation and the error message is not very helpful. Maybe either you or @hp2500 can just add this to the error message for now.

   I don't think instantiating Study-14 pipelines should be a high priority (there is no setup.py). We were actually more interested in getting the parameters of the last pipeline step. To do that, we could in principle just use mocking to replace the ContitionalImputer just so we can call pipeline[-1].get_params(). But maybe that's to specialized.

   More generally I think it would be good to optionally disable the version check if the libraries are available.
6. PGijsbers commented on Jun 28, 2019

   

   PGijsbers

   on Jun 28, 2019

   Collaborator

   More actions

   I'll make the change to automatically reinstantiate the model when run_flow_on_task is called with a flow that has no instantiated model and all dependencies are satisfied (with the right version). I think we should involve @mfeurer on if/how/when we want to allow disabling version checks and how it should behave.
7. mfeurer commented on Jul 8, 2019

   

   mfeurer

   on Jul 8, 2019

   Collaborator

   More actions

   More generally I think it would be good to optionally disable the version check if the libraries are available.

   We could create different levels of checks for this:

   • that the library is available
   • that the major version is equal
   • that the minor version is equal
   • that the exact version is equal

   Would this together with #722 solve this issue?
8. mfeurer commented on Jul 9, 2019

   

   mfeurer

   on Jul 9, 2019

   Collaborator

   More actions

   Fixed via #724. Please reopen if that's not the case.
9. mfeurer

   closed this as completedon Jul 9, 2019
10. amueller commented on Jul 22, 2019

    

    amueller

    on Jul 22, 2019

    Contributor

    More actions

    import openml
    
    task = openml.tasks.get_task(3)
    flow = openml.flows.get_flow(7684, reinstantiate=True)  # Necessary otherwise no model is made.
    
    run = openml.runs.run_flow_on_task(flow, task)

    yields

    ValueError: Trying to deserialize a model with dependency sklearn==0.19.1 not satisfied.
11. amueller

    reopened this on Jul 22, 2019
12. mfeurer

    closed this as completedin #744on Sep 26, 2019