CI runs on production server

It looks like something is wrong with the unit tests, because they upload runs to the production server: https://www.openml.org/r/8857778

Please don't :). Unless these experiments are actually useful?

@janvanrijn

@janvanrijn @mfeurer can you take a look at this? I'm still seeing lots of CI runs streaming in on production...

@ArlindKadra

@ArlindKadra could you please prioritize this?

@joaquinvanschoren

@joaquinvanschoren @mfeurer sure thing. I found out the problem.

Thanks!

@mfeurer

@mfeurer The problem is in the OpenML_Tutorial.ipynb at tests/test_examples/. We are uploading runs to the production server with Travis.

Can we let it submit to the test server? Or do we have to comment out the publish step in the tutorial?

On Wed, 7 Mar 2018 at 15:51 Arlind Kadra ***@***.***> wrote: @mfeurer <https://github.com/mfeurer> The problem is in the OpenML_Tutorial.ipynb at tests/test_examples/. We are uploading runs to the production server with Travis. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#392 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpQV2dYBxb0mA1B65jluzStR4BKxqvEks5tb_JPgaJpZM4R9YUu> .

-- Thank you, Joaquin

Yeah sure, we can go with the first option.

@ArlindKadra

After @ArlindKadra figured this out I had a brief look at how we could tackle this. All this is based on the premise that we have examples and tutorials in CI to make sure that they keep on working (which rules out the option to comment out code in the tutorial).

Work only on the test server. This appears to be pretty boring as there aren't a lot of results for tasks.
Only do the uploads to the test server, however, we would not get them if we download all previous results from the live server.
Split up the tutorial page into several smaller tutorials in which analysis of runs and upload of runs are separate scripts. Then we could redirect the uploads to the test server without making the other example boring.

@joaquinvanschoren what do you think about the last solution? You would need to use separate notebooks for presentation then.

Ci works with an API key right? Why don't we make the production API key read only?

…

On 13 Mar 2018 09:31, "Matthias Feurer" ***@***.***> wrote: After @ArlindKadra <https://github.com/arlindkadra> figured this out I had a brief look at how we could tackle this. All this is based on the premise that we have examples and tutorials in CI to make sure that they keep on working (which rules out the option to comment out code in the tutorial). 1. Work only on the test server. This appears to be pretty boring as there aren't a lot of results for tasks. 2. Only do the uploads to the test server, however, we would not get them if we download all previous results from the live server. 3. Split up the tutorial page into several smaller tutorials in which analysis of runs and upload of runs are separate scripts. Then we could redirect the uploads to the test server without making the other example boring. @joaquinvanschoren <https://github.com/joaquinvanschoren> what do you think about the last solution? You would need to use separate notebooks for presentation then. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#392 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACL7-ppwiIisv7LyX4SpxHngaCw3uT_nks5td8oagaJpZM4R9YUu> .

Wouldn't that give an exception?

One that you can predict and potentially catch

Yes, but one would have to catch that either outside of the notebook (and therefore not fully test it). I also think that we mock out the upload calls. Then the notebook will behave like before, there just will be no runs on the live server.

What does mocking out mean? Ideally you want an example where you upload a dataset/flow/run and then download it again. That should only be done on the test server. What about using environment variables? You could set them before you do the uploading part (something like OPENML_UPLOAD_SERVER=... ) and then read them in the notebook. I.e. if that variable exists then change the target server. Or: it wouldn’t hurt if we change the target server explicitly in the example? Users can always comment that out if they want to push to production.

On Wed, 14 Mar 2018 at 08:23, Matthias Feurer ***@***.***> wrote: Yes, but one would have to catch that either outside of the notebook (and therefore not fully test it). I also think that we mock out the upload calls. Then the notebook will behave like before, there just will be no runs on the live server. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#392 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpQV66IV3a7Nmro-KLQ6jpAZiZ11JVuks5teMVmgaJpZM4R9YUu> .

-- Thank you, Joaquin

FYI, this is still happening.
Can we do the testing without uploading test runs to the production server?

No one had time so far to do something about this. I just tried this, but the solution using unittest.mock did not work (see below for a snippet). I will have to check this next Tuesday.

    @mock.patch('openml._api_calls._perform_api_call')
    def test_tutorial(self, patch):
        def side_effect(*args, **kwargs):
            if (
                args[0].endswith('/run/')
                and kwargs['file_elements'] is not None
            ):
                raise ValueError('Not uploading a run!')
            else:
                return mock.DEFAULT
        patch.side_effect = side_effect
        self._test_notebook('OpenML_Tutorial.ipynb', patch)
        self.assertEqual(patch.call_count, 2)

I just opened a PR at #430.

Fix/#392 (#430)

Fixed via #430.

It's happening again:
https://www.openml.org/r/9912434

@ArlindKadra

Hm, I can't reproduce this locally... @ArlindKadra could you please look into this?

@mfeurer

@mfeurer It is from the recent pull request regarding flows that was merged into development.

openml-python/tests/test_flows/test_sklearn.py

Lines 994 to 1005 in 7c0a77d

    
           def test_openml_param_name_to_sklearn(self): 
        
               scaler = sklearn.preprocessing.StandardScaler(with_mean=False) 
        
               boosting = sklearn.ensemble.AdaBoostClassifier( 
        
                   base_estimator=sklearn.tree.DecisionTreeClassifier()) 
        
               model = sklearn.pipeline.Pipeline(steps=[ 
        
                   ('scaler', scaler), ('boosting', boosting)]) 
        
               flow = openml.flows.sklearn_to_flow(model) 
        
               task = openml.tasks.get_task(115) 
        
               run = openml.runs.run_flow_on_task(flow, task) 
        
               run = run.publish() 
        
               run = openml.runs.get_run(run.run_id) 
        
               setup = openml.setups.get_setup(run.setup_id)

The class does not extend OpenML's TestCase but the unittest one. That is why it is publishing at the main server.

openml-python/tests/test_flows/test_sklearn.py

Line 59 in 7c0a77d

class TestSklearn(unittest.TestCase):

@mfeurer

@mfeurer A solution is:
setting the config.server to the production server in the setup
changing the config.server to the test server in the test_openml_param_name_to_sklearn unittest

@ArlindKadra

Thanks @ArlindKadra for checking. I just created PR #609 to subclass all test cases from OpenML's core test case which hopefully fixes this issue.

@mfeurer

@mfeurer that is actually better, I am surprised that the sklearn tests did not make use of any tasks from the production server.

Okay fixed again via #609.

Navigation Menu

Repository navigation

CI runs on production server #392

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def test_openml_param_name_to_sklearn(self):
	scaler = sklearn.preprocessing.StandardScaler(with_mean=False)
	boosting = sklearn.ensemble.AdaBoostClassifier(
	base_estimator=sklearn.tree.DecisionTreeClassifier())
	model = sklearn.pipeline.Pipeline(steps=[
	('scaler', scaler), ('boosting', boosting)])
	flow = openml.flows.sklearn_to_flow(model)
	task = openml.tasks.get_task(115)
	run = openml.runs.run_flow_on_task(flow, task)
	run = run.publish()
	run = openml.runs.get_run(run.run_id)
	setup = openml.setups.get_setup(run.setup_id)

CI runs on production server #392

Description

Activity

joaquinvanschoren commented on Mar 7, 2018

mfeurer commented on Mar 7, 2018

ArlindKadra commented on Mar 7, 2018

joaquinvanschoren commented on Mar 7, 2018

ArlindKadra commented on Mar 7, 2018

joaquinvanschoren commented on Mar 7, 2018

ArlindKadra commented on Mar 7, 2018

mfeurer commented on Mar 13, 2018

janvanrijn commented on Mar 13, 2018

mfeurer commented on Mar 13, 2018

janvanrijn commented on Mar 13, 2018

mfeurer commented on Mar 14, 2018

joaquinvanschoren commented on Mar 14, 2018

3 remaining items

joaquinvanschoren commented on Mar 25, 2018

mfeurer commented on Mar 29, 2018

mfeurer commented on Apr 3, 2018

mfeurer commented on Apr 6, 2018

joaquinvanschoren commented on Dec 13, 2018

mfeurer commented on Dec 17, 2018

ArlindKadra commented on Dec 18, 2018

ArlindKadra commented on Dec 18, 2018

mfeurer commented on Dec 19, 2018

ArlindKadra commented on Dec 19, 2018

mfeurer commented on Jan 29, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions