Task API changed? #604

Activity

1. janvanrijn commented on Dec 7, 2018

   

   janvanrijn

   on Dec 7, 2018

   MemberAuthor

   More actions

   This is getting just crazy. The type of the parameters changes. I added two prints to this function

   def _create_task_from_xml(xml):
       """Create a task given a xml string.
   
       Parameters
       ----------
       xml : string
           Task xml representation.
   
       Returns
       -------
       OpenMLTask
       """
       dic = xmltodict.parse(xml, force_list={'oml:input'})["oml:task"]
       estimation_parameters = dict()
       inputs = dict()
       # Due to the unordered structure we obtain, we first have to extract
       # the possible keys of oml:input; dic["oml:input"] is a list of
       # OrderedDicts
       for input_ in dic["oml:input"]:
           name = input_["@name"]
           inputs[name] = input_
   
       evaluation_measures = None
       if 'evaluation_measures' in inputs:
           evaluation_measures = inputs["evaluation_measures"][
               "oml:evaluation_measures"]["oml:evaluation_measure"]
   
       task_type = dic["oml:task_type"]
       common_kwargs = {
           'task_id': dic["oml:task_id"],
           'task_type': task_type,
           'task_type_id': dic["oml:task_type_id"],
           'data_set_id': inputs["source_data"][
               "oml:data_set"]["oml:data_set_id"],
           'evaluation_measure': evaluation_measures,
       }
       if task_type in (
           "Supervised Classification",
           "Supervised Regression",
           "Learning Curve"
       ):
           # Convert some more parameters
           for parameter in \
                   inputs["estimation_procedure"]["oml:estimation_procedure"][
                       "oml:parameter"]:
               name = parameter["@name"]
               text = parameter.get("#text", "")
               estimation_parameters[name] = text
           print(type(common_kwargs))
           print('1', type(estimation_parameters), estimation_parameters)
   
           common_kwargs['estimation_procedure_type'] = inputs[
               "estimation_procedure"][
               "oml:estimation_procedure"]["oml:type"],
           common_kwargs['estimation_parameters'] = estimation_parameters,
           common_kwargs['target_name'] = inputs[
                   "source_data"]["oml:data_set"]["oml:target_feature"]
           common_kwargs['data_splits_url'] = inputs["estimation_procedure"][
                   "oml:estimation_procedure"]["oml:data_splits_url"]
           print('2', type(common_kwargs['estimation_parameters']))
       print(inputs[
               "estimation_procedure"][
               "oml:estimation_procedure"]["oml:type"])
       print(common_kwargs)
       cls = {
           "Supervised Classification": OpenMLClassificationTask,
           "Supervised Regression": OpenMLRegressionTask,
           "Clustering": OpenMLClusteringTask,
           "Learning Curve": OpenMLLearningCurveTask,
       }.get(task_type)
       if cls is None:
           raise NotImplementedError('Task type %s not supported.')
       return cls(**common_kwargs)
   

   This is the outcome. Looks like dark magic to me.

   <class 'dict'>
   1 <class 'dict'> {'number_repeats': '1', 'number_folds': '10', 'percentage': '', 'stratified_sampling': 'true'}
   2 <class 'tuple'>
   crossvalidation
   {'task_id': '3', 'task_type': 'Supervised Classification', 'task_type_id': '1', 'data_set_id': '3', 'evaluation_measure': None, 'estimation_procedure_type': ('crossvalidation',), 'estimation_parameters': ({'number_repeats': '1', 'number_folds': '10', 'percentage': '', 'stratified_sampling': 'true'},), 'target_name': 'class', 'data_splits_url': 'https://www.openml.org/api_splits/get/3/Task_3_splits.arff'}
   {'type': ('crossvalidation',), 'parameters': ({'number_repeats': '1', 'number_folds': '10', 'percentage': '', 'stratified_sampling': 'true'},), 'data_splits_url': 'https://www.openml.org/api_splits/get/3/Task_3_splits.arff'}
   
2. janvanrijn commented on Dec 7, 2018

   

   janvanrijn

   on Dec 7, 2018

   MemberAuthor

   More actions

   Someone added a comma after these lines, effectively changing them into tuples. This seems like a mistake.

   We should handle these things with a bit more care. These kind of set backs always take a long time to debug
3. mfeurer commented on Dec 7, 2018

   

   mfeurer

   on Dec 7, 2018

   Collaborator

   More actions

   This seems like a mistake.

   I think so, too. I just merged this. Nevertheless, @ArlindKadra could you please confirm this?

   We should handle these things with a bit more care.

   I didn't spot that mistake and the unittests didn't either. It seems like this information wasn't used yet? Any suggestions on how to improve this?
4. mfeurer

   closed this as completedon Dec 7, 2018
5. ArlindKadra commented on Dec 7, 2018

   

   ArlindKadra

   on Dec 7, 2018

   Member

   More actions

   @mfeurer @janvanrijn I can confirm that Jan found a bug. It was added when @mfeurer merged the pull request for more tasks from Joaquin #544
   How that did not raise a bug when a comma was added there should be fixed. We should include a unit test.