Can't download dataset metadata without downloading the dataset itself #612

Activity

1. janvanrijn commented on Jan 25, 2019

   

   janvanrijn

   on Jan 25, 2019

   Member

   More actions

   Agreed. Similar for tasks, where the split file is commonly downloaded (potentially 10 times the size of the dataset)
2. mfeurer commented on Jan 29, 2019

   

   mfeurer

   on Jan 29, 2019

   Collaborator

   More actions

   Agreed, being able to disable this would be a great addition. I think actually be a better solution than the one proposed in #346, what do you think @amueller ?
3. PGijsbers commented on Feb 13, 2019

   

   PGijsbers

   on Feb 13, 2019

   Last edited by PGijsbers

   Collaborator

   More actions

   I agree that metadata should be accessible easily without downloading the whole dataset. I think adding a parameter to the get_dataset method makes sense, e.g.:

   def get_dataset(dataset_id: int, download_data: bool=False) -> OpenMLDataset:
       """ Download the OpenML dataset representation, optionally also download actual data file. 
   
       This function is thread/multiprocessing safe.
       This function uses caching. A check will be performed to determine if the information has
       previously been downloaded, and if so be loaded from disk instead of retrieved from the server.
   
       Parameters
       ----------
       dataset_id : int
           Dataset ID of the dataset to download
       download_data : bool
           If True, also download the data file. Beware that some datasets are large and it might make
           the operation noticeably slower. Metadata is also still retrieved.
           If False, create the OpenMLDataset and only populate it with the metadata.
           The data may later be retrieved through the `OpenMLDataset.get_data` method.
   
       Returns
       -------
       dataset : :class:`openml.OpenMLDataset`
           The downloaded dataset."""
   

   We would then also have to update the get_data method to check if the data is already present in cache (and download it if not). Edit: looks like retrieve_class_labels will have issues too, we'll have to see how to work around that (if it's an issue).

   I think not downloading the data by default is fine behavior. Right now to look at the data, you always call the get_data method anyway.

   For those looking for a solution right now to download qualities metadata without downloading the actual data, use this code-snippet (tested with OpenML 0.7.0):

   from openml.datasets.functions import _get_dataset_qualities, DATASETS_CACHE_DIR_NAME
   from openml.datasets.dataset import _check_qualities
   from openml.utils import _create_cache_directory_for_id
   
   def get_qualities(dataset_id):
      did_cache_dir = _create_cache_directory_for_id(DATASETS_CACHE_DIR_NAME, dataset_id)
      qualities_xml_dict = _get_dataset_qualities(did_cache_dir, dataset_id)
      return _check_qualities(qualities_xml_dict)
   
4. PGijsbers

   self-assigned this

   on Mar 17, 2019
5. mfeurer commented on Mar 18, 2019

   

   mfeurer

   on Mar 18, 2019

   Collaborator

   More actions

   Functionality added in #644.
6. mfeurer

   closed this as completedon Mar 18, 2019