OpenML-Python ARFF reader

This is not to discredit the LIAC-ARFF reader, as I think it is useful tool that allows for parsing and unparsing of many arff files that surve a broad purpose.

However, today I discovered a bug (strings with quotes and escape characters, we use this for storing arff traces) which required me to dive into the code. We currently use it for serializing arffs to disk (either to local disk, but also to upload to OpenML). Basically, it serializes arff files in a nice way, but deserializing goes wrong.

Given the following facts:
a) It seems that active development has stopped in 2015 judging by the commits
b) The code builds upon a set of regular expressions (which is great for efficiency, but hard to adapt without the proper knowledge)
c) The unit tests are not as extensive as I would expect them to be (although this is something we can change, of course)

I can't help but wonder whether it wouldn't be better to switch to the scipy variant. We discussed this before, what were the pro's and cons again?

@mfeurer

Well, @mfeurer has made 5 commits to liac-arff in the last month. I've also opened issues and did at least one PR in the past year.
The scipy variant has issues of its own, e.g. it does not support sparse arff.
More discussion on this here: scikit-learn/scikit-learn#9908

Anyway, @mfeurer has been a lot closer to this discussion. Thoughts?

@mfeurer

Until shortly before, what I liked about the LIAC arff is that is is quite restrictive. If it parses an arff file, we are sure that it is a good arff file. However today I discovered that it does not always consistent generating arff output, which is a feature upon which we heavily rely and which potentially influences the data integrity on the server. The problem seems quite deep in the core, I wasn't able to fix it this afternoon.

Well, @mfeurer has made 5 commits to liac-arff in the last month. I've also opened issues and did at least one PR in the past year.

This is exactly the problem, right?

The scipy variant has issues of its own, e.g. it does not support sparse arff.

Can we not convert our sparse arff datasets to vanilla arff? I'm sure that would make many things in the system way easier.

.. Apparently scipy arff is also not the answer, as it can only read and not write arff. But on the other hand, how hard can it be to exploit the dump function of a csv writer and manually add a header?

This is exactly the problem, right?

Is the fact that we're working on this a problem?

Can we not convert our sparse arff datasets to vanilla arff?

Have a look at the LibSVM datasets, some of them have up to 20 million features. I think you really want to support sparse arff files.

But on the other hand, how hard can it be to exploit the dump function of a csv writer and manually add a header?

If csv == arff this would be a solution, but liac-arff needs to support sparse data and None values need to be encoded as ?.

I'll have a look at this now. Could you please elaborate on the unit tests you would like to see?

Is the fact that we're working on this a problem?

No, I think that is great. The problem that I wanted to point out is that the original developers do not seem to do any maintenance on the library anymore since 2015, which is a problem for a crucial dependency in our project.

I'll have a look at this now. Could you please elaborate on the unit tests you would like to see?

It would be great if we could get this unit test working.

FFR:

import unittest
import arff
try:
    from StringIO import StringIO
except ImportError:
    from io import StringIO

OBJ = {
    'description': '\nDummy Dataset\n\n\n',
    'relation': 'Dummy',
    'attributes': [
        ('input1', ['"mean"', '"median"', '"most_frequent"']),
        ('y', 'REAL'),
    ],
    'data': [
        ['"mean"', 0.0],
        ['"median"', 1.0],
        ['"most_frequent"', 1.0],
        ['"mean"', 0.0]
    ]
}

ARFF = '''%
% Dummy Dataset
%
%
%
@RELATION Dummy
@ATTRIBUTE input1 {"mean", "median", "most_frequent"}
@ATTRIBUTE y REAL
@DATA
'\\"mean\\"',0.0
'\\"median\\"',1.0
'\\"most_frequent\\"',1.0
'\\"mean\\"',0.0
'''

class TestLoadDump(unittest.TestCase):
    def get_dumps(self):
        dumps = arff.dumps
        return dumps

    def get_loads(self):
        loads = arff.loads
        return loads

    def test_simple(self):
        dumps = self.get_dumps()
        loads = self.get_loads()

        arff = ARFF
        obj = OBJ

        count = 0
        while count < 10:
            count += 1

            arff = dumps(obj)
            self.assertEqual(ARFF, arff)
            obj = loads(arff)
self.assertEqual(OBJ, obj)

Does the scipy function read/write both sparse and non-sparse ARFF files?

On Thu, 14 Jun 2018 at 11:57 janvanrijn ***@***.***> wrote: Is the fact that we're working on this a problem? No, I think that is great. The problem that I wanted to point out is that the original developers do not seem to do any maintenance on the library anymore since 2015, which is a problem for a crucial dependency in our project. I'll have a look at this now. Could you please elaborate on the unit tests you would like to see? It would be great if we could get this unit <https://github.com/janvanrijn/liac-arff/blob/master/tests/test_loads_dumps_escape.py> test working. FFR: import unittest import arff try: from StringIO import StringIO except ImportError: from io import StringIO OBJ = { 'description': '\nDummy Dataset\n\n\n', 'relation': 'Dummy', 'attributes': [ ('input1', ['"mean"', '"median"', '"most_frequent"']), ('y', 'REAL'), ], 'data': [ ['"mean"', 0.0], ['"median"', 1.0], ['"most_frequent"', 1.0], ['"mean"', 0.0] ] } ARFF = '''% % Dummy Dataset % % % @relation Dummy @Attribute input1 {"mean", "median", "most_frequent"} @Attribute y REAL @DaTa '\\"mean\\"',0.0 '\\"median\\"',1.0 '\\"most_frequent\\"',1.0 '\\"mean\\"',0.0 ''' class TestLoadDump(unittest.TestCase): def get_dumps(self): dumps = arff.dumps return dumps def get_loads(self): loads = arff.loads return loads def test_simple(self): dumps = self.get_dumps() loads = self.get_loads() arff = ARFF obj = OBJ count = 0 while count < 10: count += 1 arff = dumps(obj) self.assertEqual(ARFF, arff) obj = loads(arff) self.assertEqual(OBJ, obj) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#466 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpQV7UF27ZxLH7taFns0QHaOv-_I2I9ks5t8ofWgaJpZM4TzZDh> .

-- Thank you, Joaquin

the original developers do not seem to do any maintenance on the library anymore since 2015, which is a problem for a crucial dependency in our project.

That's why I can do this now ;)

renatopp/liac-arff#69

Please check if my latest pull request fixes this.

Does the scipy function read/write both sparse and non-sparse ARFF files?

From the scipy docs:

This function should be able to read most arff files. Not implemented functionality include:

    date type attributes
    string type attributes

It can read files with numeric and nominal attributes. It cannot read files with sparse data ({} in the file). However, this function can read files with missing data (? in the file), representing the data points as NaNs.

Scipy can't write anything and we actually use this function to load the splits.

These limitations were one of the reasons I chose to go with liac-arff.

Can we close this for now given the upcoming fix in #481? Also, given that this is a duplicate of #116?

Absolutely! Thanks for fixing this :)

@janvanrijn

But you need to review my PR @janvanrijn otherwise this won't be fixed ;)

You mean this one right:
#481

Navigation Menu

Repository navigation

OpenML-Python ARFF reader #466

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OpenML-Python ARFF reader #466

Description

Activity

joaquinvanschoren commented on May 5, 2018

janvanrijn commented on May 5, 2018

mfeurer commented on Jun 14, 2018

janvanrijn commented on Jun 14, 2018

joaquinvanschoren commented on Jun 14, 2018

mfeurer commented on Jun 15, 2018

mfeurer commented on Jun 19, 2018

janvanrijn commented on Jun 19, 2018

mfeurer commented on Jun 20, 2018

janvanrijn commented on Jun 20, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions