Repository navigation
Table of tests coverage #46
Description
We need to document the test coverage.
I have pushed a page tests-coverage.md to master and enabled GitHub Pages so that it shows at https://data-apis.org/array-api-tests/tests-coverage.html (it also created a page for the README). I started a basic table with statistical functions, based on #39. It's not yet completely filled out.
@honno Let me know what you think of the general styling and the categories. My thinking is each cell will have "Yes", "No", or "N/A", and if there is more to say about something we can add a footnote. The idea is based on tables you often see on Wikipedia (e.g., this one).
Feel free to edit it and to fill it out more. You can push directly to master if you want. I don't know if there is any other way to see a preview of the page without pushing to master and letting GitHub render it. You can also change the theme if you want (see https://github.com/data-apis/array-api-tests/settings/pages). I was also thinking of adding some Javascript that colors the cells green if they have "Yes" in them and red otherwise.
By the way, if you use emacs, the markdown-mode makes it really easy to edit the table. You can just press tab and enter to move between cells, and it automatically realigns them. I don't know if there are similar plugins for other editors.
Activity
honno commentedon Dec 7, 2021
IMO I think such a granular table isn't worth it. I'm going to spend time filling it out, and then 100% forget to update it in lockstep with test suite changes—at the very least it will be a finicky thing to worry about. We're already 80%+ Yes with #39 anywho. Maybe something better to do after we think the suite is complete, to verify that it indeed is.
I think it's better to just summarise the goals of the test suite i.e. the column categories you've outlined, and state that it's WIP for now. Maybe giving broad allusions to what it lacks (testing some smaller function groups, usability things like nice error messages, etc.).
asmeurer commentedon Dec 7, 2021
Well my understanding from the meeting is that we do want it. What do you think @rgommers @kgryte? If we're already close, then we should think about whether we are missing any categories. Otherwise, I think this will be useful to help us understand the remaining 20% that is missing.
Outlining other big TODOs here like better reporting is a good idea. Right now most of those ideas only live in the issue tracker, but that isn't very visible to people using the test suite, and I've often found myself re-explaining it to people.
rgommers commentedon Dec 8, 2021
@honno's point is about the level of granularity I think. This will be a huge table, perhaps it will be easier to parse if it had a single entry/row for "statistical functions" rather than one for each function. That would make sense to me. You'd implement support for a particular type of test (say promotion behavior) for a set of functions at once normally. So little info will be lost, and you may make the table ~5x shorter.
honno commentedon Dec 8, 2021
Well now I'm writing tests that that covers all relevant bases at once, and all the existing tests cover all relevant bases, so we can just say now "we need to implement set + utility + searching functions" in a sentence and not have any table or list at all.
honno commentedon Dec 13, 2021
Let's clarify what we want. My impression was we want a resource that could give Array API implementers a high-level overview of what this test suite does and doesn't. The current page does a great job doing that, so I'm thinking we just need to tweak it (remove table, readability changes, etc.) and we'll be done.
honno commentedon Dec 14, 2021
As discussed in our meeting, let's prioritise expressing the broad strokes for now, and sit with a "completed" test suite sit for a while before documenting the granular details. Good news that's mostly done, so I'll:
honno commentedon Sep 30, 2022
I think users of the test suite have gotten the jist of what the test suite is actually testing with the README as-is, so closing (feel free to re-open).