(DO NOT MERGE) Allow table columns to be hidden from print output - #2466

This is even later, and maybe controversial...

@mhvk @astrofrog @embray ?

I like the idea in principle, and think it might be useful as well when we implement virtual columns (one could imagine having Time's jd1, jd2 in hidden columns but showing the virtual Time column). I'm less sure about the implementation, though: to me, this is not a property of a Column (it does nothing if I print the column itself, right?), but belongs in the table, or perhaps in the format. I'd be happier with the table having, e.g., a dictionary like format attribute, with entries corresponding to column formats, but which could be edited, with maybe individual entries set to False (not sure at all this is practical, mostly just trying to think how to do it on the table itself).

@mhvk - I think that makes sense and would simplify the implementation. I do think that format is an intrinsic property of a column and would not want to override that, esp. since you don't want to lose the format if you hide then unhide the format.

I had originally also thought about an API like:

t = Table(..)
t.hide_columns(col1, ...)
t.unhide_columns(col1, ..)
t.unhide_columns()  # unhide all

I thought about show_columns, but to me that feels like it should be literally showing them on the screen.

Then Column could have a hidden property that is False for a bare column but looks up into the parent table for a column within a table. But now there is a question, because there is no reason you cannot have that hidden property also be settable and change the parent table settings. So this is two ways to do it. So maybe we just document the table-level methods as the correct way to do this, but setting hidden will also work.

@taldcroft - agreed on not re-using format. With your example, I see a bit more why you implemented it as you did. Also, it does seem nice to be able to do table[column].hidden = True, which implies the hidden property indeed has to be on the column, not on the table.

@mhvk - does that mean you would be OK with the original (current) implementation?

I looked into putting into Table and the most straightforward method would involve adding a keyword arg to the Table init like hidden_columns or something. It feels a bit more awkward in the API than a hidden arg for Column (which already has a bunch of stuff). I tried to come up with a clever way to make the Table.hidden_columns be stowed away within meta (out of view from users) so that no new keyword arg would be required (and no change to pickling), but couldn't come up with anything nice.

Does this affect the outputted files? (if not, could one end up in a situation where a user gets a table from a code with some columns pre-hidden) and then doesn't understand why the file output doesn't match the printed output? What about the browser-based javascript viewer?

@astrofrog - this affects the HTML output (and thus ipython notebook repr) and browser-based javascript viewer. Basically anything that uses Table.pformat to get the table representation. It does not affect anything in any other packages like io.fits or io.ascii. I didn't intend for the hidden column to affect how a table gets stored to disk, just how it gets output to the screen / browser. I see your point about some points for confusion. A pickled Table (or from an APE6 DTIF file) could have pre-hidden columns. I suppose we could issue warnings when any I/O is done with a Table that has hidden columns. Hmm...

@taldcroft - how about setting the milestone to 1.0 so as to have more time to discuss this?

@astrofrog - OK milestone is 0.4.

I re-ran the one test that stalled out.

Explicit is better than implicit.

I agree with @astrofrog that there is a large potential for confusion. It is very easy already to select a subset of columns, so that the user can easily select which columns should be printed and which should not be printed.

In [1]: from astropy.table import Table

In [2]: t = Table({'a':[1,2,3], 'b':[4,5,6], 'c':['q','w','e']})

In [3]: print t
 a   c   b 
--- --- ---
  1   q   4
  2   w   5
  3   e   6

In [5]: # This can scramble the order of columns, but if that is a concern, a list can be used
In [7]: print_cols = list(set(t.colnames) - set('b'))

In [9]: print t[print_cols]
 a   c 
--- ---
  1   q
  2   w
  3   e

Special needs like time columns would be better served by #2790.

Looking back at this, I think that in its current form there is too much potential for confusion (e.g. HTML output vs other formats, etc.).

Along the same lines as what @hamogu is suggesting, what about having a method without_columns that returns a new view of the table that excludes specific columns? E.g.:

print t.without_columns('b')

This would be shorthand for

t2 = t.copy()
t2.remove_columns('b')
print t2

except that without_columns wouldn't need to return a copy necessarily, could be a view. Anyway, just brainstorming :)

The problem with table views as such is that they are not views, but rather copies (as explicitly noted in the example from @astrofrog). So if you want to work with a subset of a table and perform mutable operations, that's a no-go.

t2 = t.without_columns('b')
print(t2)  # examine it, figure out what to do
t2['r'] = t2['a'] + 2
t2.remove_rows([1,2])
# But t is still the same

There is a reason that every spreadsheet app allows the ability to hide columns instead of just make a new sheet with a copy of a subset of columns. I want Table to have spreadsheet-like capabilities (like computed columns), so hiding is a natural feature.

Pushing this to 1.1, but it's still a good idea! 😄 It would be a simple thing to make all columns visible before any I/O operation including pickling.

Note that in current master it is now possible to make a quasi-view of a subset of columns:

t2 = Table([t['a'], t['b']], copy=False)

Here t2['a'] and t['a'] share the same data. But this is more dangerous and confusing, not less, because tables are mutable (columns are re-made as needed), and so these columns can become unbound in ways that users would not easily predict.

As I said, most people have used spreadsheets and everyone agrees it is a good thing to be able to hide/unhide columns on the spreadsheet, and nobody is freaked out because certain data are not visible.

Not 1.1-critical so removing milestone

Looked at this again after a long time (as I was reviewing "ready-for-final-review" PRs), and ended up uncomfortable. Mostly, I feel the state should not be on a Column (even if we allow table[column].hidden = True, I think it should set something on the parent table), but this may be a moot point now that we have mixin columns. If we do have it on the table, I would suggest noting the hidden columns by default in __repr__. Ideally, we'd also propagate these meta data to ecsv files, etc.

Anyway, for now removed the ready-for-final-review label...

@mhvk - I look at this PR (longingly) every so often. One thing that has changed in the meantime is moving toward encapsulating new column information in info. This is now a natural and easy place to put new column attributes.

The other problem with having Table manage the hidden status in a hidden dict is that you need to maintain that dict. Not insurmountable, but if one removes or renames a column then the hidden dict needs update. This just creates a new linkage in the code and a way for things to break. Having the hidden attribute on the actual column itself is consistent with the way other attributes are handled to this point and keeps them local and tidy.

But anyway, if I ever revive this it will be totally different. I would like to leave this open for a little longer so I see it occasionally, but this exact PR will never get merged.

Hi humans 👋 - this pull request hasn't had any new commits for approximately 3 years. I plan to close this in a month if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please close this and open an issue instead to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If you believe I commented on this issue incorrectly, please report this here.

Long live hidden table columns!

I really need this today -- I have a number of tables with a columns that are complex objects (tables, long dicts) with huge reprs where I need to access the values but never print in a full table. The solution of just selecting out the "columns I want" every time I look at the table is a pain.

I'm commenting as incentive to myself to first do the info overhaul and then this, which will then be easy. And hopefully @mhvk will come around to the viewpoint that the visibility of a column is something that belongs in info, just like name, description, format, meta, etc. It's just another column attribute after all.

I was hunting for this feature, it would be really useful for sunpy if the default display columns could be changed and then unhidden by the user, or become unhidden if explicitly indexed etc.