Skip to content

Refactor CompImageHDU to inherit from ImageHDU#15474

Merged
saimn merged 82 commits into
astropy:mainfrom
astrofrog:compimagehdu-inheritance
Sep 30, 2024
Merged

Refactor CompImageHDU to inherit from ImageHDU#15474
saimn merged 82 commits into
astropy:mainfrom
astrofrog:compimagehdu-inheritance

Conversation

@astrofrog
Copy link
Copy Markdown
Member

@astrofrog astrofrog commented Oct 12, 2023

This pull request addresses #9238 by refactoring CompImageHDU to inherit from ImageHDU instead of BinTableHDU. This includes some subtle API changes that cannot be warned about in advance using deprecation warnings or future warnings, so this PR, if accepted, would definitely warrant a major version bump. Ideally this would go in v6.0 but it depends on whether we can achieve consensus and stamp out remaining issues.

This PR isn't 100% ready (see 'Outstanding issues' below and FIXME comments in code) but I wanted to try and open this now in case it is something we can consider getting in before the v6.0 feature freeze. I doubt that the outstanding issues would cause significant code changes so it should be safe to already review things as-is.

Background

Compressed image HDUs are stored in FITS files as binary tables. The CompImageHDU was therefore developed as a a subclass of BinTableHDU. However, that implementation is quite complex because it tries to look to users like an ImageHDU - for example .header is meant to be the image header, and .data is meant to be the decompressed image data. To quote comments from the code:

# TODO: The difficulty of implementing this screams a need to rewrite this module
Bypasses `BinTableHDU._writeheader()` which updates the header with
metadata about the data that is meaningless here; another reason
why this class maybe shouldn't inherit directly from BinTableHDU...

Issue #9238 by @mhvk also suggested that CompImageHDU would be simpler if it inherited from ImageHDU.

This PR does that refactoring, and represents CompImageHDU as an ImageHDU subclass that internally stores a BinTableHDU representing the compressed data. Instead of always keeping the image and table header in sync, headers are only converted at I/O time which should in principle make things faster.

Overall, this results in a few hundred lines of code being removed. For now CompImageHeader is kept to avoid breaking API so that warnings about reserved keywords are emitted as soon as they are set. However, if we would be happy to 'break' this behavior and only warn at I/O time or when calling verify, we could get rid of the custom header subclass. If we want to keep the behavior, we can nevertheless try and avoid code duplication with the Header class by ssimplifying the header validation (see #15293).

API changes/fixes

I have tried to minimize API changes as much as possible. However, it is impossible to not have any changes at all:

  • With this PR, a compressed image HDU chdu will no longer return True for isinstance(chdu, BinTableHDU). There's no way to avoid this and also no way to do this with a gradual deprecation. However, I would hope that this is rare and that if anything people would do isinstance(chdu, CompImageHDU) which will still be True.

  • The following options have changed defaults in the CompImageHDU initializer to match ImageHDU:

    • uint=True replaces uint=False (I need to check if I can come up with an example that shows the difference here)
    • scale_back=None replaces scale_back=False (in practice I think this won't change anything)
  • The checksum on the main header is now the image checksum not the binary table checksum (I believe this was a bug: Incorrect checksum/datasum in some cases for compressed FITS images? #14753)

Fixes:

Repeating for auto-close in less readable format:

Fixes #9238
Fixes #10512
Fixes #12216
Fixes #14081
Fixes #14611
Fixes #14753

Modified tests

I've tried to change tests as little as possible but there were some unavoidable changes:

  • Many tests (probably non-optimally) were checking the contents of chdu._header which no longer exists (the equivalent is chdu._bintable.header.
  • test_compressed_header_missing_znaxis has been removed - this checked that removing ZNAXIS or ZBITPIX from the internal binary table caused accessing the compressed data property to fail. This doesn't seem needed because it relies on modifying private attributes on CompImageHDU and we now also generate the compressed header on-the-fly so there is no opportunity to remove the keywords before the data is compressed.
  • test_compressed_header_double_extname - included some lines checking state of internal/private properties. Those checks have been removed as the test is no longer relevant for this API.

Outstanding issues/open questions

  • Add back support for accessing CompImageHDU.shape (should be simple)
  • Make sure that we don't write out anything when in mode='update' if a file is simply opened and closed (test_open_comp_image_in_update_mode failure). For now I have xfailed the test to make sure the rest of the CI works properly.

For future

  • Not critical for this PR, but currently this preserves the CompImageHeader class - as noted higher up we could potentially consider removing or simplifying this in future.
  • This keeps the compressed_data property on CompImageHDU. It might be better to deprecate it and encourage people to open the file with disable_image_compression? I doubt this property is used much. We can do this in a follow-up though.
  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • Is a milestone set? Milestone must be set but we cannot check for it on Actions; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@github-actions
Copy link
Copy Markdown
Contributor

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

@astrofrog astrofrog force-pushed the compimagehdu-inheritance branch from ed65a28 to 7f7ca75 Compare October 12, 2023 11:54
Copy link
Copy Markdown
Contributor

@mhvk mhvk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@astrofrog - I went over it relatively quickly and noticed mostly little imprementation things (particularly for _image_header_to_empty_bintable where quite some code can be removed since you are now initializing a new bintable).

Unfortunately, I feel rather less able to judge the main changes... But I do like the overall idea and think it would be good to proceed for 6.0 -- hopefully, the test coverage is substantial enough...

On the API changes, I think they are all fine (for a major version bump), but note that it is possible to ensure that isinstance(chdu, BinTableHDU) keeps working (using ABCMeta and __subclasshook__. Though I think this is not really needed.

Comment thread astropy/io/fits/hdu/compressed/compressed.py Outdated
Comment thread astropy/io/fits/hdu/compressed/compressed.py Outdated
Comment thread astropy/io/fits/hdu/compressed/compressed.py Outdated
# Store a temporary backup of self.data in a different attribute;
# see below
self._imagedata = self.data
if self._bintable is not None:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused about self._tmp_bintable -- could one not just create self._bintable if it does not yet exist? I.e., have tmp_bintable instead of self._tmp_bintable and here

if self._bintable is None:
    self._bintable = tmp_bintable
else:
    self._bintable.header = self._tmp_bintable.header
    self._bintable.data = self._tmp_bintable.data

The difference is that if there was no bintable, there will now be one (since _postwriteto would likely do nothing -- which would seem fine (if not, a flag to remove it on self might still be a bit nicer)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a philosophical/deliberate choice - if a CompImageHDU wasn't backed by a _bintable prior to writing, I was going with the idea that it should also not be after writing, i.e. writing the CompImageHDU should not have any side effects on CompImageHDU. Does this make sense or do you think after writing to a file the CompImageHDU should switch to being backed by a bintable? (this could result in changes to .data if there is any quantization).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the data is fully loaded in memory, do we need to keep the bintable ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we need to, at least in update mode, as if the data isn't changed, we want to write the bintable out as-is without recompressing it. If we dropped _bintable when the data is decompressed we'd also lose the ability for the user to use the compressed_data property. I would vote to keep it as-is for now and to see in a follow-up whether there are any benefits to dropping _bintable once .data is accessed under certain circumstances.

"""
The tile shape used for the tiled compression.
# FIXME: determine how we can simplify or avoid the following methods. These
# seem to be needed for the checksum calculations to work properly.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weirdly, coverage suggests the methods are not actually called. Though it is wrong so often that I've started to just ignore it...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've now added a test that does exercise this (it is needed specifically for lazy-loading HDUs when the file object might be controlled by the user) - see bda155e (and cc @saimn FYI)

Comment thread astropy/io/fits/hdu/compressed/header.py Outdated
Comment thread astropy/io/fits/hdu/compressed/header.py Outdated
assert h2[1].header["CHECKSUM"] == "D2GXD29VD2EVD29V"
assert "DATASUM" in h2[1].header
assert h2[1].header["DATASUM"] == "113055149"
assert h2[1].header["DATASUM"] == "2189405276"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense that both checksum and datasum have changed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think checksum includes the data and the header so since the datasum has changed I think it makes sense that checksum changes too

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with this branch, comment are moved to the end and blank cards are removed:

❯ diff -u <(fitsheader tmp-ref.fits) <(fitsheader tmp-new.fits) 
--- /proc/self/fd/11	2024-06-14 19:34:45.798001800 +0200
+++ /proc/self/fd/14	2024-06-14 19:34:45.798001800 +0200
@@ -1,12 +1,12 @@
-# HDU 0 in tmp-ref.fits:
+# HDU 0 in tmp-new.fits:
 SIMPLE  =                    T / conforms to FITS standard                      
 BITPIX  =                    8 / array data type                                
 NAXIS   =                    0 / number of array dimensions                     
 EXTEND  =                    T                                                  
-CHECKSUM= 'B3dAC2Z9B2dAB2Z9'   / HDU checksum updated 2024-06-14T19:29:18       
-DATASUM = '0       '           / data unit checksum updated 2024-06-14T19:29:18 
+CHECKSUM= 'A3dAD2Z9A2bAA2Z9'   / HDU checksum updated 2024-06-14T19:29:38       
+DATASUM = '0       '           / data unit checksum updated 2024-06-14T19:29:38 
 
-# HDU 1 in tmp-ref.fits:
+# HDU 1 in tmp-new.fits:
 XTENSION= 'IMAGE   '           / Image extension                                
 BITPIX  =                   16 / My special comment                             
 NAXIS   =                    2 / My special naxis comment                       
@@ -14,10 +14,6 @@
 NAXIS2  =                  300                                                  
 PCOUNT  =                    0 / My special PCOUNT comment                      
 GCOUNT  =                    1 / My special GCOUNT comment                      
-COMMENT   FITS (Flexible Image Transport System) format defined in Astronomy and
-COMMENT   Astrophysics Supplement Series v44/p363, v44/p371, v73/p359, v73/p365.
-COMMENT   Contact the NASA Science Office of Standards and Technology for the   
-COMMENT   FITS Definition document #100 and other FITS information.             
 OBJECT  = 'NGC 1316'                                                            
 TELESCOP= 'Optical '                                                            
 EQUINOX =               1950.0 / EPOCH OF RA DEC                                
@@ -34,34 +30,12 @@
 CRPIX2  = 1.470000000000000E+02                                                 
 CROTA2  =      0.000000000E+00 /                                                
 ORIGIN  = 'AIPSGorilla (NRAOCV IPX) 15JUL94'    /                               
-CHECKSUM= 'WdAaaZ5YZbAaaZ3Y'   / HDU checksum updated 2024-06-14T19:29:18       
-DATASUM = '113055149'          / data unit checksum updated 2024-06-14T19:29:18 
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
-                                                                                
+CHECKSUM= 'N7gSP4ZSN4fSN4ZS'   / HDU checksum updated 2024-06-14T19:29:38       
+DATASUM = '113055149'          / data unit checksum updated 2024-06-14T19:29:38 
+COMMENT   FITS (Flexible Image Transport System) format defined in Astronomy and
+COMMENT   Astrophysics Supplement Series v44/p363, v44/p371, v73/p359, v73/p365.
+COMMENT   Contact the NASA Science Office of Standards and Technology for the   
+COMMENT   FITS Definition document #100 and other FITS information.             

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DATASUM is different because it was previously computed on the bintable, and now it is computed on the image array. I think I remember you raised that question a while ago, though I don't remember the outcome.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you mention this in the changelog entry:

Fix the image checksum/datasum in CompImageHDU.header to be those for the
image HDU instead of for the underlying binary table.

And the discussion I was remembering is #14753.
However I'm not convinced that CHECKSUM/DATASUM should be the ones of the (uncompressed) image.

https://github.com/astropy/astropy/blob/a32e8565405c88c333192ae4d3d3b43216fdd4be/docs/io/fits/appendix/history.rst#321-2014-03-04 mentions this in PyFITS 3.2.1:

Fixed checksums on compressed images, so that the ZHECKSUM and ZDATASUM contain a checksum of the original image HDU, while CHECKSUM and DATASUM contain checksums of the compressed image HDU. This feature was supposed to be supported in 3.2, but the support was buggy.

This seems more logical to me.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the changelog entry, I think we do want CompImageHDU.header (which is the decompressed/image header) to contain the image checksum. This should then get stored as ZHECKSUM on disk. And if one opens the file disabling the image compression, BinTableHDU.header will contain the BinTableHDU checksum, stored as CHECKSUM on disk. Does this make sense?

I'll double check that this is the actual behaviour though.

Comment thread astropy/io/fits/hdu/hdulist.py Outdated
Comment thread astropy/io/fits/hdu/hdulist.py Outdated
@astrofrog
Copy link
Copy Markdown
Member Author

@mhvk - thanks for the review! Will go through and implement shortly.

@saimn - given that there are only 9 days left until feature freeze, I am thinking maybe it would be best not to rush this in, given that we are still catching up with bug reports from the refactoring of the compression C module. The release schedule now allows us to have 7.0.0 after 6.1.0 so maybe it makes sense to plan to merge this very soon after 6.1.0 is released to allow more time for people to try out the developer version and do follow-up PRs etc. So this would mean delaying merging this by just over 6 months, and then planning to have another major release in about a year.

@saimn
Copy link
Copy Markdown
Contributor

saimn commented Oct 18, 2023

@astrofrog - yes, makes sense and seems a good plan. We should merge this kind of PR at the beginning of a release cycle so we have to test internally and fix a few things if needed.

@cmarmo
Copy link
Copy Markdown
Member

cmarmo commented Jan 15, 2024

Hello everybody,
I've taken some time to check whether this PR is solving a number of issues about CompImageHDU.
Here a list of issues I have taken into account which are still failing on main: #5999 (however the error is not the same as described in the issue itself), #10512, #12216, #14081, #14611.
After merging with main locally, I have run this PR against the examples offered in the issues above.
Four of them seem solved by this approach 🥳 .
However #14611 is not.

When running

from astropy.io import fits
import numpy as np

data = np.arange(21 * 33).reshape((21, 33)).astype(np.int32)
header = fits.Header()
hdu = fits.CompImageHDU(data, header, compression_type="RICE_1", tile_shape=(5, 6))
print(hdu.section[0, 1])

The following error is thrown

Traceback (most recent call last):
...
    print(hdu.section[0, 1])
          ~~~~~~~~~~~^^^^^^
  File "/home/cmarmo/software/astropy/astropy/io/fits/hdu/compressed/section.py", line 116, in __getitem__
    self.hdu._bintable.data,
    ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'data'

If I understand correctly this is due to _get_bin_table_without_data not initializing self._bintable here
I was tempted to add something there like

            self._bintable = self._get_bintable_without_data()
            if data is not None:
                self._bintable.data = self._add_data_to_bintable(self._bintable)

But then I hit some new obstacles in _check_compressed_header....
Is section still usable here?
Is the image or the table data which should be returned in that case?

@pllim pllim added this to the v7.0.0 milestone Jan 16, 2024
@cmarmo
Copy link
Copy Markdown
Member

cmarmo commented Jan 21, 2024

Is the image or the table data which should be returned in that case?

I am under the impression that hdu.section() should return uncompressed image data.
In that case as CompImageHDU here inherits from ImageHDU would it be helpful to have CompImageSection inheriting from Section?

@mhvk
Copy link
Copy Markdown
Contributor

mhvk commented Mar 1, 2024

@astrofrog - I saw you answered some (most?) of the comments - let me know if this needs review again (since it is a large one, and I don't really remember anything from my last review, I'll wait till you are done!)

@astrofrog
Copy link
Copy Markdown
Member Author

@mhvk will do! I still need to address the remaining comments and push my changes but ran out of time for this week :)

@astrofrog astrofrog force-pushed the compimagehdu-inheritance branch 2 times, most recently from 30d1eb5 to c6e1874 Compare March 4, 2024 14:16
@astrofrog astrofrog force-pushed the compimagehdu-inheritance branch from 1b218d3 to ee51849 Compare March 12, 2024 10:29
@astrofrog
Copy link
Copy Markdown
Member Author

pre-commit.ci autofix

@pllim
Copy link
Copy Markdown
Member

pllim commented Mar 12, 2024

doctest failure in oldest deps looks related:

___________________________ [doctest] unfamiliar.rst ___________________________
489 ..
490   EXAMPLE START
491   Accessing Compressed FITS Image HDU Headers
492 
493 The content of the decompressed HDU header may be accessed using the ``.header`` attribute::
494 
495     >>> filename = fits.util.get_testdata_filepath('compressed_image.fits')
496 
497     >>> hdul = fits.open(filename)
498     >>> hdul[1].header
Differences (unified diff with -expected +actual):
    @@ -1,7 +1,7 @@
    -XTENSION= 'IMAGE   '           / Image extension
    -BITPIX  =                   16 / data type of original image
    -NAXIS   =                    2 / dimension of original image
    -NAXIS1  =                   10 / length of original image axis
    -NAXIS2  =                   10 / length of original image axis
    -PCOUNT  =                    0 / number of parameters
    -GCOUNT  =                    1 / number of groups
    +XTENSION= 'IMAGE   '           / Image extension                                
    +BITPIX  =                   16 / data type of original image                    
    +NAXIS   =                    2 / dimension of original image                    
    +NAXIS1  =                    8 / width of table in bytes                        
    +NAXIS2  =                   10 / number of rows in table                        
    +PCOUNT  =                   60 / number of group parameters                     
    +GCOUNT  =                    1 / number of groups                               

docs/io/fits/usage/unfamiliar.rst:498: DocTestFailure

@astrofrog astrofrog force-pushed the compimagehdu-inheritance branch from be79a57 to a10f1d5 Compare March 12, 2024 14:14
@astrofrog
Copy link
Copy Markdown
Member Author

@cmarmo - thank you for investigating which bugs were fixed by this PR! I have now added regression tests for all these bugs in this PR and indeed most now pass. However, now it is #5999 which is still present, so I have xfailed that test. I started looking into it but it is a complex issue and not a regression in this PR, so I would like to keep it for a separate standalone PR.

@astrofrog
Copy link
Copy Markdown
Member Author

pre-commit.ci autofix

@astrofrog astrofrog marked this pull request as ready for review March 12, 2024 14:33
@astrofrog astrofrog requested a review from saimn as a code owner March 12, 2024 14:33
@astrofrog
Copy link
Copy Markdown
Member Author

astrofrog commented Mar 12, 2024

This PR is now complete in terms of everything I want to include/fix, and all tests pass - with the exception of #15474 (comment).

As mentioned in #15474 (comment) we will want to wait until 6.1 has been branched before merging this, but in the meantime it would be great to go through the review cycle so that we can have it ready to merge as soon as 6.1 is out, and maximize how long it lives in main for testing.

cc @saimn @mhvk

@astrofrog astrofrog requested a review from mhvk March 12, 2024 14:36
@astrofrog astrofrog force-pushed the compimagehdu-inheritance branch from c8771e6 to ae3a32d Compare March 12, 2024 15:04
@astrofrog
Copy link
Copy Markdown
Member Author

@pllim - that failure should be fixed now. It was actually an important bug which wasn't caught by the main test suite so I have added a proper test too.

@astrofrog
Copy link
Copy Markdown
Member Author

Ok there are now a few other test failures - I clearly need to work on this more before a review, switching back to draft.

@astrofrog astrofrog marked this pull request as draft March 12, 2024 15:08
@astrofrog astrofrog force-pushed the compimagehdu-inheritance branch from bda155e to a3543f1 Compare September 17, 2024 13:39
@astrofrog
Copy link
Copy Markdown
Member Author

(I rebased to pick up any recent CI changes/fixes)

@saimn
Copy link
Copy Markdown
Contributor

saimn commented Sep 19, 2024

Didn't have time to answer yet, sorry, but while using this branch and working on something else I notice that fitsinfo with a big compressed file triggers a full decompression because we now inherit ImageHDU._summary() which uses .data. So we might need to redefine ._summary() to make sure the data is not loaded for fitsinfo.

@astrofrog
Copy link
Copy Markdown
Member Author

I've started working on dropping the internal table if the full data is decompressed, there are test failures which I'll look into later, just wanted to push the work so far to not lose it.

@astrofrog
Copy link
Copy Markdown
Member Author

@saimn - ok so I've spent some more time on this to try and address the remaining issues. First, I wasn't able to get rid of the bintable completely when the data is fully decompressed, because we need to keep a reference to the bintable to know things like _data_start and so on which are used in the lazy HDU loading. If you imagine loading a file with multiple compressed HDUs, and you first access the first HDU and decompress the data, by the time you are ready to read the next HDU, you still need to know about the underlying BinTableHDU to get the offset right in case the file object has been changed in the meantime. However, I have done the following which hopefully will be sufficient:

  • 669a433 - this ensures that when update_header gets called we don't force the whole dataset to be decompressed, and also avoids a recursion when accessing CompImageHDU.data which sets CompImageHDU.data if decompression happens.
  • c84b9e4 - adds a test along the lines of the example you suggested @saimn, essentially making sure we can modify the decompressed data inplace and have it work
  • 75406db - if decompressing the full data, write it to self.data. Note that we also need to make sure we save and restore the orig_* attributes, otherwise we lose scaling information.
  • d48eb90 - this gets rid of _decompression_active and instead replaces it with _data_loaded which now indicates whether or not the data has been fully decompressed (or has been set directly via .data). This then fixes the _summary issue. This usage of _data_loaded is slightly different to on the other HDUs, but is conceptually the same since it indicates whether the data lives in CompImageHDU.data or in some object behind the scenes.

@saimn - does this seem reasonable?

Copy link
Copy Markdown
Contributor

@saimn saimn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a last look and I think we are good to go ! 🎉
Many thanks @astrofrog, not an easy one but really great to finally do this !

I just pushed a little change, removing the _data_modified attribute which is now unused. Let's wait for CI and if you can just cross check.

@saimn
Copy link
Copy Markdown
Contributor

saimn commented Sep 30, 2024

Ok, let's merge and close a few issues ! :)

@saimn saimn merged commit 9b9b0fe into astropy:main Sep 30, 2024
@astrofrog
Copy link
Copy Markdown
Member Author

Thank you for the reviews and helping push this forward!

@Cadair
Copy link
Copy Markdown
Member

Cadair commented Oct 1, 2024

Amazing work both @astrofrog and also @saimn for reviewing this monster, I am very excited to start being able to use this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

8 participants