Skip to content

TST: add regression tests for issue #12216#16019

Merged
saimn merged 2 commits into
astropy:mainfrom
neutrinoceros:io/fits/bug/remember_compression_options
Oct 7, 2024
Merged

TST: add regression tests for issue #12216#16019
saimn merged 2 commits into
astropy:mainfrom
neutrinoceros:io/fits/bug/remember_compression_options

Conversation

@neutrinoceros
Copy link
Copy Markdown
Contributor

@neutrinoceros neutrinoceros commented Feb 9, 2024

Description

Fixes #12216

I note that #12058 was listed a potential partial solution, and I haven't dug deeply into it, but found that it didn't resolve the issue. Hopefully this approach is suitable.
If not, I've listed other possible venues, see #12216 (comment)

Blocked by #15474

update 2024-10-03: the bug was fixed in #15474, so I revamped this PR to only add the test suite I wrote for it.

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 9, 2024

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • Is a milestone set? Milestone must be set but we cannot check for it on Actions; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 9, 2024

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

@neutrinoceros neutrinoceros force-pushed the io/fits/bug/remember_compression_options branch from 75ecc94 to 6cd7b50 Compare February 9, 2024 13:02
@neutrinoceros neutrinoceros marked this pull request as ready for review February 9, 2024 13:27
@neutrinoceros neutrinoceros requested a review from saimn as a code owner February 9, 2024 13:27
@saimn
Copy link
Copy Markdown
Contributor

saimn commented Feb 9, 2024

The FITS compression code is being heavily refactored in #15474, so let's ping @astrofrog :)

@saimn saimn requested a review from astrofrog February 9, 2024 18:45
@neutrinoceros
Copy link
Copy Markdown
Contributor Author

Oh, I wasn't aware, thanks for noticing ! if it helps, maybe we should switch this one to draft mode for now ?

@pllim pllim added this to the v6.0.1 milestone Feb 9, 2024
@pllim
Copy link
Copy Markdown
Member

pllim commented Feb 9, 2024

The bug this PR is fixing was reported back in 5.x. Looks like this PR is good to backport, but #15474 is not till v7 at the earliest, so if Tom R is okay with this patch, we should accept it just so we do not have to wait till v7 for a fix?

@saimn saimn modified the milestones: v6.0.1, v6.0.2 Mar 25, 2024
@neutrinoceros neutrinoceros force-pushed the io/fits/bug/remember_compression_options branch from 6cd7b50 to 0e4778e Compare April 3, 2024 13:11
@neutrinoceros
Copy link
Copy Markdown
Contributor Author

I rebased to refresh CI. Let's actually ping @astrofrog while I'm here.

@astrofrog astrofrog modified the milestones: v6.0.2, v6.1.1 Apr 4, 2024
@neutrinoceros
Copy link
Copy Markdown
Contributor Author

@astrofrog does this need to wait for #15474 ?

@neutrinoceros neutrinoceros force-pushed the io/fits/bug/remember_compression_options branch from 0e4778e to 8a179e8 Compare June 16, 2024 10:42
@astrofrog
Copy link
Copy Markdown
Member

Ideally yes otherwise it will not be a trivial conflict to resolve.

@neutrinoceros neutrinoceros changed the title BUG: fix a bug where compression options passed to fits.CompImageHDU would be lost if no data was inputted at instantiation BUG: fix a bug where compression options passed to fits.CompImageHDU would be lost if no data was inputted at instantiation (⏰ wait for #15474) Jun 16, 2024
@neutrinoceros
Copy link
Copy Markdown
Contributor Author

you got it

@neutrinoceros neutrinoceros marked this pull request as draft June 17, 2024 09:47
@saimn saimn modified the milestones: v6.1.1, v6.1.2 Jun 17, 2024
@astrofrog astrofrog removed this from the v6.1.2 milestone Jul 25, 2024
@neutrinoceros neutrinoceros changed the title BUG: fix a bug where compression options passed to fits.CompImageHDU would be lost if no data was inputted at instantiation (⏰ wait for #15474) BUG: fix a bug where compression options passed to fits.CompImageHDU would be lost if no data was inputted at instantiation Oct 2, 2024
@saimn
Copy link
Copy Markdown
Contributor

saimn commented Oct 2, 2024

@astrofrog - do you think those tests would be useful ?

@astrofrog
Copy link
Copy Markdown
Member

I guess we might as well, though not sure if we need it to be that extensive - maybe we can just check the compression options in the compressed header that was written out? (Or maybe it wouldn't be enough to illustrate the problem?)

@neutrinoceros
Copy link
Copy Markdown
Contributor Author

would the following assertion match your suggestion @astrofrog ? It appears to pass on main and fail on v6.1.4, which I think indicates that it captures the regression ?

assert hdul[1].header["BITPIX"] == hdul2[1].header["BITPIX"]

@astrofrog
Copy link
Copy Markdown
Member

Yes partially but I am thinking more along the lines of checking e.g. ZVAL?/ZNAME?/ZCMPTYPE/ZQUANTIZ which are the actual options you are trying to set.

@neutrinoceros
Copy link
Copy Markdown
Contributor Author

None of these options seemed to appear in the headers I considered while looking into this. Maybe I'm not looking in the right place ?

@astrofrog
Copy link
Copy Markdown
Member

@neutrinoceros - are you opening the file disabling the compression in order to access the header? (these will just be in the compressed header):

In [1]: import numpy as np

In [2]: from astropy.io import fits

In [3]: hdu  = fits.CompImageHDU(np.random.random((128, 128)))

In [5]: hdu.writeto('test.fits')

In [6]: hdu2 = fits.open('test.fits', disable_image_compression=True)[1]

In [7]: hdu2.header
Out[7]: 
XTENSION= 'BINTABLE'           / binary table extension                         
BITPIX  =                    8 / array data type                                
NAXIS   =                    2 / number of array dimensions                     
NAXIS1  =                   32 / width of table in bytes                        
NAXIS2  =                  128 / number of rows in table                        
PCOUNT  =                14585 / number of group parameters                     
GCOUNT  =                    1 / number of groups                               
TFIELDS =                    4 / number of fields in each row                   
TTYPE1  = 'COMPRESSED_DATA'    / label for field 1                              
TFORM1  = '1PB(122)'           / data format of field: variable length array    
TTYPE2  = 'GZIP_COMPRESSED_DATA' / label for field 2                            
TFORM2  = '1PB(0)  '           / data format of field: variable length array    
TTYPE3  = 'ZSCALE  '           / label for field 3                              
TFORM3  = '1D      '           / data format of field: 8-byte DOUBLE            
TTYPE4  = 'ZZERO   '           / label for field 4                              
TFORM4  = '1D      '           / data format of field: 8-byte DOUBLE            
ZIMAGE  =                    T / extension contains compressed image            
ZTENSION= 'IMAGE   '           / Image extension                                
ZBITPIX =                  -64 / data type of original image                    
ZNAXIS  =                    2 / dimension of original image                    
ZNAXIS1 =                  128 / length of original image axis                  
ZNAXIS2 =                  128 / length of original image axis                  
ZPCOUNT =                    0 / number of parameters                           
ZGCOUNT =                    1 / number of groups                               
ZTILE1  =                  128 / size of tiles to be compressed                 
ZTILE2  =                    1 / size of tiles to be compressed                 
ZCMPTYPE= 'RICE_1  '           / compression algorithm                          
ZNAME1  = 'BLOCKSIZE'          / compression block size                         
ZVAL1   =                   32 / pixels per block                               
ZNAME2  = 'BYTEPIX '           / bytes per pixel (1, 2, 4, or 8)                
ZVAL2   =                    4 / bytes per pixel (1, 2, 4, or 8)                
ZNAME3  = 'NOISEBIT'           / floating point quantization level              
ZVAL3   =                 16.0 / floating point quantization level              
ZQUANTIZ= 'NO_DITHER'          / No dithering during quantization               
EXTNAME = 'COMPRESSED_IMAGE'   / name of this binary table extension  

@neutrinoceros neutrinoceros force-pushed the io/fits/bug/remember_compression_options branch from 8a179e8 to 60c6d1f Compare October 3, 2024 12:29
@neutrinoceros neutrinoceros changed the title BUG: fix a bug where compression options passed to fits.CompImageHDU would be lost if no data was inputted at instantiation TST: add regression tests for issue #12216 Oct 3, 2024
@neutrinoceros
Copy link
Copy Markdown
Contributor Author

Thanks ! I cleaned up this PR and now it's all about adding my tests

@neutrinoceros neutrinoceros force-pushed the io/fits/bug/remember_compression_options branch from 60c6d1f to e052853 Compare October 3, 2024 12:33
@neutrinoceros neutrinoceros marked this pull request as ready for review October 3, 2024 12:33
"ZCMPTYPE",
"ZQUANTIZ",
]
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To save a few lines, maybe just for k in expected_zflags?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@neutrinoceros neutrinoceros force-pushed the io/fits/bug/remember_compression_options branch from e052853 to 12158ed Compare October 3, 2024 14:03
@pllim pllim added testing no-changelog-entry-needed and removed Bug Close? Tell stale bot that this issue/PR is stale labels Oct 3, 2024
@pllim pllim modified the milestones: v6.1.5, v7.0.0 Oct 3, 2024
@pllim
Copy link
Copy Markdown
Member

pllim commented Oct 3, 2024

Thanks! I updated the labels and also moved the milestone to match #15474

@pllim
Copy link
Copy Markdown
Member

pllim commented Oct 3, 2024

Ugh the wheel jobs got stuck. Maybe I need to prioritize fixing the labeled event for that job.

Copy link
Copy Markdown
Contributor

@saimn saimn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few suggestions

Comment thread astropy/io/fits/hdu/compressed/tests/test_compressed.py Outdated
Comment thread astropy/io/fits/hdu/compressed/tests/test_compressed.py Outdated
data = prng.random((50, 30)).astype("float32")

hdu = fits.CompImageHDU(**init_kwargs)
hdu.data = data.copy()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy seems useless ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. It's a remnant of the initial bug report, but it's not necessary to reproduce the problem. Thanks for catching it !

@neutrinoceros neutrinoceros force-pushed the io/fits/bug/remember_compression_options branch from 44e9a6c to 4d6a72f Compare October 4, 2024 06:58
Copy link
Copy Markdown
Contributor

@saimn saimn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, thanks @neutrinoceros !

@saimn saimn merged commit ab90d93 into astropy:main Oct 7, 2024
@neutrinoceros neutrinoceros deleted the io/fits/bug/remember_compression_options branch October 7, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compression settings ignored when creating empty CompImageHDU

4 participants