Skip to content

Added ability to define a private _validate_card method on Header subclasses#15293

Closed
astrofrog wants to merge 1 commit into
astropy:mainfrom
astrofrog:fits-header-validation
Closed

Added ability to define a private _validate_card method on Header subclasses#15293
astrofrog wants to merge 1 commit into
astropy:mainfrom
astrofrog:fits-header-validation

Conversation

@astrofrog
Copy link
Copy Markdown
Member

This adds a way for header subclasses to define a private _validate_card method that can be used to e.g. emit warnings or errors depending on the card. It is intended for use in CompImageHeader in an upcoming refactor but I thought it would be easier to propose this as a standalone PR. It is private API so does not require a changelog entry.

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Sep 8, 2023

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • Is a milestone set? Milestone must be set but we cannot check for it on Actions; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

Copy link
Copy Markdown
Contributor

@mhvk mhvk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In itself, something like this makes sense. But I think we need to think a bit about the API: here you return True/False. Another option would be to let _validate raise if it wants.

But I think I like the concept of ScienceState.validate and Attribute.convert_input (latter in combination with __get__) more, where the method returns the value, possibly adjusted to match what is needed. How about doing something similar here? I could see this being useful more broadly (even, e.g., in turning a float into a string for regular cards; don't remember how it is done now...)

@astrofrog
Copy link
Copy Markdown
Member Author

@mhvk - in the use case I have for this in compressed FITS, I want to emit a warning if a card is invalid and then proceed silently. If we followed your suggestion, would it be acceptable for _validate_card to return None to indicate that the card did not validate correctly? Or should _validate_card raise an exception that we then catch and re-emit as a warning? (which will add some complexity to the code)

@mhvk
Copy link
Copy Markdown
Contributor

mhvk commented Sep 8, 2023

@astrofrog - I think it can still up to the use case how _validate_card behaves for bad input - I think returning None and then in the code below just ignoring it would be fine. I mostly like the ability to convert value to something else that is more directly useful.

@saimn
Copy link
Copy Markdown
Contributor

saimn commented Sep 8, 2023

That's for _check_compressed_header ?
I'm not sure about this new method, since io.fits already has a verification system ;). Any reason not to use that ?

@astrofrog
Copy link
Copy Markdown
Member Author

@saimn - no this is going to be used in CompImageHeader which currently does validation on the fly as keywords are set/updated. This PR will allow that class to be dramatically simplified in my refactor and avoid any code duplication while preserving the validation (without this PR we have to duplicate several of the Header methods just to add validation). My understanding of the current verify system is that it is not going to be called every time a header key/value pair is set. Note also that this is just private API, and I wasn't intending on advertising it.

If you would prefer, I could just do this as part of the PR to refactor CompImageHDU to inherit from ImageHDU but I thought it would be cleaner to do separately.

@astrofrog
Copy link
Copy Markdown
Member Author

I've implemented @mhvk's comments but we can continue discussing whether this is needed.

An alternative would be to break existing behavior and no longer warn if a user sets/updates a keyword/value pair in a CompImageHDU header and instead do this only with the standard verification framework e.g. at read/write time. This would allow us to get rid of CompImageHDU completely but it is a change in behavior. @saimn what do you think?

Comment thread astropy/io/fits/header.py Outdated
f"(keyword, value, [comment]) tuple; got: {card!r}"
)

if self._validate_card(card) is None:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the new scheme this should become

card = self._validate_card(card)
if card is None:
    return

Otherwise any changes made would not actually be used! (same above)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or is _validate_card allowed to make in-place changes to the card? That would definitely make sense too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be allowed to return a new card so have updated following your suggestion

Comment thread astropy/io/fits/tests/test_header.py Outdated
if card.comment is not None and card.comment.lower() != card.comment:
warnings.warn("Comment should be lowercase, fixing", UserWarning)
card.comment = card.comment.lower()
return card
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, return card should not be indented (or there should be another clause with a return at the end).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@mhvk
Copy link
Copy Markdown
Contributor

mhvk commented Sep 13, 2023

I cannot really comment on the overall question, but it seems to me that perhaps that is better answered anyway after refactoring, especially if that will make CompImageHDU much simpler!

@astrofrog
Copy link
Copy Markdown
Member Author

@mhvk @saimn - I'm happy to delay this until after the refactoring PR, or could simply incorporate it as part of the refactoring PR if you prefer.

@astrofrog astrofrog force-pushed the fits-header-validation branch from 7e2f21a to 41918ee Compare September 14, 2023 19:11
@mhvk
Copy link
Copy Markdown
Contributor

mhvk commented Sep 15, 2023

@astrofrog - What is here now definitely looks good to me.

On the larger question, I guess there is a general point that one preferably should not introduce new private features without also merging something that uses it.

So, perhaps the next step is to make a PR with your refactor, build on top of this one (probably best reduced to a single commit)? That would allow us to see the usage in practice!

…classes to do validation on keyword names, values, or comments
@astrofrog astrofrog force-pushed the fits-header-validation branch from 41918ee to 4c39610 Compare September 15, 2023 13:10
@astrofrog
Copy link
Copy Markdown
Member Author

@mhvk - sounds good, I've squashed this down to a single commit and will work on opening the refactor PR.

@saimn
Copy link
Copy Markdown
Contributor

saimn commented Sep 26, 2023

Sorry @astrofrog I'm still not convinced by this addition 😅 in parallel to the existing verification system.
The verification system already does various checks and fixes, that are quite similar to the example you put in the unit tests (HeaderWithValidation), e.g. https://github.com/astropy/astropy/blob/main/astropy/io/fits/card.py#L1110

My understanding of the current verify system is that it is not going to be called every time a header key/value pair is set. Note also that this is just private API, and I wasn't intending on advertising it.

Right, this is the main difference, the verification system is not called when setting a card. Currently I think there are mainly two ways to verify the cards:

  • when reading a file and calling .verify() or when writing with output_verify, this is the verification system.
  • any other checks when creating HDUs etc. are directly emitting warnings.

All of this is not perfect and has many flaws, but I'm not sure that adding another way to validate cards will help to clarify the situation ;). And yes it's private but adding this to Header means it could be used anywhere in io.fits, which is probably my main objection to this PR. Maybe you could put it on CompImageHeader first with the refactor PR, and then we discuss more how to integrate/improve/refactor the verification system ?

@astrofrog
Copy link
Copy Markdown
Member Author

Ok sounds good, let's close this for now and I will focus on opening the refactor PR instead for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants