Skip to content

fix: add fixture for dataproc batch and better test cleanup - #8262

Merged
leahecole merged 9 commits into
mainfrom
fix_8254
Aug 18, 2022
Merged

leahecole merged 9 commits into
mainfrom
fix_8254

Conversation

@leahecole

@leahecole leahecole commented Aug 15, 2022

Copy link
Copy Markdown
Collaborator

Description

#8021 was similar but was throwing the Aborted error, this is giving a 409 AlreadyExists which was happening when it would retry but already find a batch there. I've added teardown logic to hopefully reduce the likelihood that an AlreadyExists happens.

Fixes #8254

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

@leahecole
leahecole requested a review from bradmiro August 15, 2022 14:37
@product-auto-label product-auto-label Bot added the samples Issues that are directly related to samples. label Aug 15, 2022
@leahecole leahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 15, 2022
@leahecole
leahecole marked this pull request as ready for review August 15, 2022 19:58
@leahecole
leahecole requested review from a team, rachael-ds and rafalbiegacz as code owners August 15, 2022 19:58
@leahecole leahecole added the blunderbuss: assign Instruct blunderbuss to assign someone label Aug 15, 2022
@blunderbuss-gcf blunderbuss-gcf Bot removed the blunderbuss: assign Instruct blunderbuss to assign someone label Aug 15, 2022
@leahecole leahecole changed the title fix: add second exception to backoff decorator fix: add fixture for dataproc batch and better test cleanup Aug 15, 2022
# Retry if we see a flaky 409 "subnet not ready" exception
@backoff.on_exception(backoff.expo, Aborted, max_tries=3)
def test_process(test_bucket):
def test_process(test_bucket, test_dataproc_batch):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am definitely rusty on my pytest but are you possibly creating two test buckets by referencing it in both bq_dataset and test_process?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are both defined as scope="module" so will only create the object once in this case.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! What Charlie said 😁

@leahecole
leahecole requested a review from bradmiro August 16, 2022 13:11
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 16, 2022
@engelke engelke assigned leahecole and unassigned engelke Aug 16, 2022
@leahecole

Copy link
Copy Markdown
Collaborator Author

Ugh, I don't see this failure locally but I'll debug in a bit and see if I can fix it

@leahecole
leahecole merged commit d4d26d4 into main Aug 18, 2022
@leahecole
leahecole deleted the fix_8254 branch August 18, 2022 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

samples Issues that are directly related to samples.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

composer.2022_airflow_summit.data_analytics_process_test: test_process failed

5 participants