Skip to content

Limit the size and download time of feeds fetched by the planet aggregator - #314

Merged
mraible merged 1 commit into
masterfrom
planet-fetch-limits
Nov 18, 2026
Merged

mraible merged 1 commit into
masterfrom
planet-fetch-limits

Conversation

@mraible

@mraible mraible commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

The planet feed fetcher now has one deadline for the whole fetch and a size limit for the body. Before, only the connection and response headers had timeouts.

  • planet.aggregator.fetch.maxBytes (default 5 MB) caps the body; a Content-Length over it is rejected before reading.
  • planet.aggregator.fetch.maxSeconds (default 30) covers connecting, the headers and the body, so a feed that stalls or arrives slowly is cancelled.
  • Feed URLs other than http and https fail with a FetcherException that says so. Local weblogger: feeds are handled before this, as before.

The new properties are in roller.properties, next to the other planet.aggregator.* settings.

Tests: RomeFeedFetcherLimitsTest runs an in-JVM HTTP server and covers a normal feed, oversized feeds (with and without Content-Length), a slow and a stalled feed, and a non-HTTP URL. RomeFeedFetcherLocalTest and WebloggerRomeFeedFetcherTest still pass, and mvn test passes (755 tests).

CHANGES entries for this and the related fixes will come in one PR.

…gator

The planet feed fetcher now stops reading a feed once it passes planet.aggregator.fetch.maxBytes (5 MB by default) and gives up on a feed that hasn't fully arrived within planet.aggregator.fetch.maxSeconds (30 seconds by default). Before, only the connection and response headers had timeouts, so a large or slowly sent feed could hold up the feed updater. A Content-Length over the limit is rejected before the body is read.

Feed URLs other than http and https now fail with a FetcherException that says so, instead of an IllegalArgumentException from the HTTP client.

@snoopdave snoopdave left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of fff56e2a4: The size and download limits are applied in the feed fetch path, and the new tests cover normal, oversized, slow, stalled, and non-HTTP responses. PR #326 has no substantive fix hunk missing here; its syntax differences match the Java 11 branch. I found no verified issue requiring a change.

No issues found.

@mraible
mraible requested a review from snoopdave November 18, 2026 03:21

@snoopdave snoopdave left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

@mraible
mraible merged commit 976f1fa into master Nov 18, 2026
9 checks passed
@mraible
mraible deleted the planet-fetch-limits branch November 18, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants