Skip to content

Bound weblog template rendering and keep templates and macros per weblog - #335

Open
mraible wants to merge 19 commits into
roller-6.1.xfrom
template-limits-6.1.x
Open

mraible wants to merge 19 commits into
roller-6.1.xfrom
template-limits-6.1.x

Conversation

@mraible

@mraible mraible commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

The 6.1.x version of #317. Five changes to how weblog templates render:

  • Render limits. A render stops with a logged error, and the request fails like any other rendering error (so the page isn't cached), once it writes more than rendering.maxOutputChars (10 MB by default) or runs longer than rendering.maxRenderSeconds (30 by default). The output writer, #foreach (a RollerForeach directive), method calls, property reads, iteration and inserted values all check the limits. A #foreach of more than 100,000 passes fails the render too; Velocity's own directive.foreach.max_loops ended the loop silently, so it's no longer set. The secure uberspector still applies; the new RenderLimitUberspector is chained after it.
  • What a render builds in memory. Each render also has a budget, kept in RenderSession and shared by any session started inside it: at most ten times maxOutputChars characters of strings built (string literals, +, #define blocks used as strings, string calls), and at most 1,000,000 items added to lists and maps or returned by split calls. Calls whose cost comes inside one call are checked before they run:
    • a list, map or map entry is measured as text, without building it, before it's inserted, passed to toString, String.valueOf, + or StringBuilder.append, or compared with a string by == or !=;
    • addAll, putAll, toArray and clone are checked against the budget before copying, and containsAll, removeAll and retainAll are refused past 10,000,000 comparisons;
    • numbers stop at 100,000 bits (pow, shifts, bit setters and * are checked first), and primality tests aren't offered;
    • $utils.split and String.split are refused past 100,000 pieces.
  • String calls. String.replace, concat, toUpperCase and toLowerCase, the expanding $utils methods, and StringBuilder and StringBuffer growth work out the longest string they can build before they run, counting only the characters that really grow. replaceAll and replaceFirst check the output limit before each replacement, counting the groups the replacement refers to. The regular expression methods check the deadline as they run, and an expression that repeats a part that can match an empty string, which can backtrack without reading the text, is refused. String.repeat, format, formatted, join, indent, lines and splitWithDelimiters, any other String method whose result length can't be worked out from the call, and the methods of a Class object other than getName (which would reach static methods) aren't available to templates. An error from a regular expression call shows the template error page as before.
  • Templates per weblog. While a weblog renders, #parse and #include load only that weblog's own stored templates; others are not found. RollerResourceLoader refuses to start with resource.loader.roller.cache=true, since this relies on it being off, and reads stored templates as text. The public error pages no longer show template ids (they stay in the log), and long error messages are shortened.
  • Inline macros per template. velocimacro.inline.local_scope=true with inline.replace_global=true: macros defined inline in a template are local to that template and the render it starts, so two weblogs can use the same macro name and an edited macro takes effect on the next render. Library macros (roller.vm, weblog.vm and a site's own library such as roller-custom.vm) stay global, and a page can still call macros from a template it #parses.

$utils.addNofollow adds at most rel="nofollow" to each <a href= link on 6.1.x, and is bounded that way. The bundled themes define no inline macros, use only plain regular expressions, and call no Class methods other than getName (in the error pages).

Tests: RenderLimitsTest and WeblogTemplateScopeTest, StringGrowthTest and DeadlineRegexTest. Of the 16 render tests first added in this pass, 15 failed on the code before it (checked on master); the other checks that small lists and maps still print. All 28 test classes under ui/rendering pass (158 tests), compiled for Java 11. The CHANGES entries are updated, including a call-out for the macro scoping: a custom theme that calls a macro defined in a template it doesn't #parse now prints the call as text.

A render now stops with a logged error, and the request fails as for other rendering errors, once it writes more than rendering.maxOutputChars characters (10 MB by default) or runs longer than rendering.maxRenderSeconds (30 by default). The output writer, #foreach, method calls, iteration and every inserted value check these limits, so loops that write nothing and strings built up in memory stop as well. #foreach also stops after 100,000 passes, and String.repeat is not offered to templates.

While a weblog is rendered, #parse and #include load only that weblog's own stored templates. The public error pages no longer show template ids; they stay in the log, and the message names the template instead.

Macros defined inline in a template are now local to that template and the render it starts, so two weblogs can define macros of the same name and an edited macro takes effect on the next render. Library macros stay global, and macros still reach between a page and the templates it includes.

@snoopdave snoopdave left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 86f204f6a: The fix hunks match #317; the javax/jakarta import difference is appropriate for this branch. One string method can allocate past the configured size limit before the new check runs. No build was run, as requested.

…utput limit before they run

String.replace, replaceAll, replaceFirst, concat, toUpperCase and toLowerCase, $utils.replace, and StringBuilder and StringBuffer growth methods now work out the longest string they can build before running, and the render stops with the output limit error when that is over the limit. String.format, formatted, join and indent are not offered to templates, like String.repeat, along with any other String method returning a string whose length can't be worked out from the call. Ordinary replaces and other string calls work as before.
…ide the render limits

String literals such as "$a$b", the + operator, and #define blocks or macro bodies used as strings now stop at the output limit while the string is built, through a parser (RenderLimitParser, set as parser.class) that swaps in nodes which count what they write into a buffer and check the length of two strings before + joins them. Variables set from a template may hold no string longer than the limit.

Every variable read and write and every macro call now checks the render deadline, so a recursive macro that writes nothing and calls no methods stops in time, and regular expressions given to String.matches, replaceAll, replaceFirst and split check the deadline as they run, so a slow expression stops with the render.

RollerResourceLoader now refuses to start with resource.loader.roller.cache=true, which each weblog rendering only its own templates relies on.
…he context checks

replaceAll and replaceFirst now check the output limit before each replacement instead of refusing any call whose worst case over the whole string is over the limit, so a replace with group references such as $s.replaceAll('(\d)', '<$1>') on a long string works again. The arguments of the regular expression methods are converted the way Velocity converts them.

RenderLimitContext is removed: every block render already writes to the counted output, which checks the deadline, so a macro that writes nothing still stops in time, and strings built by literals and + are now checked where they are built.
@mraible

mraible commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

A second review raised a few more points, now addressed in the latest commits:

  • The same additions as Bound weblog template rendering and keep templates and macros per weblog #317 are here: literals, + and blocks used as strings stop at the output limit while they are built, regex methods check the deadline as they run, and the stored template loader refuses to start with caching on. The decorator path is unchanged because velocityDecorator is never assigned on this branch either.

@mraible
mraible requested a review from snoopdave November 18, 2026 04:16

@snoopdave snoopdave left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐞Codex Issue: 1 blocking inline finding on the current head.

…pieces that are kept

The nodes RenderLimitParser swaps in were built after the parser had let go of the template it was parsing, so they had no template. Velocity asks a node for its template name when it reports a problem, which turned harmless templates such as #set($x = $undefined + 1) and a macro that recurses too deeply into a NullPointerException. The replacement nodes now take the template from the node they replace.

A split with the default limit of 0 drops trailing empty pieces, but the piece count included them, so a string ending in many delimiters was refused even though split would have returned few pieces or none. DeadlineRegex now does the split itself in one pass, holding empty pieces back until a piece that isn't empty follows, so trailing empty pieces are neither counted nor allocated and the expression only runs once.

The comments above the render limit settings in velocity.properties now describe each setting once.

@snoopdave snoopdave left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of a1ddd1e4d: The current code addresses both earlier review points. The fix hunks match #317; the test support file uses the expected javax import for this branch. Two string paths still allocate past the configured output limit before checking it.

Earlier comments: 2 of 2 resolved.

…ase changes

The pre-call output bound covered $utils.replace only, so $utils.escapeHTML, escapeXML, escapeJavaScript, encode, hexEncode, autoformat and the other String-returning $utils methods could build a result many times the output limit before the length check ran. StringGrowth now bounds each of them before the call by counting only the characters that method can expand, and refuses a $utils method added later that returns a string until it is listed.

The case-change bound was three times the length, which refused toUpperCase on a large ASCII string whose result fits the limit. It now adds two characters only for each character outside ASCII.

DeadlineRegex also checks the text after the last replacement against the limit before appendTail copies it, instead of after.
The escapeHTML and escapeXML bounds counted every character outside ASCII as an entity, so large text in Cyrillic, CJK and most other scripts, which those escapers leave alone, was refused. StringGrowth now finds the characters each escaper expands by escaping each char once, and counts only those.
RollerResourceLoader turned each stored template into bytes in the template encoding and read them back with the platform's default charset, which changes text outside ASCII when the two differ. The template is already a string, so it is now read with a StringReader.
RenderLimitParser moved each node's children to its replacement from the first to the last, so the child array grew by one on every move. They are now moved last first, and the array is allocated once.
A template error can quote a value from the template in full, such as a long string passed to a method. The log line and the error page now show the first 1,000 characters, and the log line stays on one line.
The bounds for toUpperCase and toLowerCase counted every character outside ASCII as tripling, and escapeHTML and escapeXML charged every escaped character with the longest entity, so large Cyrillic or CJK text, or HTML near the limit, was refused although the result would fit. Each character's growth is now looked up in a table built by running the methods on it, in the locales whose case rules differ.

The date format test now checks every locale. Its pattern used letters SimpleDateFormat rejects, so it had been checking none.
… string, and count only the groups a replacement uses

- An expression with a part that can match empty in more than one way at the same place can try a very large number of ways without reading the text, so the deadline check in DeadlineText never runs. checkPattern refuses a group or assertion that can match empty followed by a quantifier, and a group with two alternatives that can match empty. Pattern itself is used to try whether a part can match empty.
- replaceAll and replaceFirst with a null replacement now run under the deadline too, and fail as the String methods do.
- A replacement is counted by the lengths of the groups it refers to, so a replace that moves text around in a string near the limit is no longer refused.
- String.splitWithDelimiters and lines aren't offered to templates, since DeadlineRegex doesn't run them.
- The comments no longer say Velocity converts arguments. It finds no method for a call whose arguments don't fit, so the conversions for those calls are gone.
Velocity's directive.foreach.max_loops ends a loop without an error, so a page cut short by it was served and cached as if it were whole. RollerForeach now fails the render on the pass after the 100,000th, like the other limits, and the property is gone from velocity.properties. A loop of exactly 100,000 passes still completes.

The tests' check that Roller's limit, not the test's own ceiling, stopped a render followed getCause, which RenderingException doesn't set, so it never saw the ceiling. It now follows the root cause.
…ps and numbers before a call runs

The output limit stops what a template writes, but not what it keeps in memory, or what one call builds before its result is checked.

- A render may build at most ten times rendering.maxOutputChars characters of strings, counted where they are built (string literals, + and string calls), and add at most 1,000,000 items to lists and maps or get them from split calls. A session started inside another one shares its budget.
- A list or map is measured as text, without building the text, before it is inserted, passed to toString, String.valueOf or +, or compared with a string by == or !=.
- addAll and putAll are checked against the item budget before they copy anything. containsAll, removeAll and retainAll are refused when they would compare more than 10,000,000 pairs of elements.
- Numbers stop at 100,000 bits: pow, the shifts, the bit setters and * are checked before they run, and primality tests aren't offered.
- $utils.split is refused when it would return more pieces than a #foreach can visit, as String.split already was.
- A Class object offers only getName, so String.format and join can't be reached as static methods.
- An error from a regular expression call, such as a replacement naming a group that doesn't exist, shows the template error page again instead of failing the page.
- A blank or non-numeric rendering.maxOutputChars or rendering.maxRenderSeconds falls back to the default with a warning instead of failing every page, and the limit messages say which limit was reached.
addNofollow now adds nofollow to every link, by adding a rel or adding to the link's own one, so the old bound, which only counted links that already had rel="nofollow", was no longer an upper bound. Each "<a href=" now counts as gaining at most ' rel="nofollow"'.
The #335 entry now says that a long #foreach fails the render rather than stopping, and lists the budget for strings, list items and numbers and the calls templates can't use. Macro scoping gets its own entry, which says what a custom theme that calls a macro from another template sees and how to fix it.
… items

- A map entry, such as one from entrySet(), is measured as key=value like a map, so a large list inside one is refused before it is printed.
- toArray and clone on a list or map count the items they copy against the budget before they run.
- StringBuilder.append and insert count a list or map passed as an Object by its text.
- Only addAll and putAll count the items of a list passed to them, so adding a large list as one item is one item.
- Whether part of a regular expression can match empty no longer depends on sample texts: lookarounds, anchors and boundaries count as matching empty, and the part is also tried in comments mode, which an inline (?x) may have turned on.
- The tests use sizes just over the limits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants