Repository navigation
Conversation
A render now stops with a logged error, and the request fails as for other rendering errors, once it writes more than rendering.maxOutputChars characters (10 MB by default) or runs longer than rendering.maxRenderSeconds (30 by default). The output writer, #foreach, method calls, iteration and every inserted value check these limits, so loops that write nothing and strings built up in memory stop as well. #foreach also stops after 100,000 passes, and String.repeat is not offered to templates. While a weblog is rendered, #parse and #include load only that weblog's own stored templates. The public error pages no longer show template ids; they stay in the log, and the message names the template instead. Macros defined inline in a template are now local to that template and the render it starts, so two weblogs can define macros of the same name and an edited macro takes effect on the next render. Library macros stay global, and macros still reach between a page and the templates it includes.
snoopdave
left a comment
There was a problem hiding this comment.
Review of 86f204f6a: The fix hunks match #317; the javax/jakarta import difference is appropriate for this branch. One string method can allocate past the configured size limit before the new check runs. No build was run, as requested.
…utput limit before they run String.replace, replaceAll, replaceFirst, concat, toUpperCase and toLowerCase, $utils.replace, and StringBuilder and StringBuffer growth methods now work out the longest string they can build before running, and the render stops with the output limit error when that is over the limit. String.format, formatted, join and indent are not offered to templates, like String.repeat, along with any other String method returning a string whose length can't be worked out from the call. Ordinary replaces and other string calls work as before.
…ide the render limits String literals such as "$a$b", the + operator, and #define blocks or macro bodies used as strings now stop at the output limit while the string is built, through a parser (RenderLimitParser, set as parser.class) that swaps in nodes which count what they write into a buffer and check the length of two strings before + joins them. Variables set from a template may hold no string longer than the limit. Every variable read and write and every macro call now checks the render deadline, so a recursive macro that writes nothing and calls no methods stops in time, and regular expressions given to String.matches, replaceAll, replaceFirst and split check the deadline as they run, so a slow expression stops with the render. RollerResourceLoader now refuses to start with resource.loader.roller.cache=true, which each weblog rendering only its own templates relies on.
…he context checks
replaceAll and replaceFirst now check the output limit before each replacement instead of refusing any call whose worst case over the whole string is over the limit, so a replace with group references such as $s.replaceAll('(\d)', '<$1>') on a long string works again. The arguments of the regular expression methods are converted the way Velocity converts them.
RenderLimitContext is removed: every block render already writes to the counted output, which checks the deadline, so a macro that writes nothing still stops in time, and strings built by literals and + are now checked where they are built.
|
A second review raised a few more points, now addressed in the latest commits:
|
snoopdave
left a comment
There was a problem hiding this comment.
🐞Codex Issue: 1 blocking inline finding on the current head.
…pieces that are kept The nodes RenderLimitParser swaps in were built after the parser had let go of the template it was parsing, so they had no template. Velocity asks a node for its template name when it reports a problem, which turned harmless templates such as #set($x = $undefined + 1) and a macro that recurses too deeply into a NullPointerException. The replacement nodes now take the template from the node they replace. A split with the default limit of 0 drops trailing empty pieces, but the piece count included them, so a string ending in many delimiters was refused even though split would have returned few pieces or none. DeadlineRegex now does the split itself in one pass, holding empty pieces back until a piece that isn't empty follows, so trailing empty pieces are neither counted nor allocated and the expression only runs once. The comments above the render limit settings in velocity.properties now describe each setting once.
snoopdave
left a comment
There was a problem hiding this comment.
Review of a1ddd1e4d: The current code addresses both earlier review points. The fix hunks match #317; the test support file uses the expected javax import for this branch. Two string paths still allocate past the configured output limit before checking it.
Earlier comments: 2 of 2 resolved.
…ase changes The pre-call output bound covered $utils.replace only, so $utils.escapeHTML, escapeXML, escapeJavaScript, encode, hexEncode, autoformat and the other String-returning $utils methods could build a result many times the output limit before the length check ran. StringGrowth now bounds each of them before the call by counting only the characters that method can expand, and refuses a $utils method added later that returns a string until it is listed. The case-change bound was three times the length, which refused toUpperCase on a large ASCII string whose result fits the limit. It now adds two characters only for each character outside ASCII. DeadlineRegex also checks the text after the last replacement against the limit before appendTail copies it, instead of after.
The escapeHTML and escapeXML bounds counted every character outside ASCII as an entity, so large text in Cyrillic, CJK and most other scripts, which those escapers leave alone, was refused. StringGrowth now finds the characters each escaper expands by escaping each char once, and counts only those.
RollerResourceLoader turned each stored template into bytes in the template encoding and read them back with the platform's default charset, which changes text outside ASCII when the two differ. The template is already a string, so it is now read with a StringReader.
RenderLimitParser moved each node's children to its replacement from the first to the last, so the child array grew by one on every move. They are now moved last first, and the array is allocated once.
A template error can quote a value from the template in full, such as a long string passed to a method. The log line and the error page now show the first 1,000 characters, and the log line stays on one line.
The bounds for toUpperCase and toLowerCase counted every character outside ASCII as tripling, and escapeHTML and escapeXML charged every escaped character with the longest entity, so large Cyrillic or CJK text, or HTML near the limit, was refused although the result would fit. Each character's growth is now looked up in a table built by running the methods on it, in the locales whose case rules differ. The date format test now checks every locale. Its pattern used letters SimpleDateFormat rejects, so it had been checking none.
… string, and count only the groups a replacement uses - An expression with a part that can match empty in more than one way at the same place can try a very large number of ways without reading the text, so the deadline check in DeadlineText never runs. checkPattern refuses a group or assertion that can match empty followed by a quantifier, and a group with two alternatives that can match empty. Pattern itself is used to try whether a part can match empty. - replaceAll and replaceFirst with a null replacement now run under the deadline too, and fail as the String methods do. - A replacement is counted by the lengths of the groups it refers to, so a replace that moves text around in a string near the limit is no longer refused. - String.splitWithDelimiters and lines aren't offered to templates, since DeadlineRegex doesn't run them. - The comments no longer say Velocity converts arguments. It finds no method for a call whose arguments don't fit, so the conversions for those calls are gone.
Velocity's directive.foreach.max_loops ends a loop without an error, so a page cut short by it was served and cached as if it were whole. RollerForeach now fails the render on the pass after the 100,000th, like the other limits, and the property is gone from velocity.properties. A loop of exactly 100,000 passes still completes. The tests' check that Roller's limit, not the test's own ceiling, stopped a render followed getCause, which RenderingException doesn't set, so it never saw the ceiling. It now follows the root cause.
…ps and numbers before a call runs The output limit stops what a template writes, but not what it keeps in memory, or what one call builds before its result is checked. - A render may build at most ten times rendering.maxOutputChars characters of strings, counted where they are built (string literals, + and string calls), and add at most 1,000,000 items to lists and maps or get them from split calls. A session started inside another one shares its budget. - A list or map is measured as text, without building the text, before it is inserted, passed to toString, String.valueOf or +, or compared with a string by == or !=. - addAll and putAll are checked against the item budget before they copy anything. containsAll, removeAll and retainAll are refused when they would compare more than 10,000,000 pairs of elements. - Numbers stop at 100,000 bits: pow, the shifts, the bit setters and * are checked before they run, and primality tests aren't offered. - $utils.split is refused when it would return more pieces than a #foreach can visit, as String.split already was. - A Class object offers only getName, so String.format and join can't be reached as static methods. - An error from a regular expression call, such as a replacement naming a group that doesn't exist, shows the template error page again instead of failing the page. - A blank or non-numeric rendering.maxOutputChars or rendering.maxRenderSeconds falls back to the default with a warning instead of failing every page, and the limit messages say which limit was reached.
addNofollow now adds nofollow to every link, by adding a rel or adding to the link's own one, so the old bound, which only counted links that already had rel="nofollow", was no longer an upper bound. Each "<a href=" now counts as gaining at most ' rel="nofollow"'.
The #335 entry now says that a long #foreach fails the render rather than stopping, and lists the budget for strings, list items and numbers and the calls templates can't use. Macro scoping gets its own entry, which says what a custom theme that calls a macro from another template sees and how to fix it.
… items - A map entry, such as one from entrySet(), is measured as key=value like a map, so a large list inside one is refused before it is printed. - toArray and clone on a list or map count the items they copy against the budget before they run. - StringBuilder.append and insert count a list or map passed as an Object by its text. - Only addAll and putAll count the items of a list passed to them, so adding a large list as one item is one item. - Whether part of a regular expression can match empty no longer depends on sample texts: lookarounds, anchors and boundaries count as matching empty, and the part is also tried in comments mode, which an inline (?x) may have turned on. - The tests use sizes just over the limits.
The 6.1.x version of #317. Five changes to how weblog templates render:
rendering.maxOutputChars(10 MB by default) or runs longer thanrendering.maxRenderSeconds(30 by default). The output writer,#foreach(aRollerForeachdirective), method calls, property reads, iteration and inserted values all check the limits. A#foreachof more than 100,000 passes fails the render too; Velocity's owndirective.foreach.max_loopsended the loop silently, so it's no longer set. The secure uberspector still applies; the newRenderLimitUberspectoris chained after it.RenderSessionand shared by any session started inside it: at most ten timesmaxOutputCharscharacters of strings built (string literals,+,#defineblocks used as strings, string calls), and at most 1,000,000 items added to lists and maps or returned by split calls. Calls whose cost comes inside one call are checked before they run:toString,String.valueOf,+orStringBuilder.append, or compared with a string by==or!=;addAll,putAll,toArrayandcloneare checked against the budget before copying, andcontainsAll,removeAllandretainAllare refused past 10,000,000 comparisons;pow, shifts, bit setters and*are checked first), and primality tests aren't offered;$utils.splitandString.splitare refused past 100,000 pieces.String.replace,concat,toUpperCaseandtoLowerCase, the expanding$utilsmethods, andStringBuilderandStringBuffergrowth work out the longest string they can build before they run, counting only the characters that really grow.replaceAllandreplaceFirstcheck the output limit before each replacement, counting the groups the replacement refers to. The regular expression methods check the deadline as they run, and an expression that repeats a part that can match an empty string, which can backtrack without reading the text, is refused.String.repeat,format,formatted,join,indent,linesandsplitWithDelimiters, any otherStringmethod whose result length can't be worked out from the call, and the methods of aClassobject other thangetName(which would reach static methods) aren't available to templates. An error from a regular expression call shows the template error page as before.#parseand#includeload only that weblog's own stored templates; others are not found.RollerResourceLoaderrefuses to start withresource.loader.roller.cache=true, since this relies on it being off, and reads stored templates as text. The public error pages no longer show template ids (they stay in the log), and long error messages are shortened.velocimacro.inline.local_scope=truewithinline.replace_global=true: macros defined inline in a template are local to that template and the render it starts, so two weblogs can use the same macro name and an edited macro takes effect on the next render. Library macros (roller.vm,weblog.vmand a site's own library such asroller-custom.vm) stay global, and a page can still call macros from a template it#parses.$utils.addNofollowadds at mostrel="nofollow"to each<a href=link on 6.1.x, and is bounded that way. The bundled themes define no inline macros, use only plain regular expressions, and call noClassmethods other thangetName(in the error pages).Tests:
RenderLimitsTestandWeblogTemplateScopeTest,StringGrowthTestandDeadlineRegexTest. Of the 16 render tests first added in this pass, 15 failed on the code before it (checked on master); the other checks that small lists and maps still print. All 28 test classes underui/renderingpass (158 tests), compiled for Java 11. The CHANGES entries are updated, including a call-out for the macro scoping: a custom theme that calls a macro defined in a template it doesn't#parsenow prints the call as text.