Compare map keys recursively in recursive comparisons - #4417

Fixes #3835

This makes map keys follow the recursive comparison configuration by default, as discussed for 4.0.0. Keys with equal fields can match across types, ignored fields and custom comparators apply to keys, and differences hidden by a key's overridden equals are detected unless overridden equality is explicitly requested.

I gave the entry-based approach a try. It grew beyond the small change I initially expected, so I'm opening this as a draft to get feedback on the structure and tradeoffs.

Approach

The comparison works with Map.Entry objects but compares their keys and values explicitly. This preserves existing value paths and avoids treating an entry as an ordinary JDK object or swapping key/value roles when collection order is ignored. Sorted maps retain ordered comparison, including when ignored entries are filtered out. No public option or runtime dependency is added.

For unordered maps, matching has to consider the whole entry. If ignoring id makes two keys equivalent, Sam/1 -> red, Sam/2 -> blue must match Sam/3 -> blue, Sam/4 -> red. Each entry is used once, and an augmenting-path search can revise an earlier choice when a comparator allows overlapping matches. Hashes prioritize candidates but never exclude a recursive match. When a full match fails, a key-only pass retains missing-key diagnostics and value differences at their existing paths.

What made this less straightforward

• Shared references and field rules: reusing a result obtained at an ignored field could hide a real difference elsewhere. Key/value trials now have separate comparison state, inherit only ancestor cycle guards, and restore temporary type-selection locations. Diagnostic values are also compared independently.
• Memory versus repeated work: caching every candidate pair exhausted a 128 MiB heap on a 1,500-entry mismatch probe. Removing all reuse then expanded a small shared-map DAG repeatedly. The current cache retains at most 1,024 completed successful object comparisons by identity. It is disabled for field-path rules and selected fields/types, and never caches a success that relied on an ancestor cycle guard. Full matching also stops as soon as an entry cannot be matched, even after reassignment.
• Deep nesting: calling the recursive engine directly for each map value overflowed the Java stack on a 1,000-map chain. An explicit stack of comparison frames now suspends and resumes nested map comparisons and matching. This is the main source of additional internal complexity.

The cache is deliberately conservative. Path-dependent rules, cycle-dependent results and eviction can still cause expensive repeated work, potentially exponential over a shared graph. For one map, worst-case matching is O(n³ × C), where C includes nested comparison cost; this is not a bound for the entire object graph. Comparators are assumed to return stable results. Ambiguous matches also do not have a unique diagnostic pairing.

Validation

• Added 108 regression cases across the existing fields and legacy strategies; all 116 map cases pass. Coverage includes ambiguous matching, reassignment, nulls, cycles, type selection, shared references, sorted order, deep nesting and repeated expansion.
• Ran ./mvnw -pl assertj-tests/assertj-integration-tests/assertj-core-tests -am clean verify with JDK 25 and local JVM argument/encoding overrides for Windows paths. The latest run passed: 20,819 passed, 68 skipped, no failures. Before this update, an earlier run had one failure in the unchanged properties.RecursiveComparisonAssert_isEqualTo_Test.should_be_faster_the_second_time_as_the_getter_introspection_is_cached (0 ms < 0 ms); upstream subsequently disabled that flaky timing test in 7ca2715, which is now the base of this branch. The entire monorepo reactor was not run.
• Verified the new regressions fail before their fixes: four depth cases overflow without frames, and 22 sorted-map cases falsely pass before the size-filtering fix. All 116 map cases pass with the final changes.
• Spotless and license checks passed for both affected modules.
• Local probes: a shared 20-map DAG now compares its leaf twice rather than 1,048,576 times; a 1,000-map chain completes with a 1 MiB stack and 1 GiB heap; a 10,000-entry mismatch completes with a 128 MiB heap. These are regression checks, not benchmark guarantees. Temporary type-selection state is also restored after a comparator throws.

Review follow-up

The change is now split into three commits, following Joel's feedback:

1. Recursive map-key comparison and entry matching, including the existing conservative cache.
2. Frame-based execution and the deep-nesting regressions. These first two commits together preserve the previously published implementation.
3. A separate fix for different-sized sorted maps when only key or value types are selected. If the map-level size diagnostic is filtered out, ordered entry comparison continues and unmatched keys and values are checked against null. Unselected types and ignored entries remain filtered out.

The bounded cache and its limitations are unchanged; the cache discussion remains open for later review. This remains a draft for feedback on the implementation.

Check List:

• Fixes RecursiveComparisonConfiguration is not taken into account for map keys #3835
• Unit tests: YES
• Javadoc with a code example (on API only): NA — no public API added
• Code and tests follow the contributing conventions; validation caveat is noted above.