[fix](be) Backport BE fixes and refinements to branch-4.1#65754
Open
Mryange wants to merge 4 commits into
Open
[fix](be) Backport BE fixes and refinements to branch-4.1#65754Mryange wants to merge 4 commits into
Mryange wants to merge 4 commits into
Conversation
### What problem does this PR solve? Add an explicit block check to reject null column or type pointers at operator sink/get_block boundaries, while keeping the existing type compatibility check unchanged. ### Release note None (cherry picked from commit 9816a5a)
### What problem does this PR solve? `filter_by_selector` reads from the source column and writes selected rows into a destination column, so the source column should be accessed through a const interface. Root cause: the interface was non-const, and `ColumnNullable::filter_by_selector` wrote the destination nested column through a `const_cast` on the destination nullable internals. This change makes the interface const, updates the supported column implementations, removes the nullable `const_cast`, and keeps `ColumnDictionary` read-only by using a local temporary `StringRef` buffer. ### Release note None (cherry picked from commit e23b403)
### What problem does this PR solve? Integer-to-DecimalV3 casts with zero scale previously went through the generic decimal cast path even when the target decimal precision can represent the full integer input range. For example, casting an INT column to DECIMALV3(10, 0) still used the common _from_int helper, which computes decimal scaling and range-related values that are unnecessary when the scale is zero and the cast cannot narrow the integer range. Root cause: the DecimalV3 cast implementation did not have a direct fast path for non-narrowing integer casts to zero-scale decimal types. This change adds a direct zero-scale DecimalV3 path for integer and boolean inputs when the target decimal range is not narrower than the input range. The fast path writes the input value directly into the decimal native value and preserves the existing generic path for narrowing casts, non-zero-scale casts, and overflow-sensitive cases. Local optest profiling for: select sum(cast(quantity as decimalv3(10,0))) from q14_avg_expr_100m; showed the cast expression time improving from about 119.0 ms to about 109.7 ms on 100M rows, roughly an 8% reduction in this expression counter. (cherry picked from commit 5d56602)
…he#64796) ### What problem does this PR solve? Doris converts Arrow arrays into Doris columns through `DataTypeSerDe::read_column_from_arrow`. If the Arrow producer sends malformed array metadata, such as truncated validity bitmaps, truncated offsets buffers, non-monotonic string/list offsets, or offsets pointing past the child/value buffer, the existing conversion code may read invalid Arrow memory and crash BE. Root cause: the Arrow-to-Doris serde path trusted Arrow array metadata before accessing Arrow buffers. Several hot paths call `IsNull()`, `Value()`, raw value offsets, list offsets, or child arrays directly, so malformed Arrow buffers can trigger out-of-bounds reads before Doris reports a clean error. This PR adds lightweight, type-specific Arrow input validation before those buffer accesses. The checks are modeled as local preflight checks rather than full `ValidateFull()`: validity bitmap size, fixed-width data buffer size, boolean bitmap size, binary/string offsets buffer size, per-value data range, and list/map offsets monotonicity plus child length bounds. A BE config `enable_arrow_input_validation` is added and defaults to `true`. The change also fixes an existing `FixedSizeBinaryArray` sliced-read null check: the loop uses a relative index after `GetValue(start)`, but `IsNull()` expects the original Arrow row index, so it must check `start + offset_i`. | Type | Rows | Check disabled | Check enabled | Overhead | | --- | ---: | ---: | ---: | ---: | | String | 4096 | 39,352 ns | 42,032 ns | +6.8% | | String | 65536 | 604,782 ns | 624,064 ns | +3.2% | | Int64 | 4096 | 1,480 ns | 1,523 ns | +2.9% | | Int64 | 65536 | 16,160 ns | 15,883 ns | -1.7% | | Boolean | 4096 | 4,784 ns | 4,888 ns | +2.2% | | Boolean | 65536 | 79,017 ns | 80,453 ns | +1.8% | | ArrayString | 4096 | 139,747 ns | 147,793 ns | +5.8% | | ArrayString | 65536 | 2,384,377 ns | 2,477,601 ns | +3.9% | | MapStringInt | 4096 | 84,375 ns | 96,022 ns | +13.8% | | MapStringInt | 65536 | 2,742,538 ns | 2,903,358 ns | +5.9% | ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> (cherry picked from commit 43a31ac)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Backport a small group of BE correctness fixes, column refinements, decimal cast optimization, and Arrow input validation to branch-4.1.
Picked changes:
filter_by_selectorconst.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)