Skip to content

[fix](be) Backport BE fixes and refinements to branch-4.1#65754

Open
Mryange wants to merge 4 commits into
apache:branch-4.1from
Mryange:branch-4.1-pick-group-a
Open

[fix](be) Backport BE fixes and refinements to branch-4.1#65754
Mryange wants to merge 4 commits into
apache:branch-4.1from
Mryange:branch-4.1-pick-group-a

Conversation

@Mryange

@Mryange Mryange commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Backport a small group of BE correctness fixes, column refinements, decimal cast optimization, and Arrow input validation to branch-4.1.

Picked changes:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Mryange added 4 commits July 17, 2026 11:29
### What problem does this PR solve?

Add an explicit block check to reject null column or type pointers at
operator sink/get_block boundaries, while keeping the existing type
compatibility check unchanged.

### Release note

None

(cherry picked from commit 9816a5a)
### What problem does this PR solve?

`filter_by_selector` reads from the source column and writes selected
rows into a destination column, so the source column should be accessed
through a const interface. Root cause: the interface was non-const, and
`ColumnNullable::filter_by_selector` wrote the destination nested column
through a `const_cast` on the destination nullable internals. This
change makes the interface const, updates the supported column
implementations, removes the nullable `const_cast`, and keeps
`ColumnDictionary` read-only by using a local temporary `StringRef`
buffer.

### Release note

None

(cherry picked from commit e23b403)
### What problem does this PR solve?

Integer-to-DecimalV3 casts with zero scale previously went through the
generic decimal cast path even when the target decimal precision can
represent the full integer input range. For example, casting an INT
column to DECIMALV3(10, 0) still used the common _from_int helper, which
computes decimal scaling and range-related values that are unnecessary
when the scale is zero and the cast cannot narrow the integer range.

Root cause: the DecimalV3 cast implementation did not have a direct fast
path for non-narrowing integer casts to zero-scale decimal types.

This change adds a direct zero-scale DecimalV3 path for integer and
boolean inputs when the target decimal range is not narrower than the
input range. The fast path writes the input value directly into the
decimal native value and preserves the existing generic path for
narrowing casts, non-zero-scale casts, and overflow-sensitive cases.

Local optest profiling for:

select sum(cast(quantity as decimalv3(10,0))) from q14_avg_expr_100m;

showed the cast expression time improving from about 119.0 ms to about
109.7 ms on 100M rows, roughly an 8% reduction in this expression
counter.

(cherry picked from commit 5d56602)
…he#64796)

### What problem does this PR solve?

Doris converts Arrow arrays into Doris columns through
`DataTypeSerDe::read_column_from_arrow`. If the Arrow producer sends
malformed array metadata, such as truncated validity bitmaps, truncated
offsets buffers, non-monotonic string/list offsets, or offsets pointing
past the child/value buffer, the existing conversion code may read
invalid Arrow memory and crash BE.

Root cause: the Arrow-to-Doris serde path trusted Arrow array metadata
before accessing Arrow buffers. Several hot paths call `IsNull()`,
`Value()`, raw value offsets, list offsets, or child arrays directly, so
malformed Arrow buffers can trigger out-of-bounds reads before Doris
reports a clean error.

This PR adds lightweight, type-specific Arrow input validation before
those buffer accesses. The checks are modeled as local preflight checks
rather than full `ValidateFull()`: validity bitmap size, fixed-width
data buffer size, boolean bitmap size, binary/string offsets buffer
size, per-value data range, and list/map offsets monotonicity plus child
length bounds. A BE config `enable_arrow_input_validation` is added and
defaults to `true`.

The change also fixes an existing `FixedSizeBinaryArray` sliced-read
null check: the loop uses a relative index after `GetValue(start)`, but
`IsNull()` expects the original Arrow row index, so it must check `start
+ offset_i`.

| Type | Rows | Check disabled | Check enabled | Overhead |
| --- | ---: | ---: | ---: | ---: |
| String | 4096 | 39,352 ns | 42,032 ns | +6.8% |
| String | 65536 | 604,782 ns | 624,064 ns | +3.2% |
| Int64 | 4096 | 1,480 ns | 1,523 ns | +2.9% |
| Int64 | 65536 | 16,160 ns | 15,883 ns | -1.7% |
| Boolean | 4096 | 4,784 ns | 4,888 ns | +2.2% |
| Boolean | 65536 | 79,017 ns | 80,453 ns | +1.8% |
| ArrayString | 4096 | 139,747 ns | 147,793 ns | +5.8% |
| ArrayString | 65536 | 2,384,377 ns | 2,477,601 ns | +3.9% |
| MapStringInt | 4096 | 84,375 ns | 96,022 ns | +13.8% |
| MapStringInt | 65536 | 2,742,538 ns | 2,903,358 ns | +5.9% |

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

(cherry picked from commit 43a31ac)
@Mryange
Mryange requested a review from yiguolei as a code owner July 17, 2026 06:47
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange

Mryange commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 83.86% (291/347) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.51% (22690/40151)
Line Coverage 40.27% (222126/551559)
Region Coverage 36.58% (175277/479178)
Branch Coverage 37.66% (78239/207748)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants