Skip to content

fix: Correct Flink PyArrow dependency constraints#6604

Open
zelinewang wants to merge 2 commits into
feast-dev:masterfrom
zelinewang:fix/flink-pyarrow-extra
Open

fix: Correct Flink PyArrow dependency constraints#6604
zelinewang wants to merge 2 commits into
feast-dev:masterfrom
zelinewang:fix/flink-pyarrow-extra

Conversation

@zelinewang

Copy link
Copy Markdown

The Flink extra currently combines a base pyarrow>=21.0.0 requirement with a
pyarrow<21.0.0 extra requirement. The attempted extra != 'flink' exclusion
does not prevent pip or uv from evaluating both requirements, so
feast[flink] is unsatisfiable.

This change puts the shared pyarrow>=16.1.0 lower bound in the base
dependencies and adds only pyarrow<21.0.0 to the Flink extra. A normal Feast
install can still select the latest PyArrow release, while a Flink install gets
the compatible >=16.1.0,<21.0.0 intersection.

The dependency regression check now lives in a focused test module that does
not import the Flink runtime.

Which issue(s) this PR fixes

Fixes #6583.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s).
  • My PR title follows conventional commits format.

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Validated with:

  • the focused dependency test on Python 3.10;
  • dry-run uv resolution for both .[flink] and the default project;
  • a wheel build and inspection of its Requires-Dist metadata;
  • Ruff check and format verification.

Release note

fix: Make the Flink extra installable by applying its PyArrow upper bound only when the extra is selected.

@zelinewang
zelinewang requested a review from a team as a code owner July 14, 2026 20:22

@shuchu shuchu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@shuchu

shuchu commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

the CI pipeline fails, @zelinewang can you try to update the "pixi.lock" file by running the following command?

pixi lock

@ntkathole

Copy link
Copy Markdown
Member

Also run make lock-python-dependencies-all to update requirements

@zelinewang

Copy link
Copy Markdown
Author

done, ran pixi lock plus the requirements recompile for all three python versions. heads up the pybuild-deps step fails because faiss-cpu==1.14.3 has no sdist on pypi and that pin is already in master's minimal-sdist files, so I left the three -build.txt files untouched. also the fresh resolve moves pyarrow to 21.0.0 on py3.10 (24 on 3.11/3.12), both within the corrected floor

@shuchu

shuchu commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

here is the error info:

pixi install -e duckdb-tests --locked
  /home/runner/.pixi/bin/pixi install -e duckdb-tests --locked --manifest-path pyproject.toml --color always
  Error:   × Lock-file version 7 is newer than supported
    help: Maximum supported version: 6 (pixi v0.63.1)
          Cannot continue with --locked or --frozen mode as the lock-file cannot be read.
          Try running `pixi self-update` to update to the latest version.

Comment thread pixi.lock Outdated
@zelinewang

Copy link
Copy Markdown
Author

oops, my local pixi was 0.73 which writes the v7 format. redid it with an older one, back to v6 now and the diff is way smaller. also fixed the missing DCO sign-off

@zelinewang
zelinewang force-pushed the fix/flink-pyarrow-extra branch from d3fc474 to 3309da1 Compare July 17, 2026 23:53
@ntkathole

Copy link
Copy Markdown
Member

@zelinewang needs rebase on master, there are conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extra != does not work as expected

3 participants