feat: Created DocEmbedder class by patelchaitany · Pull Request #5973 · feast-dev/feast

patelchaitany · 2026-02-16T11:28:33Z

What this PR does / why we need it:

This PR adds a Document Embedder capability to Feast, allowing users to go from raw documents to embeddings stored in the online vector store in a single step. It handles chunking, embedding generation, and writing the results to the online store — providing an end-to-end ingestion pipeline for RAG workflows within Feast.

What changed:

sdk/python/feast/chunker.py

Defines the document chunking layer. Provides:

A BaseChunker abstract class that users can extend with custom chunking strategies
A built-in TextChunker that splits plain text by word count with configurable size, overlap, and thresholds
A chunk_dataframe() convenience method to chunk all documents in a DataFrame

Currently only basic text chunking is implemented. There is room for improvement — future iterations can support more advanced strategies like semantic chunking, sentence-aware splitting, or format-specific chunkers (PDF, HTML, etc.).

sdk/python/feast/embedder.py

Defines the embedding generation layer. Provides:

A BaseEmbedder abstract class with a modality-routing registry for extensibility
A MultiModalEmbedder with built-in support for text (via sentence-transformers) and image (via CLIP) embeddings
Lazy model loading and configurable batch processing

sdk/python/feast/doc_embedder.py

The high-level orchestrator that coordinates chunking, embedding, and storage. Provides:

A DocEmbedder class that runs the full pipeline: chunk -> embed -> logical layer -> write to online store
A user-defined logical layer function to map output to the FeatureView schema
Auto-generation of a default FeatureView if the user does not define their own (controlled by create_feature_view). This is a basic implementation intended as a starting point

sdk/python/feast/init.py

Updated to export DocEmbedder, LogicalLayerFn, BaseChunker, TextChunker, ChunkingConfig, BaseEmbedder, MultiModalEmbedder, and EmbeddingConfig as part of Feast's public API.

Which issue(s) this PR fixes:

Create DocEmbedder class along with RAGRetriever #5426

Misc

patelchaitany · 2026-02-16T11:53:20Z

@ntkathole @jyejare can You pls review this PR and let me know if any changes is needed.

jyejare

Great Addition @patelchaitany , this is a milestone for Feast in RAG. Glad to see multiple types of data are being supported by Embedder.

Few comments and we should be good to go.

sdk/python/tests/unit/test_doc_embedder.py

ntkathole · 2026-03-02T10:58:02Z

@patelchaitany filename typo - examples/rag-retriever/rag_feast_docebedder.ipynb should be rag_feast_docembedder.ipynb

sdk/python/feast/doc_embedder.py

sdk/python/feast/embedder.py

sdk/python/feast/chunker.py

sdk/python/feast/embedder.py

sdk/python/feast/doc_embedder.py

sdk/python/feast/embedder.py

sdk/python/feast/doc_embedder.py

pyproject.toml

sdk/python/feast/chunker.py

sdk/python/feast/doc_embedder.py

…ng them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]>

into fet-DocEmbedder

Signed-off-by: Chaitany patel <[email protected]>

devin-ai-integration

Devin Review found 1 new potential issue.

View 23 additional findings in Devin Review.

sdk/python/feast/doc_embedder.py

ntkathole

lgtm

franciscojavierarceo · 2026-03-16T13:16:03Z

sdk/python/feast/doc_embedder.py

+
+
+@runtime_checkable
+class LogicalLayerFn(Protocol):


Probably we should rename this as it's not particularly intuitive IMO

So i have two name in the mind - SchemaMapperFn or the FeatureViewMapperFn i think this two name are more intuitive if you have particular name in mind then let me know.

* - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * - Introduced DocEmbedder class for embedding documents and transforming them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <[email protected]> * resolving the merge conflict Signed-off-by: Chaitany patel <[email protected]> --------- Signed-off-by: Chaitany patel <[email protected]> Signed-off-by: aaronzuo <[email protected]>

patelchaitany requested a review from a team as a code owner February 16, 2026 11:28

This comment was marked as resolved.

Sign in to view

patelchaitany changed the title ~~feat: Created DocEmbedder class #5972~~ feat: Created DocEmbedder class Feb 16, 2026

This comment was marked as resolved.

Sign in to view

jyejare reviewed Feb 23, 2026

View reviewed changes

patelchaitany requested a review from jyejare February 24, 2026 10:01

This comment was marked as resolved.

Sign in to view

ntkathole force-pushed the fet-DocEmbedder branch from 1ef9d8e to 083eadb Compare March 2, 2026 05:15

ntkathole added the ok-to-test label Mar 2, 2026

ntkathole force-pushed the fet-DocEmbedder branch from 083eadb to bb74079 Compare March 2, 2026 07:52

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/chunker.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 2, 2026

View reviewed changes

sdk/python/feast/embedder.py Show resolved Hide resolved

ntkathole removed the ok-to-test label Mar 2, 2026

jyejare approved these changes Mar 3, 2026

View reviewed changes

patelchaitany force-pushed the fet-DocEmbedder branch from daf292f to fcc85cd Compare March 4, 2026 05:38

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch 2 times, most recently from e00ee22 to ee663f5 Compare March 5, 2026 06:31

ntkathole force-pushed the fet-DocEmbedder branch from ee663f5 to 282243b Compare March 6, 2026 15:06

ntkathole reviewed Mar 6, 2026

View reviewed changes

sdk/python/feast/chunker.py Outdated Show resolved Hide resolved

patelchaitany force-pushed the fet-DocEmbedder branch from de74a1f to 23cc11e Compare March 8, 2026 18:48

ntkathole reviewed Mar 9, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

ntkathole reviewed Mar 9, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch from cf300d8 to ae53f06 Compare March 9, 2026 10:15

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch from ae53f06 to e3b5e4e Compare March 9, 2026 11:01

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch 2 times, most recently from 05ee154 to 510c54a Compare March 9, 2026 12:22

This comment was marked as resolved.

Sign in to view

patelchaitany force-pushed the fet-DocEmbedder branch from 510c54a to 85ee6e2 Compare March 9, 2026 12:35

ntkathole force-pushed the fet-DocEmbedder branch from 85ee6e2 to 2c66417 Compare March 11, 2026 04:48

Merge branch 'fet-DocEmbedder' of https://github.com/patelchaitany/feast

7562dd0

into fet-DocEmbedder

patelchaitany force-pushed the fet-DocEmbedder branch from 720f0ce to 7562dd0 Compare March 11, 2026 05:28

This comment was marked as resolved.

Sign in to view

Merge branch 'master' into fet-DocEmbedder

f4439cc

This comment was marked as resolved.

Sign in to view

patelchaitany added 2 commits March 11, 2026 11:48

Merge branch 'fet-DocEmbedder' of https://github.com/patelchaitany/feast

d455968

into fet-DocEmbedder

Merge branch 'fet-DocEmbedder' of https://github.com/patelchaitany/feast

b4545aa

into fet-DocEmbedder

ntkathole added the ok-to-test label Mar 14, 2026

patelchaitany added 2 commits March 16, 2026 15:30

Merge remote-tracking branch 'origin/master' into fet-DocEmbedder

9fc01a8

resolving the merge conflict

30dddd7

Signed-off-by: Chaitany patel <[email protected]>

patelchaitany force-pushed the fet-DocEmbedder branch from 523bdfb to 30dddd7 Compare March 16, 2026 11:18

devin-ai-integration bot reviewed Mar 16, 2026

View reviewed changes

sdk/python/feast/doc_embedder.py Show resolved Hide resolved

ntkathole approved these changes Mar 16, 2026

View reviewed changes

ntkathole merged commit 0719c06 into feast-dev:master Mar 16, 2026
26 checks passed

franciscojavierarceo reviewed Mar 16, 2026

View reviewed changes

Conversation

patelchaitany commented Feb 16, 2026 • edited by devin-ai-integration bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

What changed:

sdk/python/feast/chunker.py

sdk/python/feast/embedder.py

sdk/python/feast/doc_embedder.py

sdk/python/feast/init.py

Which issue(s) this PR fixes:

Misc

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

patelchaitany commented Feb 16, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

jyejare left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

ntkathole commented Mar 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ntkathole left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

franciscojavierarceo Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

patelchaitany Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

patelchaitany commented Feb 16, 2026 •

edited by devin-ai-integration bot

Loading