Skip to content

feat: add TwelveLabs Marengo embedder for cross-modal video/text search#1936

Open
mohit-twelvelabs wants to merge 1 commit into
docarray:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat: add TwelveLabs Marengo embedder for cross-modal video/text search#1936
mohit-twelvelabs wants to merge 1 commit into
docarray:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

What this adds

A new opt-in helper, docarray.utils.twelvelabs.TwelveLabsEmbedder, that populates the embedding field of VideoDoc (and text queries) using TwelveLabs Marengo multimodal embeddings.

Marengo maps text and video into the same 512-dimensional vector space, so a text-query embedding can be compared directly against video embeddings. That makes it a natural fit for DocArray's find / document-index cross-modal search:

from docarray import DocList
from docarray.documents import VideoDoc
from docarray.utils.twelvelabs import TwelveLabsEmbedder

embedder = TwelveLabsEmbedder()  # reads TWELVELABS_API_KEY from the env

docs = DocList[VideoDoc]([VideoDoc(url="https://example.com/clip.mp4")])
embedder.embed_docs(docs)            # fills docs[*].embedding in place

q = embedder.embed_query("a dog catching a frisbee")  # same 512-d space

Why it helps this project

DocArray already models video as a first-class type (VideoDoc) and ships vector-search backends, but leaves embedding generation to the user. This wires up a production multimodal video embedder so users can go from VideoDoc to searchable vectors without leaving DocArray, and unlocks true cross-modal (text-to-video) retrieval.

Opt-in / non-breaking

  • Nothing imports twelvelabs unless you use the helper; it's gated behind a new twelvelabs extra (pip install "docarray[twelvelabs]") using the same import_library pattern as the other optional integrations.
  • No existing defaults or behavior change.

How it was tested

  • No-network unit tests (fake client) cover text embedding, video-segment averaging, in-place embed_docs, the TextDoc query path, and the missing-URL / missing-key error cases — these run in CI without any credentials.
  • Live tests gated on TWELVELABS_API_KEY (skipped when unset). I ran both locally against the real API with twelvelabs==1.2.8: the text-embedding test and the video-embedding test (Big Buck Bunny sample) both pass and return 512-d vectors.
  • black -S, isort --profile black, and ruff are clean on the changed files.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

@mohit-twelvelabs
mohit-twelvelabs force-pushed the feat/twelvelabs-integration branch from caed0f7 to 1320dff Compare June 24, 2026 23:02
@mohit-twelvelabs

Copy link
Copy Markdown
Author

Pushed 1320dff to address the failing CI:

  • DCO: amended the commit with Signed-off-by: Mohit Varikuti <[email protected]> — now passing.
  • check-black: reformatted docarray/utils/twelvelabs.py with black 22.10.0 (the pinned version) — the @overload stub bodies needed ... on their own line. Passing.
  • lint-ruff / import-test: the real cause was poetry.lock being out of sync after adding the twelvelabs optional dependency, so poetry install bailed with "twelvelabs (>=1.2.8) doesn't match any versions". Regenerated the lock with poetry lock --no-update (Poetry 1.8.x, matching the lock's generator) — only twelvelabs and its dep pydantic-core were added, no other churn. Both jobs are green now.

The embedder import stays lazy (import_library('twelvelabs') behind the twelvelabs extra), so from docarray import DocList, BaseDoc works without the extra installed, and the helper raises the standard pip install "docarray[twelvelabs]" hint otherwise. Verified locally: black --check, ruff, and the 6 no-network unit tests all pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant