Skip to content

Instrument Vercel AI experimental_evaluate #24692

Description

@andreiborza

[email protected] added experimental_evaluate. In [email protected], it also publishes on the ai:telemetry tracing channel with type: 'experimental_evaluate' (vercel/ai#21162). Our v7 channel subscriber ignores unknown event types, so these calls make no span today.

Scope

  • Change only the v7 channel subscriber (packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts). The function does not exist in ai v4 to v6.
  • Make one span for each call. The channel does not publish the inner model call.

Target span

  • Name: evaluate <modelId>, op gen_ai.evaluate, origin auto.vercelai.channel
  • gen_ai.operation.name: evaluate, gen_ai.provider.name (raw provider ID, for example typesafe.evaluation), gen_ai.request.model, gen_ai.response.model
  • gen_ai.usage.input_tokens, output_tokens, total_tokens
  • gen_ai.input.messages: [{ type: 'evaluation', state, questions }] (only when recordInputs is on)
  • gen_ai.output.messages: [{ type: 'evaluation', answers }] (only when recordOutputs is on)
  • Keep the shared vercel.ai.* attributes that other Vercel channel spans have.

Notes

  • @sentry/conventions has no gen_ai.evaluate op yet.
  • Add a node-integration test. It needs ai >= 7.0.111 in the test app.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions