[email protected] added experimental_evaluate. In [email protected], it also publishes on the ai:telemetry tracing channel with type: 'experimental_evaluate' (vercel/ai#21162). Our v7 channel subscriber ignores unknown event types, so these calls make no span today.
Scope
- Change only the v7 channel subscriber (
packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts). The function does not exist in ai v4 to v6.
- Make one span for each call. The channel does not publish the inner model call.
Target span
- Name:
evaluate <modelId>, op gen_ai.evaluate, origin auto.vercelai.channel
gen_ai.operation.name: evaluate, gen_ai.provider.name (raw provider ID, for example typesafe.evaluation), gen_ai.request.model, gen_ai.response.model
gen_ai.usage.input_tokens, output_tokens, total_tokens
gen_ai.input.messages: [{ type: 'evaluation', state, questions }] (only when recordInputs is on)
gen_ai.output.messages: [{ type: 'evaluation', answers }] (only when recordOutputs is on)
- Keep the shared
vercel.ai.* attributes that other Vercel channel spans have.
Notes
@sentry/conventions has no gen_ai.evaluate op yet.
- Add a node-integration test. It needs
ai >= 7.0.111 in the test app.
[email protected]addedexperimental_evaluate. In[email protected], it also publishes on theai:telemetrytracing channel withtype: 'experimental_evaluate'(vercel/ai#21162). Our v7 channel subscriber ignores unknown event types, so these calls make no span today.Scope
packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts). The function does not exist inaiv4 to v6.Target span
evaluate <modelId>, opgen_ai.evaluate, originauto.vercelai.channelgen_ai.operation.name: evaluate,gen_ai.provider.name(raw provider ID, for exampletypesafe.evaluation),gen_ai.request.model,gen_ai.response.modelgen_ai.usage.input_tokens,output_tokens,total_tokensgen_ai.input.messages: [{ type: 'evaluation', state, questions }](only whenrecordInputsis on)gen_ai.output.messages: [{ type: 'evaluation', answers }](only whenrecordOutputsis on)vercel.ai.*attributes that other Vercel channel spans have.Notes
@sentry/conventionshas nogen_ai.evaluateop yet.ai >= 7.0.111in the test app.