Skip to content

Instrument TypeSafe Jev (@typesafe-ai/sdk) #24693

Description

@andreiborza

TypeSafe's official SDK (@typesafe-ai/sdk, latest 0.6.0) calls the Jev evaluation model with TypeSafeClient.systemOne({ state, questions, model? }). Users who call Jev directly (not through Vercel AI experimental_evaluate, see #24692) get no span today, only a plain http.client span from fetch.

Scope

  • Add a new AI integration in packages/server-utils/: logic in src/ai/typesafe/, integration in src/integrations/typesafe.ts, registered in getTracingIntegrations(). Re-export it from the runtime packages.
  • Instrument TypeSafeClient.prototype.systemOne. Make one span for each call, not one span for each retry.
  • models.list() is out of scope.

Target span

Use the same shape as the Vercel AI evaluate span in #24692, so that both paths look the same in the product.

  • Name: evaluate <model>, op gen_ai.evaluate, origin auto.ai.typesafe
  • gen_ai.operation.name: evaluate, gen_ai.provider.name: typesafe
  • gen_ai.request.model (the model option, or the client default, jev-latest), gen_ai.response.model (result.model, for example jev-1.13.0)
  • gen_ai.usage.input_tokens, output_tokens, total_tokens (from result.usage.input_tokens / output_tokens)
  • gen_ai.input.messages: [{ type: 'evaluation', state, questions }] (only when recordInputs is on)
  • gen_ai.output.messages: [{ type: 'evaluation', answers }] (only when recordOutputs is on)
  • Set the span status to error when the call throws (TypeSafeError and API errors).

Notes

  • The SDK sends requests with fetch to POST /v1/systemone. It has no tracing channel, so we must patch the client.
  • @sentry/conventions has no gen_ai.evaluate op yet (same gap as Instrument Vercel AI experimental_evaluate #24692).
  • Add a node-integration test with a mocked TypeSafe API server.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions