TL;DR

• One in-process OpenTelemetry tree: AGENT turn, LLM spans from Arize's OpenAI instrumentor, TOOL spans from a thin wrapper around ^{client.tools.executeTool}. Same trace ID, free parent/child nesting, no ^traceparent plumbing to Scalekit's servers.
• Discover tools for an identifier with ^{listAvailableTools} (or ^{listScopedTools} when you have a connector filter)
• Execute with ^{tracedTools(scalekit).executeTool(...)} so each call emits an OpenInference TOOL span
• Mark resolved connector failures as ERROR (most AgentKit failures do not throw)
• Export to Arize over OTLP HTTP; keep a local span tree for demos

You wire Scalekit so your agent can call Slack, GitHub, or Calendar as a real connected user. The model decides on a tool. Scalekit runs ^executeTool. The user sees a confident "Done!" in the chat.

In Arize you open the trace and see only LLM spans. The model's request to call the tool is there: name and arguments. The actual tool run is not. Neither is the fact that Scalekit returned ^{missing_scope} while the model still said success.

That gap is not an Arize bug and not a Scalekit bug. Provider auto-instrumentors (OpenAI, Anthropic) patch the LLM client. They never see your application called Scalekit. Without an application-level TOOL span, the trace is a flat list of chat completions with a hole where reality happened.

This post shows how to close that hole with Scalekit's public API, OpenInference semantic conventions, and Arize AX as the backend, without monkey-patching the Node SDK.

By the end of this tutorial you will have:

• A clear mental model for what LLM auto-instrumentation does and does not capture.
• A copy-paste-ready ^tracedTools wrapper typed against ^{@scalekit-sdk/node}'s public signature.
• An agent loop that nests TOOL under AGENT and LLM under the same parent.
• Next.js / OpenTelemetry setup notes that prevent "TOOL spans but no LLM spans."
• A cloneable prototype: ^{ecosystem/arize-agentkit-tracing-demo}.

Honest note: The reference demo typechecks, builds, and passes offline span-contract tests against real ^{ExecuteToolResponse} envelope shapes. It has also been run end to end against a live Scalekit environment and a real Arize AX project: one trace contains ^{AGENT run_agent} (OK) with ^LLM and ^TOOL children, and the TOOL spans for failing GitHub connector calls exported as ^ERROR, the failure path this post is about. The success path (a connector call that returns cleanly and should export ^OK) has so far only been verified against the offline stub, so treat ^{isErrorResult} as validated for failures and unproven for successes until you see a green TOOL span in your own project.

Architecture Overview

Scalekit owns connected accounts, scopes, and tool execution. Arize AX owns storage, search, and evaluation of OpenInference traces. Your app owns the agent loop and the decision to emit TOOL spans in process; next to the LLM spans, under one AGENT root.

Walk through one turn:

1. You start an AGENT span for the turn.
2. The model runs under the OpenAI instrumentor → LLM span(s).
3. On ^tool_calls, you call Scalekit through ^tracedTools → TOOL span as a child of the active context.
4. Results go back to the model; another LLM span may claim success or report the error.
5. The batch exporter ships the whole tree to Arize.

Key point: Instrumenting tool execution in your process is what keeps TOOL nested under AGENT without stitching. A hypothetical Scalekit-side span exporter would need W3C ^traceparent (or equivalent) or you would get orphan tool traces.

Who Is This For

This post is for teams shipping tool-using agents on Scalekit AgentKit who already care about LLM observability: Arize, Phoenix, or any OpenInference-compatible backend, and have noticed that "we instrumented OpenAI" still leaves tool outcomes invisible. You may be debugging flaky connectors, auditing multi-tenant ^identifier usage, or building evals that need ground-truth tool success/failure next to model text. You are not looking for another OAuth tutorial; you already have connected accounts. You need the runtime story in the trace.

The Problem

Stock instrumentation gives you something like:

What you need for the failure case:

(^{OpenAI Chat Completions} is the span name the OpenInference instrumentor emits — that is the string you search for in Arize, not the ^{chat.completions.create} method name.)

Here is the same shape from a real run of the reference demo, read back out of Arize:

The AGENT span closes ^OK while two TOOL children are ^ERROR. That contrast — a turn that "succeeded" containing tools that did not — is the entire reason to put both in one trace.

Only the combined tree makes the lie obvious. Scalekit alone knows the tool failed. Arize alone knows what the model said. One trace with both is the product surface.

The Solution: A Public-API Wrapper, Not a Monkey-Patch

The tempting approach is ^{Object.getPrototypeOf(client.tools)} and patching ^executeTool. Do not ship that.

^ToolsClient is not exported from ^{@scalekit-sdk/node}, only ^{ScalekitClient} is. A prototype patch reaches into internals the SDK never promised. When those internals move, the patch stops applying and spans silently disappear. No throw, no warning. Observability that fails closed is better than observability that fails invisible.

Instead, wrap the documented call site:

If ^executeTool changes shape, TypeScript fails at build time. That is the integration contract you want for a recipe that may later become a package.

Prerequisites

• Node.js 20+
• A Scalekit environment with AgentKit and at least one Active connected account (^identifier from Dashboard → AgentKit → Connected Accounts)
• OpenAI API key or any OpenAI-compatible proxy (LiteLLM, gateway) via ^{OPENAI_BASE_URL}
• An Arize AX space: Space ID, API key, project name, and the OTLP endpoint for your region (US / EU / Canada — do not assume US)

Optional for local-only demos: omit Arize credentials; the reference app still builds an in-memory span tree in the UI.

Step 1: Install and Configure

Environment (names match the demo):

Checkpoint: You can construct ^{ScalekitClient} and list a non-empty set of tools for the identifier. An empty list here is a connected-account problem, not a tracing problem — fix it before wiring spans.

Step 2: Discover Tools for the Identifier

Keep discovery connector-agnostic. Whatever the identifier is authorized to call — GitHub, Gmail, Slack, Calendar — becomes the model's tool list. This is especially important for multi-tenant tool calling where each identifier may have a different set of authorized connectors.

Watch the signatures — these three are not interchangeable:

Reach for ^{listScopedTools} only when you actually have a provider or connection filter to pass. The reference demo defaults to ^{listAvailableTools} and falls back to ^{listTools({ filter: { identifier } })} when the available list comes back empty — common when your identifier is a connection id rather than an email.

Pass a ^{specFor(toolName)} lookup into the tracer so TOOL spans get OpenInference's ^{tool.description} and ^{tool.parameters}, not only the name.

Step 3: Wrap ^executeTool with OpenInference TOOL Spans

This is the integration surface worth extracting later. For context on why tool calling authentication requires careful runtime instrumentation — not just static config — see our dedicated guide. Conceptually:

Call site in the agent loop:

Checkpoint: Offline tests (or a deliberate failure) show TOOL spans with ^OK vs ^ERROR matching success vs failure-in-^data.

Why Error Classification Is the Hard Part

The Node SDK types success as roughly:

Connector failures (missing scope, provider 4xx, ^{{ ok: false, error: "..." }}) almost always arrive as a resolved promise with the failure inside ^data — not as a throw, and not as a top-level ^error on the envelope.

If you only mark ERROR on throw, those spans export as OK and the headline demo is dead. The default classifier should unwrap ^data and look for structural cues (^{ok === false}, ^error present, ^{statusCode >= 400}, status strings like ^failed). That is still a guess about connector payloads. Run a tool you know will fail, inspect ^result.data, and tighten ^{isErrorResult} for your connectors.

Step 4: Own the AGENT Span; Let the Instrumentor Own LLM

Construct the OpenAI client lazily after the tracer and instrumentor register. A client created at module import time can miss the patch and emit zero LLM spans while TOOL spans still work — a hole that looks like a half-broken integration.

Step 5: Register Arize / OpenTelemetry (Next.js Notes)

Order is fixed:

1. Register ^{NodeTracerProvider} (resource must include Arize project name).
2. Register instrumentors; for Next.js, call ^{manuallyInstrument} on the imported ^openai module.
3. Then construct OpenAI clients.

Sketch:

Next.js-specific gotchas (these cost people full afternoons):

In the App Router, put setup in ^{instrumentation.ts} so it runs before route modules load:

After a demo turn, ^forceFlush() the provider so the batch is more likely to appear in Arize before you refresh the UI.

Step 6: Verify the Span Contract (Before You Trust the UI)

The demo ships ^{npm run verify}: a stub Scalekit client returning real envelope shapes.

The middle rows are the ones people get wrong first.

For a live check: revoke a scope or use a bad resource id, run one agent turn, open Arize by ^trace_id, and confirm TOOL is ERROR while the final LLM content is still optimistic. That is the screenshot worth putting in an incident channel.

Privacy and Multi-Tenant Attributes

Demo spans often include full tool arguments, full results, and chat content. ^{scalekit.identifier} is frequently an email. That is useful locally and not a production default.

Before you reuse the recipe:

• Hash or redact identifiers in span attributes.
• Drop or scrub PII from ^input.value / ^output.value.
• Do not return raw span attributes to untrusted browsers (the demo does this on purpose for teaching).
• Use Arize project-side scrubbing where available.
• Prefer attributes like ^{scalekit.organization_id} / ^{scalekit.user_id} from authenticated connected-account context over free-form strings the client asserted.

Those Scalekit fields are what make tenant filters in Arize trustworthy. For a deeper look at access control in multi-tenant AI agents, see our dedicated guide.

Complete Flow (Minimal)

Clone and run the full Next.js app from the prototype README: install, copy ^.env.example → ^.env.local, ^{npm run verify}, ^{npm run dev}, open ^{http://localhost:3000}, send a prompt that forces a connected tool, then open Arize with the printed trace id.

Common Scenarios

• Internal agent for a team: Each teammate has their own Scalekit identifier; traces filter by ^{scalekit.identifier} or org id when you debug "why did Slack fail for Alice but not Bob?"
• Customer-facing product agent: Resolve identifier from your session after auth; never accept it from the browser as gospel. Tool spans show which tenant's connection failed.
• LiteLLM / gateway: Point ^{OPENAI_BASE_URL} at the proxy. OpenInference still patches the OpenAI SDK; destination host does not remove LLM spans.
• Evals: Score "tool ERROR + model claimed success" as a first-class failure mode once both sit in one trace.
• Future package: Keep ^{traced-tools.ts} on public types only so promotion from recipe → npm package does not require a rewrite.

Troubleshooting

I only see LLM spans in Arize

You never wrap ^executeTool, or the wrapper runs outside an active AGENT span and you are looking at a different filter. Confirm TOOL span kind and ^tool.name attributes. Confirm the agent actually called a tool this turn.

I only see TOOL spans, no LLM spans

Classic Next/bundler issue: instrumentor patched a different ^openai instance than the agent imports. Use ^{manuallyInstrument} + ^{serverExternalPackages: ['openai']}. Also confirm the client is created after ^register().

TOOL spans are always OK even when the connector failed

Your ^{isErrorResult} is not unwrapping ^data. Log ^{JSON.stringify(result)} for a known failure and fix the classifier.

No traces in Arize at all

Wrong region endpoint, missing project name, missing space/api headers, or batch not flushed before process exit. Local collector/UI may still show spans while export is broken — treat those as separate pipelines.

Spans not recording (^{isRecording() === false})

Duplicate ^{@opentelemetry/api} versions or TracerProvider registered in a different realm than the agent module. Align versions and init order.

Tool discovery returns nothing

Identifier has no Active connected account, or you filtered connectors incorrectly. Fix connection state in the Scalekit dashboard before debugging tracing.

If you called ^{listScopedTools}, note that its ^filter is required and the API rejects an empty one (^{no ... specified in filter}) — that reads as "no tools" but is really a bad request. With no filter to pass, use ^{listAvailableTools(identifier)}, and fall back to ^{listTools({ filter: { identifier } })} when the available list is empty.

Every TOOL span is ERROR, including calls that should have worked

The mirror image of the classifier bug: if ^{isErrorResult} is too eager it flags healthy payloads too, and a wall of red hides the real failures. Confirm against a call you know succeeded — if its span is still ERROR, log ^{JSON.stringify(result)} and tighten the structural checks.

Tradeoffs and Limitations

Known gap: Error detection remains structural until Scalekit exposes a first-class error channel on the execute envelope. Treat ^{isErrorResult} as configuration per connector family, not as a permanent universal truth.

What's Next

• Run the prototype against a failed tool and save the Arize tree as your team's definition of done for agent observability.
• Redact production attributes; keep ^scalekit.* tenant keys you actually filter on.
• Reuse the same wrapper under LangGraph, Mastra, or a raw loop — the integration is the execute boundary, not the framework. For instance, see how tool-calling auth patterns apply across different agent frameworks.
• Review credential ownership patterns to understand who should hold tokens at each layer of your agent architecture.
• Read Scalekit docs for connectors and ^executeTool.
• Read Arize OpenInference docs for span kinds and project setup.