AssemblyAI connector for AI agents

Tools your transcription agent reaches for on AssemblyAI, scoped per user.

CALL ANY TOOL

12 tools for transcription in AssemblyAI: create and list transcripts, split them into sentences and paragraphs, search words, export subtitles, get redacted audio, run speech understanding tasks, and prompt LLMs over transcripts.

assemblyai_create_chat_completion

Create chat completion

Send a prompt or chat messages to an LLM through AssemblyAI's LLM Gateway, optionally with a transcript's text inserted, to summarize, answer questions about or extract data from audio. Returns choices with the model's message (content and any tool_calls), finish_reason, usage token counts and request_id. Use this for free-form prompts over transcripts (the replacement for LeMUR). Use assemblyai_run_speech_understanding for built-in translation, speaker identification, custom formatting, summaries or action items. Get a valid model ID from assemblyai_list_llm_models. To use a transcript, it must be completed; reference it with transcript_id and the {{ transcript }} tag.

Parameters

Name

Type

Required

Description

model

string

Required

ID of the LLM to use, from assemblyai_list_llm_models, for example claude-sonnet-4-5-20250929, gpt-4.1 or gemini-2.5-flash-lite. Never guess it; check the model list. Example: gemini-2.5-flash-lite

fallback_config

object

Optional

Fallback behaviour: retry (default true) retries once after 500 ms, and depth (1 or 2, default 1) caps how many fallbacks are tried. Example: {"retry": true, "depth": 2}

fallbacks

array

Optional

Fallback models to try in order when the primary model fails. Each item needs a model and can override other request fields. Example: [{"model": "claude-sonnet-4-6"}]

max_tokens

integer

Optional

Maximum number of tokens to generate. Defaults to 1000. Example: 1500

messages

array

Optional

Conversation so far, as objects with role (system, user, assistant or tool) and content. Provide messages or prompt. To use a transcript, put the tag {{ transcript }} in a message and set transcript_id. Example: [{"role": "system", "content": "You summarize calls."}, {"role": "user", "content": "Summarize this call: {{ transcript }}"}]

model_region

string

Optional

Set to global to route to the provider's global (non-regional) endpoints for lower cost. Currently live for Anthropic Claude models. Leave empty for in-region processing. One of: `global`.

post_processing_steps

array

Optional

Steps applied to the model's answer. The only step today is json-repair, which fixes malformed JSON. Example: [{"type": "json-repair"}]

prompt

string

Optional

A single prompt, sent as one user message. Use instead of messages for one-shot requests. Can contain the {{ transcript }} tag. Example: List the action items in this meeting: {{ transcript }}

response_format

object

Optional

Constrain the answer to JSON that matches a schema: {"type": "json_schema", "json_schema": {"name": ..., "schema": {...}, "strict": true}}. Not supported by gpt-oss models. Example: {"type": "json_schema", "json_schema": {"name": "summary", "schema": {"type": "object", "properties": {"summary": {"type": "string"}}, "required": ["summary"]}, "strict": true}}

temperature

number

Optional

Randomness from 0 to 2; lower is more deterministic. Leave empty for the model's default. Example: 0.2

tool_choice

string

Optional

Which tool the model may call: none, auto, or {"type": "function", "function": {"name": "get_weather"}} to force one. Example: auto

tools

array

Optional

Functions the model may call, each {"type": "function", "function": {"name": ..., "description": ..., "parameters": <JSON Schema>}}. Tool calls come back in choices[].message.tool_calls; send results back as role tool messages. Example: [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}]

transcript_id

string

Optional

ID of a completed AssemblyAI transcript whose text replaces the first {{ transcript }} tag in the messages or prompt. Example: 0072a82b-aa22-4962-add2-6121c36c17c6

assemblyai_create_transcript

Create transcript

Submit an audio or video file at a URL to AssemblyAI for asynchronous transcription, with optional speaker labels, language detection, PII redaction and audio intelligence. Returns the new transcript object with its id and status queued. Use this to start a transcription. Poll assemblyai_get_transcript until status is completed or error, then read it with assemblyai_get_transcript_sentences, assemblyai_get_transcript_paragraphs or assemblyai_get_subtitles. The file must be reachable at a URL; uploading local files is not supported.

Parameters

Name

Type

Required

Description

audio_url

string

Required

Public or pre-signed URL of the audio or video file to transcribe (MP3, WAV, MP4, M4A, FLAC, WebM and other common formats). AssemblyAI downloads the file itself, so the URL must be reachable without login. Local file uploads are not supported by this tool. Example: https://assembly.ai/wildfires.mp3

audio_end_at

integer

Optional

Stop transcribing at this point in the file, in milliseconds. Example: 60000

audio_start_from

integer

Optional

Start transcribing at this point in the file, in milliseconds. Example: 5000

auto_highlights

boolean

Optional

Set to true to extract key phrases. Results appear in auto_highlights_result.

content_safety

boolean

Optional

Set to true to flag sensitive content (hate speech, violence, profanity and similar). Results appear in content_safety_labels.

content_safety_confidence

integer

Optional

Confidence threshold (25 to 100, default 50) for content moderation labels. Only used when content_safety is true. Example: 60

custom_spelling

array

Optional

Spelling replacements applied to the transcript. Each item maps one or more spoken forms (from, each up to 5 words) to a single replacement word (to). Example: [{"from": ["dicarlo"], "to": "DeCarlo"}]

disfluencies

boolean

Optional

Set to true to keep filler words such as um and uh in the transcript.

domain

string

Optional

Set to medical-v1 to enable Medical Mode, which improves accuracy for medications, procedures, conditions and dosages (English, Spanish, German and French). Leave empty for general audio. One of: `medical-v1`.

entity_detection

boolean

Optional

Set to true to detect named entities (people, organizations, locations, dates and so on). Results appear in entities.

filter_profanity

boolean

Optional

Set to true to mask profanity in the transcript text.

format_text

boolean

Optional

Apply casing and text formatting (numbers, dates and so on). Defaults to true. PII redaction needs it on.

iab_categories

boolean

Optional

Set to true to classify the content into IAB topic categories. Results appear in iab_categories_result.

keyterms_prompt

array

Optional

Domain words or phrases (up to 6 words each) the model should recognize, such as product names, people or technical terms. Up to 1,000 terms for universal-3-5-pro or 200 for universal-2. Example: ["AssemblyAI", "Universal-3", "diarization"]

language_code

string

Optional

Language of the audio, as an AssemblyAI language code such as en, en_us, en_uk, es, fr, de, hi, ja or zh. Leave empty to detect the language automatically. Do not combine with language_detection true. Supported codes: https://www.assemblyai.com/docs/pre-recorded-audio/supported-languages. Example: en_us

language_codes

array

Optional

Languages spoken in a code-switched file (speakers mixing languages). One of the codes must be en. Example: ["en", "es"]

language_confidence_threshold

number

Optional

Minimum confidence (0 to 1) for the automatically detected language. The transcript fails with an error when detection confidence is lower. Only valid with language detection. Example: 0.7

language_detection

boolean

Optional

Automatic language detection. Applied automatically when language_code is empty. Set to false only together with a language_code.

language_detection_options

object

Optional

Options for automatic language detection. expected_languages limits detection to the listed codes, fallback_language is used when the detected language is not in that list (auto lets the model pick), code_switching true detects speakers switching languages mid-file, and on_no_speech_detected chooses between failing (error, the default) or completing with empty text (fallback). Example: {"expected_languages": ["en", "es"], "fallback_language": "en"}

multichannel

boolean

Optional

Set to true to transcribe each audio channel separately (for example a call recorded with one speaker per channel). Words and utterances then carry a channel number.

prompt

string

Optional

Free-text context of up to 1,500 words to guide the universal-3-5-pro model, such as the topic, names, jargon or the desired style. Ignored by other models. Example: A customer support call about a billing dispute for Acme Cloud.

punctuate

boolean

Optional

Add punctuation to the transcript. Defaults to true. Speaker labels and sentiment analysis need it on.

redact_pii

boolean

Optional

Set to true to redact personal information (names, phone numbers, card numbers and so on) from the transcript text. Requires format_text true. Choose the categories with redact_pii_policies.

redact_pii_audio

boolean

Optional

Set to true to also produce a copy of the audio with the redacted PII beeped out. Requires redact_pii true. Fetch it with assemblyai_get_redacted_audio once the transcript completes.

redact_pii_audio_options

object

Optional

Options for redacted audio: return_redacted_no_speech_audio true returns a file even when the audio has no speech, and override_audio_redaction_method silence replaces PII with silence instead of a beep. Example: {"override_audio_redaction_method": "silence"}

redact_pii_audio_quality

string

Optional

File type of the redacted audio: mp3 (default) or wav. One of: `mp3`, `wav`.

redact_pii_policies

array

Optional

Categories of personal information to redact when redact_pii is true, such as person_name, phone_number, email_address, credit_card_number, us_social_security_number or location. Example: ["person_name", "phone_number", "email_address"]

redact_pii_return_unredacted

boolean

Optional

Set to true to also return the original text as unredacted_text, unredacted_words and unredacted_utterances. Requires redact_pii true.

redact_pii_sub

string

Optional

How redacted text is replaced: hash (default, replaces each character with #) or entity_name (replaces it with the category, such as [PERSON_NAME]). One of: `entity_name`, `hash`.

redact_static_entities

object

Optional

Extra exact terms to redact, as a map from a label to the list of terms. Redacted with the redact_pii_sub style, on top of PII redaction. Example: {"INTERNAL_TOOL": ["Bearclaw", "Cubclaw"]}

remove_audio_tags

string

Optional

Universal-3.5 Pro can add inline annotations such as audio event markers and speaker cues. all (the default) removes them all, speaker removes only speaker cues and keeps the rest. One of: `all`, `speaker`.

sentiment_analysis

boolean

Optional

Set to true to score the sentiment (positive, neutral, negative) of each sentence. Results appear in sentiment_analysis_results.

speaker_labels

boolean

Optional

Set to true to identify who spoke when (speaker diarization). The transcript then includes utterances with speaker labels A, B, C and so on. Required by speaker identification.

speaker_options

object

Optional

Range of possible speakers, as min_speakers_expected and max_speakers_expected (min must not exceed max); include_speaker_confidence true adds a per-word speaker confidence. Requires speaker_labels true. Do not combine with speakers_expected. Example: {"min_speakers_expected": 2, "max_speakers_expected": 4}

speakers_expected

integer

Optional

Exact number of speakers to identify. Requires speaker_labels true. Do not combine with speaker_options. Example: 2

speech_models

array

Optional

Speech models to use, in priority order: universal-3-5-pro (most accurate, supports prompt and temperature) and universal-2 (broadest language coverage). When the first model does not support the audio's language the next one is used. Omit for the default ["universal-3-5-pro", "universal-2"].

speech_threshold

number

Optional

Reject files where less than this fraction (0 to 1) of the audio is speech. Example: 0.5

speech_understanding_request

object

Optional

Optional speech understanding task to run on the finished transcript, with results returned on the transcript. The speech understanding task to run, as an object with exactly one task key. Translation: {"translation": {"target_languages": ["es", "de"], "formal": true}} (add "match_original_utterance": true with speaker labels to get per-utterance translations). Speaker identification: {"speaker_identification": {"speaker_type": "role", "known_values": ["interviewer", "candidate"]}}, or speaker_type "name" with a speakers list of objects that each have a name (or role) and an optional description. Needs a transcript made with speaker_labels true. Custom formatting: {"custom_formatting": {"date": "mm/dd/yyyy", "phone_number": "(xxx)xxx-xxxx", "email": "username@domain.com"}}. Summarization: {"summarization": {"summary_type": "bullets"}} (bullets or paragraph, optional effort low or medium). Action items: {"action_items": {"include_decisions": true}}.

temperature

number

Optional

Randomness of the universal-3-5-pro model's output, from 0 (default, most deterministic) to 1. Ignored by other models. Example: 0

assemblyai_delete_transcript

Delete transcript

assemblyai_get_redacted_audio

Get redacted audio

assemblyai_get_subtitles

Get subtitles

assemblyai_get_transcript

Get transcript

assemblyai_get_transcript_paragraphs

Get transcript paragraphs

assemblyai_get_transcript_sentences

Get transcript sentences

assemblyai_list_llm_models

List llm models

assemblyai_list_transcripts

List transcripts

assemblyai_run_speech_understanding

Run speech understanding

assemblyai_search_transcript_words

Search transcript words

Build your Agent

Same auth pattern across LangChain, OpenAI, Anthropic, and Google ADK.

import { ScalekitClient } from "@scalekit-sdk/node";
import { createReactAgent } from "@langchain/langgraph/prebuilt";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);

// AssemblyAI tools scoped to this user
const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["assemblyai"], toolNames: [
    "assemblyai_create_transcript",
    "assemblyai_get_transcript",
    "assemblyai_run_speech_understanding"] },
  pageSize: 100,
});

const agent = createReactAgent({ llm, tools });
await agent.invoke({ messages: [{ role: "user", content: "Transcribe this webinar recording and list the action items" }] });

import OpenAI from "openai";
import { ScalekitClient } from "@scalekit-sdk/node";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);
const openai = new OpenAI();

const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["assemblyai"] }, pageSize: 100,
});

const res = await openai.chat.completions.create({
  model: "gpt-5",
  messages: [{ role: "user", content: "Transcribe this webinar recording and list the action items" }],
  tools,
});

// Execute the tool call with the user's vaulted AssemblyAI API key
await sk.tools.executeTool(res.choices[0].message.tool_calls[0], "user_123");

import Anthropic from "@anthropic-ai/sdk";
import { ScalekitClient } from "@scalekit-sdk/node";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);
const anthropic = new Anthropic();

const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["assemblyai"] }, pageSize: 100,
});

const msg = await anthropic.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Transcribe this webinar recording and list the action items" }],
  tools,
});

// Tool call runs with the user's vaulted AssemblyAI API key
await sk.tools.executeTool(msg.content, "user_123");

import { Agent } from "@google/adk/agents";
import { ScalekitClient } from "@scalekit-sdk/node";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);

const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["assemblyai"] }, pageSize: 100,
});

const agent = new Agent({
  name: "transcription_agent",
  model: "gemini-2.5-pro",
  instruction: "AssemblyAI tools scoped to this user",
  tools,
});

await agent.run("Transcribe this webinar recording and list the action items");

Try these prompts

Copy any prompt into your agent. Each maps directly to a AssemblyAI tool. Click to copy, paste into your agent, done.

Transcribe

Copy the prompt

Copied

Transcribe this recording URL with speaker labels and PII redaction.

Copy the prompt

Copied

List the transcripts created in the last 7 days.

Copy the prompt

Copied

Check whether yesterday's transcript has finished processing.

Read and export

Copy the prompt

Copied

Split this transcript into paragraphs with timestamps.

Copy the prompt

Copied

Export this transcript as SRT subtitles.

Copy the prompt

Copied

Find every mention of pricing in this transcript.

Understand

Copy the prompt

Copied

Summarize this transcript and list the action items.

Copy the prompt

Copied

Translate this transcript into Spanish.

Copy the prompt

Copied

Ask an LLM which objections the customer raised in this call.

SEE HOW AUTH WORKS

Each user connects their own AssemblyAI API key once, and Scalekit sends it with every call. Keys stay vaulted, every call is scope checked, and every action is logged.

Authorize

Your user connects

AssemblyAI

once. We tie it to their identity and the meetings they approved — no shared bot account, no org-wide access

Who:

user ‘A’

when:

Once per user

access:

Limited to user

Store

Their

AssemblyAI

token lives in a vault scoped to them. User A's meetings are never reachable by an agent acting for user B, even on the same connection

vault:

encrypted

scope:

per-user

tokens:

auto-refreshed

Resolve

When your agent calls a

AssemblyAI

tool, we fetch the right token server-side. It never touches your agent, never appears in the LLM context, never shows up in your logs

speed:

~40ms

check:

before every call

seen by:

nobody

Audit

Every

AssemblyAI

tool call is logged — who triggered it, which meeting was fetched, what came back. 90 days of history, tied to the user who authorized it

history:

90 days

export:

SIEM-ready

logged:

every call

Test other agents

See the same per-user auth pattern across other transcription and AI connectors.

Support and Ops Teams

Meeting prep

Pulls agenda, participant context, and open action items before every meeting.

GTM and RevOps Teams

Sales call prep agent

Reads tomorrow's calls from Google Calendar, mines past Granola notes and Attio history for context, and delivers each rep a prep brief in Slack, scoped to the calls that rep actually owns.

Why Scalekit

Secure your agent's access. Connectors ship in minutes

01.

Shared tokens break per-user analytics

A shared AssemblyAI API key looks fine in a demo. In production every transcript looks like one service account, and you cannot tell which user triggered it. Scalekit resolves the credential of the actual user who triggered the agent, never a shared bot.

// shared key
audit → bot_service_account

// scalekit
audit → user_abc ✓

02.

Authentication is not authorization

03.

Multi-tenancy is architectural

04.

AssemblyAI today. Ten connectors tomorrow.

“Our agents act across Salesforce, Gong, Google Drive, and more, on behalf of every customer. Scalekit behind the scenes meant we can keep adding tools without ever rebuilding how credentials or tool calling work.”

Venu Madhav Kattagoni

Head of Engineering / Von

FAQs

Frequently Asked Questions

Does the agent access AssemblyAI as the user or through a shared key?

As the user. Each user connects their own AssemblyAI API key once, and Scalekit sends it with every call. Audit logs attribute every transcript the agent creates or deletes to that user, not a shared service account.

Where is the AssemblyAI API key stored?

In Scalekit's managed AES-256 token vault, namespaced per tenant. To rotate it, the user reconnects with a new API key. Revocation is a single dashboard action. Credentials never appear in prompts, logs, or LLM context.

Can I limit what the agent does in AssemblyAI?

Yes. Filter by tool name in listScopedTools to expose only what you want, for example the transcript and export tools without assemblyai_delete_transcript, which permanently deletes a transcript's data and any file uploaded for it. Scalekit also enforces scope checks before every API call.

What happens when a user revokes AssemblyAI access?

The connection is invalidated on the next tool call. Subsequent requests for that user fail closed with a clear error. Other users in the tenant remain unaffected. The event is logged for audit.

Which transcripts can the agent see?

Transcripts created with the project behind the user's AssemblyAI API key. assemblyai_list_transcripts returns the ones from the last 30 days, newest first, and assemblyai_get_transcript fetches any one of them by ID.