Algolia Crawler connector for AI agents

Tools your search ops agent reaches for on Algolia Crawler, scoped per user.

CALL ANY TOOL

20 tools for web crawling into Algolia: list and inspect crawlers, run, pause, and reindex them, crawl or test single URLs, and version crawler configurations.

algoliacrawler_cancel_task

Cancel task

Cancel a blocking task on a crawler so its schedule can resume. Returns an empty successful response when the task is cancelled. Use this when a task ran into an error and is blocking the crawler's schedule. Use get_task_status to check whether a task is still pending before cancelling. Requires a crawler id from list_crawlers and the id of the blocking task.

Parameters

Name

Type

Required

Description

string

Required

Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809.

taskID

string

Required

Unique ID (UUID) of the blocking task to cancel. Tasks that ran into an error block the crawler schedule until cancelled. Example: 98458796-b7bb-4703-8b1b-785c1080b110.

algoliacrawler_crawl_urls

Crawl urls

algoliacrawler_create_crawler

Create crawler

algoliacrawler_delete_crawl_runs

Delete crawl runs

algoliacrawler_delete_crawler

Delete crawler

algoliacrawler_get_config_version

Get config version

algoliacrawler_get_crawl_run_file

Get crawl run file

algoliacrawler_get_crawler

Get crawler

algoliacrawler_get_task_status

Get task status

algoliacrawler_get_url_stats

Get url stats

algoliacrawler_list_config_versions

List config versions

algoliacrawler_list_crawl_runs

List crawl runs

algoliacrawler_list_crawlers

List crawlers

algoliacrawler_list_domains

List domains

algoliacrawler_pause_crawler

Pause crawler

algoliacrawler_run_crawler

Run crawler

algoliacrawler_start_reindex

Start reindex

algoliacrawler_test_url

Test url

algoliacrawler_update_crawler

Update crawler

Rename a crawler or replace its entire configuration with a new one. Returns a taskId for the asynchronous update. Use this to rename a crawler, or to overwrite the whole configuration when versioning is not needed. Use update_crawler_config instead to change individual top-level settings with a new configuration version recorded. Requires a crawler id from list_crawlers.

Parameters

Name

Type

Required

Description

string

Required

Crawler ID (UUID) of the crawler to update. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809.

config

object

Optional

Complete replacement configuration for the crawler. Because this operation replaces the whole configuration, send every setting you want to keep, not only the ones you are changing; anything omitted is dropped. Changes made here are not versioned. A configuration must name the Algolia application the crawler writes to, a rate limit from 1 to 100 that sets how many concurrent tasks run per second (start low, for example 2 to 4), and a non-empty list of actions (up to 30). Each action names the target index and carries a record extractor, which is an object marked as a function whose source is a JavaScript function string that turns a crawled page into Algolia records; an action can also limit which URLs it applies to with path patterns. Add start URLs or sitemaps so the crawler knows where to begin. The Algolia API key is optional on creation (the Crawler generates one if omitted) and must never be an Admin API key. Minimal working example: {"appId": "YourApplicationID", "apiKey": "YourCrawlerIndexingApiKey", "rateLimit": 4, "startUrls": ["https://www.example.com"], "actions": [{"indexName": "example_index", "pathsToMatch": ["https://www.example.com/**"], "recordExtractor": {"__type": "function", "source": "({ url, $ }) => [{ objectID: url.href, title: $('head title').text() }]"}}]}

name

string

Optional

New name for the crawler, up to 64 characters. Provide this to rename the crawler; leave empty to keep the current name. Example: test-crawler.

algoliacrawler_update_crawler_config

Update crawler config

Build your Agent

Same auth pattern across LangChain, OpenAI, Anthropic, and Google ADK.

import { ScalekitClient } from "@scalekit-sdk/node";
import { createReactAgent } from "@langchain/langgraph/prebuilt";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);

// Algolia Crawler tools scoped to this user
const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["algoliacrawler"], toolNames: [
    "algoliacrawler_list_crawlers",
    "algoliacrawler_get_url_stats",
    "algoliacrawler_crawl_urls"] },
  pageSize: 100,
});

const agent = createReactAgent({ llm, tools });
await agent.invoke({ messages: [{ role: "user", content: "Which URLs failed in the last crawl of our docs crawler?" }] });

import OpenAI from "openai";
import { ScalekitClient } from "@scalekit-sdk/node";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);
const openai = new OpenAI();

const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["algoliacrawler"] }, pageSize: 100,
});

const res = await openai.chat.completions.create({
  model: "gpt-5",
  messages: [{ role: "user", content: "Which URLs failed in the last crawl of our docs crawler?" }],
  tools,
});

// Execute the tool call with the user's vaulted Algolia Crawler username and password
await sk.tools.executeTool(res.choices[0].message.tool_calls[0], "user_123");

import Anthropic from "@anthropic-ai/sdk";
import { ScalekitClient } from "@scalekit-sdk/node";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);
const anthropic = new Anthropic();

const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["algoliacrawler"] }, pageSize: 100,
});

const msg = await anthropic.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Which URLs failed in the last crawl of our docs crawler?" }],
  tools,
});

// Tool call runs with the user's vaulted Algolia Crawler username and password
await sk.tools.executeTool(msg.content, "user_123");

import { Agent } from "@google/adk/agents";
import { ScalekitClient } from "@scalekit-sdk/node";

const sk = new ScalekitClient(env.SCALEKIT_ENV_URL, env.SCALEKIT_CLIENT_ID, env.SCALEKIT_CLIENT_SECRET);

const { tools } = await sk.tools.listScopedTools("user_123", {
  filter: { connectionNames: ["algoliacrawler"] }, pageSize: 100,
});

const agent = new Agent({
  name: "search_ops_agent",
  model: "gemini-2.5-pro",
  instruction: "Algolia Crawler tools scoped to this user",
  tools,
});

await agent.run("Which URLs failed in the last crawl of our docs crawler?");

Try these prompts

Copy any prompt into your agent. Each maps directly to a Algolia Crawler tool. Click to copy, paste into your agent, done.

Crawlers

Copy the prompt

Copied

List the crawlers on our Algolia account.

Copy the prompt

Copied

Is the docs crawler running, reindexing, or blocked?

Copy the prompt

Copied

Pause the blog crawler while we fix its config.

Crawl runs and health

Copy the prompt

Copied

Show crawl runs for the docs crawler from the last 7 days.

Copy the prompt

Copied

Break down the last crawl by URL status and reason.

Copy the prompt

Copied

Check whether the reindex task is still pending.

Recrawls and config

Copy the prompt

Copied

Recrawl these 5 pricing page URLs.

Copy the prompt

Copied

Test crawl this URL and show the records it would extract.

Copy the prompt

Copied

List the config versions of the docs crawler and who changed them.

SEE HOW AUTH WORKS

Each user connects their own Algolia Crawler username and password once; Scalekit sends them with every call. Credentials stay vaulted, every call is scope checked, and every action is logged.

Authorize

Your user connects

Algolia Crawler

once. We tie it to their identity and the meetings they approved — no shared bot account, no org-wide access

Who:

user ‘A’

when:

Once per user

access:

Limited to user

Store

Their

Algolia Crawler

token lives in a vault scoped to them. User A's meetings are never reachable by an agent acting for user B, even on the same connection

vault:

encrypted

scope:

per-user

tokens:

auto-refreshed

Resolve

When your agent calls a

Algolia Crawler

tool, we fetch the right token server-side. It never touches your agent, never appears in the LLM context, never shows up in your logs

speed:

~40ms

check:

before every call

seen by:

nobody

Audit

Every

Algolia Crawler

tool call is logged — who triggered it, which meeting was fetched, what came back. 90 days of history, tied to the user who authorized it

history:

90 days

export:

SIEM-ready

logged:

every call

Test other agents

See the same per-user auth pattern across other search and crawling connectors.

Engineering Teams

DevOps assistant agent

Polls GitHub for failing checks and stale PRs, opens Linear issues for the ones that need work, and posts a daily digest to Slack. It acts as the engineer, not a shared service account.

Engineering Teams

Engineering standup agent

Pulls commits from GitHub and GitLab, tracks issue movement in Jira, and posts a per-engineer standup brief to Slack. Each engineer's activity is read on their own delegated OAuth.

Engineering Teams

Slack triage

Polls Slack for new messages, classifies bugs and support requests with a LangGraph router, files GitHub issues or Zendesk tickets, and confirms in the thread.

Engineering Teams

Auto release notes agent

Reads merged GitHub PRs, groups them into structured release notes, publishes the page to Notion, and announces the release in Slack. Every call runs on the engineer's own delegated OAuth.

Why Scalekit

Secure your agent's access. Connectors ship in minutes

01.

Shared tokens break per-user analytics

One shared Algolia Crawler login looks fine in a demo. In production every crawler change looks like one service account, and you cannot tell which user triggered it. Scalekit resolves the credential of the actual user who triggered the agent, never a shared bot.

// shared login
audit → bot_service_account

// scalekit
audit → user_abc ✓

02.

Authentication is not authorization

03.

Multi-tenancy is architectural

04.

Algolia Crawler today. Ten connectors tomorrow.

“Our agents act across Salesforce, Gong, Google Drive, and more, on behalf of every customer. Scalekit behind the scenes meant we can keep adding tools without ever rebuilding how credentials or tool calling work.”

Venu Madhav Kattagoni

Head of Engineering / Von

FAQs

Frequently Asked Questions

Does the agent access Algolia Crawler as the user or as a shared key?

As the user. Each user connects their own Algolia Crawler username and password once, and Scalekit sends them with every call. Audit logs attribute every action to that user, not a shared service account.

Where are the Algolia Crawler credentials stored?

In Scalekit's managed AES-256 token vault, namespaced per tenant. To rotate them, reconnect with new credentials. Revocation is a single dashboard action. Credentials never appear in prompts, logs, or LLM context.

Can I limit what the agent does in Algolia Crawler?

Yes. Filter by tool name in listScopedTools to expose only what you want. Scalekit also enforces scope checks before every API call.

What happens when a user revokes Algolia Crawler access?

The connection is invalidated on the next tool call. Subsequent requests for that user fail closed with a clear error. Other users in the tenant remain unaffected. The event is logged for audit.

Can the agent delete crawlers or crawl history?

Only if you expose those tools. Of the 20 tools, 9 read, 9 write, and 2 are destructive: algoliacrawler_delete_crawler and algoliacrawler_delete_crawl_runs. Leave them out of listScopedTools and the agent can still inspect crawlers, run reindexes, and test URLs.

On this page

Tool and actions Framework Compatibility Starter Prompt Auth Lifecycle Try Other Agents Why Scalekit