# SmartChats

> SmartChats is an open-source, voice-first personal AI assistant: an agent you talk to that keeps your logs, metrics, todos and knowledge graph in your own database. Self-host it with one command (`curl -fsSL https://smartchats.ai/install | bash`, or the Docker image sheunaluko/smartchats) or use the hosted app at smartchats.ai. Apache-2.0 licensed, code at https://github.com/sheunaluko/smartchats.

Every page below is plain markdown. For agents installing or operating SmartChats for a user, start with "For AI agents". For agents working on the code, read AGENTS.md in the repository.


---

Source: https://smartchats.ai/docs

# SmartChats

SmartChats is a voice-first AI agent. You talk to it, and it can
remember what you said, search the web, write and run code, and draw
charts from your data. It reads and writes your data over the
[Model Context Protocol](https://modelcontextprotocol.io) too, so other
tools can use the same data.

The same agent runs two ways:

- **Cloud:** sign up at [smartchats.ai](https://smartchats.ai). There is
  no server to run.
- **Self-hosted:** install with one command, then run `smartchats setup`
  and `smartchats start` on your laptop or a VPS. Same app, same agent.
  Your data stays in a database on your machine. See
  [Quick Start](/docs/quickstart).

Your data can move both ways. `smartchats data export` and
`smartchats data import` (and the matching MCP tools) copy everything
between cloud and local.

---

## Voice logging and recall

The core use is simple. Through the day, talk to SmartChats from your
phone or any device. Log thoughts, exercise, ideas, anything, in plain
speech. It stores them as logs, metrics, todos and knowledge-graph
facts.

Later, ask the built-in agent to recall it. Summarize the week. Find
recurring themes. Compare what you logged on Tuesday with what you said
on Friday. Draft a journal entry from your raw logs.

If you prefer another tool for the recall side, point Claude (or any
MCP client) or the `smartchats` CLI at your account. It is the same
data.

```
   phone (voice in, all day)
              │
              ▼
       SmartChats stores
       what you said
              │
              └─────────── your data (cloud or local) ─────────┐
                                       │                        │
                                       ▼                        ▼
                              speak to SmartChats'       (optional) Claude
                              agent for recall           or any MCP client
```

### The other direction: bring data in, then talk to it

Any agent that speaks MCP can also write into your account. For example:
point a coding agent at a folder of Markdown notes and have it write
them into SmartChats as logs, metrics and knowledge-graph facts through
the MCP server. Then ask SmartChats about them by voice.

```
   your existing data              agent reads and structures it
   (Markdown, PDFs, exports)                    │
              │                                 ▼
              ▼                        MCP server writes into
        any agent that                 your SmartChats account
        speaks MCP                              │
                                                ▼
                                       speak with it (SmartChats'
                                       agent or any MCP client)
```

---

## Who it is for

#### Knowledge worker

Notes, decisions and half-formed ideas end up scattered across apps.
Speak them to SmartChats as they happen, then ask its agent to recall
them. It can pull patterns across weeks, list unfinished threads and
connect last month's logs to this week's.

#### Developer

SmartChats gives an agent a typed, open data layer:

- **MCP both ways.** Read and write a user's logs, metrics, todos and
  knowledge graph from any MCP client.
- **One contract.** The `DataAPI` interface is the same in the web app,
  the CLI and the MCP server. Add a query builder once and every
  consumer can use it.
- **Public source.** The CLI works against any deployment: cloud, local
  or your own fork.
- **Code execution.** The agent writes and runs JavaScript in a sandbox
  in the browser, with workspace state that lasts across turns.
- **Live agent messaging.** A coding agent can talk with you through
  the app with the [hub](/docs/hub) or the [MCP bridge](/docs/relay).

See [Architecture](/docs/architecture) for how the pieces fit.

#### Quantified self

Log workouts, sleep, mood, water or weight by voice. "20 pull-ups, 30
push-ups, swam 30 minutes" becomes three structured metrics.

- **Charts on request.** Ask for weekly running distance or a month of
  macros and the chart renders in the chat.
- **Your own categories.** Search is full-text, vector and structured.
- **Export any time.** `smartchats data export` writes every log,
  metric and entity to a JSON bundle.

#### MCP client

If you build an MCP client, or use one such as Claude Desktop,
SmartChats is a data source:

- **33 tools.** Read, search, insert, update and delete logs, metrics,
  todos and knowledge-graph entries. Vector search on logs, entities and
  relations. A read-only `run_query` for raw SurrealQL. Bundle import
  and export.
- **Authenticated.** The cloud target acts as you. The local target is
  your single-user install.
- **Same data layer as the app and the CLI.**

See [MCP Server](/docs/mcp) for setup and the full tool list.

#### Researcher

SmartChats is a working system for studying voice interaction, agent
memory and knowledge graphs over time.

- **Inspectable.** The source is public. Fork and change any of it.
- **Self-hostable.** Run the whole stack on your machine.
- **Testable.** The schema is versioned, the layered runner
  (`smartchats-test`) covers lint, build, unit, integration and e2e, and
  data round-trips through export and import.
- **Measurable voice.** The vie voice engine logs the evidence it tunes
  from, and has a replay harness. See [Voice engine](/docs/voice-engine).

---

## Get started

  - [Quick Start](/docs/quickstart)
  - [Self-Hosting](/docs/self-host)
  - [Architecture](/docs/architecture)
  - [MCP Server](/docs/mcp)

---

## What's in the stack

- **The app.** A Next.js app in TypeScript that runs in your browser,
  with desktop and mobile layouts. The agent loop (cortex) streams
  structured output, runs code in a sandbox and keeps a knowledge
  graph.
- **Voice.** Speech recognition and voice activity detection in the
  browser (tivi), and streaming text-to-speech from the server. The vie
  voice engine is the default turn-taking engine. Before the main model
  answers, a small server-side turn judge decides whether you have
  finished speaking. See [Voice engine](/docs/voice-engine).
- **Model providers.** OpenAI, Anthropic, Gemini, xAI and Together AI
  for chat, OpenAI and Azure for speech. Bring your own keys. See
  [Providers](/docs/providers).
- **The local server** (`smartchats-local-server`). Serves the app and
  the `/local-api` routes against SurrealDB, including media upload and
  storage through any S3-compatible bucket such as SeaweedFS. See
  [Local server](/docs/local-server).
- **The CLI** (`smartchats`). Sets up, starts and stops the local stack,
  signs in to the cloud, and imports and exports data. See
  [CLI Reference](/docs/cli).
- **The MCP server** (`smartchats-mcp`). Gives any MCP client your data
  over stdio. See [MCP Server](/docs/mcp).
- **The hub and relay** (`smartchats-hub`, `smartchats-mcp-bridge`).
  Let coding agents on your machines talk with you through the app. See
  [Hub](/docs/hub) and [Relay](/docs/relay).

The source for all of it is in the public repo, under the Apache
License 2.0. See [Architecture](/docs/architecture#open-and-closed) for
what is open and what is closed.

---

## See also

- [GitHub issues](https://github.com/sheunaluko/smartchats/issues) for bugs and feature requests.
- [LinkedIn (maintainer)](https://www.linkedin.com/in/sheun-aluko/) for design discussions.
- [Contributing](/docs/contributing) for conventions and the dev workflow.

---

Source: https://smartchats.ai/docs/quickstart

# Quick Start

Get SmartChats running on your machine in a few minutes.

## Installation

#### macOS & Linux

```bash
curl -fsSL https://smartchats.ai/install | bash
```

The installer downloads a release for macOS (Apple Silicon) or Linux (x64 or arm64) into `~/.smartchats` and then runs `smartchats setup` (shown under [Quick start](#quick-start)). The release bundles the CLI, the server, SurrealDB and the media storage (SeaweedFS). You don't need Node, Bun or Docker.

> **Note:** **Linux users.** The installer needs `tar` and either `curl` or `wget` (preinstalled on most distros). If `tar` is missing: `sudo apt install tar` on Debian/Ubuntu, `sudo dnf install tar` on Fedora.

> **Warning:** **Intel Macs.** There is no binary release for Intel Macs, and the installer exits with a message. Use the npm package instead (see the Package Managers tab). It needs Node 24+.

To check that smartchats was installed successfully, open a new terminal window and run:

```bash
smartchats --version
# Output: 0.x.y
```

If you see `command not found`, add the install directory to your `PATH`. The installer symlinks the CLI into `/usr/local/bin` when that directory is writable. Otherwise it adds `~/.smartchats/bin` to `~/.zshrc`, `~/.bashrc` or your fish config.

```bash
export PATH="$HOME/.smartchats/bin:$PATH"
```

Installer flags go after `bash -s --`:

```bash
curl -fsSL https://smartchats.ai/install | bash -s -- --version v0.4.0 --non-interactive
```

| Flag | Default | Notes |
|---|---|---|
| `--version <tag>` | `latest` | Install a specific release tag |
| `--prefix <dir>` | `~/.smartchats` | Install location |
| `--non-interactive` | off | Don't run `smartchats setup` after installing |
| `--no-path` | off | Don't touch `/usr/local/bin` or shell rc files |

#### Windows

```powershell
iwr -useb https://smartchats.ai/install.ps1 | iex
```

> **Warning:** **Windows support is experimental.** The installer and the `windows-x64` release exist, but they have not yet been tested on real Windows machines. Expect problems, and please report them on [GitHub](https://github.com/sheunaluko/smartchats/issues). If it doesn't work for you, use WSL2 with the macOS & Linux install line.

The installer needs PowerShell 5 or later and Windows 10 1803 or later (for `tar.exe`). Only x64 is supported. Windows on ARM exits with an error because SurrealDB does not ship a Windows ARM build.

It installs to `%USERPROFILE%\.smartchats`, adds `%USERPROFILE%\.smartchats\bin` to your user `PATH`, and then runs `smartchats setup`.

To check that smartchats was installed successfully, open a new PowerShell window and run:

```powershell
smartchats --version
# Output: 0.x.y
```

To pass parameters, download the script first:

```powershell
iwr -useb https://smartchats.ai/install.ps1 -o install.ps1
.\install.ps1 -Version v0.4.0 -NonInteractive
```

| Parameter | Default | Notes |
|---|---|---|
| `-Version <tag>` | `latest` | Also read from `SMARTCHATS_VERSION` |
| `-Prefix <dir>` | `$HOME\.smartchats` | Also read from `SMARTCHATS_PREFIX` |
| `-NoPath` | off | Don't change the user `PATH` |
| `-NonInteractive` | off | Don't run `smartchats setup` after installing |

#### Package Managers

If you have Node 24 or later:

```bash
npm install -g smartchats-ai
```

This installs the CLI only. It doesn't include the Bun runtime or SurrealDB. `smartchats setup` checks for both and prints the install command for anything missing. It also needs `git`, because it clones the SmartChats source to `~/.smartchats/cli/source` and builds it on first start.

Try without installing:

```bash
npx smartchats-ai setup
```

For most users the `curl` installer is simpler.

#### Docker

Each release publishes an all-in-one image (SurrealDB, media storage and the server) for `linux/amd64` and `linux/arm64`:

```bash
docker pull sheunaluko/smartchats:latest   # or :v0.4.0
```

The same image is on GitHub's registry as `ghcr.io/sheunaluko/smartchats-aio`.

To build it yourself instead, from a clone: `docker build -f Dockerfile.aio -t smartchats-aio .` (the build installs a published release, not your checkout. Pin one with `--build-arg SMARTCHATS_VERSION=v0.4.0`).

Put your keys in `~/.smartchats/.env`, the same file a native install reads:

```bash
mkdir -p ~/.smartchats
cat > ~/.smartchats/.env <<EOF
TOGETHER_API_KEY=...          # default chat model + voice turn judge
AZURE_SPEECH_KEY=...          # default voice
AZURE_SPEECH_REGION=eastus
EOF
chmod 600 ~/.smartchats/.env
```

Then run the container:

```bash
docker run -d --name smartchats -p 3000:3000 \
  -v ~/.smartchats/.env:/root/.smartchats/.env:ro \
  -v ~/.smartchats/data:/data \
  sheunaluko/smartchats:latest
```

Open [http://localhost:3000](http://localhost:3000). To the container your browser is another device, so the first visit asks you to pair: enter the code from `docker logs smartchats` (or run `docker exec smartchats smartchats pair`). You do this once per browser. Photos and files go through the same port to the media storage inside the container. The database and media live in the `/data` volume.

To check that it started, run:

```bash
docker logs smartchats
# Output: ... Stack up. Open: http://localhost:3000
```

> **Warning:** **You need at least one LLM provider key.** The container starts without one, but the agent cannot reply. `docker logs smartchats` shows a `NO API KEYS CONFIGURED` box when no key is found. See [Required environment](/docs/self-host#required-environment).

Ways to supply keys:

| Method | When to use |
|---|---|
| `-v ~/.smartchats/.env:/root/.smartchats/.env:ro` | Same file as a native install. Edit it, then `docker restart smartchats` |
| `--env-file ~/.smartchats/.env` | Read once when the container is created |
| `-e OPENAI_API_KEY=$OPENAI_API_KEY` | One-off runs, CI |

From a clone, `bin/aio -d` builds the image if it is missing and runs it the same way, forwarding provider keys from your shell.

For more, see [Self-Hosting](/docs/self-host#docker-alternative).

## Upgrading

Once installed, smartchats can upgrade itself:

```bash
smartchats upgrade
```

This re-runs the installer in place. Your data directory, `.env` and `config.json` are kept. Restart the stack afterwards with `smartchats restart`.

| Flag | Default | Notes |
|---|---|---|
| `--version <tag>` | `latest` | Install a specific release, for example `v0.4.0` |
| `-y, --yes` | off | Skip the confirmation prompt |

**npm users:** run `npm install -g smartchats-ai@latest` instead. `smartchats upgrade` prints this and exits 1.

**Source checkouts:** run `git pull` in the repo.

**Docker:** `docker pull sheunaluko/smartchats:latest` (or rebuild with `docker build --no-cache -f Dockerfile.aio -t smartchats-aio .`), then remove and re-run the container. Data in the `/data` volume is kept.

## Uninstalling

There is no uninstall command. To remove a `curl` install, stop the stack and delete the install directory:

```bash
smartchats stop
rm -rf ~/.smartchats
rm -f /usr/local/bin/smartchats
```

`~/.smartchats` holds your data (`data/`), keys (`.env`), config and logs. Export your data first if you want to keep it (see [Backup + restore](/docs/self-host#backup--restore)).

If the installer edited your shell rc file, delete the `# Added by SmartChats installer` line and the `PATH` line after it. If you signed in to the cloud, also delete `~/.smartchats-mcp/credentials.json`.

**Windows:** delete `%USERPROFILE%\.smartchats` and remove `%USERPROFILE%\.smartchats\bin` from your user `PATH`.

**npm:** `npm uninstall -g smartchats-ai`. The clone at `~/.smartchats/cli/source` stays until you delete it.

**Docker:** `docker rm -f smartchats` and `docker rmi sheunaluko/smartchats:latest`. Delete `~/.smartchats/data` to remove your data too.

## Quick start

After installing, run the guided setup (the installers run it for you):

```bash
smartchats setup
```

The wizard will:

1. Check your system. A binary install checks free disk only. A source or npm install also checks for `bun` and `surreal`.
2. Ask for each provider key in [Providers & API Keys](/docs/providers), starting with Together and Azure, the two the default setup uses (see [Which keys matter](#which-keys-matter)). Press enter to skip any of them.
3. Write the `.env` file, start the stack and print [http://localhost:3000](http://localhost:3000). To use it from your phone, open the same address with your computer's network name or IP and pair once with the code from `smartchats pair`.

```bash
# Check what's running:
smartchats status

# Stop the stack:
smartchats stop

# Tail logs:
smartchats logs -f

# Add or change provider API keys later (interactive):
smartchats env
# Then `smartchats restart` to pick up the changes.
```

### Which keys matter

`setup` and `env` ask for every key in [Providers & API Keys](/docs/providers#providers), starting with the two the default setup uses: Together (the default chat model and the voice turn judge) and Azure Speech (the default voice). Every key is optional, but the agent needs at least one chat provider to reply.

- **Without Together**, another provider's model is used if it has a key, and the server skips the turn judge, so a pause mid-sentence ends the turn.
- **Without Azure**, replies are spoken with an OpenAI voice if OpenAI has a key.
- **Without OpenAI**, semantic search over your logs, knowledge graph and memory is off: OpenAI's embeddings are currently the only ones SmartChats uses. Entries saved without a key are not embedded later, so add it early if you want them searchable.

To change keys later, run `smartchats env`, or edit `~/.smartchats/.env` (`<repo>/.env` for a source install), then `smartchats restart`.

That's it. Talk to the agent.

## What's next

- **[CLI Reference](/docs/cli)**: every command and flag.
- **[Self-Hosting](/docs/self-host)**: env-var management, running on a VPS, Docker.
- **[Architecture](/docs/architecture)**: how voice, agent runtime, knowledge graph and sandboxed execution fit together.
- **[MCP](/docs/mcp)**: expose smartchats to Claude Desktop and other MCP-aware tools.

## Use the hosted SaaS instead

Don't want to run infrastructure? Sign up at [smartchats.ai](https://smartchats.ai). Same stack, managed billing, no install. Free tier included.

---

Source: https://smartchats.ai/docs/self-host

# Self-Hosting

This page is for running SmartChats yourself: on your laptop, a VPS, a Raspberry Pi, anywhere with a shell.

For most people, [Quick Start](/docs/quickstart) already covers it. This page goes further: env-var management, running on a VPS, Docker, and what runs inside the stack.

## TL;DR

```bash
curl -fsSL https://smartchats.ai/install | bash
smartchats setup
```

That gives you a self-hosted SmartChats on your machine. Everything below is for cases where you need more control.

## What you need

- **macOS (Apple Silicon) or Linux** with a shell. Intel Macs use the npm package. Windows is experimental (see [Quick Start](/docs/quickstart)).
- **An OpenAI API key.** `setup` treats it as the required key. It covers chat models, embeddings and OpenAI voices.
- Optional: Anthropic, Google (Gemini), xAI, Together AI, Azure Speech and Serper keys for more models, the turn verdict, Azure voices and web search.
- **3 GB free disk.** `setup` and `doctor` warn below that.

The installer bundles the CLI, the server (both compiled with Bun, runtime included) and the SurrealDB binary. No Node, no Docker, no system-level deps required.

## The stack, briefly

`smartchats start` launches **two** processes:

```
                  ┌──────────────────────────────────────┐
                  │  smartchats-server  (port 3000)      │
                  │                                      │
                  │   • serves the SPA at /              │
                  │   • handles /local-api/* (health,    │
                  │     llm, tts, embeddings, data,      │
                  │     usage, keys, tools, insights,    │
                  │     storage)                         │
                  └─────────────────┬────────────────────┘
                                    │  ws://127.0.0.1:8000/rpc
                                    ▼
                  ┌──────────────────────────────────────┐
                  │  surreal  (port 8000, loopback only) │
                  │  ~/.smartchats/data/surreal.db       │
                  └──────────────────────────────────────┘
                          ▲
                          │
                   browser at :3000
```

The browser only needs port 3000. SurrealDB binds to `127.0.0.1`. The server serves the static export of the Next.js app directly, so there is no Next.js runtime.

## Where things live

| Path | Purpose |
|---|---|
| `~/.smartchats/bin/` | Installed binaries (`smartchats`, `smartchats-server`, `surreal`) |
| `~/.smartchats/app/out/` | The static SPA the server serves |
| `~/.smartchats/data/surreal.db` | SurrealDB storage (rocksdb). Change the directory with `start --data-dir` |
| `~/.smartchats/run/pids.json` | Live process IDs, ports and start times |
| `~/.smartchats/logs/` | Per-process logs (`surreal.log`, `server.log`) |
| `~/.smartchats/.env` | API keys and config for a binary install. A source install uses `<repo>/.env` |
| `~/.smartchats/config.json` | CLI preferences (last port, resolved source root). `$XDG_CONFIG_HOME/smartchats/config.json` if that variable is set |

`smartchats home` prints the install root (binary install) or the resolved source root.

## Required environment

The stack boots without any keys, but **the agent cannot reply** until at least one LLM provider key is configured. `smartchats start` prints a `NO API KEYS CONFIGURED` box when it can't find any.

| Variable | Required? | Effect if missing |
|---|---|---|
| `OPENAI_API_KEY` | **Strongly recommended** | OpenAI models, embeddings and OpenAI voices unavailable |
| `ANTHROPIC_API_KEY` | Optional | Claude models can't be selected |
| `GOOGLE_API_KEY` *(or `GEMINI_API_KEY`)* | Optional | Gemini models can't be selected |
| `XAI_API_KEY` | Optional | Grok models can't be selected |
| `TOGETHER_API_KEY` *(or `TOGETHERAI_API_KEY`)* | Recommended for voice | The default turn verdict is skipped and every voice turn is answered |
| `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | Recommended for voice | The default voice provider (Azure) fails. Pick an OpenAI voice instead |
| `SERPER_API_KEY` | Optional | Agent's web-search tool is a no-op |
| `SMARTCHATS_API_KEY` | Optional | Bearer token for scripts and agents on other machines ([Other devices and scripts](#other-devices-and-scripts)) |
| `SMARTCHATS_AUTH` | Optional | `off` lets every device in without pairing |

> **Note:** **`setup` and `env` only prompt for OpenAI, Anthropic, Google and Serper.** Add the others to the `.env` file by hand, then run `smartchats restart`.

## Environment configuration

`smartchats start` reads the `.env` file at the install root (or repo root), merges in your shell environment, and passes the result to the server. A variable set in your shell wins over the same name in `.env`, so `OPENAI_API_KEY=sk-... smartchats start` overrides the file for one run.

Each provider has a `SMARTCHATS_`-prefixed name and one or more plain names. The server checks them in the order below and uses the first one set. If none is set, it falls back to a key stored in SurrealDB with `POST /local-api/keys`.

| Provider | Checked in this order |
|---|---|
| OpenAI | `SMARTCHATS_OPENAI_API_KEY`, `OPENAI_API_KEY` |
| Anthropic | `SMARTCHATS_ANTHROPIC_API_KEY`, `ANTHROPIC_API_KEY` |
| Google (Gemini) | `SMARTCHATS_GOOGLE_API_KEY`, `GOOGLE_API_KEY`, `GEMINI_API_KEY` |
| xAI | `SMARTCHATS_XAI_API_KEY`, `XAI_API_KEY` |
| Together AI | `SMARTCHATS_TOGETHER_API_KEY`, `TOGETHER_API_KEY`, `TOGETHERAI_API_KEY` |
| Azure Speech key | `SMARTCHATS_AZURE_SPEECH_KEY`, `AZURE_SPEECH_KEY` |
| Azure Speech region | `SMARTCHATS_AZURE_SPEECH_REGION`, `AZURE_SPEECH_REGION` |
| Serper (web search) | `SMARTCHATS_SERPER_API_KEY`, `SERPER_API_KEY` |

Azure Speech is read from the environment only. A database-stored Azure key is not used for voice.

### Server settings

These are read by `smartchats-server`. `smartchats start` sets the `SURREAL_*`, `SMARTCHATS_HOST`, `SMARTCHATS_PORT` and `SMARTCHATS_STATIC_DIR` values itself, so setting them in `.env` has no effect under `start`. They matter when you run the server another way, such as `smartchats dev`.

| Variable | Server default | Set by `start` to |
|---|---|---|
| `SMARTCHATS_HOST` | `127.0.0.1` | `0.0.0.0` |
| `SMARTCHATS_PORT` | `4242` | `--port` (default `3000`) |
| `SMARTCHATS_STATIC_DIR` | unset (API only) | `~/.smartchats/app/out` |
| `SURREAL_URL` | `ws://127.0.0.1:8000/rpc` | `ws://127.0.0.1:<surreal-port>/rpc` |
| `SURREAL_NS` | `smartchats` | `smartchats` |
| `SURREAL_DB` | `main` | `main` |
| `SURREAL_USER` | `root` | `root` |
| `SURREAL_PASSWORD` | `root` | `root` |

Media storage. `smartchats start` runs SeaweedFS and sets these for the server, so you only set them to use your own S3-compatible bucket instead (all four are needed). See [Storage](/docs/local-server#storage) for the bind address and the browser-facing endpoint:

| Variable | Default | Notes |
|---|---|---|
| `SMARTCHATS_S3_ENDPOINT` | set by `start` | e.g. `https://<account>.r2.cloudflarestorage.com` |
| `SMARTCHATS_S3_ACCESS_KEY_ID` | unset | |
| `SMARTCHATS_S3_SECRET_ACCESS_KEY` | unset | |
| `SMARTCHATS_S3_BUCKET` | unset | |
| `SMARTCHATS_S3_REGION` | `us-east-1` | Signing region |
| `SMARTCHATS_S3_LINKS` | `server` | `direct`: browsers get presigned bucket URLs instead of links through the server ([Storage](/docs/local-server#storage)) |

### Where the `.env` file lives

| Install path | `.env` location |
|---|---|
| `curl \| bash` install | `~/.smartchats/.env` |
| Source checkout, or npm install | `<repo>/.env` (npm installs clone to `~/.smartchats/cli/source`) |
| Docker container | `/root/.smartchats/.env` (mount the host file there, see below) |

Ways to write keys:

```bash
smartchats env                   # interactive walkthrough (recommended)
smartchats env --list            # show what's configured (masked)
vi ~/.smartchats/.env            # direct edit, needed for xAI / Together / Azure
```

`smartchats setup` also writes the file on first run. After editing, run `smartchats restart` (or `docker restart smartchats`) to pick up the changes.

To check which keys the CLI sees, run:

```bash
smartchats env --list
# Output:
#   ✓ OpenAI                   sk-V…vGiE  (.env $OPENAI_API_KEY)
#   ✗ Anthropic (Claude)       not set  (optional)
```

### Sharing one `.env` between bare-metal and Docker installs

The Docker container reads `/root/.smartchats/.env` (its install root), and the bare-metal install reads `~/.smartchats/.env` on the host. **Mounting the host file into the container reuses the same config**:

```bash
docker run -d --name smartchats \
  -p 3000:3000 \
  -v ~/.smartchats/.env:/root/.smartchats/.env:ro \
  -v ~/.smartchats/data:/data \
  sheunaluko/smartchats:latest
```

Edit `~/.smartchats/.env` later, then:

```bash
docker restart smartchats
```

### Adding keys after the stack is running

- **`smartchats env` on the host** (recommended): interactive prompts, writes to `~/.smartchats/.env`. Then `smartchats restart`, or `docker restart smartchats` if you mounted the file into the container.
- **Edit the file directly**: `vi ~/.smartchats/.env`, then restart. This is the only way to add xAI, Together and Azure keys.
- **Store a key in SurrealDB** with the keys API. No restart needed, and an environment key for the same provider still wins:

```bash
curl -X POST http://localhost:3000/local-api/keys \
  -H 'content-type: application/json' \
  -d '{"keys":{"together":"..."}}'
```

The API accepts `openai`, `anthropic`, `google`, `xai`, `together` and `azure`. `GET /local-api/keys/providers` shows which providers are available and where each key came from.

### Other devices and scripts

The computer running SmartChats needs nothing. A phone or another computer pairs once: it shows **Pair this device**, and you enter the code from

```bash
smartchats pair
# Pairing code: 482 913   (valid 5 min, one device)
```

or from Settings → Devices → **Pair a new device**. `smartchats devices` lists paired devices and `smartchats devices revoke <id>` removes one.

**In Docker** your own browser is another device to the container, so it pairs once too. While nothing is paired, the container prints a code at startup (`docker logs smartchats`, valid 15 minutes); for a new one run `docker exec smartchats smartchats pair`.

Scripts and agents on other machines can send `Authorization: Bearer $SMARTCHATS_API_KEY` instead, when you set that variable. `SMARTCHATS_AUTH=off` turns all of this off. Details: [Security](/docs/local-server#security).

## Running on a VPS

The same install and commands work on a VPS. For a public deploy, put a reverse proxy in front for TLS and access control.

### Install on the VPS

```bash
ssh user@vps
curl -fsSL https://smartchats.ai/install | bash
```

The installer runs the interactive `smartchats setup`. For a scripted install, write `~/.smartchats/.env` first and use the non-interactive form:

```bash
curl -fsSL https://smartchats.ai/install | bash -s -- --non-interactive
smartchats setup --no-prompt
```

### Restrict access to port 3000

`smartchats start` always binds the server to `0.0.0.0`, so port 3000 is reachable on every interface. There is no flag to change this. Block the port at the firewall and let only the reverse proxy reach it:

```bash
# ufw example: allow SSH and HTTPS, deny direct access to 3000
sudo ufw allow OpenSSH
sudo ufw allow 443/tcp
sudo ufw deny 3000/tcp
sudo ufw enable
```

SurrealDB already binds to `127.0.0.1` only.

### Front it with nginx + Let's Encrypt

Keep `Host $host` and `X-Forwarded-For` in the config: they are how the server tells visitors from this machine ([Security](/docs/local-server#security)). Visitors then pair like any other device; `auth_basic` adds a second lock if you want one.

```nginx
server {
  listen 443 ssl http2;
  server_name smartchats.your-domain.com;

  ssl_certificate     /etc/letsencrypt/live/.../fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/.../privkey.pem;

  auth_basic           "SmartChats";
  auth_basic_user_file /etc/nginx/.htpasswd;

  location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}
```

### Auto-start on boot (systemd)

Run the stack in the foreground so systemd supervises it. On SIGTERM, `start --foreground` stops both processes.

```ini
# /etc/systemd/system/smartchats.service
[Unit]
Description=SmartChats local stack
After=network.target

[Service]
Type=simple
User=YOUR_USER
WorkingDirectory=/home/YOUR_USER
ExecStart=/home/YOUR_USER/.smartchats/bin/smartchats start --foreground --no-prompt
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now smartchats
sudo systemctl status smartchats
```

### Periodic backup

```bash
#!/bin/bash
# /etc/cron.daily/smartchats-backup  (runs as root, so use full paths)
mkdir -p /home/YOUR_USER/backups
/home/YOUR_USER/.smartchats/bin/smartchats data export \
  /home/YOUR_USER/backups/$(date +%Y-%m-%d).json --target=local
```

## Docker (alternative)

There is no published image. Build one from `Dockerfile.aio` in a clone of the repo:

```bash
git clone https://github.com/sheunaluko/smartchats.git
cd smartchats
docker build -f Dockerfile.aio -t smartchats-aio:latest .
```

Each release also publishes this image as `sheunaluko/smartchats:<version>` and `:latest` on Docker Hub, and as `ghcr.io/sheunaluko/smartchats-aio` on GitHub's registry (linux/amd64 and linux/arm64), so building is optional. The Dockerfile runs `curl … install | bash` against `debian:bookworm-slim`, so the image contains a published release (pin one with `--build-arg SMARTCHATS_VERSION=v0.4.0`). Its entrypoint is `smartchats start --data-dir=/data --foreground --no-prompt`, and `/data` holds the database and media. Media storage runs inside the container and the browser reaches it through port 3000, so that is the only port to publish.

```bash
docker run -d --name smartchats \
  -p 3000:3000 \
  -v ~/.smartchats/.env:/root/.smartchats/.env:ro \
  -v ~/.smartchats/data:/data \
  sheunaluko/smartchats:latest
```

The `.env` mount is the same file the bare-metal CLI reads (`~/.smartchats/.env`), so one config works for both install modes. See [Sharing one `.env` between bare-metal and Docker installs](#sharing-one-env-between-bare-metal-and-docker-installs).

> **Warning:** **Don't run the container and a native stack on the same data directory at once.** Both store the database at `<data-dir>/surreal.db`, so `~/.smartchats/data` works for either, one at a time.

**`-e` flags** also work for one-off runs:

```bash
docker run --rm -p 3000:3000 \
  -e TOGETHER_API_KEY=$TOGETHER_API_KEY \
  -v ~/.smartchats/data:/data \
  sheunaluko/smartchats:latest
```

`bin/aio` in a clone wraps the build and run steps (`--rebuild`, `-d`, `--port`, `--data-dir`, `--tag`). It names the container `smartchats`, which is the name `smartchats doctor` checks.

Docker is useful when you need process isolation or container orchestration. For laptop self-hosting, the native install is simpler.

## Backup + restore

```bash
# Snapshot (writes a single JSON file)
smartchats data export ~/snapshot.json --target=local

# Wipe and restore (the import upserts every row by id)
smartchats stop
rm -rf ~/.smartchats/data
smartchats start
smartchats data import ~/snapshot.json --target=local
```

The fresh stack boots with an empty database, and the import writes the rows back. `data --target=local` always connects to `ws://localhost:8000/rpc`, so it needs the default `--surreal-port`.

## Troubleshooting

> **Warning:** **Port 3000 already in use** → another process is bound to it. Either stop it (`lsof -i:3000` to find it), or `smartchats start --port 3030`.

> **Warning:** **`smartchats start` exits with "surreal binary not found"** → in a binary install, `~/.smartchats/bin/surreal` is missing and there is no `surreal` on `PATH`. Re-run the installer. In a source or npm install, install SurrealDB with the command `start` prints.

> **Warning:** **The agent doesn't reply** → no provider API keys reachable. Run `smartchats env --list` to see what the CLI found, add keys to `.env`, then `smartchats restart`.

> **Note:** **Diagnosing a running stack** → `smartchats status` for a health snapshot, `smartchats logs -f` to tail per-process output, `smartchats logs --proc server -n 200` for just the server's last 200 lines.

> **Note:** **Iterating on the source?** → `smartchats dev` (Next.js dev server with hot reload, the local server and SurrealDB in Docker) instead of `smartchats start`. See [Contributing](/docs/contributing).

## See also

- **[Quick Start](/docs/quickstart)**: install and first run.
- **[CLI Reference](/docs/cli)**: every command and flag.
- **[Architecture](/docs/architecture)**: the open / cloud boundary and what's inside the stack.
- **[Contributing](/docs/contributing)**: modifying SmartChats itself.

---

Source: https://smartchats.ai/docs/agents

# For AI Agents

This page is for AI agents that install, configure or operate SmartChats on behalf of a user. Everything here runs without prompts and reports in machine-readable form. Plain-markdown copies of every docs page are at `https://smartchats.ai/docs/<page>.md`, indexed in [`/llms.txt`](https://smartchats.ai/llms.txt) and combined in [`/llms-full.txt`](https://smartchats.ai/llms-full.txt). If you are working on the SmartChats code itself, read [`AGENTS.md`](https://github.com/sheunaluko/smartchats/blob/main/AGENTS.md) in the repository instead.

## What you are installing

A local stack of three processes, started and stopped together by the `smartchats` CLI:

| Process | Port | Role |
|---|---|---|
| SurrealDB | `127.0.0.1:8000` | The user's data (logs, metrics, todos, knowledge graph, sessions) |
| SeaweedFS | `127.0.0.1:8333` | Media storage (photos, videos, files), reached only through the server. Optional |
| smartchats-server | `0.0.0.0:3000` | The web app at `/app` and the API at `/local-api/*` |

The user talks to the agent in a browser at `http://localhost:3000/app`. The model provider keys are the user's own.

## Install without prompts

macOS (Apple Silicon) and Linux (x64, arm64):

```bash
curl -fsSL https://smartchats.ai/install | bash -s -- --non-interactive
```

This installs to `~/.smartchats` and links `smartchats` into `/usr/local/bin` when it is writable (otherwise it adds `~/.smartchats/bin` to the shell rc). Add `--no-path` to touch neither, then call `~/.smartchats/bin/smartchats` directly. Add `--version v0.4.0` to pin a release. Intel Macs have no binary release: use `npm install -g smartchats-ai` (Node 24+).

Docker, anywhere:

```bash
docker run -d --name smartchats -p 3000:3000 \
  -v ~/.smartchats/.env:/root/.smartchats/.env:ro \
  -v ~/.smartchats/data:/data \
  sheunaluko/smartchats:latest
```

## Configure keys

Keys live in `~/.smartchats/.env` (mode 0600). Write them with the user's consent and never echo them back. With the keys in the environment, `setup --no-prompt` saves them and starts the stack:

```bash
TOGETHER_API_KEY=... AZURE_SPEECH_KEY=... AZURE_SPEECH_REGION=eastus \
  smartchats setup --no-prompt
```

Or write the file and start:

```bash
umask 077
cat >> ~/.smartchats/.env <<'EOF'
TOGETHER_API_KEY=...
AZURE_SPEECH_KEY=...
AZURE_SPEECH_REGION=eastus
EOF
smartchats start --no-prompt
```

| Key | Needed for | Without it |
|---|---|---|
| `TOGETHER_API_KEY` | The default chat model (DeepSeek V4.1 Flash) and the voice turn judge | Another provider's model is used if one has a key. Voice turns end on silence with no judge |
| `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` | The default voice | An OpenAI voice is used if OpenAI has a key, otherwise no speech |
| `OPENAI_API_KEY` | GPT models, OpenAI voices, embeddings (semantic search over the user's logs) | Semantic search is off |
| `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `XAI_API_KEY` | Claude, Gemini, Grok models | Those models aren't offered |
| `SERPER_API_KEY` | The agent's web search tool | Web search is a no-op |

At least one chat provider key (Together, OpenAI, Anthropic, Google or xAI) is needed for the agent to reply. Restart after changing keys: `smartchats restart`. Every variable is listed in [Providers & API Keys](/docs/providers).

## Verify

Each command exits 0 on success:

```bash
smartchats status --json
# {"up": true, "procs": [{"name": "surreal", "pid": 1234, "port": 8000, "pidAlive": true, "portHealthy": true, ...}, {"name": "seaweedfs", ...}, {"name": "server", ...}]}

smartchats doctor --json
# {"exit": 0, "results": [{"name": "LLM provider keys", "status": "pass", "severity": "warn", "note": "found: Together, Azure ..."}, ...], "port": 3000}

curl -s http://localhost:3000/local-api/health
# {"ok": true, "checks": {"data": {"ok": true, ...}, "providers": {"ok": true, ...}}}

curl -s http://localhost:3000/local-api/keys/providers
# {"llm": [{"provider": "together", "available": true, ...}, ...], "tts": [{"provider": "azure", "available": true, ...}, ...]}
```

From a clone of the repository, `node scripts/smoke-stack.mjs http://localhost:3000` exercises the stack the way a user would: the app loads, a database query, an LLM reply, and a media upload and download. It prints one line per check and exits non-zero if any fails.

> **Note:** **From the machine running SmartChats** these need no credentials. From anywhere else `/local-api` answers `401 pairing_required`: use `Authorization: Bearer $SMARTCHATS_API_KEY` (set it on the server), or pair with a code from `smartchats pair`. In Docker, run commands inside the container: `docker exec smartchats smartchats status`.

## Operate

| Task | Command |
|---|---|
| Start / stop / restart | `smartchats start --no-prompt`, `smartchats stop`, `smartchats restart` |
| Pair a phone or another computer | `smartchats pair --json` → `{"code", "expires_at"}`; list and revoke with `smartchats devices [--json]`, `smartchats devices revoke <id>` |
| Logs | `smartchats logs --proc server -n 100` (or `surreal`, `seaweedfs`), files in `~/.smartchats/logs/` |
| Run in the foreground (systemd, containers) | `smartchats start --foreground --no-prompt` |
| Other ports | `smartchats start --port 3100 --surreal-port 8100` (media: `SMARTCHATS_S3_PORT=8433`) |
| Back up the user's data | `smartchats data export ~/smartchats-backup.json` |
| Restore | `smartchats data import ~/smartchats-backup.json --dry-run`, then without `--dry-run` |
| Upgrade | `smartchats upgrade -y` |

Process IDs and ports are in `~/.smartchats/run/pids.json`. The database is in `~/.smartchats/data/surreal.db` and media in `~/.smartchats/data/seaweedfs/`.

## When something is wrong

| Symptom | Likely cause | Fix |
|---|---|---|
| `status` shows `server` with `portHealthy: false` | The server failed to start | `smartchats logs --proc server -n 50`. Often a port in use: start with `--port` |
| Health `providers.ok` is false, the agent never replies | No chat provider key | Add one to `~/.smartchats/.env`, then `smartchats restart` |
| Replies work but there's no voice | No Azure key and no OpenAI key | Add `AZURE_SPEECH_KEY` + `AZURE_SPEECH_REGION` (or `OPENAI_API_KEY`) |
| Uploads fail in the app, `createUpload` answers 501 | Media storage is off | `smartchats doctor` shows why (no `weed` binary, or `SMARTCHATS_STORAGE=off`) |
| Every `/local-api` call returns 401 `pairing_required` | The caller is not the machine running SmartChats (another device, or the host of a Docker container) | Pair (`smartchats pair`, then enter the code in the app), or send `Authorization: Bearer $SMARTCHATS_API_KEY` |
| The app shows **Pair this device** | Same | Give the user the code from `smartchats pair` (Docker: `docker exec smartchats smartchats pair`) |
| `smartchats: command not found` after install | PATH not updated in this shell | Use `~/.smartchats/bin/smartchats`, or open a new shell |

## Give the user's agent their data

SmartChats exposes the user's data to other agents (Claude Desktop, Claude Code, Cursor and others) through an MCP server with tools to read and write logs, metrics, todos and the knowledge graph. See [MCP Server](/docs/mcp).

## Ground rules

- The data is the user's: logs, journal entries and transcripts are personal. Don't copy them anywhere the user didn't ask for.
- Keys stay in `~/.smartchats/.env` with mode 0600. Don't print them, commit them, or pass them on the command line where other users can see them.
- The server listens on all interfaces. Other devices must pair, but don't expose port 3000 to the internet without TLS and a reverse proxy that forwards `Host` and `X-Forwarded-For` ([Security](/docs/local-server#security)). Pair a device only when the user asks.

## See also

- [Quick Start](/docs/quickstart)
- [Providers & API Keys](/docs/providers)
- [Local Server](/docs/local-server)
- [CLI Reference](/docs/cli)

---

Source: https://smartchats.ai/docs/providers

# Providers & API keys

A self-hosted SmartChats server calls model and service providers with your own API keys. This page lists every provider it can use, the environment variables it reads, and what each feature needs.

## Providers

| Provider | Env vars (first match wins) | Used for | Needed by default? |
|---|---|---|---|
| Together AI | `SMARTCHATS_TOGETHER_API_KEY`, `TOGETHER_API_KEY`, `TOGETHERAI_API_KEY` | Open-weight chat models, including the default model. The default voice turn verdict. | **Yes**: default chat model and turn verdict |
| Azure Speech | `SMARTCHATS_AZURE_SPEECH_KEY`, `AZURE_SPEECH_KEY` plus `SMARTCHATS_AZURE_SPEECH_REGION`, `AZURE_SPEECH_REGION` | Text to speech (the default voice) | **Yes**: default voice |
| OpenAI | `SMARTCHATS_OPENAI_API_KEY`, `OPENAI_API_KEY` | GPT chat models, embeddings (semantic search), OpenAI text to speech | **For semantic search**: the only embeddings SmartChats uses today |
| Anthropic | `SMARTCHATS_ANTHROPIC_API_KEY`, `ANTHROPIC_API_KEY` | Claude chat models | No |
| Google (Gemini) | `SMARTCHATS_GOOGLE_API_KEY`, `GOOGLE_API_KEY`, `GEMINI_API_KEY` | Gemini chat and image models | No |
| xAI | `SMARTCHATS_XAI_API_KEY`, `XAI_API_KEY` | Grok chat models | No |
| Serper | `SMARTCHATS_SERPER_API_KEY`, `SERPER_API_KEY` | The agent's web search tool | No |
| Your own S3-compatible bucket | `SMARTCHATS_S3_ENDPOINT`, `SMARTCHATS_S3_ACCESS_KEY_ID`, `SMARTCHATS_S3_SECRET_ACCESS_KEY`, `SMARTCHATS_S3_BUCKET`, optional `SMARTCHATS_S3_REGION` (default `us-east-1`) | Media storage, instead of the SeaweedFS that `smartchats start` runs | No |

The variable names come from `packages/smartchats-local-server/src/config.ts`. For each provider the server checks, in order:

1. `SMARTCHATS_<PROVIDER>_API_KEY`
2. the provider's usual name, such as `TOGETHER_API_KEY` (and any alias listed above)
3. a key stored in the database with `POST /keys`

The `SMARTCHATS_` name always wins. Use it when another tool on the machine already sets the plain name to a different key.

### Chat models by provider

Each model in the registry (`packages/cortex/src/model_registry.ts`) belongs to one provider, and needs that provider's key:

| Provider | Models |
|---|---|
| Together AI | DeepSeek V4.1 Flash (the default), GLM, Kimi, Qwen, MiniMax, gpt-oss |
| OpenAI | GPT-4.1, GPT-5 family, GPT-6 family, o4-mini |
| Anthropic | Claude Haiku, Sonnet, Opus and Fable |
| Google | Gemini Flash, Flash Lite and Pro, plus an image model |
| xAI | Grok |

The app's preset menu greys out choices whose provider has no key. It reads availability from `GET /local-api/keys/providers`.

## Setting keys

### With the CLI

The CLI writes keys to a `.env` file that `smartchats start` loads on boot:

- **Binary install** (the `curl … | bash` installer): `~/.smartchats/.env`. Docker mounts the same file.
- **Source install** (a clone of the repo): `.env` at the repo root.

```bash
smartchats env          # walk the providers and set or update each key
smartchats env --list   # show what is configured, masked
smartchats setup        # first run: also prompts for keys
smartchats restart      # pick up changes on a running stack
```

`smartchats env` and `setup` prompt for OpenAI, Anthropic, Google and Serper. For any other provider, add the line to the `.env` file yourself. `smartchats start` passes every variable in that file to the server, and a variable already set in your shell wins over the file.

```ini
# ~/.smartchats/.env
TOGETHER_API_KEY=...
AZURE_SPEECH_KEY=...
AZURE_SPEECH_REGION=eastus
XAI_API_KEY=...
```

### With POST /keys

Keys can also be stored in the database, through the BYO Keys section of the app's settings or directly:

```bash
curl -X POST http://127.0.0.1:3000/local-api/keys \
  -H 'Content-Type: application/json' \
  -d '{"keys": {"together": "..."}}'
```

| Route | What it does |
|---|---|
| `POST /local-api/keys` | Stores keys for `openai`, `anthropic`, `google`, `xai`, `together` or `azure` |
| `GET /local-api/keys` | Masked preview of stored keys only (not env vars) |
| `DELETE /local-api/keys/:provider` | Removes a stored key |
| `GET /local-api/keys/providers` | Which LLM and TTS providers have a usable key, and from where |

These work as shown from the machine running SmartChats. From another machine, add `-H "Authorization: Bearer $SMARTCHATS_API_KEY"` (set that variable on the server first).

> **Warning:** **Env vars shadow stored keys.** A key from the environment is used before a stored one. `POST /keys` saves the key anyway and returns a `warning` naming the providers it will ignore at call time.

> **Warning:** **Some keys are env-only.** Text to speech reads the Azure and OpenAI keys from the environment only, so an Azure key stored with `POST /keys` does not enable the voice. The Serper key and the S3 settings also come only from the environment.

## What each feature needs

| Feature | Needs | Without it |
|---|---|---|
| Chat with the default model | Together | The request fails with `no together API key configured`, naming `SMARTCHATS_TOGETHER_API_KEY` and `POST /keys`. Pick a model from a provider you have a key for. |
| Chat with any other model | That model's provider | Same error, naming that provider |
| Voice turn verdict (default judge) | The verdict model's provider. Together for the default `deepseek-v4.1-flash` | The server skips the verdict and answers the turn, as if it had said COMPLETE. Turns are never held, so a pause mid-sentence ends the turn. |
| Default voice (Azure Emma) | Azure key **and** region | With an OpenAI key, the reply is spoken with an OpenAI voice instead. With neither, there is no speech. |
| OpenAI voices | OpenAI | Falls back to Azure if Azure is configured |
| Embeddings (`text-embedding-3-small`) | OpenAI | The embeddings route returns an error. Saved logs are written without a vector, and search by meaning does not work. Adding a key later does not embed entries saved before it. |
| Web search | Serper | The search tool returns `search disabled: SMARTCHATS_SERPER_API_KEY or SERPER_API_KEY not set` |
| Media storage | No key: `smartchats start` runs SeaweedFS (bundled with binary installs) | Without SeaweedFS and without your own `SMARTCHATS_S3_*` bucket, storage routes return 501 and uploads are off. |

> **Note:** **The turn verdict needs token probabilities.** Judge v2 decides from the model's probability that the user is done, read from the first token's logprobs. In llm-service only the Together adapter returns them today. With a verdict model from another provider, the verdict is decided from the sampled word and its threshold has no effect. A verdict is one short call per voice turn, well under a cent with the default model. See [Voice engine](/docs/voice-engine).

## See also

- [Voice engine](/docs/voice-engine): how the turn verdict uses these keys
- [Self-Hosting](/docs/self-host): env-var management, bearer-token auth and Docker
- [CLI Reference](/docs/cli): `smartchats env` and `smartchats setup`
- [TURN_VERDICT.md](https://github.com/sheunaluko/smartchats/blob/main/packages/llm-service/TURN_VERDICT.md): the verdict's providers and self-hosting note

---

Source: https://smartchats.ai/docs/cli

# CLI Reference

`smartchats` is the command-line interface for the SmartChats stack. One tool covers the whole lifecycle: install, configure, launch, observe, stop, upgrade, sign in to the cloud, and move data around.

Cloud auth is shared with the MCP server. Log in once with `smartchats login` and any tool reading `~/.smartchats-mcp/credentials.json` is authenticated too.

## Install

See [Quick Start](/docs/quickstart). The short version:

```bash
curl -fsSL https://smartchats.ai/install | bash
```

## Top-level

```bash
smartchats <command> [options]
```

If you run `smartchats` with no arguments:

- **No config and no source reachable** → runs `setup` (guided first run).
- **Otherwise** → runs `start` on the last-used port.

If the first argument is a flag other than `--help` or `--version`, the CLI routes to the legacy `launch` command (for older `npx smartchats --no-prompt` scripts).

To check the installed version, run:

```bash
smartchats --version
# Output: 0.x.y
```

`-v` and `-V` work too. Use `smartchats help <command>` or `smartchats <command> --help` for per-command details.

## Lifecycle

### `smartchats setup`

Guided first run. Detects the install, checks the system, prompts for API keys, writes `.env`, and starts the stack.

```bash
smartchats setup [options]
```

| Flag | Default | Notes |
|---|---|---|
| `--no-prompt` | off | Use existing env / `.env` values. Skip all prompts |
| `--no-start` | off | Collect config only. Don't start the stack |
| `--repo-path <path>` | `~/.smartchats/cli/source` | Where to clone the source (source and npm installs only) |

What it walks through, in order:

1. **Install**: a binary install uses `~/.smartchats`. Otherwise it locates a source checkout (`$SMARTCHATS_HOME`, a directory walk, `config.smartchatsHome`) or clones one to `~/.smartchats/cli/source` (`$XDG_DATA_HOME/smartchats/cli/source` if that variable is set).
2. **System check**: free disk (warns below 3 GB). Source and npm installs also need `bun` and `surreal`, and setup stops with an install command if either is missing.
3. **API keys**: walks OpenAI, Anthropic, Google and Serper. For each, it shows what it found in your environment or `.env` and lets you keep or replace it.
4. **Write + start**: writes `.env` and runs `start`. The stack is up at [http://localhost:3000](http://localhost:3000) when the wizard exits.

### `smartchats env`

Interactively configure provider API keys. Same prompt loop as the key step of `setup`, with no system check and no start. Use it to rotate keys or enable a provider later.

```bash
smartchats env [options]
```

| Flag | Default | Notes |
|---|---|---|
| `-l, --list` | off | Show what's configured (masked). Don't prompt |

It walks OpenAI, Anthropic, Google (Gemini) and Serper. For each one it shows what's already set and lets you keep it, replace it, or remove it. It writes to `~/.smartchats/.env` for a binary install (the same file you mount into Docker at `/root/.smartchats/.env`) or `<repo>/.env` for a source install.

> **Note:** **`env` doesn't prompt for xAI, Together AI or Azure Speech.** Add `XAI_API_KEY`, `TOGETHER_API_KEY`, `AZURE_SPEECH_KEY` and `AZURE_SPEECH_REGION` to the `.env` file by hand. See [Required environment](/docs/self-host#required-environment).

If the stack is running, restart afterwards to pick up the new values:

```bash
smartchats restart           # bare-metal install
docker restart smartchats    # docker install
```

To check what's configured without prompts, run:

```bash
smartchats env --list
# Output:
#   ✓ OpenAI                   sk-V…vGiE  (env $OPENAI_API_KEY)
#   ✓ Anthropic (Claude)       sk-a…iQAA  (.env $ANTHROPIC_API_KEY)
#   ✓ Google (Gemini)          AIza…Uvbg  (env $GEMINI_API_KEY)
#   ✗ Serper (web search)      not set  (optional)
```

### `smartchats start`

Launches the local stack as detached child processes. Re-running while the stack is up and healthy prints the URL and exits.

```bash
smartchats start [options]
```

| Flag | Default | Notes |
|---|---|---|
| `--port <n>` | 3000 | App port (browser connects here) |
| `--surreal-port <n>` | 8000 | SurrealDB port (binds to `127.0.0.1`) |
| `--data-dir <path>` | `~/.smartchats/data` | SurrealDB storage. The database is `<data-dir>/surreal.db` |
| `--rebuild` | off | Source mode only: run the workspace build before starting. Ignored in a binary install |
| `-f, --foreground` | off | Stay attached. Ctrl-C or SIGTERM stops both processes |
| `--no-prompt` | off | Skip the "configure keys now?" prompt that appears when no provider key is found. Use in scripts, CI and Docker |

The stack is two processes: `surreal` and `smartchats-server`. The server hosts the static SPA and the API at `/local-api/*` on the same port. It always binds to `0.0.0.0`.

`start` reads `.env` from the install root (or repo root) and your shell environment. A shell variable wins over the same name in `.env`.

When detached (the default), `start` returns 0 once both processes report ready. PIDs are written to `~/.smartchats/run/pids.json`.

### `smartchats stop`

Stops the running stack. Sends SIGTERM, waits, then sends SIGKILL.

```bash
smartchats stop [--timeout <ms>]
```

| Flag | Default | Notes |
|---|---|---|
| `--timeout <ms>` | 5000 | How long to wait after SIGTERM before SIGKILL |

### `smartchats restart`

`stop`, then `start`. All `start` flags are forwarded.

```bash
smartchats restart [start-options...]
```

### `smartchats status`

Shows what's running, on what ports, with what health. Exits 0 if the stack is up and serving, 1 otherwise.

```bash
smartchats status [--json]
```

To check the stack, run:

```bash
smartchats status
# Output:
#   surreal  pid ✓ 12345  port ✓ 8000   up 2m 14s
#   server   pid ✓ 12346  port ✓ 3000   up 2m 14s
#
# ✔ Stack healthy. Open: http://localhost:3000
```

`--json` prints `{ "up": ..., "procs": [...] }`, or `{ "up": false, "reason": "no-pid-file" }` when nothing is running.

### `smartchats logs`

Tails per-process log files (`~/.smartchats/logs/surreal.log`, `~/.smartchats/logs/server.log`).

```bash
smartchats logs [options]
```

| Flag | Default | Notes |
|---|---|---|
| `-f, --follow` | off | Stream new output (Ctrl-C to stop) |
| `--proc <surreal\|server\|all>` | all | Which process(es) to show |
| `-n, --lines <n>` | 50 | Initial line count |

### `smartchats dev`

Hot-reload development stack. Runs `bin/devserve` in the smartchats source, so it needs a source checkout.

```bash
smartchats dev [-- <devserve-args>]
```

Anything after `--` goes to `bin/devserve` unchanged. It has two target modes:

| `--target` | Notes |
|---|---|
| `surreal` (default) | SurrealDB in a Docker container on port 8000, plus the local server and the Next.js dev server with hot reload |
| `aio` | Runs `bin/aio`: the all-in-one Docker image. Add `--rebuild` to rebuild it |

Other `devserve` flags: `--port=<n>` (Next.js port, default 3000), `--no-server`, `--no-app`, `--no-prebuild`, `--no-surreal`, `--clean`. Run `bin/devserve --help` for details.

```bash
smartchats dev -- --target=aio --rebuild
```

See [Contributing](/docs/contributing) for the development workflow.

### `smartchats upgrade`

Upgrades a binary install by re-running the installer in place. Your data directory, `.env` and `config.json` are kept.

```bash
smartchats upgrade [options]
```

| Flag | Default | Notes |
|---|---|---|
| `--version <tag>` | `latest` | Install a specific release, for example `v0.4.0` |
| `-y, --yes` | off | Skip the confirmation prompt |

It runs `curl -fsSL https://smartchats.ai/install | bash -s -- --non-interactive --version <tag> --prefix <install-root>`. Run `smartchats restart` afterwards to use the new binaries.

For an npm install it prints `npm install -g smartchats-ai@latest` and exits 1. For a source checkout it tells you to `git pull` and exits 1.

## Devices

The computer running SmartChats needs no pairing. A phone or another computer shows **Pair this device** and asks for a code. See [Security](/docs/local-server#security).

### `smartchats pair`

Prints a 6-digit code for one device, valid 5 minutes.

```bash
smartchats pair [--json] [--port <n>]
# Pairing code: 482 913   (valid 5 min, one device)

docker exec smartchats smartchats pair    # Docker
```

`--json` prints `{"code": "482913", "expires_at": 1760000000000}`.

### `smartchats devices`

```bash
smartchats devices [--json]          # id, name, paired, last used
smartchats devices revoke <id>       # the device must pair again
```

## Cloud

### `smartchats login`

Browser sign-in to the SmartChats cloud. Clears any cached token first, then caches the new credentials at `~/.smartchats-mcp/credentials.json`, the same file the MCP server reads.

```bash
smartchats login
```

### `smartchats logout`

Clears cached cloud credentials.

```bash
smartchats logout
```

### `smartchats whoami`

Shows the signed-in cloud user (uid, email, project, credentials file). If no credentials are cached, it starts the browser sign-in.

```bash
smartchats whoami
```

### `smartchats data`

Moves user data between deployments as a JSON bundle. Use it for backups, debugging, and moving between local and cloud.

```bash
smartchats data export <file> [options]
smartchats data import <file> [options]
```

| Flag | Default | Notes |
|---|---|---|
| `--target=<cloud\|local>` | `local` | `cloud` uses your `smartchats login` credentials. `local` connects to `ws://localhost:8000/rpc` as root |
| `--file=<path>` | none | Bundle path. The first positional argument works too |
| `--tables=t1,t2` | see below | Export: tables to export. Import: subset of the bundle's tables to write |
| `--include-sensitive` | off | Export only: also export `byo_api_keys` and `usage_records` |
| `--page-size=<n>` | 100 | Export only: rows per page |
| `--dry-run` | off | Import only: read and count rows, don't write |

Each flag also accepts a space instead of `=` (for example `--target cloud`).

Without `--tables`, export writes `logs`, `metrics`, `user_entities`, `user_relations`, `user_data`, `app_data`, `sessions`, `cortex`, `cortex_dynamic_functions`, `smartchats_apps` and `smartchats_app_installs`. Telemetry tables are never exported. The bundle file is written with `0600` permissions.

Import upserts every row by id, so it's safe to re-run. Rows are written as the user of the target deployment.

```bash
smartchats data export ~/backup.json --target=cloud
smartchats data import ~/backup.json --target=local --dry-run
smartchats data import ~/backup.json --target=cloud --tables=logs,metrics
```

> **Note:** **`--target=local` always uses port 8000.** If you started the stack with a different `--surreal-port`, the local target can't reach it.

## Meta

### `smartchats doctor`

Environment health check. Diagnoses what's installed, what's running, and what's reachable.

```bash
smartchats doctor [--port <n>] [--json]
```

| Flag | Default | Notes |
|---|---|---|
| `--port <n>` | 3000 | Port to probe for the running stack |
| `--json` | off | Print the full result set as JSON |

Checks (in order):

1. **SmartChats context**: where is the source root?
2. **bun installed**: critical.
3. **surreal installed**: critical.
4. **Free disk ≥ 3 GB**: warning only.
5. **Docker installed**: informational.
6. **Image `smartchats-aio:latest` present**: informational.
7. **Container `smartchats` running**: informational.
8. **`http://localhost:<port>` responds**: critical when the `smartchats` container is running.
9. **Response is SmartChats**: sanity check (catches port collisions).
10. **LLM provider keys**: warning if no OpenAI, Anthropic or Google key is found.

Exits 0 if every critical check passes, 1 otherwise.

> **Warning:** **`doctor` looks for `bun` and `surreal` on `PATH`, even in a binary install.** A binary install doesn't need a separate `bun`, so this check can fail on a working install. `doctor` also reads `.env` from the source root only, so use `smartchats env --list` to check keys in a binary install.

### `smartchats home`

Prints where smartchats lives and how the CLI found it.

```bash
smartchats home [--path-only]
```

In a binary install it prints the install root and the server, surreal and SPA paths. `--path-only` prints just one path (the install root in a binary install, otherwise the source root), or exits 1 with no output if neither is found:

```bash
cd "$(smartchats home --path-only)"
```

Source root resolution order: `$SMARTCHATS_HOME` → walking up from the current directory looking for `Dockerfile.aio` → `config.smartchatsHome` (written after an auto-clone).

### `smartchats help <command>`

Per-command help. Same as `<command> --help`.

## Legacy

### `smartchats launch`

Hidden from `--help` but still callable. Builds and runs the Docker all-in-one image (`smartchats-aio:latest`) from the smartchats source, cloning it first if needed. Use `smartchats start` for new installs.

```bash
smartchats launch [options]
```

Flags: `--no-prompt`, `--rebuild`, `-d, --detached`, `--port <n>`, `--data-dir <path>`, `--tag <tag>`, `--test`, `--repo-path <path>`. Run `smartchats launch --help` for details.

## Environment

| Variable | Default | Purpose |
|---|---|---|
| `SMARTCHATS_HOME` | unset | Path to a smartchats source checkout. Overrides the directory walk and config fallback |
| `SMARTCHATS_CONFIG_FILE` | unset | Full path to the CLI config file |
| `XDG_CONFIG_HOME` | unset | If set, CLI config lives at `$XDG_CONFIG_HOME/smartchats/config.json` (else `~/.smartchats/config.json`) |
| `XDG_DATA_HOME` | unset | If set, auto-clones go to `$XDG_DATA_HOME/smartchats/cli/source` |
| `SMARTCHATS_REPO_URL` | `https://github.com/sheunaluko/smartchats.git` | Source URL for the auto-clone |
| `SMARTCHATS_CREDENTIALS_FILE` | `~/.smartchats-mcp/credentials.json` | Cloud credentials location |
| `SMARTCHATS_API_KEY` | unset | Bearer token the server accepts from scripts and agents on other machines ([Security](/docs/local-server#security)) |
| `SMARTCHATS_AUTH` | `pair` | `off`: no pairing, every device gets in |
| `SMARTCHATS_LOCAL_HOST` | `127.0.0.1` | Dev mode only: host the Next.js `/local-api/*` rewrite forwards to |
| `SMARTCHATS_LOCAL_PORT` | `4242` | Dev mode only: port for that rewrite |
| `NO_COLOR` | unset | Disables CLI color output |

Provider keys are read from your environment and the `.env` file. A shell variable wins over `.env`. For each provider, the first name set wins, in this order:

| Provider | Names |
|---|---|
| OpenAI | `SMARTCHATS_OPENAI_API_KEY`, `OPENAI_API_KEY` |
| Anthropic | `SMARTCHATS_ANTHROPIC_API_KEY`, `ANTHROPIC_API_KEY` |
| Google (Gemini) | `SMARTCHATS_GOOGLE_API_KEY`, `GOOGLE_API_KEY`, `GEMINI_API_KEY` |
| xAI | `SMARTCHATS_XAI_API_KEY`, `XAI_API_KEY` |
| Together AI | `SMARTCHATS_TOGETHER_API_KEY`, `TOGETHER_API_KEY`, `TOGETHERAI_API_KEY` |
| Azure Speech | `SMARTCHATS_AZURE_SPEECH_KEY`, `AZURE_SPEECH_KEY` (plus `SMARTCHATS_AZURE_SPEECH_REGION`, `AZURE_SPEECH_REGION`) |
| Serper | `SMARTCHATS_SERPER_API_KEY`, `SERPER_API_KEY` |

The CLI prompts for the OpenAI, Anthropic, Google and Serper keys. The server reads all of them. See [Self-Hosting](/docs/self-host#environment-configuration) for server settings.

## Files

| Path | Purpose |
|---|---|
| `~/.smartchats/bin/` | Installed binaries |
| `~/.smartchats/app/out/` | Static SPA bundle |
| `~/.smartchats/data/surreal.db` | SurrealDB rocksdb store |
| `~/.smartchats/run/pids.json` | Live process state for the running stack |
| `~/.smartchats/logs/{surreal,server}.log` | Per-process append logs |
| `~/.smartchats/config.json` | CLI preferences (last port, resolved source root) |
| `~/.smartchats/.env` | Provider keys and config, binary install |
| `<repo>/.env` | Provider keys and config, source or npm install |
| `~/.smartchats/cli/source/` | Default auto-clone location when no source is found |
| `~/.smartchats-mcp/credentials.json` | Cloud credentials, shared with the MCP server |

## See also

- **[Quick Start](/docs/quickstart)**: install and first run.
- **[Self-Hosting](/docs/self-host)**: long-form deployment guide.
- **[Contributing](/docs/contributing)**: modifying SmartChats itself.
- **[MCP](/docs/mcp)**: expose smartchats to Claude Desktop and other MCP-aware tools via the shared credential store.

---

Source: https://smartchats.ai/docs/architecture

# Architecture

This page gives an overview of the whole system first, then goes into
the data layer, which most contributions touch.

## Request flow

The browser app does most of the work: it runs the agent loop, the voice
engine and the code sandbox. A server (the local server when you
self-host, the hosted cloud otherwise) holds the API keys, calls the
model providers, and owns the database and media storage.

```
 browser: SmartChats app
 ┌───────────────────────────────────────────────┐
 │ mic ─► tivi (VAD, speech recognition)         │
 │          │                                    │
 │          ▼                                    │
 │        vie (turn-taking) ── turn ready ──┐    │
 │                                          ▼    │
 │        cortex (agent loop) ◄── typed text     │
 │          │                                    │
 │          ▼                                    │
 │        SmartChatsBackend (local or cloud)     │
 └──────────┬────────────────────────────────────┘
            │ HTTP, NDJSON stream back
            ▼
 server: smartchats-local-server (/local-api/*)
 ┌───────────────────────────────────────────────┐
 │ /llm/streamWithTTS                            │
 │   1. turn verdict (small model): done talking?│
 │        INCOMPLETE ─► end response, keep       │
 │                      listening                │
 │   2. main model ─► llm-service providers      │
 │   3. sentences ─► TTS ─► audio frames         │
 │ /data  ─► SurrealDB                           │
 │ /storage ─► S3 bucket (e.g. SeaweedFS)        │
 │ /tools ─► web search, URL fetch               │
 └───────────────────────────────────────────────┘
```

## The pieces

### The app and its shells

`apps/smartchats` is a Next.js app built as a static single-page app.
Its UI is a set of shells that share one store: the default desktop
shell and a mobile shell, picked in settings, plus shells mounted on
their own routes, `/sail` and `/voicelab`. Built-in mini apps
(log, metrics and knowledge-graph explorers, todos and others) live in
`app/apps/`. The app talks to its server only through the
`SmartChatsBackend` interface described [below](#the-facade-pattern-in-practice).

### cortex: the agent loop

`packages/cortex` is the agent framework. It composes the system prompt,
calls the model through the backend, parses structured output, runs the
functions and code the model asks for in a sandbox, and keeps workspace
state across turns. It runs in the browser and in Node. Hosts inject
the sandbox and the transport. It also holds the model
catalog, presets and voices.

### llm-service: providers, streaming and the turn verdict

`packages/llm-service` is the provider layer used by both servers. It
normalizes OpenAI, Anthropic, Gemini, xAI and Together AI into one
request and stream shape, with structured output, effort levels and
bring-your-own keys. Text-to-speech adapters cover OpenAI and Azure.

The combined LLM and TTS handler streams NDJSON: text deltas, then
audio for each sentence as soon as it is ready. When a voice request
carries a `verdict` block, the handler first makes one small model call
that judges whether the user has finished speaking, and writes that
verdict frame before any delta. On INCOMPLETE it ends the response
without starting the main call. See [Providers](/docs/providers).

### tivi and vie: voice in and turn-taking

`packages/tivi` provides the voice interface as React hooks: voice
activity detection, browser speech recognition, the TTS playback queue
and calibration. `packages/vie` is the voice engine and the default
turn-taking engine. It decides when a turn has ended from VAD and
recognizer signals, asks a judge (the server verdict by default, or the
local Smart Turn model), and tunes its timing per device from logged
evidence. The older engine stays behind the `voiceEngine: 'legacy'`
setting. See [Voice engine](/docs/voice-engine).

### smartchats-local-server

`packages/smartchats-local-server` is the self-hosted Express server.
It serves the app and the API under `/local-api/*`: `health`, `llm`,
`tts`, `embeddings`, `data`, `usage`, `keys`, `tools`, `insights` and
`storage`. Data lives in SurrealDB. Media uploads go straight from the
browser to an S3-compatible bucket through signed links, with a `media`
table tracking each object. `smartchats start` runs SeaweedFS as that
bucket and sets the `SMARTCHATS_S3_*` variables, or you point them at
your own. See
[Local server](/docs/local-server).

### smartchats-hub, the relay and the MCP bridge

These let coding agents talk with you through the app.
`smartchats-relay-cf` is the WebSocket relay between agents and your
browser, partitioned by user. `smartchats-relay-protocol` defines its
message envelope and `smartchats-relay-connect` handles connect,
reconnect and re-auth. `smartchats-mcp-bridge` is an MCP server one
agent runs to reach the relay. `smartchats-hub` is a per-machine daemon
that holds one relay connection for every local agent, refreshes the
token once, keeps a durable inbox per agent and serves MCP on
`127.0.0.1`. See [Hub](/docs/hub) and [Relay](/docs/relay).

### smartchats-tools

`packages/smartchats-tools` holds the agent's I/O helpers, such as web
search through Serper and URL fetch with readable-text extraction. They
are pure functions with no host dependencies, so the local server's
`/tools/*` routes and the cloud use the same code.

### MCP

`packages/smartchats-mcp` exposes the data layer to MCP clients over
stdio. See [MCP Server](/docs/mcp).

## The data layer

The data layer follows three rules:

1. **One contract for data access.** Every consumer (browser app, CLI,
   MCP server) talks to data through the same `DataAPI` interface.
2. **Two implementations.** Cloud (your smartchats.ai account) and local
   (your own install). Both implement the same contract.
3. **One home for each kind of change.** Query builders, multi-step
   operations and schema changes each have a set place, and consumers
   pick them up.

A new feature is usually one small change in one place. If one feature
touches five files, look for the existing home first.

### Layers

```
                    smartchats web app                                 what users see
                            │
                            ▼
                  getBackend().data.query(...)
                            │
       ┌────────────────────┴──────────────────┐
       │                                       │
   smartchats-cli                       smartchats-mcp                  other consumers
       │                                       │
       └────────┬──────────────────────────────┘
                ▼
    operations.importBundle / exportBundle                              multi-step ops
                ▼
     queries.X(args) → { query, variables }                             pure builders (no I/O)
                ▼
                DataAPI.query                                           contract (smartchats-backend)
              ╱            ╲
   makeCloudDataAPI       makeLocalDataAPI                              factories (smartchats-database)
        │                       │
   cloud-client              createClient
   (hosted cloud)            (your local SurrealDB)
```

| Layer | Responsibility | Lives in |
|---|---|---|
| Web app | Voice and chat UI, agent loop, KG views | `apps/smartchats` |
| CLI | Setup, start and stop, login, data import and export | `packages/smartchats-cli` |
| MCP | Data tools for MCP clients | `packages/smartchats-mcp` |
| Operations | Multi-step work: bundle import and export | `packages/smartchats-database/src/operations/` |
| Query builders | Pure functions: args → `{ query, variables }`, no I/O | `packages/smartchats-database/src/queries/` |
| `DataAPI` contract | Two methods: `query()` and `healthCheck()` | `packages/smartchats-backend/src/types.ts` |
| Cloud factory | Wraps `smartchats-cloud-client` and cloud auth | `packages/smartchats-database/src/data_api.ts` |
| Local factory | Wraps the SDK-direct `createClient` | `packages/smartchats-database/src/data_api.ts` |

### Adding a query the agent can run

Three files, about 20 lines.

1. **Add a builder** in `packages/smartchats-database/src/queries/your_topic.ts`:

   ```ts
   export function findRecentByTag(args: { tag: string; limit?: number }): QuerySpec {
     return {
       query: 'SELECT * FROM logs WHERE tags CONTAINSALL [$tag] ORDER BY ts DESC LIMIT $limit',
       variables: { tag: args.tag, limit: args.limit ?? 20 },
     };
   }
   ```

2. **Expose it to MCP** in `packages/smartchats-mcp/src/tools.ts`:

   ```ts
   server.tool(
     'find_recent_by_tag',
     'Find recent logs tagged with a given label',
     { tag: z.string(), limit: z.number().optional() },
     async (args) => runAndFormat(queries.findRecentByTag(args), handle),
   );
   ```

3. **Expose it to the CLI** as a subcommand in `packages/smartchats-cli/src/commands/`:

   ```ts
   const result = await handle.data.query(queries.findRecentByTag(args));
   console.log(JSON.stringify(result.rows, null, 2));
   ```

The same builder works against the cloud (after `smartchats login`) and
against a local install. The builder has no transport-specific code.

### Adding a multi-step operation

If the work is more than one query, for example looping over a bundle
and writing each row with progress and error reporting, put it in
`operations/`:

```ts
// packages/smartchats-database/src/operations/your_op.ts
import type { DataAPI } from 'smartchats-backend';

export async function yourOp(data: DataAPI, args: YourOpArgs): Promise<YourOpResult> {
  // loops, error aggregation, progress callbacks
  await data.query(...);
}
```

The CLI and MCP then both call `operations.yourOp(handle.data, args)`.
See `import_bundle.ts` and `export_bundle.ts` for real examples.

## The facade pattern in practice

Two interfaces wrap the backends.

### 1. `SmartChatsBackend` (in the web app)

The web app uses the full `SmartChatsBackend` interface from
`packages/smartchats-backend/src/types.ts`. Besides `data` it has `llm`,
`tts`, `embeddings`, `usage`, `keys`, `billing`, `tools`,
`integrations`, `insights` and `storage`, plus `capabilities`,
`health()` and `providers()`.

```ts
// apps/smartchats/src/lib/backend_facade.ts
const backend = getBackendInstance();

await backend.llm.streamWithTTS(...);         // chat completion with speech
await backend.tts.stream(...);                // text-to-speech
await backend.embeddings.embed(...);          // embedding vectors
await backend.data.query(...);                // SurrealDB read/write
await backend.storage.createUpload(...);      // media upload links
await backend.insights.emit(events);          // telemetry
const health = await backend.health();        // aggregate health probe
const report = await backend.providers();     // which LLM / TTS providers can serve
```

`BackendFacadeProvider` puts the active backend into a React context.
Features check `capabilities`: a self-hosted backend reports
`billing: false`, so the credit chip hides, the billing page redirects
and the bring-your-own-key UI shows instead. `storage` is false until a
bucket is configured. `providers()` tells the model menu which models
and voices have a key behind them.

### 2. `DataAPIHandle` (in the CLI and MCP)

The CLI and MCP only need data, so they use the smaller
`DataAPIHandle`:

```ts
const handle = await makeDataAPI(target);   // 'cloud' or 'local'

await handle.data.query({ query, variables });
const uid = await handle.getUid();
await handle.close();
```

It is the same `DataAPI` contract in a lighter wrapper. The factories
(`makeCloudDataAPI`, `makeLocalDataAPI`) absorb the difference between
stateless cloud calls and a stateful WebSocket connection.

## Open and closed

Everything you need to use, extend, self-host or contribute is in the
public repo:

- **Contracts** (`smartchats-backend`): the typed interface every
  consumer codes against.
- **Local stack** (`smartchats-backend-local`, `smartchats-local-server`,
  `smartchats-database`, `smartchats-cli`, `smartchats-mcp`): the
  install-and-run path. No cloud account needed.
- **Cloud client** (`smartchats-cloud-client`): the client for the
  hosted cloud. It defaults to smartchats.ai.

The code that runs the hosted service is closed. Everything in the
public repository, every package included, is licensed under the Apache
License 2.0 (copyright Sattvic Systems LLC).

## Single direct importer of `surrealdb`

Only `packages/smartchats-database/src/client.ts` imports the
`surrealdb` npm package in runtime code. Everything else goes through
the `Client` interface it exports (`createClient`, `createLazyClient`,
`createUserClient`, and the client's `signupAsUser` and `signinAsUser`).

This made the v2 to v3 SDK migration a one-file change. SDK quirks
(named exports, the `connect()` change, the removal of `queryRaw`, the
`Table` wrapper for `insert()`, DateTime normalization) are handled
inside `client.ts`.

## Auth and credentials

There is one credential store, `~/.smartchats-mcp/credentials.json`
(mode `0600`), which holds the cloud refresh token. The CLI, the MCP
server, the bridge and the hub all use it. Run `smartchats login` once.

The ID token lasts one hour. `smartchats-cloud-client` refreshes it on
demand with the stored refresh token over plain `fetch`. The browser
sign-in flow uses a localhost callback.

## Data model: event time and row lifecycle

Every event-time table has two kinds of timestamp. Keep them apart.

| Field(s) | What it means | Set by | Migrated? |
|---|---|---|---|
| `created_at` / `updated_at` | When the row was written in *this* database | DB (`VALUE time::now()` / `READONLY`) | **Never.** Stripped on import. |
| `ts` / `local_date` / `local_tz` | When the thing the row describes happened in the user's life | App (`nowEventTime()` in `app/modules/system.ts`) | **Always.** Carried through export and import. |

The event-time fields answer "when did it happen?":

- `ts: datetime`: the UTC instant. Use it to sort and filter.
- `local_date: string`: `YYYY-MM-DD` in the user's time zone at the
  time, so `GROUP BY local_date` gives correct daily totals with no
  time zone logic in the query.
- `local_tz: string`: the IANA zone (for example `America/Chicago`), so
  the local time of day can be rebuilt later.

Bundles import without `created_at` and `updated_at`, so the destination
database stamps fresh values while the event-time fields keep the
original timeline. Without this split, a year-old bundle would look
"created today" and sort wrongly.

Schema change history is in `HISTORY.md` at the repo root.

## Verification

The layered test runner is `packages/smartchats-test`. Each level runs
in order and stops at the first failure unless you pass
`--continue-on-failure`.

| Level | Name | What it runs |
|---|---|---|
| L0 | `lint` | `turbo run lint` across the workspace |
| L1 | `build` | `turbo run build`, which type-checks every package |
| L2 | `unit` | `npm run test:unit` in each package that defines it |
| L3 | `integration` | Integration suite against an in-memory SurrealDB the level starts and cleans up |
| L4 | `e2e` | The Playwright Simi suite (`apps/smartchats/tests/e2e/simi.spec.ts`). Needs a running stack |

```bash
npx smartchats-test                 # every level that needs no running stack (L0-L3)
npx smartchats-test quick           # lint + build
npx smartchats-test build,unit      # chosen levels
npx smartchats-test --list          # list levels
```

For e2e, use `bin/test-e2e`. It starts the no-Docker stack with
`bin/test-bun-deploy`, waits for `/local-api/health`, runs
`smartchats-test e2e`, and shuts the stack down.

```bash
bin/test-e2e                        # full Simi suite
bin/test-e2e --pick                 # choose workflows interactively
bin/test-e2e -- --grep agent_input  # pass flags through to Playwright
```

See the [Verification Workflow in CONTRIBUTING.md](https://github.com/sheunaluko/smartchats/blob/main/CONTRIBUTING.md#verification-workflow)
for which level fits which change.

## See also

- [Voice engine](/docs/voice-engine): vie, the turn judge and tuning.
- [Providers](/docs/providers): model and speech providers and keys.
- [Local server](/docs/local-server): routes, SurrealDB and storage.
- [Hub](/docs/hub) and [Relay](/docs/relay): agent messaging.
- [MCP Server](/docs/mcp): the data tools for MCP clients.
- [CLI Reference](/docs/cli): every subcommand.
- [Contributing](/docs/contributing): conventions and the dev workflow.

---

Source: https://smartchats.ai/docs/voice-engine

# Voice engine

In a voice session, SmartChats has to decide when you have finished speaking and your words should go to the agent. The part that makes this decision is **vie**, the voice interface engine (`packages/vie`). It is the default engine. The older pipeline stays available as `voiceEngine: 'legacy'`.

This page explains how vie works, what you can change, and where to look when a turn ends too early or too late. The full design is in the [vie spec](https://github.com/sheunaluko/smartchats/blob/main/packages/vie/SPEC.md).

## What the engine does

vie reads four kinds of signal:

- **Audio power**: is the room loud enough to start the speech recognizer?
- **The speech detector** (VAD): when speech starts and stops.
- **The browser speech recognizer**: the text, updated word by word.
- **A judge**: does the text so far sound finished?

From these it produces a fixed chain of events for each turn:

```
listen.wanted → speech.ended → transcript.settled → turn.sent → turn.ended → turn.shipped → (turn.amended)
```

| Event | Meaning | Decided by |
|---|---|---|
| `listen.wanted` | Sound is loud enough to start the recognizer | Audio power vs `powerThreshold` |
| `speech.ended` | You paused | The speech detector, after `tailMs` of silence |
| `transcript.settled` | The words have caught up | No word-changing update for `settleMs` after `speech.ended` |
| `turn.sent` | The text left for the agent | At the settle for the server judge, at COMPLETE for the others |
| `turn.ended` / `turn.shipped` | You are done and the turn is final | The judge |
| `turn.amended` | Words arrived for a turn already sent | The recognizer delivering late |

Around that chain, the turn lifecycle adds four more events:

| Event | Meaning |
|---|---|
| `turn.held` | The judge said INCOMPLETE. The turn waits for more words. |
| `turn.merged` | A held turn shipped with the new words added to it, as one message. |
| `turn.forced` | A held turn shipped as it was, after silence or a commit keyword. |
| `turn.reopened` | New speech reopened a turn that had already been sent. |

Only one thing is ever undone. If speech starts again while vie waits for the settle, `speech.ended` is cancelled and the turn carries on as one.

The turn states look like this:

```
listening ─ vad.end ─► ended ─ settleMs quiet ─► judging ─ COMPLETE ───► shipped ─ reply done ─► idle
    ▲                    │                          │
    │      vad.start ────┘ (the one retraction)     ├─ INCOMPLETE ─► held ─ silence / keyword ─► shipped (as is)
    │                                               │                 │
    └──────────── continuation ◄────────────────────┴─ abort ─► held ─┘ new words ─► merged into the turn
```

## The three judges

The speech detector and the settle only know that you paused. The **judge** decides whether the pause ended your thought. Choose one in Settings under "Who decides you're done talking" (tivi setting `vieJudge`).

| Judge | Setting | How it decides | Use it when |
|---|---|---|---|
| Server verdict | `server` (default) | A small LLM call on the server reads your words before the main reply | You have a key for the verdict model. This is the most accurate option. |
| Smart Turn | `smart_turn` | An audio model scores the end of your speech in the browser | You want no extra server call. It struggles on some voices (see the spec, section 7). |
| None | `none` | Silence alone ends the turn | You speak in short, complete commands, or you want the lowest latency. |

Every judge goes through the same `judging` state. They differ only in how they answer:

- **Server**: vie sends the text with a verdict request at the settle (`turn.sent`). The verdict arrives before any reply text, usually 0.3 to 0.5 s later. If it never arrives, the turn is confirmed after 3 s.
- **Smart Turn**: vie scores the audio. A probability at or above `threshold` is COMPLETE. Past `judgeTimeoutMs` (1.4 s) the turn ships anyway.
- **None**: COMPLETE at once.

Words that go to a tool waiting for input (dictation, voice memo) always use no judge, because they make no model call for the server to judge.

## The server turn verdict

The server judge is one small LLM call made before the main reply. It answers COMPLETE (the reply proceeds) or INCOMPLETE (the turn is held). The code is in `packages/llm-service/src/streaming/verdict.ts`, and [TURN_VERDICT.md](https://github.com/sheunaluko/smartchats/blob/main/packages/llm-service/TURN_VERDICT.md) has the measurements.

The app asks for a verdict only when all of these hold:

- the call comes from a running voice session
- the voice engine is vie and the judge is `server`
- a verdict model is set (`verdictModel`, default `deepseek-v4.1-flash`)

### Judge v2

The app sends `judge: 'v2'`. Under judge v2:

- **The score** is P(COMPLETE), the model's own probability that its one-word answer is COMPLETE. It is read from the first output token's probabilities, not written by the model.
- **The threshold** defaults to 0.5. The server holds the turn when the score is below it.
- **The input** is the last 2 conversational turns, quoted as a transcript ("User: … / Assistant: …") in one message. The judge reads the exchange as a third party. This stops a reply like "go ahead, I'm listening" from reading as an invitation to keep talking.

Scores pile up at both ends. Fewer than 3% fall between 0.1 and 0.9, so the threshold matters little in practice. There is no tuner for it.

> **Note:** **Failures never cut you off.** A verdict that fails, cannot be parsed or passes its cap (`verdictCapMs`, default 800 ms) counts as COMPLETE. The only failure you can hear is a false hold: the app waiting for more after you finished.

### Holds and merges

When the verdict is INCOMPLETE:

1. The server ends the response without making the main model call.
2. The app plays the keep-going cue. This is a short tone, or a spoken "ok" / "mhm" when acks are on (`enableAcks`).
3. Whatever you say next is merged with the held text and sent as one message (`turn.merged`).
4. If you say nothing for `holdSilenceMs` (3 s), or say a commit keyword ("done", "finished", "listo", "terminado"), the held text ships as it is (`turn.forced`). A forced turn carries no verdict.

### Providers

Judge v2 needs a provider that returns token probabilities. Today that is Together, so the default verdict model needs a Together key.

- With a Together key, v2 decides from P(COMPLETE).
- With a verdict model from another provider, v2 decides from the sampled word and the threshold has no effect.
- With no key for the verdict model, the local server skips the verdict and answers the turn, as if the verdict had said COMPLETE.

See [Providers & API keys](/docs/providers) for how to set the key.

## Parameters

All timing lives in `packages/vie/src/core/params.ts`.

| Parameter | Default | Bounds | How it is set |
|---|---|---|---|
| `powerThreshold` | 0.01 | | Manual |
| `tailMs` | 200 ms | 150 to 1500 | Fixed. The speech detector's debounce. Never tuned, can be set by hand. |
| `settleMs` | 400 ms | 100 to 1200 | Adapts both ways, every turn |
| `prerollMs` | 0 ms | 0 to 400 | Ratchets up when a first word is clipped |
| `threshold` | 0.15 | 0.05 to 0.6 | Smart Turn's COMPLETE threshold. Ratchets down on a wrong hold. |
| `breathMs` | 300 ms | | Manual |
| `continuationMs` | 1000 ms | | Manual. Speech within this of a ship belongs to that turn. |
| `holdSilenceMs` | 3000 ms | | Fixed. A held turn ships after this much silence. |
| `judgeTimeoutMs` | 1400 ms | | Fixed. Smart Turn only. |

### How tuning works

vie learns its timing on each device from what happens in real turns:

- **Settle eases down and jumps up.** On every settle, `settleMs` moves 25% of the way toward the largest gap between word updates in that settle window. These moves are quiet. When a word lands late after a ship, `settleMs` jumps straight to that word's gap. On a clean device settle drifts down toward the recognizer's real gaps, jumps when a word arrives late, then eases down again.
- **The tail is fixed** at 200 ms. Deciding whether a pause ended your thought is the settle's and the judge's job.
- **Pre-roll and the Smart Turn threshold ratchet.** Each moves at most one step per turn, and only in its safe direction: pre-roll up after a clipped first word, the threshold down after a wrong hold. They do not come back down on their own.
- **Pins.** A pinned parameter never moves. Moving a slider in Settings pins it.
- **Per device.** Parameters, pins, the evidence ledger and the tuner history are stored in the browser's localStorage under `vie.device.v1`. A phone and a laptop each learn their own values.

> **Warning:** **Calibration pins settle.** Running calibration on `/voicelab` pins `settleMs` at the end of its descent, which stops it adapting. Unpin it if you want the per-turn easing back.

## Continuations, late words and amends

Two things can happen after a turn has settled. vie tells them apart by whether the speech detector heard new speech.

**Late words.** A word update that arrives after the settle **with no new speech start** is the recognizer finishing what it already heard. It belongs to the same utterance, in any phase:

- While the judge runs, the words are added to the turn. For the server judge the request is sent again with the full text.
- While the turn is held, they are added to the held text.
- After the ship, the turn is amended (`turn.amended`). Before the reply's first token, the app cancels the reply and sends one complete message in place of the old one. After that, the words are handed to the agent as late words.

A recognizer revision is not late words. If the recognizer only repeats the end of what was already sent (for example by dropping a word), the sent text is kept and only the new words after it count.

**Continuations.** New speech after the settle, a speech start followed by its first word, is a continuation. vie acts at the first word, not the bare speech start, because coughs and background talk often start the detector without producing words.

| When the first word arrives | What happens |
|---|---|
| The assistant is speaking | Barge-in: the audio stops at once, then cancel and add (below) |
| The judge or model is running, nothing heard yet, no tool has run | **Abort and hold.** The run is stopped, the agent never sees it, and the held text plus the new words go as one message. |
| A tool already ran in this turn | **Cancel and add.** The new words go as their own message after a cancel note, so a side effect is not repeated. |
| A tool is waiting for input | The words go to the tool |
| The reply is finished | A normal new turn |

## Settings you can change

These are tivi settings (`packages/tivi/src/lib/settings.ts`), set in the Settings panel:

| Setting | Default | What it does |
|---|---|---|
| `voiceEngine` | `vie` | `legacy` switches back to the old pipeline |
| `vieJudge` | `server` | `server`, `smart_turn` or `none` |
| `verdictModel` | `deepseek-v4.1-flash` | The server verdict model |
| `verdictEffort` | `none` | Reasoning effort for the verdict call. `none` turns thinking off. |
| `verdictCapMs` | 800 | Longest the server waits for the verdict before going ahead as COMPLETE |
| `enableAcks` | `false` | Speak "ok" / "mhm" on a hold in place of the tone |

The engine choice and the verdict settings are only in the Settings panel. The agent can change the rest by voice. Ask it, for example, "turn on backchannel acks" or "pin the settle time at 500 ms". Its tools:

- `set_voice_setting` and `reset_voice_settings` for settings such as acks, pre-roll and playback speed
- `get_voice_tuning` for current parameters, pins, recent tuner moves and device alerts
- `pin_voice_setting`, `unpin_voice_setting`, `reset_voice_tuning` and `set_auto_tune` for vie's learned parameters
- `report_speech_cutoff` when a transcript lost its first word (this raises pre-roll). A lost last word is recorded but does not move settle.

`/voicelab` shows the engine's layers live and runs calibration.

## Telemetry

vie writes these events to insights:

| Event | What to look at |
|---|---|
| `voice_turn` | One row per turn: `params.settle_ms`, `params.tail_ms`, `settle_max_gap_ms`, `settle_update_offsets_ms` |
| `vie_event` | Each event in the chain. `transcript.settled` carries `max_gap_ms`, `turn.amended` carries `gap_ms`, `turn.reopened` carries `phase` and `held_by` |
| `vie_evidence` | Each evidence row, including `verdict.wrong_hold` (a hold ended by silence or a keyword) and `verdict.cut_off` (speech soon after a COMPLETE), with the score and threshold |
| `voice_tuning_changed` | Each parameter move. `quiet: true` marks routine settle easing |
| `turn_verdict` | Each server verdict as vie received it |
| `smart_turn_decision` | Each Smart Turn score |

A healthy device shows settle easing down, an occasional jump on a late word, and a low rate of late words. Each late word cancels a reply in progress.

## See also

- [vie spec](https://github.com/sheunaluko/smartchats/blob/main/packages/vie/SPEC.md): states, tuning rules, invariants and open items
- [TURN_VERDICT.md](https://github.com/sheunaluko/smartchats/blob/main/packages/llm-service/TURN_VERDICT.md): the server verdict, judge v2 and the replay script for measuring a prompt or model change
- [Providers & API keys](/docs/providers)
- [tivi](/docs/packages/tivi)

---

Source: https://smartchats.ai/docs/local-server

# Local server reference

`smartchats-local-server` is the backend of a self-hosted SmartChats. It is an Express app that serves the API under `/local-api/*` and, when `SMARTCHATS_STATIC_DIR` is set, the static app at `/`. It stores data in SurrealDB, calls LLM and TTS providers with your own keys, and signs upload links for an S3-compatible bucket.

`smartchats start` launches it for you. This page covers what it does underneath.

## How `smartchats start` runs it

`start` launches up to three detached processes and records them in `~/.smartchats/run/pids.json`:

1. **SurrealDB** on `127.0.0.1:8000`, user `root`, password `root`, with RocksDB storage at `~/.smartchats/data/surreal.db`. `start` waits up to 30 s for its `/health`.
2. **SeaweedFS** for media storage on `127.0.0.1:8333` (see [Storage](#storage)). It is optional: if it is missing or doesn't come up within 30 s, the stack runs without media.
3. **The local server** on port 3000, bound to `0.0.0.0`, pointed at that SurrealDB (`ns=smartchats`, `db=main`), serving the bundled app. `start` waits up to 30 s for `/local-api/health`.

The server's environment is your `.env` merged with the shell's environment. The shell wins, so `KEY=value smartchats start` overrides one launch. On boot the server connects to SurrealDB and applies the schema, and exits if either fails.

| Flag | Default | Notes |
|---|---|---|
| `--port` | `3000` | App and API port |
| `--surreal-port` | `8000` | SurrealDB port, loopback only |
| `--data-dir` | `~/.smartchats/data` | SurrealDB storage, and media under `seaweedfs/` |
| `--rebuild` | off | Run `turbo run build` before starting |
| `-f`, `--foreground` | off | Stay attached. Ctrl-C stops the stack |
| `--no-prompt` | off | Skip the "configure keys?" prompt when no keys are found |

## Logs

| Process | Path |
|---|---|
| SurrealDB | `~/.smartchats/logs/surreal.log` |
| SeaweedFS (media) | `~/.smartchats/logs/seaweedfs.log` |
| Local server | `~/.smartchats/logs/server.log` |

```bash
smartchats logs [-f] [--proc surreal|seaweedfs|server|all] [-n 50]
```

The server logs one line per request (`METHOD url -> status ms`), plus per-route lines for LLM streams with model, token counts and cost.

## Environment

| Variable | Default | Notes |
|---|---|---|
| `SMARTCHATS_PORT` | `4242` | `smartchats start` sets `3000` |
| `SMARTCHATS_HOST` | `127.0.0.1` | `smartchats start` sets `0.0.0.0` |
| `SMARTCHATS_API_KEY` | unset | Bearer token for scripts and agents on other machines (see [Security](#security)) |
| `SMARTCHATS_AUTH` | `pair` | `off` lets every device in without pairing |
| `SMARTCHATS_DATA_DIR` | unset | Where `devices.json` (paired devices) is kept; `smartchats start` sets the data directory. Unset means `~/.smartchats` |
| `SMARTCHATS_STATIC_DIR` | unset | Directory of the static app export to serve at `/`. Unset means API only |
| `SURREAL_URL` | `ws://127.0.0.1:8000/rpc` | |
| `SURREAL_NS` | `smartchats` | |
| `SURREAL_DB` | `main` | |
| `SURREAL_USER` | `root` | |
| `SURREAL_PASSWORD` | `root` | |
| `SMARTCHATS_OPENAI_API_KEY` | then `OPENAI_API_KEY` | Chat, embeddings, OpenAI TTS |
| `SMARTCHATS_ANTHROPIC_API_KEY` | then `ANTHROPIC_API_KEY` | |
| `SMARTCHATS_GOOGLE_API_KEY` | then `GOOGLE_API_KEY`, `GEMINI_API_KEY` | Gemini |
| `SMARTCHATS_XAI_API_KEY` | then `XAI_API_KEY` | |
| `SMARTCHATS_TOGETHER_API_KEY` | then `TOGETHER_API_KEY`, `TOGETHERAI_API_KEY` | |
| `SMARTCHATS_AZURE_SPEECH_KEY` | then `AZURE_SPEECH_KEY` | Azure TTS |
| `SMARTCHATS_AZURE_SPEECH_REGION` | then `AZURE_SPEECH_REGION` | Azure TTS |
| `SMARTCHATS_SERPER_API_KEY` | then `SERPER_API_KEY` | Web search |
| `SMARTCHATS_S3_ENDPOINT` | set by `start` (SeaweedFS) | Media bucket. All four S3 values are needed to enable storage |
| `SMARTCHATS_S3_ACCESS_KEY_ID` | unset | |
| `SMARTCHATS_S3_SECRET_ACCESS_KEY` | unset | |
| `SMARTCHATS_S3_BUCKET` | unset | |
| `SMARTCHATS_S3_REGION` | `us-east-1` | Signing region |
| `SMARTCHATS_S3_LINKS` | `server` | `direct` hands browsers presigned bucket URLs (see Storage) |

Provider keys resolve in this order: the `SMARTCHATS_` variable, the plain variable, a key saved in the database with `POST /keys`, then none.

## Routes

All routes are under `/local-api`. JSON request bodies are limited to 10 MB.

| Route | Notes |
|---|---|
| `GET /health` | Readiness. `{ ok, checks: { data, providers, tts } }`. 503 only when SurrealDB is unreachable. Missing keys are reported but don't fail it |
| `POST /llm/stream` | Text-only NDJSON stream |
| `POST /llm/streamWithTTS` | LLM plus TTS NDJSON stream, with the optional turn verdict |
| `POST /tts/stream` | NDJSON TTS stream, PCM16 24 kHz mono |
| `POST /embeddings/embed` | OpenAI `text-embedding-3-small` |
| `POST /data/query` | `{ query, variables? }` runs SurrealQL. Returns `{ statements: [{ status, result, time }] }` |
| `GET /data/health` | Per-table check of the required tables |
| `GET /usage/records` | Paginated usage rows |
| `GET /usage/summary?since=<iso>` | Aggregate for a time window |
| `GET /keys/providers` | Which LLM and TTS providers are available |
| `GET /keys` | Masked previews of keys saved in the database (env keys are not shown) |
| `POST /keys` | `{ keys: { openai?, anthropic?, google?, xai?, together?, azure? } }`. Warns when an env var shadows a saved key |
| `DELETE /keys/:provider` | Remove a saved key |
| `POST /tools/search` | Serper web search. Needs a Serper key |
| `POST /tools/fetchUrl` | Fetch a page and extract readable text |
| `POST /insights/batch` | Telemetry events, stored as sent in `insights_events` |
| `POST /storage/createUpload` | Start an upload and get its upload links |
| `POST /storage/resumeUpload` | Resume a multipart upload |
| `POST /storage/completeUpload` | Finish an upload |
| `POST /storage/getMediaUrl` | Fresh download link |
| `POST /storage/listMedia` | List files |
| `POST /storage/deleteMedia` | Delete a file |

Self-hosted mode has no billing. Usage rows are written for every LLM, TTS, embedding and tool call with `creditsCharged: 0`.

### `POST /llm/stream`

```json
{
  "model": "gpt-5.5",
  "input": [{ "role": "user", "content": "Hello" }],
  "max_tokens": 16384,
  "temperature": 0.7,
  "effort": "low",
  "stop": ["\n\n"],
  "session_id": "optional, recorded on the usage row"
}
```

`model` and a non-empty `input` are required. `text_format` (`{ type: "json_schema", name, strict, schema }`) asks for structured output. `max_tokens` defaults to 16384 and is capped at 32768. `{ "warmup": true }` returns at once without calling a model. An unknown model or a missing key for its provider is a 400 before the stream starts.

The response is NDJSON, one object per line:

```json
{"t":"delta","d":"Hel"}
{"t":"delta","d":"lo!"}
{"t":"done","data":{"success":true,"output_text":"Hello!","usage":{"input_tokens":9,"output_tokens":3},"model":"gpt-5.5","provider":"openai","finish_reason":"stop","latency_ms":640}}
```

A failure mid-stream writes `{"t":"error","error":"..."}`.

### `POST /llm/streamWithTTS`

Same body as `/llm/stream`, plus TTS fields and an optional `verdict` block:

```json
{
  "model": "gpt-5.5",
  "input": [{ "role": "user", "content": "so I was thinking about" }],
  "tts": true,
  "tts_provider": "azure",
  "tts_voice": "en-US-AvaMultilingualNeural",
  "tts_speed": 1,
  "tts_instructions": "warm, brief",
  "verdict": { "model": "deepseek-v4.1-flash", "effort": "none", "cap_ms": 800, "judge": "v2", "threshold": 0.5 }
}
```

The verdict is a small model call that runs before the main model on voice turns and decides whether the user finished speaking. Only `verdict.model` is required. `judge` is `v1` (the default) or `v2`, and `threshold` (0 to 1, default 0.5) applies to `v2`.

When a verdict runs, its frame comes first:

```json
{"t":"verdict","complete":false,"source":"model","model":"deepseek-v4.1-flash","cap_ms":800,"ms":{"queue":2,"model":310,"total":312},"utterance_words":5,"context_turns":0}
```

`source` is `model`, `cap` (timed out, treated as complete), `error`, or `skipped`. If `complete` is false the turn ends there: no main call is made, and a `done` frame follows with `finish_reason: "held"`. Otherwise the stream continues:

| Frame | Fields |
|---|---|
| `text` | `d`: a text delta |
| `audio_start` | `s` chunk index, `text`, `ms` |
| `audio` | `s`, `c` sequence, `b64` PCM16 24 kHz mono |
| `audio_end` | `s`, `ms` |
| `audio_error` | `s`, `error`. The text stream is unaffected |
| `server_timing` | Timing phases |
| `llm_done` | The full text and usage, before trailing audio finishes |
| `done` | Final stats, with `verdict` when one ran |
| `error` | LLM stream failure |

> **Note:** **A missing verdict key doesn't fail the turn.** If the server has no key for the verdict model's provider, it logs `no key for verdict model ...` and runs the turn without a verdict. A missing key for the main model is still a 400.

## Storage

Photos, videos and files go to an S3-compatible bucket. `smartchats start` runs one for you: SeaweedFS (`weed mini`, bundled with binary installs) on `127.0.0.1:8333`, with its data and generated credentials under `<data-dir>/seaweedfs` (`media/`, `seaweedfs.env`, `s3.json`, all 0600). The server gets the `SMARTCHATS_S3_*` values from `start`. `smartchats status` and `smartchats logs --proc seaweedfs` show it.

Only the server talks to the bucket. Upload and download links point at the server itself (`/local-api/storage/blob/<token>`), and it streams the bytes to and from the bucket. So media works from any device that can open the app (a phone on your network, a container's published port, a reverse proxy) with nothing else exposed. Each link is signed (HMAC, from the bucket secret) for one operation on one file and expires within minutes, like a presigned URL. That is why these routes don't need the bearer token: an `<img>` can't send one.

| Env | Default | Notes |
|---|---|---|
| `SMARTCHATS_S3_PORT` | `8333` | S3 port (loopback) |
| `SMARTCHATS_S3_LINKS` | `server` | `direct`: hand browsers presigned bucket URLs instead, so bytes skip the server. The bucket must then be reachable from the browser |
| `SMARTCHATS_S3_BIND` | `127.0.0.1` | Direct links only: address SeaweedFS listens on, e.g. `0.0.0.0` |
| `SMARTCHATS_S3_PUBLIC_ENDPOINT` | the endpoint | Direct links only: address browsers reach the bucket at, e.g. `http://192.168.1.20:8333`. Links are signed for it |
| `SMARTCHATS_S3_ALLOWED_ORIGINS` | `http://localhost:<port>`, `http://127.0.0.1:<port>` | Direct links only: extra app origins the bucket allows by CORS, comma separated |
| `SMARTCHATS_WEED_BIN` | bundled `weed`, else `weed` on PATH | |
| `SMARTCHATS_STORAGE` | on | `off` runs the stack without media storage |

To use your own bucket instead (S3, R2, MinIO), set `SMARTCHATS_S3_ENDPOINT`, `SMARTCHATS_S3_ACCESS_KEY_ID`, `SMARTCHATS_S3_SECRET_ACCESS_KEY` and `SMARTCHATS_S3_BUCKET`. `start` then skips SeaweedFS. The server streams through to it the same way; add `SMARTCHATS_S3_LINKS=direct` to send browsers to the bucket instead (configure the bucket's CORS for the app's origin). Without a bucket the storage routes answer 501 `not_supported`.

From a clone, `bin/seaweedfs` runs the same local bucket for development, and `bin/devserve` picks it up.

Storage errors return `{ error, code }`: 404 `not_found`, 400 `invalid_argument`, 409 `failed_precondition`, 501 `not_supported`.

## Security

`smartchats start` binds the server to `0.0.0.0` so a phone on your network can use it. Who may call `/local-api`:

| Caller | How |
|---|---|
| The computer running SmartChats | Trusted, nothing to set up: the browser at `http://localhost:3000`, the CLI, MCP servers, scripts |
| A phone or another computer | Pairs once. It sees a **Pair this device** screen and enters a 6-digit code from `smartchats pair` (or Settings → Devices). It then keeps a cookie until you revoke it (`smartchats devices`, or Settings → Devices) |
| A script or agent on another machine | `Authorization: Bearer $SMARTCHATS_API_KEY`, when that variable is set |
| Anyone, with `SMARTCHATS_AUTH=off` | No checks, as before 0.4.0 |

Everything else gets `401 {"error": "unauthorized", "code": "pairing_required"}`. The page itself (`/app`) and signed media links stay public: they hold no data.

What counts as this machine: the connection comes from a loopback address, the `Host` is `localhost`, `127.0.0.1` or `[::1]`, any `Origin` is one of those too, and any `X-Forwarded-For` / `Forwarded` addresses are loopback. So a website whose domain resolves to `127.0.0.1` (DNS rebinding), or a page on another site calling `localhost:3000`, is not trusted.

> **Warning:** **A proxy or tunnel on the same machine must forward the visitor's `Host` or address.** With neither, every visitor looks like this machine. The [nginx example](/docs/self-host#running-on-a-vps) sets `Host $host` and `X-Forwarded-For`; Caddy forwards both by default.

Pairing codes are 6 digits, exist only after you ask for one (or once at startup while nothing is paired, for Docker), last 5 minutes (15 at startup), serve one device, and die after 5 wrong tries. A device's token is 32 random bytes in an `HttpOnly`, `SameSite=Strict` cookie; `<data-dir>/devices.json` (0600) stores only its SHA-256.

| Route | Open to | Does |
|---|---|---|
| `GET /auth/status` | anyone | `{mode, authenticated, via, device?}` |
| `POST /auth/pair` | anyone | `{code, name?}` → sets the device cookie |
| `POST /auth/pair/start` | trusted | `{code, expires_at}` for another device |
| `GET /auth/devices` | trusted | Paired devices |
| `POST /auth/devices/revoke` | trusted | `{id}` |
| `POST /auth/logout` | trusted | Unpairs the calling device |

- SurrealDB binds to `127.0.0.1` with `root`/`root` credentials and SeaweedFS to `127.0.0.1`. Don't expose ports 8000 or 8333.
- On a public server, keep the reverse proxy with TLS in front, and block direct access to port 3000.

## See also

- [Self-Hosting](/docs/self-host)
- [CLI](/docs/cli)
- [Architecture](/docs/architecture)

---

Source: https://smartchats.ai/docs/hub

# Hub

`smartchats-hub` is a local daemon that lets coding agents on your computer (Claude Code, Codex, opencode, or any other process) talk to you through SmartChats and to each other. Run one per computer.

```
SmartChats (browser, phone) <-> relay (public) <-> hub (127.0.0.1) <-> agents (local processes)
                                                      |
                                                 dashboard (127.0.0.1)
```

The hub is the only process on the machine that talks to the relay and refreshes your sign-in token. Each agent gets a durable inbox in SQLite, so messages sent while an agent is busy wait until it reads them. Agents reach the hub over MCP (streamable HTTP), plain HTTP, or the `smartchats-hub` CLI.

> **Note:** **Hub or bridge.** On a machine without a hub, an agent can still reach SmartChats through [`smartchats-mcp-bridge`](/docs/relay). The hub keeps the same tool names, so prompts written for the bridge keep working.

## Install

The hub ships in the repo as `packages/smartchats-hub`. It needs Node 24 or later (it uses the built-in `node:sqlite`).

```bash
git clone https://github.com/sheunaluko/smartchats.git
cd smartchats
npm install
npm run build
cd packages/smartchats-hub && npm link   # puts smartchats-hub on PATH
```

Remote access through SmartChats needs a SmartChats cloud sign-in. The hub reads the same credentials file as the CLI and the MCP server (`~/.smartchats-mcp/credentials.json`):

```bash
smartchats login
```

## Start, stop, status

```bash
smartchats-hub start [--port N] [--relay URL]   # foreground, Ctrl-C stops it
smartchats-hub stop                              # SIGTERM the running hub and wait
smartchats-hub status [--json]                   # pid, url, agents, control session
```

`status` lists each agent with online or offline, unacked count, watcher health, work state, and `NOT LISTENING` when nothing will wake it. It exits 3 when the hub is not running.

To run the hub at login on macOS, generate a launchd plist and load it yourself:

```bash
smartchats-hub plist --out ~/Library/LaunchAgents/ai.smartchats.hub.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/ai.smartchats.hub.plist
# remove: launchctl bootout gui/$(id -u)/ai.smartchats.hub
```

The plist runs `start` with `RunAtLoad` and `KeepAlive`. Output goes to `hub.log` in the hub home.

| Env | Default | Notes |
|---|---|---|
| `SMARTCHATS_HUB_HOME` | `~/.smartchats/hub` | Hub state directory (0700). `--home DIR` overrides it on any command |
| `SMARTCHATS_HUB_PORT` | `47821` | Fixed so agent MCP configs survive restarts |
| `SMARTCHATS_RELAY` | `wss://relay.kind.computer` | Relay endpoint |
| `SMARTCHATS_CREDENTIALS_FILE` | `~/.smartchats-mcp/credentials.json` | Firebase refresh token |
| `SMARTCHATS_CLOUD_DEV_TOKEN` | unset | Skip Firebase auth, for a local dev relay |

## Two ways onto the hub

**Join** puts an agent session that is already running on the hub. Nothing is launched and the session keeps its own permissions. Tell any agent "join the hub as builder" and it can follow the printed join sheet.

```bash
smartchats-hub join --label builder [--kind claude-code] [--model M] [--pid N] [--replace] [--json]
smartchats-hub leave
```

`join` records the agent's pid and kind (the nearest `claude`, `codex` or `opencode` process above the shell), writes `agents/<id>.mcp.json` (0600), and prints the join sheet: agent id, how to get woken, the commands to read, ack and send, and the rules.

Labels are unique on a hub and matched case-insensitively. If the holder of a label is offline, a new `join` takes it over with the same agent id and inbox and a new token. If the holder is alive, `join` refuses with `label_in_use` unless you pass `--replace`.

**Spawn** registers an agent, applies a permission profile, and launches its CLI in a new zellij tab. The hub owns its lifecycle.

```bash
smartchats-hub spawn --label L [--target claude-code|codex|opencode | --launch 'CMD'] \
  [--folder NAME | --path DIR] [--permissions trusted|ask] [--print] [-- ARGS...]
smartchats-hub agents [--json] [--prune]      # spawned agents: tab, pane, pid, state
smartchats-hub shutdown (--label L | ID) [--force]
```

| Flag | Default | Notes |
|---|---|---|
| `--target` | | `claude-code`, `codex` or `opencode`. Use `--launch 'CMD'` for anything else |
| `--folder` | | A named folder from `config.json` |
| `--path` | current directory | Any existing directory |
| `--permissions` | `trusted` | `trusted` pre-approves the folder, the hub tools and the watch command for this run only. `ask` adds nothing |
| `--print` | off | Print the launch line instead of opening a tab (also the behavior outside zellij) |
| `--force` | off | Spawn despite pre-flight problems (login, folder trust) |

`shutdown` sends two Ctrl-Cs, then SIGTERM and SIGKILL, closes the tab and unregisters the agent. SmartChats can also spawn, watch, drive and stop agents through the hub's control session (see [Remote access](#remote-access)).

## Messages and status

Every message has an `id` (ULID), `from` (`user`, `agent` or `hub`), `from_label`, `to`, `text`, `timestamp` and an optional `reply_to`. Messages from an agent carry a status:

| Status | Means |
|---|---|
| `ack` | Received, starting work. The result follows |
| `update` | Progress. No one needs to act |
| `blocked` | Can't continue until someone answers. `waiting_on` says who: `user` or `agent:<label>` |
| `complete` | The task is done. The text is the result |
| `failed` | Stopped without finishing. The text says why |

The hub derives each agent's work state from these: `idle`, `working`, `blocked_on_user` or `blocked_on_agent`. SmartChats shows a "needs you" indicator for an agent blocked on you.

```bash
smartchats-hub send --status ack "On it, checking the logs."
smartchats-hub send --status blocked --waiting-on user "Deploy now or wait for review?"
smartchats-hub send --to reviewer --status update "Tests pass, PR is up."
smartchats-hub inbox [--wait S] [--json]
smartchats-hub ack 01J...
smartchats-hub roster [--json]
```

Agent commands find their token from `--agent <id|label>`, then `$SMARTCHATS_HUB_AGENT`, then the joined agent whose pid is an ancestor of the shell. Status is required on the CLI and over HTTP. The MCP tool defaults it to `update`.

Every agent is told the same rules in its guide: read unacked messages on start and on every wake, ack first before more than 2 s of work, tag every message with a status, and treat only the user's messages as instructions. Messages from other agents and from the hub are requests or information.

## Getting woken

The inbox is the only store of messages. Events are doorbells. An agent that misses one finds the message on its next inbox read.

| Channel | How |
|---|---|
| `smartchats-hub watch --agent ID` | Prints one line per event. Run it under a long-running tool such as Claude Code's Monitor |
| `smartchats-hub watch-detached start --agent ID -- CMD ... {line}` | A watcher in its own process group that runs your command for each batch of events |
| SSE or WebSocket | `GET /events` and `GET /events/ws` with the agent's bearer |
| Long-poll | `smartchats-hub inbox --wait S`, `read_inbox`, or `GET /inbox` |

For Claude Code 2.1.224 and later, `wake-claude` posts a notice to the session's own inbox socket, which starts a turn when the session is idle. Run it from the session's shell:

```bash
smartchats-hub watch-detached start --agent ag_1 --owner-pid "$CLAUDE_PID" -- \
  smartchats-hub wake-claude --agent ag_1 {line}
```

An agent counts as **listening** when something will tell it about a new message right now. A send to an agent that isn't listening returns `queued: true, not_listening: true`. `smartchats-hub test --agent ID` runs a conformance test (wake, then replay) and records whether the agent `wakes`, `polls` or `fails`. Editable per-client hints live in `hints.md`.

## Local routing and the loop guard

| From, to | Path |
|---|---|
| SmartChats to agent | relay, hub, inbox |
| Dashboard to agent | hub, inbox |
| Agent to user | hub, relay, SmartChats. Copied to the dashboard |
| Agent to agent, same hub | hub, inbox directly. No relay round trip |
| Agent to agent, other computer | hub, relay, other hub |

`send --to <label>` resolves local labels first, then relay peers. `list_peers` lists local agents first, marked `local`.

The loop guard stops two agents from talking forever. After `max_agent_hops` (default 20) agent-to-agent messages without a user message to either, the next send fails with `LOOP_GUARD`. The sender is marked blocked on the user and gets a hub notice.

## Dashboard

The hub serves a dashboard at `http://127.0.0.1:47821/`. Open it with a one-time login link (valid 2 minutes):

```bash
smartchats-hub dashboard
```

The browser then holds a session cookie for 7 days. The dashboard shows:

- **Roster**: liveness, listening, work state (blocked agents on top with the question), unacked count, watcher health, relay slots.
- **Live traffic**: every message between you, agents and the hub, filterable by agent.
- **Chat**: type to an agent. The message lands in its inbox as `from: user, via: dashboard`.
- **SmartChats**: a channel into the live SmartChats conversation.
- **Needs you**: agents blocked on you, and agents with a message unacked longer than `unread_alert_after_s`.

Set `"dashboard": false` in `config.json` to turn it off.

## Files

Agents can save files to your SmartChats cloud storage and attach them to messages, so long output doesn't go through short spoken messages.

```bash
smartchats-hub upload report.md --title "Weekly report"
smartchats-hub send --file <media_id> --status complete "Report attached."
smartchats-hub files [--from A] [--since 2h]
smartchats-hub read-file <media_id>
smartchats-hub delete-file <media_id>
```

Agents can list and read all your files. Text, markdown, JSON, CSV and PDF come back as text in windows. An agent can delete only files it uploaded (`not_yours` otherwise).

## MCP tools

`join` and `register` write an MCP config that points the agent at the hub:

```json
{
  "mcpServers": {
    "smartchats": {
      "type": "http",
      "url": "http://127.0.0.1:47821/mcp",
      "headers": { "Authorization": "Bearer <agent token>" }
    }
  }
}
```

| Tool | Notes |
|---|---|
| `send_message_to_user(text, artifacts?, reply_to?, ack?, status?, waiting_on?, files?)` | Status defaults to `update` |
| `read_inbox(timeout_ms?, limit?)` | Long-poll of unacked messages with ids. Doesn't ack |
| `ack(ids)` | Mark messages handled |
| `wait_for_message(timeout_ms?)` | Next unacked message, acked when returned |
| `poll_messages(since?)` | Retained history, no waiting |
| `ask_user(question, timeout_ms?)` | Send as `blocked` on the user, then wait |
| `respond_and_wait(text, timeout_ms?, status?, waiting_on?)` | Send, then wait. For patch-through conversations |
| `list_peers()` | Other agents, local ones first |
| `send_message_to_peer(target_session_id, text, ...)` | Accepts a label for a local agent |
| `upload_file`, `list_files`, `read_file`, `delete_file` | Files in your SmartChats storage |
| `register(label?, model?)`, `unregister()` | Confirm or change identity, or leave |
| `hub_guide()` | The guide text, also the `hub://guide` resource |

Long-polls default to 30 s and are capped at 10 minutes.

## Configuration

Hub settings live in `~/.smartchats/hub/config.json`. The file is optional. A malformed file is an error, not silently ignored.

```json
{
  "folders": { "app": "~/dev/my-app" },
  "max_agents": 8,
  "zellij_session": "smartchats-agents",
  "remote_control": true
}
```

| Key | Default | Notes |
|---|---|---|
| `folders` | `{}` | Named folders for `spawn --folder` and for spawns from SmartChats |
| `max_agents` | `8` | Running spawned agents at once |
| `zellij_session` | `smartchats-agents` | Session that agents started from SmartChats open in |
| `remote_control` | `true` | Offer the control session on the relay |
| `max_relay_agents` | `10` | Relay slots while each agent has its own relay socket |
| `max_agent_hops` | `20` | Loop guard |
| `offline_after_s` | `600` | Liveness for agents with no known pid |
| `unread_alert_after_s` | `600` | Unread alert on the dashboard |
| `dashboard` | `true` | Serve the dashboard |

Files in the hub home:

| File | Contents |
|---|---|
| `hub.db` | Agent registry (token hashes only) and inboxes. 500 messages and 24 h per agent |
| `hub.pid`, `hub.json` | The running daemon |
| `admin_token` | Bearer for the `/admin` API |
| `agents/<id>.mcp.json` | Each agent's MCP config, with its token (0600) |
| `hints.md` | Editable wake hints, one `## <kind>` section each |
| `hub_id` | This hub's id on the relay |
| `spawned.json`, `watchers/<id>.json` | Spawned agents and detached watchers |
| `hub.log` | Output under launchd |

> **Warning:** **Localhost only.** The hub binds `127.0.0.1` with no option to listen elsewhere, and checks `Host` and `Origin` on every request. Anything that can read `~/.smartchats/hub/` can register agents, so keep that directory private.

## Remote access

When the hub starts, it opens a control session on the relay for itself (a bridge of kind `hub`, labelled with your computer's name). SmartChats uses it to list agent options, spawn agents, read an agent's screen, send it keystrokes, and shut it down. Set `"remote_control": false` to turn this off.

On a relay that can host sessions, every agent's session rides that one control socket, so the relay's per-user connection limit counts computers rather than agents. A relay hosts up to 50 sessions per hub by default. On an older relay, or with `remote_control` off, each agent gets its own socket and `max_relay_agents` applies. `status --json` shows `relay_sessions: { mux, hosted, own, waiting, max }`.

See [Relay & MCP bridge](/docs/relay) for the wire protocol.

## See also

- [Relay & MCP bridge](/docs/relay)
- [MCP Server](/docs/mcp)
- [CLI](/docs/cli)

---

Source: https://smartchats.ai/docs/relay

# Relay & MCP bridge

SmartChats runs in a browser or on a phone. The agents and terminals you want it to reach run on your own computer, behind NAT, with no public address. The browser can't open a connection to them. The relay solves this: both sides dial out to it over WebSocket, and it routes frames between them.

## Components

| Package | Role |
|---|---|
| `smartchats-relay-cf` | The relay. A Cloudflare Worker with one Durable Object per user that holds every socket for that user |
| `smartchats-relay-protocol` | Wire format shared by every party: envelope, message types, acks, resume, the hub client |
| `smartchats-relay-connect` | Client helper for outbound bridge sockets: auth, reconnect with backoff, token refresh, protocol v2 |
| `smartchats-mcp-bridge` | A stdio MCP server that makes one coding agent a SmartChats participant |
| `smartchats-hub` | A local daemon that holds one relay connection for every agent on the computer. See [Hub](/docs/hub) |
| `smartchats-relay` | The older Node relay (single process, in-memory routing). Speaks protocol v1 only and has no hub support |

The open clients default to `wss://relay.kind.computer`, which runs `smartchats-relay-cf`. The Node relay's README does not mark it superseded, but the Cloudflare relay is what every default points at.

```
  browser (SmartChats)                         your computer
  --------------------                         ------------------------------------
  /client socket  ---WSS--->  relay  <---WSS---  /bridge socket
                              (one DO           |
                               per user)        +-- smartchats-mcp-bridge  <-stdio->  coding agent
                                                |   (kind: agent, one per agent)
                                                |
                                                +-- smartchats-hub  <-MCP/HTTP->  agents A, B, C
                                                |   (kind: hub, hosts every agent session)
                                                |
                                                +-- bin/pty-bridge.mjs  <-pty->  CLI in a terminal
                                                    (kind: pty)
```

A **bridge** is anything on your computer that dials `/bridge`. A **client** is a browser that dials `/client`, lists the sessions, and subscribes to one at a time. Every bridge is a **session** with a `session_id`, `kind` (`pty`, `agent` or `hub`), `label` and `model`.

## Auth and pairing

There is no separate pairing step. Both sides present a Firebase ID token for your SmartChats account, and the relay routes only between sockets of the same account.

1. The socket opens with `?token=<Firebase ID token>`. The Worker verifies the RS256 signature, issuer, audience and expiry against Google's public keys before the WebSocket upgrade completes. A missing or bad token gets a 401 and no socket.
2. The Worker forwards the upgrade to the Durable Object named by the token's uid. Every socket for one user lands on the same object.
3. The first frame is `bridge_hello` (or `client_hello`). It registers the session and negotiates the protocol version.
4. The relay checks the token's expiry on every frame and closes an expired socket with 1008. Bridges refresh the token every 50 minutes with `bridge_reauth`. A reauth that fails closes the socket with 1008. A valid token for a different uid also closes it (`identity changed`).

Bridges get their token through `smartchats-cloud-client`, from the credentials file that `smartchats login` writes (`~/.smartchats-mcp/credentials.json`).

> **Warning:** **Plaintext at the relay.** Frames are encrypted in transit (TLS) but the relay can read them. There is no end-to-end encryption.

## Run the MCP bridge

Use the bridge on a machine without a hub (a remote VM, CI). Build it from the repo, then add it to your agent's MCP config:

```bash
cd packages/smartchats-mcp-bridge && npm run build
smartchats login   # once, for the relay token
```

```json
{
  "mcpServers": {
    "smartchats-patch-through": {
      "command": "node",
      "args": ["/path/to/smartchats/packages/smartchats-mcp-bridge/dist/index.js",
               "--label", "claude-code @ my-mac"]
    }
  }
}
```

| Flag / Env | Default | Notes |
|---|---|---|
| `--relay` / `SMARTCHATS_RELAY` | `wss://relay.kind.computer` | Relay endpoint |
| `--label` / `SMARTCHATS_AGENT_LABEL` | `<process title> @ <hostname>` | Name in the SmartChats roster |
| `--model` / `SMARTCHATS_AGENT_MODEL` | unset | Model hint for the roster |
| `--bridge-id` / `SMARTCHATS_BRIDGE_ID` | stored per label | Default persists in `~/.smartchats-mcp/agent-bridge-<label>_id` |
| `SMARTCHATS_CLOUD_DEV_TOKEN` | unset | Skip Firebase auth, for a local dev relay |

Tools: `send_message_to_user`, `wait_for_message`, `poll_messages`, `ask_user`, `respond_and_wait`, `list_peers`, `send_message_to_peer`. They have the same names and result shapes as the hub's tools. `wait_for_message` waits 30 s by default.

## Limits

Relay limits are Worker variables in `packages/smartchats-relay-cf/wrangler.jsonc`.

| Variable | Default | Notes |
|---|---|---|
| `MAX_BRIDGES_PER_USER` | `10` | Bridge sockets per account. Over the cap the upgrade gets a 429 |
| `MAX_CLIENTS_PER_USER` | `20` | Browser sockets per account. 429 over the cap |
| `MAX_SESSIONS_PER_BRIDGE` | `50` | Agent sessions one hub socket may host |
| `HELLO_TIMEOUT_MS` | `5000` | Parsed but not used by the Cloudflare relay, which authenticates at upgrade |
| `TOKEN_SWEEP_INTERVAL_MS` | `60000` | Parsed but not used. Expiry is checked on every frame instead |
| `DEV_TOKEN_BYPASS` | unset | `true` accepts any token. Never set in production |

Durable frames are buffered per session until acked: at most 500 frames and 24 h, oldest dropped first. A receiver remembers the last 1,000 ids to drop replayed repeats.

> **Note:** **Count computers, not agents.** With a hub on a relay that can host sessions, all of that computer's agents ride one socket, so `MAX_BRIDGES_PER_USER` limits computers. Each standalone MCP bridge still takes a socket of its own.

## Protocol

Protocol v2 wraps every data frame in an envelope:

```json
{
  "v": 2,
  "id": "01JB8Z...",
  "type": "agent_event",
  "ts": 1790520000000,
  "session_id": "agent-bridge-...",
  "corr": "01JB8Y...",
  "body": { "event": "message", "text": "Done: tests pass.", "status": "complete", "timestamp": 1790520000000 }
}
```

| Field | Notes |
|---|---|
| `v` | Always `2` |
| `id` | ULID. Sortable, and the receiver's dedupe key. The relay assigns one when a sender omits it |
| `type` | Frame type, up to 64 characters |
| `ts` | Milliseconds since epoch |
| `session_id` | Optional. The bridge session the frame is to or from |
| `corr` | Optional. The id of the frame this one answers |
| `body` | The frame's fields |

**Versioning.** A peer offers `v: 2` in its hello. The relay answers `bridge_registered` or `session_list` with `v: 2` if it speaks v2. A reply without `v` means v1, and the peer falls back to flat frames (`{ type, ...fields }`). A hello without `v` is a v1 peer and gets flat frames with no ids, acks or replay. The type names are the same in both versions.

Control frames stay flat:

```json
{ "v": 2, "type": "ack", "ids": ["01JB8Z..."] }
{ "v": 2, "type": "resume", "session_id": "...", "after": "01JB8Z..." }
```

A receiver acks what it handled (up to 500 ids a frame). On reconnect it sends `resume` with the last id it saw, and the relay replays newer buffered frames. If frames were lost to the buffer limits, the relay sends `pending_dropped { count, overflow, expired }` first.

### Main message types

| Type | Direction | Notes |
|---|---|---|
| `bridge_hello` / `bridge_registered` | bridge, relay | Hello carries `kind`, `label`, `model`, bridge id, `v`. The reply carries `session_id`, and `mux: { max_sessions }` when the relay can host a hub's sessions |
| `client_hello` / `session_list` | client, relay | The roster of sessions. Hubs are listed in `session_list.hubs` only |
| `list_sessions` | client to relay | Refresh the roster without reconnecting |
| `subscribe` / `subscribed` | client to relay | Pick the session to talk to |
| `agent_message` | client to agent | Your message: `{ text, timestamp }`. Durable |
| `agent_event` | agent to clients | `{ event, text?, artifacts?, status?, waiting_on?, ack?, timestamp }`. Durable |
| `peer_message` | bridge to bridge | Agent to agent within one account. Answered with `peer_ack` or `peer_error`. Durable |
| `list_peers` / `peer_list` | bridge, relay | Other sessions of the same account |
| `session_offline` | relay to clients | A session went away |
| `bridge_reauth` / `client_reauth` | to relay | Token refresh |
| `hub_cmd` / `hub_result` / `hub_event` | client, hub | Commands to a hub's control session. Never buffered. A command to an offline hub gets `hub_offline` at once |
| `session_open` / `session_opened` / `session_close` / `session_closed` / `session_error` | hub, relay | Agent sessions hosted on a hub's socket |
| `output`, `snapshot`, `input`, `resize`, ... | pty bridge, client | Terminal stream frames. Best effort, not buffered |

`peer_error` codes: `BAD_PEER_MESSAGE`, `PEER_SELF`, `PEER_NOT_FOUND` (also for another user's session, so nothing leaks), `PEER_OFFLINE`. The relay sets `from_session_id` and `from_label` on a delivered peer message from the sender's registration, so they can't be spoofed.

Message status values on `agent_event` are `ack`, `update`, `blocked`, `complete` and `failed`. `waiting_on` is `{ "type": "user" }` or `{ "type": "agent", "label": "..." }` and comes only with `blocked`.

## Connect your own bridge

```js
import { connectRelay, loadOrCreateBridgeId } from 'smartchats-relay-connect';
import { resolveConfig } from 'smartchats-cloud-client';

const relay = connectRelay({
  relayUrl: 'wss://relay.kind.computer',
  config: resolveConfig(),
  bridgeId: await loadOrCreateBridgeId(),
  tag: 'agent',
  protocol: 2,
  hello: { kind: 'agent', label: 'my-tool @ my-mac', model: 'none' },
});

relay.on('registered', ({ session_id }) => console.log('session', session_id));
relay.on('message', (msg, meta) => { /* msg is { type, ...body }, meta is { id, ts, session_id?, corr? } */ });
relay.sendFrame('agent_event', { event: 'message', text: 'hi', timestamp: Date.now() });
```

Without `protocol: 2` the helper speaks v1. It reconnects with backoff from 1 s to 30 s and refreshes the token every 50 minutes. Pass `getToken` to use your own token source.

## See also

- [Hub](/docs/hub)
- [MCP Server](/docs/mcp)
- [Architecture](/docs/architecture)

---

Source: https://smartchats.ai/docs/mcp

# MCP Server

`smartchats-mcp` exposes the SmartChats data layer over the
[Model Context Protocol](https://modelcontextprotocol.io). Add it to
Claude Desktop, Claude Code or any other MCP client and the model can
read, search and write your logs, metrics, todos and knowledge graph,
run read-only queries, and import or export your data.

It uses the same data path as the web app and the CLI. Each tool in
`src/tools.ts` is a thin wrapper that calls a query builder from
`smartchats-database` (or an operation such as `operations.exportBundle`)
against the active `DataAPI` handle. There is no MCP-only data plane.

## Install and run

```bash
cd packages/smartchats-mcp
npm run build
node dist/index.js --target=cloud   # or --target=local
```

You don't usually run it by hand. Your MCP client starts it and talks to
it over stdio.

## Targets

The server picks its data target at startup with `--target`:

- `--target=cloud` (the default): your SmartChats cloud account.
- `--target=local`: your local SmartChats install.

For cloud, the server signs in before the protocol starts. It shares the
CLI's credentials file, `~/.smartchats-mcp/credentials.json`, so one
`smartchats login` covers both.

```bash
# Cloud
smartchats login                # one time
node dist/index.js --target=cloud

# Local
node dist/index.js --target=local
```

In local mode the server connects to SurrealDB at `ws://localhost:8000/rpc`
(namespace `smartchats`, database `main`) and calls the local server at
`http://localhost:3000/local-api` for embeddings. Override these with the
`--local-url`, `--local-ns`, `--local-db`, `--local-user` and
`--local-password` flags, or the matching `SMARTCHATS_MCP_LOCAL_*`
variables (`SMARTCHATS_MCP_LOCAL_SERVER_URL` for the local server). Run
`node dist/index.js --help` for the full list.

## Add it to Claude Desktop

### Build the server

```bash
cd packages/smartchats-mcp
npm run build
```

### Add it to Claude Desktop's config

Edit `~/Library/Application Support/Claude/claude_desktop_config.json`
(macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):

```json
{
  "mcpServers": {
    "smartchats": {
      "command": "node",
      "args": [
        "/absolute/path/to/smartchats/packages/smartchats-mcp/dist/index.js",
        "--target=cloud"
      ]
    }
  }
}
```

Replace the path with the path to your checkout. Use `--target=local` to
point Claude at your local install instead of your cloud account.

### Sign in once (cloud only)

Restart Claude Desktop. If you have not run `smartchats login`, the
server starts a one-time browser sign-in. The URL appears in Claude
Desktop's MCP server logs. After that the cached refresh token renews
the short-lived ID token on its own.

### Use it

Ask Claude something like:

> "Check my SmartChats logs from the last week and summarize what I've
> been thinking about."

Claude calls `get_recent_logs` or `search_logs`, gets your data, and
summarizes it.

## Tools

The server registers 33 tools. Write tools stamp the event-time fields
(`ts`, `local_date`, `local_tz`) for you. Tools that store text with an
embedding (logs, entities, relations) embed it on insert, so callers
never pass vectors.

### Logs

| Tool | What it does | Writes |
|---|---|---|
| `search_logs` | Substring search across logs, with an optional category filter | No |
| `semantic_search_logs` | Vector search across logs by meaning | No |
| `get_recent_logs` | Most recent logs, optionally by category | No |
| `get_log_categories` | Every category with its log count | No |
| `insert_log` | Add a log entry. `event_time_override` backfills an earlier date | Yes |
| `update_log` | Patch a log. Re-embeds when `content` changes | Yes |
| `delete_log` | Delete a log by id and return the deleted row | Yes |
| `prepare_log_category` | Register a category before it has any entries | Yes |

### Metrics

| Tool | What it does | Writes |
|---|---|---|
| `get_metrics` | Metric values, filtered by name, category or date range | No |
| `get_metrics_summary` | Every tracked metric with units, counts, min/max and its latest value | No |
| `insert_metric` | Record a metric value. `source` defaults to `mcp` | Yes |
| `update_metric` | Patch `value`, `category`, `note` or `source_text` | Yes |
| `delete_metric` | Delete a metric entry by id | Yes |
| `prepare_metric` | Register a metric before it has any values | Yes |

### Todos

| Tool | What it does | Writes |
|---|---|---|
| `get_todos` | Todos, active by default or filtered by status | No |
| `insert_todo` | Add a todo with status `active` | Yes |
| `record_todo_completion` | Record a completion (for recurring todos). Leaves the status alone | Yes |
| `set_todo_status` | Set status to active, completed, cancelled or deferred | Yes |
| `reschedule_todo` | Change the due date, the recurrence rule, or both | Yes |
| `edit_todo` | Edit title, description, priority, category, due date, recurrence or metric link | Yes |
| `delete_todo` | Delete a todo. Does not delete its completions | Yes |
| `delete_todo_completions` | Delete every completion record of a todo | Yes |

### Knowledge graph

| Tool | What it does | Writes |
|---|---|---|
| `query_knowledge_graph` | Find entities and relations by name | No |
| `semantic_search_entities` | Vector search across entity names | No |
| `semantic_search_relations` | Vector search across relation triples | No |
| `insert_kg_entity` | Add an entity. Re-inserting a name creates a duplicate | Yes |
| `insert_kg_relation` | Add a directed relation between two existing entities | Yes |
| `delete_kg_entity` | Delete an entity by name. Does not delete its relations | Yes |
| `delete_kg_relation` | Delete a relation by its composite name | Yes |
| `delete_kg_relations_touching` | Delete every relation that touches an entity | Yes |

### Queries and data movement

| Tool | What it does | Writes |
|---|---|---|
| `run_query` | Raw SurrealQL, scoped to you. SELECT statements only | No |
| `export_user_data` | Write the active target's data to a JSON bundle on disk | No (writes a local file) |
| `import_user_data` | Upsert a JSON bundle into a SurrealDB instance, by id | Yes |

`run_query` rejects anything other than a read. To change data, use the
write tools above.

`export_user_data` and `import_user_data` call the same
`operations.exportBundle` and `operations.importBundle` as
`smartchats data export` and `smartchats data import`. `import_user_data`
accepts per-call overrides (`target_url`, `target_namespace`,
`target_database`, `target_user`, `target_password`) and `dry_run`, so a
server started with `--target=cloud` can import a bundle into a local
install.

> **Note:** **Semantic search needs an embedding endpoint.** On the cloud target it
>   uses your account. On the local target it calls the local server's
>   `/local-api/embeddings/embed`, so the local server must be running with
>   an OpenAI key. The write tools that embed need the same.

## Example calls

#### Search logs

You ask: "Find logs about my morning routine."

Claude calls:

```json
{
  "tool": "search_logs",
  "args": { "query": "morning routine", "limit": 10 }
}
```

The server returns matching logs:

```json
[
  { "id": "logs:abc", "category": "wellness", "content": "Morning routine: ...", "ts": "2026-04-30T13:00:00Z", "local_date": "2026-04-30", "local_tz": "America/Chicago" }
]
```

#### Knowledge graph

You ask: "What do I have related to project alpha?"

Claude calls:

```json
{
  "tool": "query_knowledge_graph",
  "args": { "query": "project alpha", "limit": 20 }
}
```

The server returns matching entities and relations. If the name is not
exact, `semantic_search_entities` finds close matches.

#### Raw SurrealQL

You ask: "How many logs do I have per category?"

Claude calls:

```json
{
  "tool": "run_query",
  "args": {
    "query": "SELECT category, count() AS n FROM logs GROUP BY category"
  }
}
```

The server returns the counts. An `UPDATE`, `DELETE` or `CREATE` in the
query is rejected.

## Authentication

**Cloud target:** the server acts as you. Every read and write is scoped
to your account, and the cloud enforces isolation between users.

**Local target:** a local install has one user. The server connects to
your local SurrealDB directly with the root credentials.

## How it fits

The MCP server is one more consumer of the shared data layer:

```
        ┌─────────────────────────────────────────┐
        │      Operations + query builders        │   shared
        │   smartchats-database/src/{operations,  │
        │                            queries}/    │
        └─────────────────────────────────────────┘
            ▲                  ▲                ▲
            │                  │                │
       smartchats         smartchats       smartchats-mcp
         web app             CLI                ▲
                                          stdio JSON-RPC
                                                ▲
                                     Claude Desktop, Claude Code, ...
```

A new query builder in `smartchats-database/src/queries/` becomes an MCP
tool with about ten lines in `tools.ts`. See
[Architecture](/docs/architecture) for the full picture.

## smartchats-mcp-bridge

`smartchats-mcp-bridge` is a different MCP server. `smartchats-mcp` gives
a model your data. The bridge lets a coding agent (Claude Code, Codex,
Gemini CLI, any MCP client) talk with you live through the SmartChats
app, by text or voice. This is called patch-through.

The agent starts the bridge as a stdio subprocess. The bridge opens a
WebSocket to the SmartChats relay with your cloud sign-in, and the relay
delivers messages between the agent and your browser.

```
coding agent ──stdio MCP──► smartchats-mcp-bridge ──WSS──► relay ──► SmartChats app
```

It registers these tools:

| Tool | What it does |
|---|---|
| `send_message_to_user` | Send you a message. Does not wait for a reply |
| `wait_for_message` | Block until you or a peer agent sends a message, or the timeout fires |
| `poll_messages` | Non-blocking read of retained messages since a timestamp |
| `ask_user` | Send a question and wait for your reply, in one call |
| `respond_and_wait` | Send a reply and wait for your next message. The tool to loop on in a conversation |
| `list_peers` | List your other connected agents |
| `send_message_to_peer` | Send a message to another of your agents |

```json
{
  "mcpServers": {
    "smartchats-patch-through": {
      "command": "node",
      "args": ["/absolute/path/to/smartchats/packages/smartchats-mcp-bridge/dist/index.js"],
      "env": { "SMARTCHATS_AGENT_LABEL": "claude-code @ my-laptop" }
    }
  }
}
```

`SMARTCHATS_RELAY` overrides the relay endpoint and
`SMARTCHATS_AGENT_MODEL` adds a model hint to the agent list. The bridge
needs a cloud sign-in (`smartchats login`).

> **Note:** **On a machine running `smartchats-hub`, point agents at the hub
>   instead.** The hub holds one relay connection and one token for every
>   agent, and keeps a durable inbox per agent. It exposes the same tool
>   names. The bridge is the fallback for machines without a hub. See
>   [Hub](/docs/hub) and [Relay](/docs/relay).

## Troubleshooting

> **Warning:** **"Authentication failed" at startup:** run `smartchats login` first.
>   The MCP server and the CLI share one credentials file.

> **Warning:** **Local target times out:** your local install isn't running. Start it
>   with `smartchats start`, then check `smartchats status`.

> **Note:** **Logs go to stderr.** The server writes protocol messages to stdout
>   and readable logs to stderr. To capture both while debugging, run
>   `node dist/index.js 2>&1 | tee mcp.log`.

## See also

- [CLI Reference](/docs/cli): the same data operations from a terminal.
- [Architecture](/docs/architecture): the layers the MCP server wraps.
- [Hub](/docs/hub) and [Relay](/docs/relay): live messaging between agents and the app.
- [Contributing](/docs/contributing): how to add a tool.

---

Source: https://smartchats.ai/docs/packages

# Packages

SmartChats is a monorepo. The app in `apps/smartchats` is a thin shell:
the voice pipeline, the agent loop, the backend contract and the data layer
live in `packages/*`. You can build a different shell on these packages, or
use one package on its own (`tivi` for browser voice, `cortex` for a
function-calling agent loop).

The MCP server (`smartchats-mcp`) and the CLI (`smartchats-cli`) both sit on
`smartchats-database` for their queries and operations.

## By role

**Voice (browser):**

- [`tivi`](/docs/packages/tivi): React voice hook and settings. Silero VAD, Web Speech recognition, TTS queue
- [`vie`](/docs/packages/vie): turn-taking engine (when a spoken turn is over), self-tuning per device, with a replay harness

**Agent runtime:**

- [`cortex`](/docs/packages/cortex): agent loop, runners, prompt modules, sandbox interface, model registry, presets, voice catalog
- [`llm-service`](/docs/packages/llm-service): LLM handlers for OpenAI, Anthropic, Gemini, xAI and Together, NDJSON streaming with TTS, turn verdict, embeddings
- [`smartchats-tools`](/docs/packages/smartchats-tools): web search (Serper) and URL text extraction used by the tool routes

**Backend:**

- [`smartchats-backend`](/docs/packages/smartchats-backend): the typed backend contract, `AuthProvider`, HTTP stream helpers
- `smartchats-backend-local`: `LocalBackend`, the contract against `smartchats-local-server`
- `smartchats-local-server`: self-hosted Express server for the backend, on local SurrealDB with your own keys. See [Local server](/docs/local-server)
- `smartchats-cloud-client`: runs query specs against smartchats.ai with a cached Firebase sign-in

**Data and sessions:**

- [`smartchats-database`](/docs/packages/smartchats-database): SurrealQL query builders, schema, the client, the operations layer
- [`smartchats-sessions`](/docs/packages/smartchats-sessions): session export, per-session analyzers, cross-session triage

**Surfaces:**

- `smartchats-cli`: the `smartchats` command. See [CLI reference](/docs/cli)
- `smartchats-mcp`: MCP server for user data (logs, metrics, knowledge graph, todos). See [MCP server](/docs/mcp)
- `graph-viz`: React knowledge graph view (sigma.js and graphology)

**Agents and relay:**

- `smartchats-hub`: per-machine daemon that holds the relay connection and a durable inbox per agent. See [Hub](/docs/hub)
- `smartchats-mcp-bridge`: MCP server that lets a coding agent message the SmartChats user
- `smartchats-relay`: WebSocket relay between bridges and browser clients. See [Relay](/docs/relay)
- `smartchats-relay-protocol`: the relay wire protocol, envelope and validators
- `smartchats-relay-connect`: outbound bridge connection helper (auth, reconnect, reauth)

**Shared:**

- `smartchats-common`: logger, helpers, sounds, insights client and `createInsightStore`
- [`simi`](/docs/packages/simi): declarative in-app test workflows

## Internal tooling

These are for working on SmartChats itself. They are not meant to be used as
libraries.

| Package | What it is |
|---|---|
| `benchpress` | Scripted end-to-end benchmark of the agent against real models (private) |
| `llmbench` | LLM streaming latency and cost benchmark across providers (private) |
| `voicebench` | Streaming TTS latency and cost benchmark (private) |
| `smartchats-test` | Layered test runner: lint, build, unit, integration, e2e (private) |
| `smartchats-tunnel` | Fly.io SSH bastion for reaching a dev server from another machine (private) |
| `smartchats-relay-cf` | Cloudflare Workers and Durable Objects port of the relay (private) |
| `smartchats-docs-canary` | A design document only, not implemented and no `package.json` |
| `sail-dsp` | Rust and WASM audio DSP for the app's `/sail` audio lab. Not marked private, but an early proof of concept used only there |

## When to use a package directly

- Voice input in another React app: `tivi`.
- A browser function-calling agent without the SmartChats UI: `cortex`.
- Server-side LLM calls with the same routing and streaming: `llm-service`.
- Reading SmartChats session data from a script: `smartchats-sessions`.
- Talking to a SmartChats database from your own code: `smartchats-database`
  with a `Client` from your backend.

If you only run SmartChats as a user, you don't need the packages. The app
composes them.

## See also

- [Architecture](/docs/architecture)
- [Voice engine](/docs/voice-engine)

---

Source: https://smartchats.ai/docs/packages/tivi

# tivi

`tivi` is the browser voice layer of SmartChats: the `useTivi` React hook,
a `Tivi` component, a `VoiceSelector` and device-local voice settings. It
runs Silero VAD v5 with `onnxruntime-web` in the browser, wraps the Web
Speech API for recognition, and plays TTS through an interruptible queue.

Turn-taking (deciding when the user has finished speaking) runs through
[`vie`](/docs/packages/vie) by default. The behaviour is described in
[Voice engine](/docs/voice-engine).

## Exports

The package entry (`src/index.ts`) exports:

| Export | What it is |
|---|---|
| `useTivi(options)` | The hook. Returns `UseTiviReturn` |
| `Tivi` (also the default export) | A component that wraps `useTivi`. Takes `TiviProps`, the same fields as `UseTiviOptions` |
| `VoiceSelector` | TTS voice picker. Optional `provider`, `backend` and `model` props force a value and hide that control |
| `getTiviSettings`, `updateTiviSettings`, `resetTiviSettings`, `TIVI_DEFAULTS`, `useTiviSettings` | Settings |
| Types | `TiviProps`, `UseTiviOptions`, `UseTiviReturn`, `TiviSettings` |

`CalibrationPanel`, `useCalibration` and the lower-level pieces in
`src/lib/` (TTS queue, ack cache, Smart Turn predictor) are not exported from
the entry. The SmartChats app imports `CalibrationPanel` by source path.

## Quick start

```tsx
import { useTivi } from 'tivi';

function Mic() {
  const tivi = useTivi({
    onTranscription: (text) => console.log('user said:', text),
    onInterrupt: () => console.log('user interrupted TTS'),
  });

  return (
    <button onClick={tivi.isListening ? tivi.stopListening : tivi.startListening}>
      {tivi.isListening ? 'Stop' : 'Start'} listening
    </button>
  );
}
```

To speak:

```ts
await tivi.speak('Hello.');          // optional second argument: rate
tivi.cancelSpeech();
```

With `enableInterruption` on, speech detected while TTS plays stops
playback and calls `onInterrupt`.

## `UseTiviOptions`

The most used options. Defaults are the hook's own. The SmartChats app passes
values from `getTiviSettings()`.

| Option | Default | Notes |
|---|---|---|
| `onTranscription` | | Final transcript callback |
| `onInterrupt`, `onAudioLevel`, `onError` | | Callbacks |
| `language` | `'en-US'` | Recognition language |
| `mode` | `'responsive'` | See [Modes](#modes) |
| `positiveSpeechThreshold` | 0.8 | VAD probability that starts speech |
| `negativeSpeechThreshold` | 0.6 | VAD probability below which speech ends |
| `minSpeechStartMs` | 150 | Sustained speech before speech-start fires |
| `powerThreshold` | 0.01 | Power level that starts recognition in `responsive` mode |
| `prerollMs` | 0 | Audio before the trigger fed to the recognizer (Chrome 135+) |
| `redemptionMs` | 500 | Silence before VAD speech-end |
| `enableInterruption` | `true` | Speech can stop TTS |
| `ttsCallFn`, `ttsStreamCallFn`, `firebaseTtsCallable` | | Injected TTS backends |
| `voiceEngine` | `'vie'` | `'vie'` or `'legacy'` |
| `vieJudge` | `'server'` | `'smart_turn'`, `'server'` or `'none'` |
| `vieProfile`, `onVieEvent`, `onVieTelemetry`, `onVieTunerMove` | | vie profile store and event hooks |
| `semanticEndpointing` | `'gate'` | Legacy engine only: `'off'`, `'shadow'` or `'gate'` |
| `enableAcks` | | Play a cached backchannel ("ok", "mmhmm") when a turn is held |

See [`types.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/tivi/src/lib/types.ts)
for the full list, including the queue, recognizer and legacy state machine
callbacks.

`UseTiviReturn` includes `isListening`, `isSpeaking`, `transcription`,
`interimResult`, `audioLevelRef`, `speechProbRef`, `error`, `mode`,
`startListening`, `stopListening`, `speak`, `cancelSpeech`,
`clearTranscription`, `pauseVADProcessing`, `resumeVADProcessing`,
`warmupVAD`, `ttsQueue`, `getVie` and `voiceEngine`.

## Modes

`TiviMode` controls how speech recognition starts:

| Mode | Recognition starts on |
|---|---|
| `guarded` | VAD speech detection. Filters noise, may miss the first word |
| `responsive` | Audio power above `powerThreshold`. Lower latency |
| `continuous` | Always running, restarted automatically. Uses more resources |

Interruption is a separate switch (`enableInterruption`).

## Settings

Settings are device-local, stored in `localStorage` under `tivi-settings`.
`TIVI_DEFAULTS` holds the defaults:

| Setting | Default | Notes |
|---|---|---|
| `positiveSpeechThreshold` | 0.5 | |
| `negativeSpeechThreshold` | 0.35 | |
| `minSpeechStartMs` | 150 | |
| `powerThreshold` | 0.01 | |
| `prerollMs` | 0 | |
| `mode` | `'responsive'` | `'guarded'`, `'responsive'` or `'continuous'` |
| `enableInterruption` | `true` | |
| `playbackRate` | 1.0 | TTS speed, 0.5 to 2.0 |
| `language` | `'en-US'` | |
| `ttsProvider` | `'cloud'` | `'browser'` (Web Speech) or `'cloud'` (backend TTS). `'openai'` is a legacy value |
| `ttsBackend` | `'firebase'` | `'local'` or `'firebase'` |
| `openaiVoice` | `'en-US-EmmaMultilingualNeural'` | Voice id for the active cloud provider |
| `openaiModel` | `'gpt-4o-mini-tts'` | |
| `aiPreset` | `'default'` | A `cortex` preset id |
| `ttsCloudProvider` | `'azure'` | Derived from the preset |
| `voiceEngine` | `'vie'` | `'vie'` or `'legacy'` |
| `vieJudge` | `'server'` | Who judges the end of a turn under vie |
| `verdictModel` | `'deepseek-v4.1-flash'` | Server verdict model |
| `verdictEffort` | `'none'` | Effort for the verdict call |
| `verdictCapMs` | 800 | Longest the server waits for the verdict |
| `enableAcks` | `false` | Backchannel acknowledgements |
| `redemptionMs` | 500 | |
| `semanticEndpointing` | `'gate'` | Legacy engine |
| `smartTurnThreshold` | 0.15 | Legacy engine |
| `maxAwaitCommitMs`, `proxyDebounceMs`, `incompleteSilenceMs`, `incompleteFallbackMinP` | 1400, 100, 3000, 0.05 | Legacy engine |
| `agentMessagePolicy` | `''` | How the app handles coding-agent messages. Empty means the default policy |

Fields ending in `_applied` are one-time migration markers.

```ts
import { getTiviSettings, updateTiviSettings, resetTiviSettings, useTiviSettings } from 'tivi';

const current = getTiviSettings();          // a copy
updateTiviSettings({ playbackRate: 1.2 });  // partial merge, saved and broadcast
resetTiviSettings();

// in a component:
const { settings, updateSettings, resetSettings } = useTiviSettings();
```

## Common pitfalls

> **Warning:** **Model files must be served from `/onnx/`.** tivi loads
>   `/onnx/silero_vad_v5.onnx`, and Smart Turn weights from
>   `/onnx/smart-turn-v3.2-gpu.onnx` (or `-cpu.onnx`). The SmartChats prebuild
>   script (`apps/smartchats/scripts/prebuild.mjs`) copies them into `public/`.
>   Another project has to do the same.

> **Warning:** **iOS Safari needs a user gesture to start audio.** Call
>   `startListening` from a click handler, not on mount.

> **Warning:** **`speechProbRef` is a ref, not state.** It keeps its last value when VAD
>   pauses. Reset it yourself if a meter binds to it.

## See also

- [Voice engine](/docs/voice-engine) for turn-taking behaviour
- [vie](/docs/packages/vie) for the turn-taking API
- [llm-service](/docs/packages/llm-service#turn-verdict) for the server verdict
- Source: [`packages/tivi/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/tivi/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/vie

# vie

`vie` (Voice Interface Engine) is the turn-taking engine behind SmartChats
voice: it decides when a spoken turn is over and ships the text. It is the
default engine in [`tivi`](/docs/packages/tivi) (`voiceEngine: 'vie'`).

This page is the API reference. What the engine does turn by turn, the
judges and the tuning rules are described in
[Voice engine](/docs/voice-engine).

## Layout

| Entry | Holds |
|---|---|
| `vie/core` | Pure code: signals to events, actions and evidence, the tuner, calibration, probe interpretation. No DOM, timers or clock |
| `vie/runtime` | `createVieController` (real clock, timers, audio ring, judge), the device profile store, the browser probe, notice text |
| `vie/replay` | Recorded signals run through the core with a fake clock |
| `vie` | Re-exports all three |

The package has no runtime dependencies.

## Controller

```ts
import { createVieController, getProfileStore } from 'vie';

const vie = createVieController({
  profile: getProfileStore(),
  smartTurn: () => false,
  judgeSource: () => 'server',
  speechThreshold: () => 0.5,
  judge: () => null,
  actions: {
    startRecognizer: (trigger) => recognizer.start(),
    ship: (text, { turn, via, p }) => sendToAgent(text),
    amend: (text, meta) => replaceMessage(meta.sentText, text),
    cue: (p, turn, source) => playAck(),
  },
});

vie.onVadStart();
vie.onUpdate(key, text, isFinal);
vie.onVadEnd(null);
```

### `ControllerDeps`

| Field | Notes |
|---|---|
| `profile` | A `ProfileStore`. Params, pins and the evidence ledger are read from and saved to it |
| `smartTurn` | `() => boolean`. Whether the browser Smart Turn judge is on |
| `judgeSource` | `() => 'smart_turn' \| 'server' \| 'none'`. Default: `smartTurn() ? 'smart_turn' : 'none'` |
| `speechThreshold` | `() => number`. VAD probability that counts a frame as speech |
| `judge` | `() => JudgeLike \| null`. `JudgeLike` has `predict(audio, signal?)` resolving to `{ probability, latency_ms }` |
| `gates` | `() => TunerGates`. Which tuners the device allows |
| `actions` | `startRecognizer`, `ship`, `amend`, `cue`, and optional `continuation` and `applyParam` |
| `onEvent`, `onEvidence`, `onTunerMove`, `onTelemetry` | Hooks |
| `now`, `judgeMaxSamples`, `judgeMinSamples`, `log` | Optional |

### `VieController`

| Method | Does |
|---|---|
| `onPower(level)`, `onRecStart(trigger, prerollActiveMs)`, `onRecEnd()`, `onVadStart()`, `onVadEnd(tailElapsedMs)`, `onUpdate(key, text, isFinal)` | Signals in |
| `pushFrame(samples, prob)` | Feeds the audio ring the judge reads |
| `setAgentSpeaking(speaking)` | Suppresses listening while the agent talks |
| `onVerdict(result)` | The server verdict for the last ship (`complete`, `source`, `model`, `verdictMs`, optional `score`, `threshold`, `judge`) |
| `pendingVerdictTurn()` | The turn whose verdict is pending, or `null` |
| `mergeContinuation(turn)` | After the app aborts the previous run, send it merged with `turn` |
| `reportCutoff(where, heard)` | Records an agent-reported cut-off as evidence |
| `applyParam(key, value)` | Applies a pinned or calibrated value to the running session |
| `stop()` | Drops everything pending |
| `view()`, `params()`, `log()`, `metrics()`, `ringSeconds()`, `lastJudgeInput()` | Read state |

## Engine

`createEngine({ params, smartTurn, judgeSource? })` returns an `Engine` with
`step(signal): EngineOutput` and `view()`. Each step returns `events`,
`actions`, `evidence` and `wakeAt` (when the engine next needs a `tick`). The
controller drives it. Replay and tests drive it directly.

- `Signal` types: `power`, `vad.start`, `vad.end`, `rec.start`, `rec.end`,
  `rec.update`, `judge.verdict`, `judge.error`, `verdict.result`, `merge`,
  `agent.speaking`, `tick`, `stop`.
- `Action` kinds: `start_recognizer`, `run_judge`, `cancel_judge`, `ship`,
  `amend`, `cue`, `discard`, `continuation`.
- `EventName`: `probe.done`, `listen.wanted`, `speech.ended`,
  `transcript.settled`, `turn.ended`, `turn.shipped`, `turn.sent`,
  `turn.reopened`, `turn.amended`, `turn.held`, `turn.merged`, `turn.forced`.

## Parameters

`Params` and `DEFAULT_PARAMS`:

| Param | Default | Tuned |
|---|---|---|
| `powerThreshold` | 0.01 | No |
| `prerollMs` | 0 | Yes, 0 to 400 |
| `tailMs` | 200 | Bounded 150 to 1500, not tuned since rules v2 |
| `settleMs` | 400 | Yes, 100 to 1200 |
| `threshold` | 0.15 | Yes, 0.05 to 0.6 (Smart Turn) |
| `breathMs` | 300 | No |
| `continuationMs` | 1000 | No |
| `holdSilenceMs` | 3000 | No |
| `judgeTimeoutMs` | 1400 | No |

`TUNED_KEYS`, `BOUNDS`, `CONSERVATIVE`, `SAFE_DIRECTION`, `clampParam`,
`TUNING_RULES_VERSION` (2) and `Pins` are exported with them.

## Evidence and tuner

Each evidence row (`EvidenceRow`) judges one event: `name`, `ms`, `turn`,
`judges`, `measure`, the `params` in force and `source` (`'engine'`,
`'agent'` or `'calibration'`). Names: `onset.margin`, `onset.slack`,
`tail.early`, `settle.late_words`, `settle.slack`, `settle.gap`,
`judge.wrong_hold`, `ship.followed_by_speech`, `verdict.wrong_hold`,
`verdict.cut_off`.

| Export | Does |
|---|---|
| `freshLedger()`, `addRow(ledger, row)`, `noteTurn(ledger)` | The per-device evidence ledger (capped at `LEDGER_CAP`, 300 rows) |
| `rate(ledger, name, window?)` | Share of recent turns with that evidence |
| `watchMetrics(ledger, window?)` | Summary metrics over the last `DISPLAY_WINDOW` (20) turns |
| `applyEvidence(row, params, pins, gates, state)` | The tuner. Returns `{ params, state, move, skipped? }` |
| `freshTunerState()` | Empty tuner state |
| `moveAlert`, `turnAlerts`, `pushAlerts` | Which tuner changes are worth announcing (`TuningAlert`) |
| `formatTuningNotice`, `formatTuningAlert`, `tuningChangedPayload`, `tuningAlertPayload` | Text and telemetry payloads for those |

A `TunerMove` has `param`, `from`, `to`, `reason`, `evidence`, `ms` and `turn`.

## Device profile

`DeviceProfile` holds `params`, `pins`, `autoTune`, the evidence `ledger`,
`tuner` state, the last `probe` and `calibration` result, and dismissed
alerts. It is stored under `PROFILE_KEY` (`vie.device.v1`).

```ts
import { getProfileStore, createProfileStore } from 'vie';

const profile = getProfileStore();          // shared store on localStorage
profile.setParam('settleMs', 600, true);    // set and pin
profile.unpin('settleMs');
profile.resetTuning();
```

`ProfileStore` methods: `get`, `update`, `subscribe`, `getSnapshot`,
`setParam(key, value, pin?)`, `pin`, `unpin`, `resetTuning(seed?)`.
`createProfileStore(store, seed?, now?)` takes any `KeyValueStore` (or
`null` for memory only).

## Probe and calibration

- `collectCapabilities(inputs?)` reads what the browser supports
  (recognizer, track start, AudioWorklet, WASM SIMD and threads, platform).
  `interpretProbe(caps, at)` turns that into `gates`, `alerts` and a
  `signature`.
- Calibration: `startCalibration(params, gates, at, script?)`,
  `observeItem(state, params, observation)`, `summarize(state)`,
  `judgeItem`, `DEFAULT_SCRIPT`.

## Replay

```ts
import { replay, fromSessionExport } from 'vie';

const fixture = fromSessionExport(sessionExport);
const result = replay(fixture, { params: { settleMs: 600 } });
result.score;   // { exact, lostWordsAtShip, recoveredByAmend, shipAfterEndP50Ms, holds, ... }
```

`replay(fixture, opts?)` runs a `Fixture` (`signals` plus `truth`, the text
per turn) with optional `params`, `smartTurn` and a `judge` callback, and
returns `turns`, `events`, `actions`, `evidence` and `score`. `drive(engine, fixture, opts?)`
does the same with an engine you built. `fromSessionExport` builds a fixture
from a session export. Interim word boundaries are approximate.

## See also

- [Voice engine](/docs/voice-engine) for the behaviour
- [tivi](/docs/packages/tivi), which hosts the controller
- [llm-service](/docs/packages/llm-service#turn-verdict) for the server verdict
- Source: [`packages/vie/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/vie/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/cortex

# cortex

`cortex` is the function-calling agent loop behind SmartChats. It runs an
LLM with tools, executes the code the model writes in an injected sandbox,
and reports progress as events. It also holds the model registry, agent
presets and the voice catalog that the other packages read.

The production agent runs in the browser. The core has no DOM references:
the sandbox and the LLM call are passed in, so the same loop runs under
Node in the package's own tests.

## The pieces

```
                ┌─────────────────────────────┐
                │           Cortex            │  run loop, message history
                └──────┬──────────────────────┘
                       │
        ┌──────────────┼────────────────┐
        ▼              ▼                ▼
   ┌─────────┐  ┌──────────────┐  ┌───────────────┐
   │ Runner  │  │ SystemContext│  │ ProcessManager│
   │ (LLM    │  │   Manager    │  │  (child       │
   │  call + │  │  (prompt     │  │   agents and  │
   │  parse) │  │   modules)   │  │   processes)  │
   └─────────┘  └──────────────┘  └───────────────┘
        │
        ▼
   ┌──────────────┐
   │  Sandbox     │  runs generated code (SandboxExecutor)
   │  Executor    │
   └──────────────┘
```

## Quick orientation

```ts
import { Cortex, StreamingRunnerV3, SystemContextManager, createIntroModule, createStreamingCodeGenModule } from 'cortex';

const scm = new SystemContextManager();
scm.add_module(createIntroModule('SmartChats'));
scm.add_module(createStreamingCodeGenModule());

const cortex = new Cortex({
  model: 'claude-haiku-5-5',
  name: 'my-agent',
  sandbox: mySandboxExecutor,   // implements SandboxExecutor
  scm,
  llmCallFn: myLlmCall,         // non-streaming call, used by child processes
});

cortex.setRunner(new StreamingRunnerV3({ streamingLlmCallFn: myStreamingCall }));

cortex.on('event', (evt) => console.log(evt.type, evt));

const reply = await cortex.chat('add 2 + 2');
```

### `CortexOps`

| Field | Notes |
|---|---|
| `model` | Registry key |
| `name` | Agent name |
| `sandbox` | A `SandboxExecutor`. Required |
| `scm` | A `SystemContextManager`. When set, it replaces `functions` and `additional_system_msg` |
| `functions`, `additional_system_msg` | Required when `scm` is not given |
| `provider` | Optional. Otherwise read from the registry |
| `llmCallFn` | `({ model, input, schema?, schema_name?, signal?, effort? }) => Promise<any>` |
| `runner` | Default `SynchronousRunner` |
| `insights` | Telemetry client (optional) |
| `utilities` | Platform helpers passed to generated code |
| `apiBaseUrl` | Optional API base URL |

## `Cortex`

`Cortex` extends `EventEmitter` and emits one event name, `'event'`, with a
`type` field. Types include `turn_start`, `turn_complete`, `thought`,
`thought_chunk`, `response_chunk`, `response_complete`, `stream_chunk`,
`stream_end`, `context_status` and `workspace_update`. With a process
manager it also forwards `process_spawned`, `process_output`,
`process_state_change`, `process_complete`, `process_agent_event`,
`process_needs_input` and `process_idle_batch`.

Commonly used methods:

| Method | Does |
|---|---|
| `chat(text, maxLoops = 4)` | Adds the user text and runs the runner. Resolves with the reply |
| `run_llm(loop = 6)` | Runs the runner on the current history |
| `setRunner(runner)` | Swaps the runner and rebuilds the system prompt for its output format |
| `cancel()` | Aborts the running turn |
| `add_user_text_input(text)` | Appends a user turn |
| `snapshot()` / `restore(s)` | Save and restore the context (`ContextSnapshot`) |
| `getContextStatus()` | Token use against the model's context window |
| `getUsage()` / `resetUsage()` | Accumulated `UsageStats` (tokens, `costUsd`, `callCount`) |
| `initProcessManager(sandboxFactory)` | Enables child processes |

## Runners

A runner owns the LLM call and parses the model output into spoken text,
code and thoughts.

| Runner | Format | Used by |
|---|---|---|
| `StreamingRunnerV3` | JSON, parsed while it streams by `JsonStreamParser` | The app's main agent |
| `StreamingRunner` | Delimited text (`<<<RESPONSE>>>`, `<<<CODE>>>`, `<<<THOUGHTS>>>`), parsed by `StreamParser` | Earlier streaming protocol |
| `SynchronousRunner` | Structured JSON (`CodeOutput` schema), parsed after the full response | Default when no runner is given |

Child processes forked by `ProcessManager` run an internal non-streaming
runner (`SynchronousRunnerV2`, not exported) that uses the delimited
protocol through `llmCallFn`.

`StreamingRunnerV3` takes `{ streamingLlmCallFn }`, a function that
receives `{ model, input, max_tokens?, temperature?, effort?, text_format?, signal? }`
and returns `{ stream, data }`. Because the `response` field streams first,
the voice layer can start speaking before the model finishes writing code.

## SystemContextManager

The system prompt is built from context modules. Each module has an `id`,
`name` and `position` (0 to 100), and any of `system_msg`, `functions`,
`output_instructions`, `output_structure`, `state` and a `beforeBuild` hook.

```ts
import { SystemContextManager, createIntroModule, createCodeGenModule, createKnowledgeGraphModule } from 'cortex';

const scm = new SystemContextManager();
scm.add_module(createIntroModule('SmartChats'));
scm.add_module(createCodeGenModule());
scm.add_module(createKnowledgeGraphModule());
scm.add_module({
  id: 'my_tools',
  name: 'My Tools',
  position: 50,
  system_msg: 'You can use these tools.',
  functions: [/* Function objects */],
});

const { system_prompt, functions, output_structure } = scm.build();
```

Methods: `add_module`, `remove_module`, `update_module`, `update_state`,
`get_module`, `list_modules`, `has_module`, `clone`, `get_usage`, `build`
and `build_messages(conversation)`. `build_messages` calls each module's
`beforeBuild` and appends module `state` as a trailing message.

Module factories:

| Factory | Arguments |
|---|---|
| `createIntroModule` | `(agentName?, model?)` |
| `createCodeGenModule` | none |
| `createStreamingCodeGenModule` | none |
| `createDynamicFunctionsModule` | none |
| `createKnowledgeGraphModule` | none |
| `createResponseGuidanceModule` | `(guidance?)` |
| `createCodeOutputModule` | none |
| `createStreamingOutputModule` | none |

## Sandbox

`SandboxExecutor` is an interface:

```ts
interface SandboxExecutor {
  execute(code: string, context: Record<string, any>, timeout: number): Promise<SandboxResult>;
  initializePersistent(): Promise<void>;
  reset(): Promise<void>;
  destroy(): void;
  abort?(reason: string): void;
  setupEventStream?(handler: (event: SandboxRuntimeEvent) => void): () => void;
  syncWorkspace?(workspace: Record<string, any>): void;
}
```

`SandboxResult` has `ok`, `data`, `error`, `executionId`, `logs` and
`events`. `DEFAULT_SANDBOX_TIMEOUT` is exported. The app ships an iframe
implementation (`apps/smartchats/app/src/IframeSandbox.ts`). The package
itself ships none.

## ProcessManager

`initProcessManager(sandboxFactory)` lets the agent run work that outlives
one turn: child agents and background code. The class is internal (only its
types are exported). It forks with `fork`, lists with `ps`, reads output with
`read`, stops with `kill` and forwards input with `sendInput`.

In the app, the agent reaches these through the functions `fork_process`,
`ps`, `read_process`, `kill_process` and `send_process_input`, defined in
`apps/smartchats/app/modules/process.ts`.

## Model registry

`MODEL_REGISTRY` covers Anthropic, OpenAI, Google Gemini, xAI and
Together-hosted models: context window, max output, dated price periods,
effort levels, tier, release and shutdown dates.

```ts
import { getModelInfo, calculateCost } from 'cortex';

const info = getModelInfo('claude-haiku-5-5');
// { id, provider, contextWindow, maxOutputTokens, prices, effort?, inputPricePer1M, outputPricePer1M, cachedInputPricePer1M, cacheWritePricePer1M, ... }

const usd = calculateCost('claude-haiku-5-5', { input_tokens: 1000, output_tokens: 500 });
```

Prices are USD per million tokens. The flat `*PricePer1M` fields are the base
rates in force now. Billing uses `calculateCost(model, usage, provider?, at?, opts?)`
or `getPricesAt`, which apply the price period for a date and long-context
tiers. `usage` takes `input_tokens`, `output_tokens`, `cached_input_tokens`
and `cache_creation_input_tokens`.

Other registry exports: `getRegisteredModels`, `getModelsByProvider`,
`getModelByTier`, `listTieredModels`, `getSelectableModels`,
`isSelectableModel`, `resolveSelectableModel`, `isModelRetired`,
`isModelDisabled`, `getEffortSpec`, `supportsEffort`, `EFFORT_ORDER`,
`getModelMaker`, `isThirdPartyHosted`, `modelsReleasedSince`, and
`GPT4O_MINI_TTS_PRICING` with `estimateGpt4oMiniTtsCost`.

## Presets, voices and catalog

- `AGENT_PRESETS`, `DEFAULT_PRESET_ID`, `getPreset`, `getPresetOrDefault`,
  `findMatchingPreset`, `resolvePresetIdForAvailability`. An `AgentPreset`
  bundles `aiModel`, `ttsProvider` (`'openai' | 'azure'`) and `ttsVoice`.
- `VOICE_CATALOG`, `listVoicesForProvider`, `listVoiceIdsForProvider`,
  `findVoice`, `listAllVoices`.
- `describeModel`, `listModelCatalog`, `MODEL_STRENGTHS`: the agent-facing
  view of the registry.
- Token counting: `countTokens`, `countTokensEstimate`, `countMessageTokens`,
  `getTokenBreakdown`.

## See also

- [llm-service](/docs/packages/llm-service), which routes calls by the registry
- [tivi](/docs/packages/tivi) for the voice layer the streaming runner feeds
- Source: [`packages/cortex/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/cortex/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/llm-service

# llm-service

Provider-agnostic LLM request and streaming handlers. One handler per
provider (OpenAI, Anthropic, Google Gemini, xAI and Together AI) plus a
router that picks the handler from the model name. Every handler returns
the same `LLMResponse` shape.

The package also holds the server-side streaming pieces that both the
local server (`/llm/streamWithTTS`) and the cloud functions use: NDJSON
framing for LLM and combined LLM+TTS streams, the TTS adapter registry
(OpenAI and Azure), the turn verdict, OpenAI embeddings and the effort
mapping.

## What's in the package

| Module | Purpose |
|---|---|
| `types.ts` | `LLMRequest`, `LLMResponse`, `LLMUsage`, `LLMStreamRequest`, `LLMStreamResponse`, `Message`, `Provider` |
| `providers/` | One handler per provider (`anthropic`, `openai`, `gemini`, `xai`, `together`) plus streaming variants. `providers/index.ts` routes by model |
| `effort.ts` | Maps an effort level to each provider's request fields |
| `streaming/` | Response splitter, NDJSON writer, LLM and LLM+TTS orchestrators, turn verdict |
| `tts_providers/` | Server TTS adapters (`azure`, `openai`), the registry, speed mapping |
| `embeddings.ts` | `openaiEmbedding` |
| `tts_tokens.ts` | `countGpt4oMiniTtsInputTokens` |

## Requests

```ts
import { handleLLMRequest } from 'llm-service';

const response = await handleLLMRequest({
  model: 'claude-haiku-5-5',
  input: [
    { role: 'system', content: 'Be brief.' },
    { role: 'user', content: 'Hello.' },
  ],
  max_tokens: 1024,
});

response.output_text;   // string
response.usage;         // { input_tokens, output_tokens, cached_input_tokens?, cache_creation_input_tokens? }
response.provider;      // 'anthropic'
```

`handleLLMRequest` calls `getProviderForModel(model)`, which reads the
provider from `cortex`'s model registry (`getModelInfo`), then calls the
matching handler. The per-provider handlers `handleAnthropicRequest`,
`handleOpenAIRequest`, `handleGeminiRequest` and `handleOpenAIStreamRequest`
are also exported.

### `LLMRequest`

| Field | Type | Notes |
|---|---|---|
| `model` | `string` | Registry key, for example `claude-haiku-5-5` or `deepseek-v4.1-flash` |
| `input` | `Message[]` | `{ role: 'system' \| 'user' \| 'assistant', content: string }` |
| `max_tokens` | `number?` | |
| `temperature` | `number?` | Dropped for effort-capable Anthropic models (`anthropicDropsTemperature`) |
| `effort` | `EffortLevel?` | See [Effort](#effort). Omitted means the provider default |
| `schema`, `schema_name` | `object?`, `string?` | Structured output. When `schema` is set the structured endpoint is used |
| `apiKey` | `string?` | Bring-your-own key. Overrides the server key for this call |
| `top_logprobs` | `number?` | Ask for the N most likely first tokens. Only the Together handler returns them |

`LLMResponse` carries `output_text`, `usage`, `model`, `provider`,
`finish_reason`, `latency_ms`, optional `top_logprobs` (`{ token, logprob }[]`)
and optional `raw`. `usage.output_tokens` includes reasoning tokens.

## Streaming

```ts
import { handleLLMStreamRequest } from 'llm-service';

const { stream, aggregated } = handleLLMStreamRequest({
  model: 'claude-haiku-5-5',
  input: [{ role: 'user', content: 'Count to five.' }],
  max_tokens: 2048,
});

for await (const delta of stream) process.stdout.write(delta);
const final = await aggregated;   // LLMResponse with full usage
```

`handleLLMStreamRequest` is synchronous and returns an `LLMStreamResponse`:
`stream` yields text deltas and `aggregated` resolves with the full response.
`LLMStreamRequest` has the same `model`, `input`, `max_tokens`,
`temperature`, `effort` and `apiKey` fields as `LLMRequest`, plus `stop`
and `text_format` (a JSON schema for structured streaming output).

### NDJSON orchestrators

Two helpers frame a stream as newline-delimited JSON on an HTTP response
(Express and Firebase `onRequest` responses both fit `NdjsonStreamResponse`).

```ts
import { streamLlmToNdjson, streamLlmTtsToNdjson } from 'llm-service';

// Text only. Writes { t: 'delta', d } lines, then done.
await streamLlmToNdjson({ res, llmStream });

// Text plus audio.
const result = await streamLlmTtsToNdjson({
  res,
  llmStream,
  tts: (opts) => adapter.stream(opts),
  voice: 'en-US-AvaMultilingualNeural',
  firstChunkWordThreshold: 8,
  firstChunkTimeThresholdMs: 0,
});
```

| `streamLlmTtsToNdjson` option | Notes |
|---|---|
| `res` | The response. `beginNdjsonStream` is called on it before the first frame |
| `llmStream` | An `LLMStreamResponse` the caller already resolved |
| `tts` | A `TtsStreamFn` (`(opts) => AsyncIterable<Buffer>`). Omit for text only |
| `voice` | Required with `tts` |
| `firstChunkWordThreshold` | First TTS chunk fires after this many words at a sentence boundary |
| `firstChunkTimeThresholdMs` | Alternate time trigger. `0` disables |
| `funcReceivedMs` | Handler entry time for the first `server_timing` frame |
| `emitServerTiming` | Default `true` |
| `onBeforeDone` | Called after all TTS settles. Its return value is merged into the `done` frame (cloud adds billing here) |
| `signal` | Aborts like a client disconnect |

The combined stream writes `text`, `audio_start`, `audio` (PCM16 24 kHz
mono, base64), `audio_end`, `audio_error`, `server_timing`, `llm_done`,
`error` and `done` frames (the `NdjsonFrame` union). If the client
disconnects, the orchestrator stops writing and aborts TTS, but still drains
the LLM stream so usage is complete. The result reports `aggregated`,
`totalTtsChars`, `ttsChunkCount`, `msTotal`, `aggregateOk` and
`clientDisconnected`.

Lower-level pieces are exported too: `ResponseSplitter`, `wordCount`,
`nthWordEndPosition`, `findBoundaryAfter`, `beginNdjsonStream`,
`writeNdjsonLine`, `openaiTtsStream` and `TTS_TARGET_BYTES` (6400).

## Turn verdict

A small LLM call the host makes before the main model runs, to decide
whether the user has finished speaking. The client sends a `verdict` block
on the request body. The host writes the verdict frame before any delta
and, on INCOMPLETE, ends the response without starting the main call.

```ts
import { parseVerdictRequest, runTurnVerdict, verdictFrame, writeNdjsonLine } from 'llm-service';

const verdict = parseVerdictRequest(req.body);   // null when absent or malformed
if (verdict) {
  const v = await runTurnVerdict({ input: req.body.input, verdict, receivedAtMs });
  writeNdjsonLine(res, verdictFrame(v));          // { t: 'verdict', complete, source, ... }
  if (!v.complete) return res.end();
}
```

| `verdict` field | Notes |
|---|---|
| `model` | Registry key of the verdict model. Required |
| `effort` | Effort for the verdict call. `'none'` turns thinking off |
| `cap_ms` | Longest wait before proceeding as COMPLETE. Default `VERDICT_DEFAULT_CAP_MS` (800), minimum 50 |
| `judge` | `'v1'` (default) or `'v2'` |
| `threshold` | Judge v2 only, 0 to 1. Holds when P(COMPLETE) is below it. Default 0.5 |

Judge v1 shows the last 4 turns as chat turns and the sampled word decides.
Judge v2 quotes the last 2 turns as a transcript, asks for `top_logprobs`
and decides from P(COMPLETE) on the first token. It falls back to the word
when the provider returns no probabilities (only Together returns them
today).

`runTurnVerdict` never throws. A timeout, failure or unparsed answer counts
as COMPLETE, and `source` says why: `'model'`, `'cap'`, `'error'` or
`'skipped'` (no user utterance to judge). The result also carries timings
(`ms.queue`, `ms.model`, `ms.total`), `usage`, `raw`, `prob_score`,
`utterance_words`, `context_turns` and `shown` (what the judge was shown).

Also exported: `sliceVerdictContext`, `toVerdictTurn`, `parseVerdict` and
`VERDICT_SYSTEM_PROMPT`. How the client uses the verdict is covered in
[Voice engine](/docs/voice-engine).

## Effort

`effort` takes an `EffortLevel` from `cortex`: `'none'`, `'minimal'`,
`'low'`, `'medium'`, `'high'`, `'xhigh'` or `'max'`. A level must be one the
model registry lists for that model. `'none'` means thinking off. It is
accepted for Anthropic, OpenAI and Together, and rejected for xAI and Gemini.

| Export | Does |
|---|---|
| `resolveEffort(adapter, model, effort)` | Validates. Returns `{ level }`, `{ thinkingOff: true }` or `null` |
| `effortBody(adapter, model, effort)` | The request fragment to spread into the SDK call, for example `{ output_config: { effort } }` on Anthropic |
| `isEffortLevel(v)` | Type guard |
| `anthropicDropsTemperature(model)` | True for Anthropic models that reject `temperature` |
| `EffortNotSupportedError` | Thrown for an unknown or unsupported level |

`EffortAdapter` is `'anthropic' | 'openai_responses' | 'openai_chat' | 'gemini' | 'xai' | 'together'`.

## TTS adapters

```ts
import { resolveAdapter, getProviderStatus, DEFAULT_TTS_PROVIDER } from 'llm-service';

const config = { openaiKey, azureKey, azureRegion };   // TtsAdapterConfig
const adapter = resolveAdapter(config, 'azure');        // ServerTtsAdapter | null
for await (const pcm of adapter!.stream({ text: 'Hi.', voice: 'en-US-AvaMultilingualNeural' })) { /* ... */ }
```

`TtsProviderId` is `'openai' | 'azure'` and `DEFAULT_TTS_PROVIDER` is
`'azure'`. `resolveAdapter` falls back to the default when the requested
provider can't be built, and returns `null` when neither can. Adapters are
built on first use and cached. `getProviderStatus(config)` lists each
provider with `available`, `isDefault` and a `reason` when unavailable.

A `ServerTtsAdapter` has `name`, `stream(opts)` (PCM16 24 kHz mono
buffers), `listVoices()` and `estimateCost(opts)`. `TtsStreamOpts` takes
`text`, `voice`, `speed`, `instructions` (OpenAI only), `onTiming` and
`signal`.

Speed helpers: `TTS_SPEED_RANGES` (OpenAI 0.25 to 4.0, Azure 0.5 to 2.0),
`clampTtsSpeed(provider, speed)`, `azureProsodyRate`, `buildAzureSsml` and
`openaiSpeedInstruction`.

## Embeddings

```ts
import { openaiEmbedding, EMBEDDING_MODEL } from 'llm-service';

const { embedding, dimensions, inputTokens, costUsd } =
  await openaiEmbedding({ apiKey, text: 'hello', dimensions: 512 });
```

`EMBEDDING_MODEL` is `text-embedding-3-small`. Options are `apiKey`
(required), `text`, `dimensions` and `signal`. Failures throw
`OpenAIEmbeddingError`.

## TTS token counting

OpenAI's `gpt-4o-mini-tts` bills by input tokens. `countGpt4oMiniTtsInputTokens(text)`
returns that count, and `cortex` exports the rates:

```ts
import { countGpt4oMiniTtsInputTokens } from 'llm-service';
import { GPT4O_MINI_TTS_PRICING } from 'cortex';

const tokens = countGpt4oMiniTtsInputTokens(text);
const inputCostUsd = tokens * GPT4O_MINI_TTS_PRICING.inputPricePer1M / 1_000_000;
```

## Provider keys

Without `apiKey`, each handler reads its key from the environment:
`ANTHROPIC_API_KEY` and `OPENAI_API_KEY` (through the SDK defaults),
`GEMINI_API_KEY`, `XAI_API_KEY` and `TOGETHER_API_KEY` (or
`TOGETHERAI_API_KEY`). The TTS registry does not read the environment. The
host passes keys in `TtsAdapterConfig`.

> **Note:** A model missing from the registry still routes: `getModelInfo` matches a
> registry key by prefix, then infers the provider from the name (`claude-`,
> `gemini-`, `grok-`, an `org/Model` id for Together, otherwise OpenAI) and
> uses that provider's default pricing.

## See also

- [cortex](/docs/packages/cortex) for the model registry and pricing
- [Voice engine](/docs/voice-engine) for how the app uses the turn verdict
- Source: [`packages/llm-service/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/llm-service/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/smartchats-database

# smartchats-database

The **data layer** for SmartChats. Pure SurrealQL query builders, an
SDK client, the local schema and a multi-step operations layer. The
query builders do no I/O.

The principle: **separate query construction from query execution**.
Builders return `QuerySpec` values (a query string and variables). The
runtime hands those specs to a `Client` that knows how to talk to a
real SurrealDB instance. This lets the same query code run in the
browser (through the backend's `DataAPI`), in Node (CLI and MCP, direct
SDK), or in tests.

## Layered model

```
your code  →  operations.X(data, opts)   multi-step (exportBundle, importBundle)
                       │
                       ▼
              queries.Y(args)            pure: args → { query, variables }
                       │
                       ▼
     data.query(spec)  or  client.runQuery(spec)    I/O happens here
            │                        │
     DataAPI (backend)        Client (createClient: direct
     cloud or local           SurrealDB connection)
```

## What's in the package

| Module | Purpose |
|---|---|
| `queries/*` | Pure `QuerySpec` builders, one file per domain (sessions, logs, knowledge_graph, todos, metrics, media, byo_keys and more). Exported at the top level and as the `queries` namespace |
| `operations/*` | `exportBundle` and `importBundle`, plus `DEFAULT_EXPORT_TABLES`, `SENSITIVE_TABLES`, `NEVER_EXPORT_TABLES` |
| `schema/*` | The local schema: `LOCAL_DDL`, `LOCAL_SCHEMA_VERSION`, `LOCAL_SCHEMA_MIGRATIONS`, `applyLocalSchema`. Exported as the `schema` namespace |
| `client.ts` | `createClient`, `createLazyClient`, `createUserClient`: direct SurrealDB connections |
| `data_api.ts` | `makeDataAPI`, `makeCloudDataAPI`, `makeLocalDataAPI`. Node only, imported from `smartchats-database/data-api` |
| `id.ts` | `stringifyRecordId`, `parseRecordIdArg` |
| `types.ts` | `QuerySpec`, `AuditFields`, `EventTimeFields` |

## Quick start

### Building a query

```ts
import { queries } from 'smartchats-database';

const spec = queries.listSessions({ limit: 50 });

// spec is { query: "SELECT ... FROM sessions ...", variables: {...} }
```

The builder has zero side effects. You can log it, snapshot-test it,
inspect what would run, without touching a database.

### Running a query (Node, direct SurrealDB)

```ts
import { createClient, queries } from 'smartchats-database';

const client = createClient({
  url: 'ws://127.0.0.1:8000/rpc',
  namespace: 'smartchats',
  database: 'main',
  auth: { username: 'root', password: 'root' },
});
await client.connect();

const sessions = await client.runQuery(queries.listSessions({ limit: 10 }));
```

`createLazyClient` connects on the first query. `createUserClient` connects
with a record-access token so per-user permissions apply. A `Client` also
has `runRaw(query, variables)` for per-statement results, `insert` and
`query`.

For a `DataAPI` without the app, Node code can use the factories on the
`data-api` subpath:

```ts
import { makeDataAPI } from 'smartchats-database/data-api';

const { data, close } = await makeDataAPI('local');   // or 'cloud'
```

### Running a query (browser, through `DataAPI`)

The browser app doesn't talk to SurrealDB directly. It goes through
the backend contract:

```ts
import { getBackend } from '@/lib/backend';
import { queries } from 'smartchats-database';

const backend = getBackend();
const { rows } = await backend.data.query(queries.listSessions({ limit: 10 }));
```

Same `QuerySpec`. The cloud backend sends it to a Cloud Function that runs
it against SurrealDB. The local backend sends it to the local server's
`/data/query` route. The call site doesn't change.

## Operations

Some workflows need several queries + ordering + error handling.
Those live in `operations/`:

```ts
import { operations } from 'smartchats-database';

const { bundle, perTable, blockedTables } = await operations.exportBundle(backend.data, {
  source: 'cloud',
  userId,
});

const result = await operations.importBundle(otherBackend.data, bundle);
```

Both take a `DataAPI`. `exportBundle` options include `tables` (default
`DEFAULT_EXPORT_TABLES`), `includeSensitive`, `pageSize` and `onProgress`.
`importBundle` options include `tables` and `concurrency`. A `Bundle` is
`{ version: 1, exportedAt, source, userId, tables, ... }`. This is how data
moves between cloud and local.

## Schema

The local schema lives in `packages/smartchats-database/src/schema/local.ts`.
`schema.applyLocalSchema(db, opts)` applies it idempotently, and the local
server calls it at startup. The cloud's per-user schema is not part of this
package.

> **Note:** **SurrealDB v3.** Notable v3 differences:
>   - `type::record` replaces `type::thing` for record constructors
>   - `type::is_string(x)` (not `type::is::string(x)`) for type predicates
>   - The `logs` table is SCHEMALESS. `content` can be NONE, a string,
>     or a record link, so builders that match on `content` guard with
>     `type::is_string(content)`

## Why no I/O at the query layer?

1. **Tests.** Tests on `QuerySpec` outputs catch query changes
   without a database.
2. **Reuse.** The same query runs in the browser (through `DataAPI`),
   in Node (direct SDK) and in tests.
3. **Cloud vs local symmetry.** Both backends accept `QuerySpec`.
   Adding a new query means writing one builder, not two.

## See also

- [smartchats-backend](/docs/packages/smartchats-backend) for `DataAPI`
- [smartchats-sessions](/docs/packages/smartchats-sessions)
- Source: [`packages/smartchats-database/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/smartchats-database/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/smartchats-backend

# smartchats-backend

The typed backend contract for SmartChats: the `SmartChatsBackend`
interface, the shared error type, HTTP helpers for NDJSON streams, the
`AuthProvider` interface and media storage helpers.

Two packages implement it:

- `smartchats-backend-local` (`LocalBackend`, `LocalAuthProvider`), which
  talks to `smartchats-local-server`
- `smartchats-backend-firebase` (closed source), which talks to the cloud
  Firebase Functions

The app imports types from this package. Its bootstrap code picks which
implementation to construct.

## What's in the package

| Module | Purpose |
|---|---|
| `types.ts` | `SmartChatsBackend`, `BackendCapabilities`, every sub-API (`LLMAPI`, `TTSAPI`, `EmbeddingsAPI`, `DataAPI`, `UsageAPI`, `KeysAPI`, `BillingAPI`, `ToolsAPI`, `IntegrationsAPI`, `InsightsAPI`, `StorageAPI`), `BackendError` |
| `auth_provider.ts` | `AuthProvider`, `AuthUser`, `AuthCapabilities`, `SignInMethod`, `EmailCredentials` |
| `http/` | `postStream`, `readNdjson`, `base64ToPcmBuffer`, `statusToBackendErrorCode`, `toServerArgs`, `createLLMStreamResult`, `createLLMTTSStreamResult` |
| `storage.ts` | Media upload limits, MIME allow list, key and cursor helpers, argument validators |

## The contract

```ts
interface SmartChatsBackend {
  readonly capabilities: BackendCapabilities;
  readonly id: string;               // "firebase" | "local" | user-defined

  readonly llm: LLMAPI;              // stream(), streamWithTTS(), warmup?()
  readonly tts: TTSAPI;              // stream(), warmup?()
  readonly embeddings: EmbeddingsAPI;
  readonly data: DataAPI;            // query(), healthCheck()
  readonly usage: UsageAPI;
  readonly keys: KeysAPI;            // BYO provider keys
  readonly billing: BillingAPI;      // throws 'not_supported' when !capabilities.billing
  readonly tools: ToolsAPI;          // search(), fetchUrl()
  readonly integrations: IntegrationsAPI;
  readonly insights: InsightsAPI;    // emit(events)
  readonly storage: StorageAPI;      // throws 'not_supported' when !capabilities.storage

  health(): Promise<HealthReport>;
  providers(): Promise<ProvidersReport>;
}
```

LLM and TTS are streaming only. `llm.streamWithTTS(args)` takes
`LLMCallArgs` plus `voice`, `speed`, `instructions`, `tts_provider` and an
optional `verdict` block (`{ model, effort?, cap_ms?, judge?, threshold? }`),
and yields `LLMTTSEvent`s: `text`, `verdict`, `text_end`, `audio_start`,
`audio`, `audio_end` and server timing events. See
[llm-service](/docs/packages/llm-service#turn-verdict) for the verdict.

### `BackendCapabilities`

All flags are booleans. The UI uses them to show or hide features.

| Flag | Meaning |
|---|---|
| `billing` | Credit and tier billing is active (billing UI, checkout) |
| `byoKeys` | Server-side bring-your-own key management |
| `embeddings` | Embedding generation |
| `search` | Web search tool |
| `urlFetch` | URL fetch and readability parsing |
| `integrations` | Third-party integrations (Google Docs, Gmail, Calendar) |
| `insights` | Telemetry ingestion |
| `storage` | Media storage (`StorageAPI`) |

`providers()` is separate from capabilities: it reports which LLM and TTS
providers can serve requests right now, given server keys and saved BYO keys.

## `BackendError`

```ts
class BackendError extends Error {
  constructor(
    code: BackendErrorCode,
    message: string,
    retryable: boolean = false,
    details?: unknown,
  );
}
```

`BackendErrorCode` is one of `insufficient_credits`, `provider_error`,
`not_supported`, `network_error`, `rate_limited`, `invalid_request`,
`not_found`, `server_error`, `aborted` or `unknown`.
`statusToBackendErrorCode(status)` maps 401 to `invalid_request`, 402 to
`insufficient_credits`, 429 to `rate_limited`, 5xx to `server_error` and
anything else to `provider_error`.

## `postStream`

```ts
import { postStream, readNdjson } from 'smartchats-backend';

const response = await postStream(url, body, headers, signal);   // Promise<Response>
for await (const frame of readNdjson(response)) {
  // one parsed object per line; malformed lines are skipped
}
```

`postStream(url, body, headers, signal?)` POSTs a JSON body and returns the
`Response`. Headers, including auth, are the caller's. Failures become
`BackendError`s:

- aborted (by `signal` or `AbortError`) becomes `aborted`, not retryable
- a network error becomes `network_error`, retryable
- a non-2xx status becomes the mapped code, retryable when the status is 500 or above

Retryable errors are retried twice, after 250 ms and 750 ms, unless the
signal has aborted. Retries happen before any body is read.

> **Warning:** Safari reports an aborted fetch as `TypeError: Load failed`, not
>   `AbortError`. `postStream` checks `signal.aborted` as well as the error
>   name. Code that copies the pattern should do the same.

`createLLMStreamResult(response)` and `createLLMTTSStreamResult(...)` turn a
streaming response into the `LLMStreamResult` and `LLMTTSStreamResult`
shapes. `toServerArgs(args)` converts `LLMCallArgs` to the server payload
(it builds `text_format` from `schema` and `schema_name`).

## Authentication

Auth is not part of `SmartChatsBackend`. A backend gets its bearer token
when it is constructed. The app uses a separate `AuthProvider`:

```ts
interface AuthProvider {
  readonly capabilities: AuthCapabilities;   // { required: boolean, methods: SignInMethod[] }
  getCurrentUser(): AuthUser | null;         // { uid, email, displayName }
  getIdToken(): Promise<string | null>;
  onAuthChange(callback: (user: AuthUser | null) => void): () => void;
  signIn(method: SignInMethod, credentials?: EmailCredentials): Promise<void>;
  signOut(): Promise<void>;
  sendPasswordReset?(email: string): Promise<void>;
}
```

`SignInMethod` is `'google' | 'email' | 'anonymous'`. `EmailCredentials`
is `{ email, password, signup? }`. `sendPasswordReset` is optional: a
provider without email accounts leaves it out and the UI hides the link.

`LocalAuthProvider` (in `smartchats-backend-local`) reports
`required: false`, always returns one local user and makes sign-in and
sign-out no-ops. `LocalBackend` takes `{ baseUrl, apiKey? }` and sends
`Authorization: Bearer <apiKey>` when a key is set.

## Storage helpers

`storage.ts` is shared by client and server so both enforce the same rules:
`STORAGE_ALLOWED_MIME`, `isAllowedMime`, `kindForMime`, `buildMediaKey`,
`usesMultipart` and `planParts` (multipart above
`STORAGE_MULTIPART_THRESHOLD_BYTES`, 50 MiB), thumbnail helpers,
`encodeMediaCursor` and `decodeMediaCursor`, and the validators
`validateCreateUploadArgs` and `validateListMediaArgs`.

## Other exports

`SMARTCHATS_REQUIRED_TABLES`, `SMARTCHATS_DEFAULT_LOCAL_PORT` (4242),
`SMARTCHATS_DEFAULT_LOCAL_URL` and `LLM_PROVIDERS` (`openai`, `anthropic`,
`google`, `xai`, `together`, `azure`).

## See also

- [llm-service](/docs/packages/llm-service) for the server side of the streams
- [smartchats-database](/docs/packages/smartchats-database) for what goes through `data.query`
- Source: [`packages/smartchats-backend/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/smartchats-backend/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/smartchats-sessions

# smartchats-sessions

Session export and analysis for SmartChats. It pulls a session's
insights events from a SurrealDB-backed deployment into a `SessionBundle`,
runs analyzers over it (errors, transcript, traces, performance,
executions), and merges failures across many sessions for triage. A second
set of analyzers (`analysis_db`) queries the database directly for cost,
errors, slow calls and other audits.

The bundle analyzers do no I/O. Each is a pure function over a
`SessionBundle`, paired with a formatter, so you can call them from your
own scripts as well as through the npm scripts.

## What's in the package

| Module | Purpose |
|---|---|
| `export.ts` | `buildSessionBundle(client, filter)`, `findSessions`, `findCandidateSessions`, `exportSessionToFile`, `exportRecentSessionsToFiles` |
| `analysis/transcript.ts` | User/agent transcript reconstruction, optionally with thoughts + code |
| `analysis/errors.ts` | Error extraction + per-session signature grouping |
| `analysis/performance.ts` | LLM/exec/voice latency stats, cache hit rate, turn pacing |
| `analysis/traces.ts` | Causality trees built from `trace_id` / `parent_event_id` |
| `analysis/executions.ts` | Per-execution detail (code, function calls, result) |
| `analysis/inspect.ts` | Drill into a specific event or trace |
| `analysis/triage_errors.ts` | Cross-session signature merge and handled state |
| `analysis_db/` | Database-side audits: `queryCostBy*`, `queryErrors`, `querySlowFunctionCalls`, `queryFunctionCallHistogram`, `queryFunctionCallsByArgs`, `queryUsersActivity`, `queryContextGrowth`, `queryIssues`, `queryTtsTiming*`, and `liveMonitor` for polling |
| `queries.ts` | Query specs for session events and session search |
| `cli/find_cli.ts` | The `find-sessions` argument parser and runner |
| `scripts/` | The npm script entry points |

## Quick start: CLI

```bash
# Pull bundles from the local instance (from the repo root)
bin/save_session --app smartchats                 # latest session
bin/save_session --app smartchats --tag simi      # filter by tag
bin/save_session --app smartchats --last 5        # several sessions

cd packages/smartchats-sessions

# Single-session deep dive
npm run analyze:summary      -- <bundle.json>
npm run analyze:transcript   -- <bundle.json>     # --with-code, --markdown
npm run analyze:errors       -- <bundle.json>
npm run analyze:performance  -- <bundle.json>
npm run analyze:traces       -- <bundle.json>
npm run analyze:inspect      -- <bundle.json> --event=<id>
```

## Quick start: programmatic

```ts
import {
  analyzeErrors, formatErrors,
  analyzeTranscript, formatTranscript,
  buildTraceTrees, formatTraces,
} from 'smartchats-sessions';
import fs from 'node:fs';

const bundle = JSON.parse(fs.readFileSync('bundle.json', 'utf8'));

// Pure analysis: returns structured data
const errors = analyzeErrors(bundle);
console.log(errors);  // { session_id, total, by_signature: {...}, records: [...] }

// Formatter renders it for humans
console.log(formatErrors(errors));
```

Every analyzer follows the same shape: `analyzeX(bundle)` returns a
typed result and `formatX(result)` returns a string. The trace analyzer is
named `buildTraceTrees`, and `inspectEvent` / `inspectTrace` drill into one
event or trace.

## Cross-session error triage

Across many sessions: what is failing, grouped by signature, with
already-fixed signatures hidden?

```bash
cd packages/smartchats-sessions

npm run triage:errors                              # ~/.smartchats/session_bundles → ./triage/<ts>/
npm run triage:errors -- --since 7d --min-count 2
npm run triage:errors -- --dry-run                 # preview without writing

# Once you ship a fix:
npm run triage:mark -- <report-slug> --status fixed --commit <sha>

# View handled-state:
npm run triage:mark -- --list
```

**State file**: `data/triage/handled.json` at the repo root, tracked in
git so the "already fixed" record is shared. Override the path with
`--state` or `SMARTCHATS_TRIAGE_STATE_FILE`. Statuses are `fixed`,
`wontfix` and `investigating`.

**Regression rule**: signatures marked `fixed` re-surface as
`REGRESSION:` reports when a session whose `end_time > fixed_at` hits
the same signature. Catches stale fixes that didn't reach production.
`wontfix` entries land in `wontfix_summary.md` so they're not silently
invisible.

## Discovering sessions to pull

Before you pull a bundle, you usually want to find candidates:

```bash
cd packages/smartchats-sessions
npm run find-sessions -- --has-error --since 7d

# Filter flags:
#   --app, --tag, --has-error, --has-event-tag, --has-event-type,
#   --missing-event-type, --min-events, --max-events, --min-duration,
#   --max-duration, --since, --until, --limit
# Formats:
#   --format=table | json | ids   (pipe ids to save_session)
```

Pair with `bin/save_session`:

```bash
npm run find-sessions -- --has-error --since 7d --format=ids \
  | xargs -I{} bin/save_session --session-id {}
```

## Adding a new analyzer

The pattern is documented in `packages/smartchats-sessions/src/analysis/README.md`.
Briefly:

1. Write `analyzeX(bundle: SessionBundle): XResult`, a pure function.
2. Write `formatX(result: XResult, opts?): string`, a pure formatter.
3. Add a script in `scripts/` and an `npm run analyze:x` entry if you want one.
4. Re-export from `index.ts`.

The analyzers don't share state. They don't need to be ordered. Each
one is independently testable with a fixture bundle.

## Open vs closed boundary

This package never imports cloud credentials. The hosted service keeps
its own admin export tool for cross-user triage, because it needs root
credentials. It reuses `findCandidateSessions` and `buildSessionBundle`
from this package.

## When to use this package directly

- You want a custom analyzer (latency distribution, knowledge graph write
  patterns, anything not in the default set)
- You're building a dashboard that reads SmartChats session data
- You're automating triage (a CI job that pulls fresh failures)

## See also

- [smartchats-database](/docs/packages/smartchats-database) for the `Client` the exporters take
- [simi](/docs/packages/simi), whose runs are tagged `simi`
- Analyzer conventions: `packages/smartchats-sessions/src/analysis/README.md`
- Source: [`packages/smartchats-sessions/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/smartchats-sessions/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/smartchats-tools

# smartchats-tools

Shared helpers for the agent's web tools: Serper web search and URL fetch
with Readability text extraction. The local server's `/tools/search` and
`/tools/fetchUrl` routes and the matching cloud functions all call these.

The helpers take keys and options as arguments and return data. They read
no environment variables and do no auth or billing. Each host adds those.

## `serperSearch`

```ts
import { serperSearch, normalizeOrganic } from 'smartchats-tools';

const r = await serperSearch({ apiKey, query: 'surrealdb hnsw index', numResults: 5 });
r.organic;     // Serper's organic results ({ title, link, snippet, ... })
r.costUsd;     // credits × $0.001

const results = normalizeOrganic(r.organic);   // SearchResult[]: { title, url, snippet, extra? }
```

| Option | Notes |
|---|---|
| `apiKey` | Serper key. Required |
| `query` | Required, non-empty |
| `numResults` | Default 10 |
| `signal` | Optional abort |

The result (`SerperSearchResult`) has `organic`, optional `knowledgeGraph`
and `answerBox`, `credits` (1 when Serper doesn't report it) and `costUsd`.
Other Serper fields are dropped.

A missing key, an empty query or a non-2xx response throws `SerperError`
(with `status` for HTTP errors). Network failures and aborts throw the
underlying error.

`normalizeOrganic` maps results to the `SearchResult` shape from
`smartchats-backend`, keeping unknown fields under `extra`.

## `extractReadableText`

```ts
import { extractReadableText } from 'smartchats-tools';

const { title, text, htmlBytes, readabilitySucceeded } =
  await extractReadableText('https://example.com/post', { maxChars: 20000 });
```

| Option | Default | Notes |
|---|---|---|
| `maxChars` | none | Text longer than this is cut and ends with `[truncated]` |
| `timeoutMs` | 15000 | Fetch timeout. Ignored when `signal` is given |
| `userAgent` | `Mozilla/5.0 (compatible; SmartChats/1.0; +https://smartchats.ai)` | |
| `signal` | | Replaces the timeout signal |

It fetches the page (following redirects), parses it with `linkedom`, adds a
`<base href>` so relative links resolve, and runs Mozilla Readability. When
Readability finds no article it falls back to the body text and
`readabilitySucceeded` is `false`. A page with no text returns empty strings
rather than throwing.

An empty URL, a non-2xx response, a network failure or a parse failure
throws `ExtractError` (`status` for HTTP errors, `cause` for the original
error).

## See also

- [smartchats-backend](/docs/packages/smartchats-backend), whose `ToolsAPI` (`search`, `fetchUrl`) these back
- Source: [`packages/smartchats-tools/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/smartchats-tools/src/index.ts)

---

Source: https://smartchats.ai/docs/packages/simi

# simi

`simi` (Simulation Interface) runs declarative test workflows inside the
app. A workflow is a plain object with a list of steps: dispatch a store
action, assert on state, wait for a condition or capture a value. The runner
executes the steps in order and emits insights events as it goes.

Each run tags its session with `simi`, the app name, the workflow id and the
workflow's own tags, so test sessions go through the same telemetry and
triage tools as real ones.

## Exports

| Export | What it is |
|---|---|
| `defineWorkflow(workflow)` | Identity helper that type-checks a `SimiWorkflow` |
| `executeWorkflow(workflow, dispatch, getState, getClient, opts?)` | The runner. Resolves with a `RunResult` |
| `KEEP_MODEL_TAG` | `'keep-model'`. Workflows with this tag ignore `modelOverride` |
| Types | `SimiWorkflow`, `WorkflowStep`, `ActionStep`, `AssertStep`, `WaitForStep`, `CaptureStep`, `Resolver`, `StateResolver`, `FindResolver`, `EvalResolver`, `ResultResolver`, `RunOpts`, `RunResult`, `StepResult` |

## A workflow

```ts
import { defineWorkflow } from 'simi';

export const basicChatFlow = defineWorkflow({
  id: 'basic_chat_flow',
  app: 'smartchats',
  tags: ['smoke', 'chat'],
  setupWorkflows: ['complete_onboarding'],
  steps: [
    { waitFor: 'state.aiModel !== "" && state.agent !== null', timeout: 10000 },
    { action: 'sendMessageAsync', args: ['What is 2 + 2?'], timeout: 30000, wait: 500 },
    { assert: 'state.chatHistory.length >= 2', message: 'Chat should have user + assistant messages' },
  ],
});
```

`SimiWorkflow` fields: `id`, `app`, `tags?`, `setupWorkflows?` and `steps`.

## Steps

| Step | Fields | Does |
|---|---|---|
| `ActionStep` | `action`, `args?`, `id?`, `wait?`, `timeout?` | Dispatches a store action. `wait` sleeps afterwards (divided by `RunOpts.speed`). The step fails if the action's promise outlasts `timeout` |
| `AssertStep` | `assert`, `message?` | Evaluates the expression. Falsy fails the workflow |
| `WaitForStep` | `waitFor`, `timeout?` | Polls the expression until it is truthy or the timeout passes |
| `CaptureStep` | `capture`, `as?` | Evaluates the expression and records the value in a `simi_capture` event |

Expressions are JavaScript strings evaluated with `state` (the store state)
and `results` (earlier action results by step label) in scope.

## `$resolve` arguments

Action `args` can hold resolver objects that are filled in at step time:

```ts
{ action: 'editChatMessage', args: [
  { $resolve: 'state', path: 'chatHistory.0.id' },
  'edited content',
] }

{ action: 'deleteSession', args: [
  { $resolve: 'find', path: 'sessions', match: { title: 'Untitled' } },
] }

{ action: 'apply', args: [
  { $resolve: 'result', step: 'compute_args', path: 'value' },
] }
```

| Resolver | Fields | Reads |
|---|---|---|
| `state` | `path` | A dotted path in the store state |
| `find` | `path`, `match?`, `index?` | An item in the array at `path` |
| `result` | `step`, `path?` | The return value of an earlier `ActionStep` with that `id` |
| `eval` | `fn(state, results?)` | Whatever the function returns |

## Setup workflows

`setupWorkflows` lists workflow ids that run before `steps`. They are looked
up in `RunOpts.workflowRegistry`. If one is missing or fails, the workflow
fails with that error. Setup workflows should be idempotent.

## `RunOpts`

| Option | Notes |
|---|---|
| `speed` | Divides every step `wait`. Default 1 |
| `modelOverride` | Runs the workflow on this model: dispatches `updateSettings({ aiModel })` first and replaces models pinned by `updateSettings` steps. Falls back to `globalThis.__SMARTCHATS_E2E_MODEL__` |
| `workflowRegistry` | Workflows by id, for `setupWorkflows` |

## Running workflows in the app

`createInsightStore` in `smartchats-common` compiles each workflow passed in
its `workflows` config and mounts them on `window.__<app>__.simi`, keyed by
workflow id:

```js
await window.__smartchats__.simi.workflows.basic_chat_flow();
window.__smartchats__.simi.list();   // every workflow id
```

The Playwright e2e specs in `apps/smartchats/tests/e2e/` call workflows the
same way. The app's workflows live in `apps/smartchats/app/simi/workflows/`
and are registered in `apps/smartchats/app/simi/index.ts`.

The runner emits `simi_workflow_start`, `simi_step`, `simi_resolve`,
`simi_capture`, `simi_model_override` and `simi_workflow_complete`. To pull
recent runs, use `bin/save_session --app smartchats --tag simi`.

## See also

- [smartchats-sessions](/docs/packages/smartchats-sessions) for analyzing the sessions a run produces
- Source: [`packages/simi/src/index.ts`](https://github.com/sheunaluko/smartchats/blob/main/packages/simi/src/index.ts)

---

Source: https://smartchats.ai/docs/contributing

# Contributing

**Licensing of contributions.** SmartChats is operated on an open core
model: this repository is open, and SmartChats Cloud is a closed product
built on it. The repository is licensed under Apache-2.0, and by
contributing you agree that your contribution is licensed under it too
(section 5) and that Sattvic Systems LLC may use it in SmartChats Cloud
and its other products. The full terms are in
[CONTRIBUTING.md](https://github.com/sheunaluko/smartchats/blob/main/CONTRIBUTING.md#licensing-of-contributions).

Questions? Join the [Discord](https://discord.gg/HYHwZwKqW2).

The developer guide is
[CONTRIBUTING.md on GitHub](https://github.com/sheunaluko/smartchats/blob/main/CONTRIBUTING.md).
It covers setup, the architecture, the verification workflow, the
conventions, the PR workflow and a section for AI coding agents.

In short:

1. **Clone, install, run `bin/preflight`.** It lints and builds the
   whole monorepo through turbo. If it passes, your environment works.
2. **Adding a query the agent can run** takes three files and about 20
   lines. See [Architecture](/docs/architecture#adding-a-query-the-agent-can-run).
3. **Run `bin/preflight` before opening a PR.** For UI changes, run
   `bin/devserve` (or `smartchats dev`) for hot reload and check the
   change in a browser.

## Running from source

The user install (`curl | bash` from [Quick Start](/docs/quickstart))
ships compiled binaries. For development, use the source:

```bash
git clone https://github.com/sheunaluko/smartchats.git
cd smartchats
npm install
```

Two ways to run the stack while you work:

```bash
# Hot-reload Next.js dev server + Express + SurrealDB container
bin/devserve

# Or through the CLI, which calls bin/devserve:
cd packages/smartchats-cli
npm link    # points the `smartchats` command at your source
smartchats dev
```

`bin/devserve` is the normal dev loop: hot reload and TypeScript watch,
with no container rebuild per change. `smartchats start` is the end-user
path, and is useful for checking production behavior.

> **Warning:** **Don't run `npm run dev` at the repo root.** Turbo would start every
>   package's dev script, including ones you don't need. The MCP server's
>   stdio script is named `dev:stdio` for this reason, so it no longer
>   starts a sign-in flow, but `bin/devserve` or `smartchats dev` still
>   start only the processes you need.

## Useful bin/ scripts

| Script | What it does |
|---|---|
| `bin/preflight` | Lint and workspace build (which type-checks). Run before every PR. |
| `bin/devserve` | Hot-reload dev stack: Next.js dev, the Express server and a SurrealDB container |
| `bin/aio` | Build and run the all-in-one Docker image from `Dockerfile.aio` |
| `bin/test-bun-deploy` | Smoke-test the no-Docker path: native SurrealDB plus the server under Bun |
| `bin/test-e2e` | Start the `bin/test-bun-deploy` stack, run the Simi e2e suite against it, then stop it |
| `bin/seaweedfs` | Run a local SeaweedFS S3 bucket for media storage. `bin/seaweedfs env` prints the `SMARTCHATS_S3_*` exports |
| `bin/release` | Prepare a release tag: bump the CLI version, commit, and create an annotated tag. Does not push |
| `bin/checkpoint "<msg>"` | `git add -A && git commit -m "checkpoint: <msg>"` |
| `bin/save_session` | Export session bundles from a local install for triage |

The layered test runner is `packages/smartchats-test` (levels L0 lint,
L1 build, L2 unit, L3 integration, L4 e2e). See
[Architecture: Verification](/docs/architecture#verification).

## Repo layout (high level)

```
smartchats/
├── apps/
│   ├── smartchats/                # Next.js app (chat UI, voice, KG), static SPA
│   └── site/                      # This docs site
├── packages/
│   ├── cortex/                    # Agent loop
│   ├── llm-service/               # Model + TTS providers, NDJSON streaming, turn verdict
│   ├── tivi/                      # Voice I/O hooks (VAD, speech recognition, TTS queue)
│   ├── vie/                       # Voice engine (turn-taking)
│   ├── smartchats-backend/        # Backend contract (types only)
│   ├── smartchats-backend-local/  # Local-target backend adapter
│   ├── smartchats-cloud-client/   # Client for the hosted cloud
│   ├── smartchats-cli/            # `smartchats` command (setup, start, etc.)
│   ├── smartchats-database/       # SurrealDB queries, operations, DataAPI factories
│   ├── smartchats-local-server/   # Express server: serves the SPA + /local-api/*
│   ├── smartchats-mcp/            # MCP server
│   ├── smartchats-mcp-bridge/     # MCP bridge from a coding agent to the relay
│   ├── smartchats-hub/            # Per-machine agent hub
│   ├── smartchats-test/           # Layered test runner
│   └── ...
├── bin/
│   ├── preflight                  # Lint + build
│   ├── devserve                   # Hot-reload dev stack
│   ├── aio                        # Docker all-in-one image
│   ├── test-e2e                   # e2e suite against a fresh stack
│   ├── checkpoint                 # `git add -A && git commit -m "checkpoint: …"`
│   └── ...
└── CONTRIBUTING.md                # The developer guide
```

## Architecture invariants

These matter. Discuss before changing them:

- **Single direct importer of `surrealdb`.** Only
  `packages/smartchats-database/src/client.ts` imports the npm package
  in runtime code. Everything else uses its `Client` interface.
- **One credential store.** `~/.smartchats-mcp/credentials.json` is
  shared by the CLI, MCP server, bridge and hub. Don't add another
  auth file.
- **The `DataAPI` contract is the read/write boundary.** Don't add
  another dispatcher layer.
- **Per-statement errors must surface.** Never return a silent `[]` on
  `status: ERR`.

See [Architecture: Open and closed](/docs/architecture#open-and-closed)
for the package-level split.

## For AI coding agents

Humans and AI coding agents build SmartChats together. The same rules
and review apply to both. What helps agent contributions land:

- **Read the existing code first.** Most patterns you need already
  exist. Search, then write. Nearly every "I need to add X" has one
  home.
- **Make small changes.** A feature usually touches one builder, one
  tool wrapper and one CLI command. A sprawling diff is a sign you are
  working against the structure.
- **Run `bin/preflight` before you call it done.** Don't rely on CI for
  your own change. For UI work, run `bin/devserve` or `smartchats dev`
  and check the change in a browser.
- **Confirm destructive actions with the user first:** deleting data,
  force-pushing, changing live cloud state, killing shared processes.
- **When tests fail, find the cause.** Read the log, reproduce it and
  fix the root cause. Don't write it off as a flaky environment.
- **Note architecture ideas instead of refactoring as you go.** Raise
  them in an issue or a discussion. See [Architecture](/docs/architecture)
  for the current design.
- **Match the surrounding code and docs** in tone, style and comment
  density.

> **Note:** **Re-read this section at the start of each session.** It is the
>   quickest way to load the project conventions.

## See also

- **Email (preferred for contribution proposals):** [shay@smartchats.ai](mailto:shay@smartchats.ai)
- **LinkedIn (maintainer):** [linkedin.com/in/sheun-aluko](https://www.linkedin.com/in/sheun-aluko/)
- **GitHub issues:** [github.com/sheunaluko/smartchats/issues](https://github.com/sheunaluko/smartchats/issues) for bugs, feature requests and questions
- **GitHub discussions:** [github.com/sheunaluko/smartchats/discussions](https://github.com/sheunaluko/smartchats/discussions) for design conversations
- [Architecture](/docs/architecture) and [CLI Reference](/docs/cli)

For larger architectural changes, please email before opening an issue
or PR.