Automated Video Generator

Local Voice Cloning & Realistic TTS — Full Documentation

This document is the authoritative reference for adding realistic, human-like, cloned voices to the Automated Video Generator. Every claim here was verified against the projects’ actual GitHub/HuggingFace license files and model weights (licenses, sizes, and maintenance status are real, not marketing copy).

1. Why this matters for the project

The generator’s default voice engine is Edge-TTS (Microsoft cloud, free but remote, no cloning, slightly robotic). To meet the project’s zero-cost, self-hosted, privacy-first, and realistic human-voice cloning goals, we use local-first TTS engines that:

2. The three providers already wired into the code

The pipeline already supports these via TTS_PROVIDER in .env. The wiring lives in:

Provider key Engine Default port Clones? Code status
voicebox jamiepine/Voicebox (multi-engine) — vendored in-repo at src/speech/ 17493 Yes (zero-shot) ✅ Default & recommended
xtts Coqui XTTS API server 8020 Yes (3s clip) ⚠️ See §6 caveat
openai-local Kokoro (OpenAI-compatible server) 8880 No (presets) ✅ Default narrator

2.1 Voicebox (default & recommended clone engine — vendored in-repo)

Voicebox is now vendored directly in the repo at src/speech/ (MIT license, clean copy of jamiepine/voicebox, commit 52f8d8d, 2026-07-20). No separate clone or external install is needed.

2.2 Kokoro (default narrator — openai-local)

2.3 XTTS (Coqui) — ⚠️ documented but NOT recommended for new builds

3. Full open-source voice-model comparison (license-verified)

All sizes and licenses below were pulled live from GitHub / HuggingFace.

# Model License Size Clones? Langs Quality Maintained? RAM fit (6 GB box) Interface
1 Kokoro Apache-2.0 312 MB ❌ presets 8 Good ✅ ✅ Comfortable Lib / ONNX / HTTP
2 F5-TTS MIT 1.3 GB ✅ 10s ref EN (+forks) High ✅ ⚠️ on-demand Lib / CLI
3 Qwen3-TTS (0.6B) Apache-2.0 1.7 GB ✅ + delivery 10 High ✅ ⚠️ on-demand HF / server
4 Chatterbox MIT ~1.8 GB ✅ few-sec 23 High ✅ ⚠️ on-demand + HF gated (login) HTTP API / MCP
5 GPT-SoVITS MIT ~2–3 GB ✅ 1–2s few-shot 50+ Very high ✅ ❌ heavy GUI / API
6 Voicebox (jamiepine) MIT bundles 7 ✅ few-sec 23 High (best-of-7) ✅ ⚠️ run light engine REST + MCP + headless
7 OpenVoice MIT ~1 GB ✅ EN/JP/KR/ZH Good ❌ stale Apr 2025 ⚠️ edge but dead Lib
8 RVC MIT ~1 GB ❌ conversion only many High (VC) ✅ ⚠️ wrong job GUI / API
9 VibeVoice-ASR MIT 15.9 GB n/a (transcribe) 50+ High (ASR) ✅ ❌ impossible here HF / server
10 VibeVoice-Realtime-0.5B MIT 1.9 GB ❌ fixed voices EN + 9 exp. Good ✅ ⚠️ edge, no clone HF
11 Coqui XTTS MPL-2.0 ~1.5 GB ✅ 16 Good ❌ archived 2y ⚠️ Lib
12 Fish Speech Non-commercial ❌ ~1 GB ✅ many High ✅ ⚠️ —
13 OmniVoice Studio AGPL ❌ — ✅ — Good ✅ ⚠️ GUI
14 ElevenLabs / Azure Proprietary/paid ❌ cloud ✅ many High — n/a Cloud API

License verdict (hard rule: MIT / Apache only)

Capability notes

4. RAM-fit strategy for the low-spec laptop

This dev machine: Total ≈ 5.86 GB, Free ≈ 390 MB, 280+ processes. Model size is the deciding filter.

Run jamiepine/voicebox as a headless local voice microservice, with Kokoro as the default narrator and a clone engine (Chatterbox Turbo / Qwen3-TTS) loaded on demand. All MIT/Apache, fully local, MCP + REST ready.

This is the cleanest agentic integration: one server, one REST/MCP surface, no per-model plumbing. Your orchestrate.ts simply calls POST /speak with a cloned profile.

Standalone fallback (if you skip the heavy Voicebox app):

Summary recommendation:

6. Rejected / do-not-use (with reasons)

7. Code structure (current implementation)

8. Setup & verification

This manual clone is **no longer needed** — Voicebox is vendored at
`src/speech/` in this repo. For reference, the upstream setup was:

```bash
cd src
../venv/Scripts/python.exe -m speech.main --host 127.0.0.1 --port 17493

in another terminal, generate with a cloned profile via REST

curl -X POST http://127.0.0.1:17493/generate
-H “Content-Type: application/json”
-d ‘{“text”:”Hello world”,”profile_id”:”","language":"en"}'


MCP: Voicebox ships a built-in MCP server (`backend/mcp_server/`) — register it
with any MCP-aware agent (Claude Code, Cursor, Cline) to let the agent speak in
your cloned voice.

### 8.2 Kokoro (default narrator)

```bash
docker run -p 8880:8880 ghcr.io/remsky/kokoro-fastapi:latest
# set .env: TTS_PROVIDER=openai-local, OPENAI_LOCAL_TTS_URL=http://localhost:8880/v1

8.3 Project wiring check

# unit tests (no live API)
npm run test:unit

# generate voices only (no render) — audio lands in workspace/staging/<job_id>/audio/
npm run generate -- --segment

9. Decision checklist (pick your stack)

10. Voicebox headless integration (implemented)

Voicebox runs as a separate, lifecycle-managed headless backend (its own Python process), NOT embedded in the Node pipeline. The pipeline owns its RAM lifecycle so no TTS engine stays resident during Remotion render / asset-fetch.

10.1 Install — in-repo vendored Voicebox

The Voicebox backend is vendored at src/speech/ in this repo. No separate clone needed. Just set up the project’s Python venv:

# From the project root
uv venv --python 3.11 venv

# Install CUDA pytorch for GPU acceleration
uv pip install --python venv/Scripts/python.exe \
  --index-url https://download.pytorch.org/whl/cu126 \
  "torch==2.13.0+cu126" "torchaudio==2.11.0+cu126"

# Install the Voicebox backend deps
uv pip install --python venv/Scripts/python.exe \
  -r src/speech/requirements.txt

CPU only: If you don’t have an NVIDIA GPU, use --extra-index-url https://download.pytorch.org/whl/cpu instead of the --index-url above. –index-url https://download.pytorch.org/whl/cu126 “torch==2.13.0+cu126” “torchaudio==2.11.0+cu126”

Verify: .venv/Scripts/python.exe -c “import torch; print(torch.cuda.is_available())” -> True

```

Why GPU matters here: The /models/load endpoint is Qwen-only (it calls load_model_async → qwen-tts-{size}). Kokoro and the clone engines load lazily on first /speak via get_tts_backend_for_engine. Kokoro-82M (hexgrad/Kokoro-82M, ~350 MB) is the lightest and the recommended narration engine; it loads into VRAM on a CUDA box. Qwen 1.7B (3.6 GB) needs GPU VRAM — it will NOT fit a 4 GB card alongside the OS, so use Kokoro for narration.

10.2 Run headless (no GUI = saves RAM)

cd src
../venv/Scripts/python.exe -m speech.main \
  --host 127.0.0.1 --port 17493 --data-dir ../workspace/cache/voicebox

src/lib/voicebox-lifecycle.ts ensureBackend() spawns exactly this command (with PYTHONPATH= cleared) if the port isn’t answering — so the pipeline can own the full lifecycle.

10.3 Synthesis flow (verified end-to-end)

Voicebox is profile-based: every generation needs a voice profile. Create a Kokoro PRESET profile (no reference audio needed) once:

curl -X POST http://127.0.0.1:17493/profiles -H "Content-Type: application/json" \
  -d '{"name":"Narrator (Kokoro Heart)","voice_type":"preset",
       "preset_engine":"kokoro","preset_voice_id":"af_heart","default_engine":"kokoro"}'
# -> {"id":"<PROFILE_ID>", ...}   (set VOICEBOX_PROFILE_ID=<PROFILE_ID>)

Then the pipeline drives it (see src/lib/api-tts-provider.ts):

  1. POST /speak { text, profile, engine:"kokoro", language } → { id, status:"generating" }
  2. poll GET /generate/{id}/status (SSE stream) until status:"completed"
  3. GET /audio/{id} → WAV (24 kHz mono PCM) written to the job’s audio dir

No manual /models/load is needed — the engine loads lazily into VRAM on the first /speak. This was proven on this box: a 5.3 s WAV synthesized via Kokoro-82M on the RTX 3050 (819 MB VRAM used, system RAM untouched).

10.4 Env vars

| Var | Default | Purpose | | :— | :— | :— | | TTS_PROVIDER | voicebox | set voicebox (default) or edge-tts | | VOICEBOX_API_URL | http://127.0.0.1:17493 | backend base URL | | VOICEBOX_BACKEND_DIR | src/ (vendored speech/ package) | project root dir (for spawn) | | VOICEBOX_PYTHON | <dir>/.venv/Scripts/python.exe | interpreter | | VOICEBOX_ENGINE | kokoro | engine for narration (chatterbox-turbo for clone) | | VOICEBOX_PROFILE_ID | — | required — the Kokoro preset or cloned profile | | VOICEBOX_API_URL is also auto-set to ws:///HTTP by lifecycle if spawned |

10.5 RAM/VRAM budget on this box (RTX 3050 4 GB VRAM)

10.6 Lifecycle (managed automatically by the pipeline)

src/lib/speech-backend.ts:

src/lib/api-tts-provider.ts generateVoiceoverWithVoicebox():

10.7 Clone YOUR real voice (end-to-end, verified on this box)

Voicebox is profile-based: narration runs through a voice profile. For a cloned voice you register a cloned profile, give it a ~10-30s reference clip of your real voice, then point the pipeline at that profile. Verified working on the RTX 3050: a Chatterbox-Turbo clone generated a 4.96s clip in your (placeholder) voice using ~3.8 GB VRAM.

One-time registration (backend must be running, see §10.2):

# 1. Create a CLONED profile backed by chatterbox-turbo (MIT, ~4GB VRAM footprint)
curl -X POST http://127.0.0.1:17493/profiles -H "Content-Type: application/json" \
  -d '{"name":"My Real Voice","voice_type":"cloned","default_engine":"chatterbox_turbo"}'
# -> {"id":"<CLONE_PROFILE_ID>", ...}

# 2. Upload a clean 10-30s clip of YOUR voice + verbatim transcript
curl -X POST http://127.0.0.1:17493/profiles/<CLONE_PROFILE_ID>/samples \
  -F "file=@C:/path/to/your-voice.wav" \
  -F "reference_text=the exact words you spoke in the clip"

Or use the bundled setup script (does both steps + writes .env):

node scripts/setup-voicebox-clone.mjs C:/path/to/your-voice.wav "verbatim transcript of the clip"

Then .env already has (set during integration):

TTS_PROVIDER=voicebox
VOICEBOX_API_URL=http://127.0.0.1:17493
VOICEBOX_ENGINE=chatterbox_turbo
VOICEBOX_PROFILE_ID=<your cloned profile id>

The agentic pipeline (src/lib/voice-generator.ts → generateVoiceoverWithVoicebox) now narrates EVERY scene in your cloned voice. Each scene: POST /speak {text, profile, engine:”chatterbox_turbo”} → poll status → GET /audio/{id} → WAV.

VRAM note: Chatterbox-Turbo is ~4 GB; on the 4 GB RTX 3050 it loads at ~3.8 GB (OS + CUDA overhead leaves little headroom). It fits, but do NOT also run the Remotion GPU render on the dGPU simultaneously — let Voicebox use the dGPU, Remotion uses the iGPU. If you hit CUDA OOM, switch VOICEBOX_ENGINE to chatterbox (multilingual, ~3.2 GB) or use the lighter Kokoro narrator (VOICEBOX_ENGINE=kokoro + a Kokoro preset profile — no cloning, ~0.8 GB VRAM).

This site is open source. Improve this page.