I made a data-centre speech engine run on a desktop card, then taught it new voices without recompiling it
Qwen3-TTS is an open speech model that can speak in built-in voices, copy a voice from a short recording, or design one from a description. Its reference library ran at less than half of real time on my graphics card. The fast engine written for it ran only on data-centre GPUs, and only with its fixed speakers. These notes cover how I got that engine onto a desktop card, how a cloned voice gets in without rebuilding anything, and what I learned when the same voices started singing songs and narrating illustrated films.
- A local studio and OpenAI-compatible API for Qwen3-TTS, running under WSL2 on Windows and sharing its graphics cards with a language model and image generation.
- Four patches on a pinned commit move a CUDA-Graph serving engine from H100-only to a desktop RTX 3090.
- A cloned voice is written into spare rows of a table the captured GPU graph already reads, so it is usable at once with no re-capture.
- Every part of a take is transcribed and compared with its text while the next part is generated, and a part that says something else is taken again.
- The same voices sing songs, perform audio scenes with sound effects and narrate films, and two MCP servers let coding agents in other projects drive all of it.
01Why I built it
I wanted spoken audio I could make on my own machine: narration, character lines, short scenes, with voices I had cloned or designed myself. Qwen3-TTS could do all of that on paper. Through its reference Python library it ran at 0.45 times real time on an RTX 3090, so a ten-second line took more than twenty seconds. That is fine for a demo and useless for listening while you write.
A much faster serving engine existed for the same model. It records its GPU work once as CUDA Graphs, a fixed sequence of operations it replays for every request, and that is where its speed comes from. It was also built for H100 data-centre cards, refused to start on anything else, and only knew the model's built-in speakers.
So the first goal was narrow: that engine, on a desktop card, under WSL2, with my own voices, behind the same speech API that OpenAI clients already speak. The card also had to stay shared with a local language model and image generation, because that is what the rest of the machine does all day.
02Architecture at a glance
Everything installs into one folder inside WSL: a pinned Python, the engine at a pinned commit with its patches, the models and every cache. Deleting that folder uninstalls it. One Windows command installs what is missing, starts the studio and opens the browser, and closing that window stops everything the studio started.
The split exists because the pieces have very different lifetimes. The engines take a minute to capture their graphs and should survive a studio restart, so they run in their own sessions and the studio adopts them. The labs hold large models for short jobs, so they start when needed and leave when idle. The studio itself restarts in about five seconds, and the engines stay loaded while it does.
03How it grew
The project took 30 working days across ten weeks. September was the speech studio, October was almost entirely songs, productions and films (190 of the 239 commits), and November added a robot singer and better use of two cards.
Each new layer reused what the one below already did. Songs are sung by the voices the speech studio clones, productions cast through the same voice lab, and films are timed to a production's audio. That is why the speech core had to be dependable first: every later stage inherits its failures.
- 14 September. The contained WSL2 install, the patched engine and a web page, then cloned and designed voices and the OpenAI-compatible API the same day.
- 15 to 16 September. Tone and pause directions inside the text, emotion clips for cloned voices, the speech check and loudness leveling.
- 29 September. Engines that skip work they never read and keep a large text table out of graphics memory, with a benchmark to measure each change.
- 2 to 3 October. The music lab: songs sung in studio voices, harmonies, and a rebuild of the whole music path after a feature was reported fixed four times.
- 17 October. A second graphics card: speech on one, music on the other.
- 23 to 24 October. Voice casting from a description and audition lines, then productions: whole scenes with sound effects.
- 25 to 27 October. Pictures and films, layered shots that play as a moving comic, and short local video clips as layers.
- 29 October. The preflight that checks a film's storyboard before any GPU work.
- 17 to 19 November. A robot singer, short logo stings, and engines that spill to the free card.
04Four patches on a pinned engine
I did not fork the engine. The installer checks out a pinned upstream commit and applies four patch files, checking first whether each is already applied so a re-run changes nothing. A stamp of the patches and lockfiles decides whether anything needs doing at all, so a normal start skips the installer in about a second.
The first patch lifts the H100-only checks. The engine already had a bf16 path for cards without FP8, and the guards were what kept it off a 3090. The second moves the engine to newer PyTorch, Triton and FlashInfer releases with prebuilt kernels, so the first start compiled nothing. The third keeps a 622 MB text table in ordinary memory: each step reads only a few of its rows, and the graphics memory it frees goes to the other tenants of the card.
The fourth patch is the one worth explaining, and it gets its own section. Together they took a request from 0.45 times real time in the reference library to five and a half to seven times, with first audio 50 to 80 milliseconds after the engine starts. Those are the project's own README measurements on one RTX 3090.
05New voices without a new graph
A captured CUDA Graph is frozen. It replays the same operations on the same memory every time, which is why it is fast and why it cannot learn a new speaker. The obvious way to add a cloned voice is to change the model's inputs and capture again, which costs a restart and about a minute each time.
The way out came from reading how the two checkpoints differ. The built-in checkpoint looks a speaker id up in an embedding table. The checkpoint that clones takes a 2,048-number speaker vector in exactly the same position of the prompt. The captured graph reads that table in place, by address. So if a cloned voice's vector is written into a row of that table, and the row is given a speaker name, the graph picks it up on the next request without knowing anything changed.
The rows come from the top of the codec vocabulary, a range the model never samples and that holds no control or language ids in this checkpoint family. The studio registers voices into those rows when an engine starts, and evicts the least recently used one when they run out.
Cloning from reference audio and its transcript works the same way. Each reference frame is normally the sum of sixteen codebook embeddings. Those sums go into further spare rows, so the whole prompt is still a plain sequence of ids, which is the only thing the frozen graph knows how to read.
06What happens to one take
Text is split into parts wherever the tone changes, a pause is asked for, or an emotion clip takes over. Plain text is packed into chunks of up to 2,000 characters, so the delivery flows across sentences. Each part is one engine request. Audio goes to a recorder on disk before any listener hears it. A crash leaves a partial file that the next start repairs, and queued jobs resume on their own.
The model samples its output, and very occasionally it never says it has finished. During bring-up a ten-token sentence at seed 0 produced 22 seconds of gibberish, while seeds 1 to 7 were fine. Each segment now gets a budget of three times its estimated length, never less than eight seconds. A segment that runs past it is cut out of the recording, listeners are told to rewind, and it is tried again with a new seed, up to twice.
While the next part is being generated, Whisper transcribes the last one and the studio compares the words with the text. Four or more words that are not in the text, or 30 per cent wrong, means the voice said something else. That part and everything after it are rewound and taken again, and the closest take is kept. Each part is also levelled to the same perceived loudness and nudged toward the voice's normal tone, so a switch from calm to excited does not jump in volume.
07Sharing two graphics cards
Speech is a small tenant. The two engines together need about thirteen gigabytes, and the rest of the machine wants the same cards for a language model, image generation, song models and video. For a while every tenant decided for itself whether to be loaded, and the result was predictable: engines kept warm forever, a film's second drawer finding its card full, spilling into system memory and stalling.
Now one module says what each stage of a job needs, and entering a stage asks every other tenant to step aside through its own release hook. Every load first reserves its memory with a shared GPU arbiter, and the studio checks the claim on the card afterwards, because the arbiter once answered that a load fit while the card was short. Every decision is written to a log, with who held each megabyte.
In November the last rigid piece went. A speech engine has a home card, and if that card is full it may use another card that has the room as it stands, without stopping anything there. That came from a day when a language model filled one card and every speech request was refused while the other card had 22 GB free.
08The same voices sing
Once voices existed, songs were the natural next question. A song starts in ACE-Step, an open music model that produces the backing and a generated singer together. The vocal is split from the backing, re-sung in a cloned or designed voice by Seed-VC, a voice-conversion model, and mixed back. Every take is kept and scored by how closely it matches the voice.
Harmonies were the hard part. They are planned from the converted lead, note by note, in the key the backing actually plays rather than the key it was asked for. A plan check then looks for clashes, unisons, big leaps and notes that drop out on held words, and repairs them before anyone sings.
Both choices came from songs that went wrong. On one, the original singer's vocal had no pitch on a held syllable. Lines planned from it cut out and jumped three octaves, while lines planned from the converted lead held a steady fifth and third. The key mattered too: the music model does not always keep the key it is asked for, and one song asked for in F minor played all three choruses in B-flat minor.
In November I added a robot singer that sings in machine time, with glitches that stay in key and a small director that places them phrase by phrase. It came with a check that measures each singer's own track against the key, so a sour held note is found by measurement rather than by ear.
09Scenes and films
A production is a whole audio scene from one request. Each role reuses a voice cast before or casts a new one: four designed candidates read the character's own audition line, the best is measured and accepted, and its emotion clips are made. Sound effects are searched in a free (CC0) library by sound and description together, or generated. The scene is performed, effects drop under speech, and the mix is mastered to a set loudness.
A film adds pictures. A local language model directs shots along the audio timeline: framing, camera move, how each shot arrives, and the two or three things it must show. An image model draws each shot, and a vision model answers one yes or no question per must-see item. Characters are drawn alone and cut out over a background drawn once per place, so the film plays as a moving comic, and ffmpeg edits it with the audio and captions.
Films were where most of the hours went. The pipeline is long, every stage uses a model that can be confidently wrong, and a fault often shows only after many shots are drawn. Most of the lessons below come from here.
10Built to be called
I use the studio from a browser, but most of its work is requested by other programs. The public API speaks the OpenAI speech format, so any OpenAI client works by changing its base URL, and it serves six audio formats, streaming as the audio is made. It also takes async jobs, batches, voice cloning and design.
Two MCP servers put the studio in a coding agent's tool list: 42 tools for voices, scenes, sound effects and films, and 19 for songs. A game project can cast a character and voice its lines without knowing anything about engines or cards. Long jobs return an id to follow rather than holding a call open, because a scene or a film can take many minutes.
11Testing without a GPU
The tests run against a stand-in engine that speaks the real engine's HTTP and WebSocket protocol and plays a tone as long as the text. Tags in the text make it fail on request: refuse before streaming, return 503 once, drop the connection, hang after one chunk, or never stop at a chosen seed. That is how the runaway guard, the retries and the reattach logic are tested without a card.
Counting def test_ lines gives 1,124 across 119 files. The studio's suite takes over eleven minutes on one machine, so it runs across a small fleet of machines in about two minutes. A page self-test drives the real page in a browser harness, and a new scenario has to fail on the old page before a page fix counts.
Unit tests on fakes are not the end of it. Real runs kept finding what fakes could not: a lab started three times, productions ten to twelve decibels too quiet, a hung test. So every stage writes its timings to a log, and a pipeline report shows where a real run spent its time.
12Where it stands
The speech studio is the stable core. Songs, productions and films work end to end and are still changing, with each run adding a check for whatever it got wrong.
Some limits remain. Expressive mode follows any instruction but sounds less natural with a cloned voice. Cloning from reference audio is still a beta. The music lab depends on Seed-VC, which is GPL-3.0, so it is installed separately and runs as its own process. No licence has been chosen for the project's own code yet.
13Ten lessons
Each of these is written down in the project's own lessons file or a commit message, with the measurement that exposed it. Most share a shape: a check that passed while the thing it stood for was wrong.
01Fixed four times, broken the whole time
A song-writing button failed for a whole day while it was reported fixed four times. The real cause was a 422: three singers were picked and the request allowed two. A status answer arriving after the refusal reopened the progress box, frozen at zero, so it looked busy. The music journal had no entry at all, because the request never got in. An audit of the music path then found 35 more of the same kinds.
Now: Every limit lives in one module that the API, the page and the tests read, and a page fix must come with a scenario that fails on the old page.
never call it fixed until the exact failing case runs and passes in front of you.
02The film passed while its picture froze
A film's final check read the file's length, which was the audio's length, and passed. The picture had frozen at 0:34 and stayed frozen to the end.
Now: The check measures the picture track's own length against the audio, and samples frames across the film to prove they differ.
check the thing itself, never a number that usually travels with it.
03A pipe nobody read
One sound-effect generator took more than fifteen minutes for a clip. Its output went to a pipe that nothing read, and once the pipe's 64 KB buffer filled, the process blocked and waited forever.
Now: Every generator writes to a log file of its own run, and that generator takes 46 to 65 seconds a clip.
a slow process may be a stuck one; look at what it is waiting on before tuning it.
04Busy is not dead
The studio stopped itself at the end of two films in a row. The last step copied about 170 MB to a Windows folder at about 10 MB a second, on the event loop, so the health check went silent and the session guard took that for a crash. My first explanation blamed something else entirely, and it was wrong; the guard's own log said the studio had stopped answering.
Now: The copy runs in a thread, and the guard stops things only on a refused port or a five-minute hang.
read the log of the thing that acted before naming a cause.
05The arbiter said it fits
Locking new voices to their timbre waited more than four minutes, twice, while the GPU arbiter kept answering that the work fit. The arbiter counted claims, not memory, and could not see about 3.8 GB that the music lab held without a claim.
Now: Every claim is checked on the card after the answer, and topped up by the difference between what the arbiter believes and what is really free.
a scheduler's view of a resource is a model; measure the resource.
06One machine, one budget
The whole machine froze during a film. Memory measured inside WSL said 70 GB was free while Windows was paging at 4,235 pages a second. A frozen run also left an empty record behind, which then stopped the next film.
Now: Heavy stages wait for room on both sides, keeping 16 GB for the desktop, WSL is capped at 64 GB, and every record is written atomically.
a virtual machine's free memory is not the host's free memory.
07The negative prompt nobody read
Backgrounds for a space film kept coming out as desert. The image model reads only 77 tokens of a negative prompt and silently drops the rest, and each picture's own terms came after a long shared list, so words like ground and desert were never read. Six of ten seeds drew desert in that order, and none or one of ten with the picture's own terms first. A two-seed proof with a short negative had passed before the film still failed.
Now: Each picture's own terms come first, and a drawing fix is proved on the film's real prompts with ten or more seeds.
prove a fix on the real inputs, at a sample size that could fail.
08Told once, checked never
One film of nearly twelve minutes lost about six hours to reruns, each fault found only after twenty shots were drawn. The language model had been told what not to do and did it anyway: asked for an empty place, it wrote "no creature", and a negation in an image prompt draws exactly what it names.
Now: A preflight reads the storyboard in about 40 ms, with no GPU, against every rule that ever broke a film, and a new fault class adds a check and a test.
a rule given to a model has to be checked in code as well.
09Targets set by hope
Casting stopped on a character because no candidate was fast enough. The "fast" pace band wanted 3.3 to 4.8 words a second, and the voice designer's fastest read was 2.61. Child voices failed the same way against pitch bands set for adults.
Now: Bands come from measured reads, per kind of speaker, and a production accepts the best candidate and reports what it missed.
a threshold written before measuring is a guess, and it fails the first real case.
10Effects nobody heard
A dragon, a wizard and an orc were given voice effects at 0.35 to 0.45 of their strength. For the dragon that was less than a semitone of change, and I heard three plain voices. The whole scene had to be performed again and the film's pictures re-timed onto the new audio.
Now: Effects default to their designed strength, a role without one gets it from its look and brief, and the spec check warns under 0.6.
sample ten seconds before spending ten minutes.
14What I would do differently
Most of the cost in this project was rework, not compute. With hindsight, five habits would have saved most of it.
- Put every limit in one place on day one. The song-writing failure was a limit known to the API and not to the page. One shared module would have made that bug impossible rather than findable.
- Schedule the cards as one plan from the start. Letting each tenant decide when to load produced weeks of one-off waits and leases. One module that says what each stage needs replaced all of them.
- Check the artefact, not a proxy. A file length, a claim count and a self-consistent record each passed while the real thing was wrong.
- Sample before spending. A ten-second listen to each voice effect and a six-clip stretch before a forty-clip film found every problem the full runs found, in a fraction of the time.
- Write every intermediate to disk first. A review error once lost twelve drawn shots that lived only in memory. Keeping each candidate on disk before it is judged makes every stage resumable.