Qwen3-TTS Studio · engineering notes

I made a data-centre speech engine run on a desktop card, then taught it new voices without recompiling it

Qwen3-TTS is an open speech model that can speak in built-in voices, copy a voice from a short recording, or design one from a description. Its reference library ran at less than half of real time on my graphics card. The fast engine written for it ran only on data-centre GPUs, and only with its fixed speakers. These notes cover how I got that engine onto a desktop card, how a cloned voice gets in without rebuilding anything, and what I learned when the same voices started singing songs and narrating illustrated films.

The short version
239commits, 14 September to 19 November 2026
5.5 to 7xreal time per request, the project's own README measurement
0.45xreal time for the reference library on the same card, same README
4patches on a pinned engine commit
61MCP tools across two servers
1,124def test_ lines counted across 119 test files

01Why I built it

I wanted spoken audio I could make on my own machine: narration, character lines, short scenes, with voices I had cloned or designed myself. Qwen3-TTS could do all of that on paper. Through its reference Python library it ran at 0.45 times real time on an RTX 3090, so a ten-second line took more than twenty seconds. That is fine for a demo and useless for listening while you write.

A much faster serving engine existed for the same model. It records its GPU work once as CUDA Graphs, a fixed sequence of operations it replays for every request, and that is where its speed comes from. It was also built for H100 data-centre cards, refused to start on anything else, and only knew the model's built-in speakers.

So the first goal was narrow: that engine, on a desktop card, under WSL2, with my own voices, behind the same speech API that OpenAI clients already speak. The card also had to stay shared with a local language model and image generation, because that is what the rest of the machine does all day.

02Architecture at a glance

Everything installs into one folder inside WSL: a pinned Python, the engine at a pinned commit with its patches, the models and every cache. Deleting that folder uninstalls it. One Windows command installs what is missing, starts the studio and opens the browser, and closing that window stops everything the studio started.

Patched engineThe CUDA-Graph serving engine, run twice: one copy on the built-in checkpoint, one on the checkpoint that clones voices.
StudioA FastAPI app with a plain JavaScript page. It schedules requests onto engine slots, records audio, keeps the library in SQLite and serves the OpenAI-compatible API.
Voice labA separate service started on demand for voice creation: quality checks on a recording, speaker embeddings, designed voices and the Whisper speech check.
Music labACE-Step for the song and a generated singer, a vocal split, and Seed-VC to re-sing the vocal in a studio voice.
Productions and filmsCasting, sound effects, a performed scene, then shots directed by a language model, drawn, reviewed and edited into a video.
GPU placementEvery engine and worker claims its memory from a shared GPU arbiter before loading, and moves to another card when its own is full.
MCP serversTwo servers, 42 tools for voices, scenes and films and 19 for songs, so an agent in another project can use the studio directly.

The split exists because the pieces have very different lifetimes. The engines take a minute to capture their graphs and should survive a studio restart, so they run in their own sessions and the studio adopts them. The labs hold large models for short jobs, so they start when needed and leave when idle. The studio itself restarts in about five seconds, and the engines stay loaded while it does.

03How it grew

The project took 30 working days across ten weeks. September was the speech studio, October was almost entirely songs, productions and films (190 of the 239 commits), and November added a robot singer and better use of two cards.

Each new layer reused what the one below already did. Songs are sung by the voices the speech studio clones, productions cast through the same voice lab, and films are timed to a production's audio. That is why the speech core had to be dependable first: every later stage inherits its failures.

04Four patches on a pinned engine

I did not fork the engine. The installer checks out a pinned upstream commit and applies four patch files, checking first whether each is already applied so a re-run changes nothing. A stamp of the patches and lockfiles decides whether anything needs doing at all, so a normal start skips the installer in about a second.

The first patch lifts the H100-only checks. The engine already had a bf16 path for cards without FP8, and the guards were what kept it off a 3090. The second moves the engine to newer PyTorch, Triton and FlashInfer releases with prebuilt kernels, so the first start compiled nothing. The third keeps a 622 MB text table in ordinary memory: each step reads only a few of its rows, and the graphics memory it frees goes to the other tenants of the card.

The fourth patch is the one worth explaining, and it gets its own section. Together they took a request from 0.45 times real time in the reference library to five and a half to seven times, with first audio 50 to 80 milliseconds after the engine starts. Those are the project's own README measurements on one RTX 3090.

05New voices without a new graph

A captured CUDA Graph is frozen. It replays the same operations on the same memory every time, which is why it is fast and why it cannot learn a new speaker. The obvious way to add a cloned voice is to change the model's inputs and capture again, which costs a restart and about a minute each time.

The way out came from reading how the two checkpoints differ. The built-in checkpoint looks a speaker id up in an embedding table. The checkpoint that clones takes a 2,048-number speaker vector in exactly the same position of the prompt. The captured graph reads that table in place, by address. So if a cloned voice's vector is written into a row of that table, and the row is given a speaker name, the graph picks it up on the next request without knowing anything changed.

The rows come from the top of the codec vocabulary, a range the model never samples and that holds no control or language ids in this checkpoint family. The studio registers voices into those rows when an engine starts, and evicts the least recently used one when they run out.

Cloning from reference audio and its transcript works the same way. Each reference frame is normally the sum of sixteen codebook embeddings. Those sums go into further spare rows, so the whole prompt is still a plain sequence of ids, which is the only thing the frozen graph knows how to read.

Read the data path, not just the API. The engine had no notion of adding a speaker. The fix was visible only by following where the speaker id ends up inside the captured graph.

06What happens to one take

Text is split into parts wherever the tone changes, a pause is asked for, or an emotion clip takes over. Plain text is packed into chunks of up to 2,000 characters, so the delivery flows across sentences. Each part is one engine request. Audio goes to a recorder on disk before any listener hears it. A crash leaves a partial file that the next start repairs, and queued jobs resume on their own.

The model samples its output, and very occasionally it never says it has finished. During bring-up a ten-token sentence at seed 0 produced 22 seconds of gibberish, while seeds 1 to 7 were fine. Each segment now gets a budget of three times its estimated length, never less than eight seconds. A segment that runs past it is cut out of the recording, listeners are told to rewind, and it is tried again with a new seed, up to twice.

While the next part is being generated, Whisper transcribes the last one and the studio compares the words with the text. Four or more words that are not in the text, or 30 per cent wrong, means the voice said something else. That part and everything after it are rewound and taken again, and the closest take is kept. Each part is also levelled to the same perceived loudness and nudged toward the voice's normal tone, so a switch from calm to excited does not jump in volume.

07Sharing two graphics cards

Speech is a small tenant. The two engines together need about thirteen gigabytes, and the rest of the machine wants the same cards for a language model, image generation, song models and video. For a while every tenant decided for itself whether to be loaded, and the result was predictable: engines kept warm forever, a film's second drawer finding its card full, spilling into system memory and stalling.

Now one module says what each stage of a job needs, and entering a stage asks every other tenant to step aside through its own release hook. Every load first reserves its memory with a shared GPU arbiter, and the studio checks the claim on the card afterwards, because the arbiter once answered that a load fit while the card was short. Every decision is written to a log, with who held each megabyte.

In November the last rigid piece went. A speech engine has a home card, and if that card is full it may use another card that has the room as it stands, without stopping anything there. That came from a day when a language model filled one card and every speech request was refused while the other card had 22 GB free.

08The same voices sing

Once voices existed, songs were the natural next question. A song starts in ACE-Step, an open music model that produces the backing and a generated singer together. The vocal is split from the backing, re-sung in a cloned or designed voice by Seed-VC, a voice-conversion model, and mixed back. Every take is kept and scored by how closely it matches the voice.

Harmonies were the hard part. They are planned from the converted lead, note by note, in the key the backing actually plays rather than the key it was asked for. A plan check then looks for clashes, unisons, big leaps and notes that drop out on held words, and repairs them before anyone sings.

Both choices came from songs that went wrong. On one, the original singer's vocal had no pitch on a held syllable. Lines planned from it cut out and jumped three octaves, while lines planned from the converted lead held a steady fifth and third. The key mattered too: the music model does not always keep the key it is asked for, and one song asked for in F minor played all three choruses in B-flat minor.

In November I added a robot singer that sings in machine time, with glitches that stay in key and a small director that places them phrase by phrase. It came with a check that measures each singer's own track against the key, so a sour held note is found by measurement rather than by ear.

09Scenes and films

A production is a whole audio scene from one request. Each role reuses a voice cast before or casts a new one: four designed candidates read the character's own audition line, the best is measured and accepted, and its emotion clips are made. Sound effects are searched in a free (CC0) library by sound and description together, or generated. The scene is performed, effects drop under speech, and the mix is mastered to a set loudness.

A film adds pictures. A local language model directs shots along the audio timeline: framing, camera move, how each shot arrives, and the two or three things it must show. An image model draws each shot, and a vision model answers one yes or no question per must-see item. Characters are drawn alone and cut out over a background drawn once per place, so the film plays as a moving comic, and ffmpeg edits it with the audio and captions.

Films were where most of the hours went. The pipeline is long, every stage uses a model that can be confidently wrong, and a fault often shows only after many shots are drawn. Most of the lessons below come from here.

10Built to be called

I use the studio from a browser, but most of its work is requested by other programs. The public API speaks the OpenAI speech format, so any OpenAI client works by changing its base URL, and it serves six audio formats, streaming as the audio is made. It also takes async jobs, batches, voice cloning and design.

Two MCP servers put the studio in a coding agent's tool list: 42 tools for voices, scenes, sound effects and films, and 19 for songs. A game project can cast a character and voice its lines without knowing anything about engines or cards. Long jobs return an id to follow rather than holding a call open, because a scene or a film can take many minutes.

11Testing without a GPU

The tests run against a stand-in engine that speaks the real engine's HTTP and WebSocket protocol and plays a tone as long as the text. Tags in the text make it fail on request: refuse before streaming, return 503 once, drop the connection, hang after one chunk, or never stop at a chosen seed. That is how the runaway guard, the retries and the reattach logic are tested without a card.

Counting def test_ lines gives 1,124 across 119 files. The studio's suite takes over eleven minutes on one machine, so it runs across a small fleet of machines in about two minutes. A page self-test drives the real page in a browser harness, and a new scenario has to fail on the old page before a page fix counts.

Unit tests on fakes are not the end of it. Real runs kept finding what fakes could not: a lab started three times, productions ten to twelve decibels too quiet, a hung test. So every stage writes its timings to a log, and a pipeline report shows where a real run spent its time.

12Where it stands

The speech studio is the stable core. Songs, productions and films work end to end and are still changing, with each run adding a check for whatever it got wrong.

Some limits remain. Expressive mode follows any instruction but sounds less natural with a cloned voice. Cloning from reference audio is still a beta. The music lab depends on Seed-VC, which is GPL-3.0, so it is installed separately and runs as its own process. No licence has been chosen for the project's own code yet.

13Ten lessons

Each of these is written down in the project's own lessons file or a commit message, with the measurement that exposed it. Most share a shape: a check that passed while the thing it stood for was wrong.

01Fixed four times, broken the whole time

A song-writing button failed for a whole day while it was reported fixed four times. The real cause was a 422: three singers were picked and the request allowed two. A status answer arriving after the refusal reopened the progress box, frozen at zero, so it looked busy. The music journal had no entry at all, because the request never got in. An audit of the music path then found 35 more of the same kinds.

Now: Every limit lives in one module that the API, the page and the tests read, and a page fix must come with a scenario that fails on the old page.

never call it fixed until the exact failing case runs and passes in front of you.

02The film passed while its picture froze

A film's final check read the file's length, which was the audio's length, and passed. The picture had frozen at 0:34 and stayed frozen to the end.

Now: The check measures the picture track's own length against the audio, and samples frames across the film to prove they differ.

check the thing itself, never a number that usually travels with it.

03A pipe nobody read

One sound-effect generator took more than fifteen minutes for a clip. Its output went to a pipe that nothing read, and once the pipe's 64 KB buffer filled, the process blocked and waited forever.

Now: Every generator writes to a log file of its own run, and that generator takes 46 to 65 seconds a clip.

a slow process may be a stuck one; look at what it is waiting on before tuning it.

04Busy is not dead

The studio stopped itself at the end of two films in a row. The last step copied about 170 MB to a Windows folder at about 10 MB a second, on the event loop, so the health check went silent and the session guard took that for a crash. My first explanation blamed something else entirely, and it was wrong; the guard's own log said the studio had stopped answering.

Now: The copy runs in a thread, and the guard stops things only on a refused port or a five-minute hang.

read the log of the thing that acted before naming a cause.

05The arbiter said it fits

Locking new voices to their timbre waited more than four minutes, twice, while the GPU arbiter kept answering that the work fit. The arbiter counted claims, not memory, and could not see about 3.8 GB that the music lab held without a claim.

Now: Every claim is checked on the card after the answer, and topped up by the difference between what the arbiter believes and what is really free.

a scheduler's view of a resource is a model; measure the resource.

06One machine, one budget

The whole machine froze during a film. Memory measured inside WSL said 70 GB was free while Windows was paging at 4,235 pages a second. A frozen run also left an empty record behind, which then stopped the next film.

Now: Heavy stages wait for room on both sides, keeping 16 GB for the desktop, WSL is capped at 64 GB, and every record is written atomically.

a virtual machine's free memory is not the host's free memory.

07The negative prompt nobody read

Backgrounds for a space film kept coming out as desert. The image model reads only 77 tokens of a negative prompt and silently drops the rest, and each picture's own terms came after a long shared list, so words like ground and desert were never read. Six of ten seeds drew desert in that order, and none or one of ten with the picture's own terms first. A two-seed proof with a short negative had passed before the film still failed.

Now: Each picture's own terms come first, and a drawing fix is proved on the film's real prompts with ten or more seeds.

prove a fix on the real inputs, at a sample size that could fail.

08Told once, checked never

One film of nearly twelve minutes lost about six hours to reruns, each fault found only after twenty shots were drawn. The language model had been told what not to do and did it anyway: asked for an empty place, it wrote "no creature", and a negation in an image prompt draws exactly what it names.

Now: A preflight reads the storyboard in about 40 ms, with no GPU, against every rule that ever broke a film, and a new fault class adds a check and a test.

a rule given to a model has to be checked in code as well.

09Targets set by hope

Casting stopped on a character because no candidate was fast enough. The "fast" pace band wanted 3.3 to 4.8 words a second, and the voice designer's fastest read was 2.61. Child voices failed the same way against pitch bands set for adults.

Now: Bands come from measured reads, per kind of speaker, and a production accepts the best candidate and reports what it missed.

a threshold written before measuring is a guess, and it fails the first real case.

10Effects nobody heard

A dragon, a wizard and an orc were given voice effects at 0.35 to 0.45 of their strength. For the dragon that was less than a semitone of change, and I heard three plain voices. The whole scene had to be performed again and the film's pictures re-timed onto the new audio.

Now: Effects default to their designed strength, a role without one gets it from its look and brief, and the spec check warns under 0.6.

sample ten seconds before spending ten minutes.

14What I would do differently

Most of the cost in this project was rework, not compute. With hindsight, five habits would have saved most of it.