Writing

urdu-realtime-1: a voice model that talks in Urdu

An experiment in speech-to-speech AI: a model that can hear and talk in the same moment, and starts a reply within 200 ms.

Two waveforms from one test call. The upper gray waveform is the caller. The lower green waveform is the model.

Every day, over 200 million South Asians talk in Urdu or a related regional language. Many voice products can say Urdu words. Few of them sound like a real conversation. The rhythm is wrong, the pauses are wrong, and the reply comes too late.

urdu-realtime-1 is my experiment to fix this problem. I want a voice model that can hold a spoken Urdu conversation that feels real. The model is in training now. It is not a finished product.

Speech in, speech out

Most voice assistants use a chain of three steps. First, they change speech into text. Next, they write a reply as text. Then, they change the reply back into speech. Each step adds delay, and the person must wait.

urdu-realtime-1 does not use this chain. The model is full duplex, and it goes directly from speech to speech. While the caller talks, the model continues to listen, and it can talk back in the same moment, like two people on a phone line. There is no text step to wait for, so a reply can start within 200 ms.

How I train the model

I started from an open model in the Moshi family, which already supports full-duplex speech. That model could not read or write Urdu script, so I added a tokenizer for Urdu.

Then I trained the model in stages. The first data set is Urdu podcasts with two speakers. I checked the transcripts of these podcasts before I used them. The second data set is recorded telephone calls. Urdu text gives the model more vocabulary and grammar.

Most of the training runs on a small cluster of NVIDIA DGX Spark computers that I operate.

Problems and fixes

Words are only one part of a conversation. Rhythm, short pauses and turns are also important, and they were the most difficult part of this work. I found and corrected three problems during development.

Problem 1: in early training runs, the model often said nothing. I found the cause in the podcast data. Podcast hosts talk for long periods, so the model learned to wait.

Problem 2: one version talked over the caller. I added a turn controller, which now decides when the model takes its turn.

Problem 3: callers mix English words into Urdu. When those words appeared in Latin script, the text output of the model had errors. The model now uses Urdu script for all of its text.

Four sample calls

I made four short test calls with the model. The video plays all four calls, with sound. Each call shows the waveform of the caller above the waveform of the model.

Four test calls in Urdu, 2 min 20 s. Turn on the sound.

In the first call, the caller orders chicken karahi with no tomatoes.

In the second call, the model plays a shop that checks a shoe order with a customer. It gives the price, 2,000 rupees, and asks if it can send the order. Then it says that the shoes arrive in two to three days.

In the third call, the caller discusses a business idea: a car that uses water as fuel. In the fourth call, the caller and the model talk about tasks in the house.

After the sample calls

After these calls, I trained the model on more telephone conversations. I also used reinforcement learning. This step teaches the model to answer the question that the caller asks.

Why a designer trains a voice model

A voice interface has time in it. The delay before a reply, and the length of a pause, change how a person speaks to the system. When the reply is slow, people speak in short commands. When the reply is fast and the turns feel natural, people speak as they do to another person.

I cannot design that experience with a slow model. Thus, I train the model myself, and I test each version in real calls in Urdu.

NextMagic Tap: touch input on a laptop with no touchscreen3 min read

See the design process these pieces come from.