fiato

A tiny voice that knows how to sigh.

 

Loading the model

Teacher, then student

Five sentences from the test set, which the model never trained on. Hear Maya-1 read each one, then fiato. The student keeps the words and the voice; its sighs and gasps are fainter than the teacher's.

The model

A transformer reads phonemes and drives a small synthesizer: pulses for the voice, noise for the breath. There is no neural vocoder, which is why it is this small.

Open the model on Hugging Face

The dataset

Two voices described to Maya-1 in a sentence each, reading books, conversation, questions, numbers and dates. Every clip was checked for wrong words and a drifting voice.

Open the dataset on Hugging Face

How it was made

Maya-1 drawn as a very large circle and fiato as a small dot, to scale by parameter count, with the dataset between them.
  1. Describe two voicesMaya-1 turns a sentence of description into a voice. One clip of each becomes the anchor.
  2. Let the teacher talkEach prompt starts with the anchor's audio, so Maya-1 carries on in the same voice for every sentence.
  3. Keep the good clipsWhisper checks the words, a speaker model checks the voice, and an audio classifier checks each sigh and gasp is really there.
  4. Train the studentAn aligner labels how long every phoneme and every sigh lasts. The model trains from scratch on one GPU.