Cutting dead air without a speech model, and ums with one

The first recording this was measured on ran 28.61 seconds, and 15.47 of those seconds had nobody speaking in them. It was not a bad recording. There was dead air at the start while the speaker found the window, one long pause in the middle, and a tail after the last word. Cutting those out by hand, on every recording you make, is where the hour goes that you thought you were saving by not editing.

In Cilevi that pass is a button. Press Find silences, in the SILENCE section at the top of the Captions tab, and every quiet stretch becomes a small chip on the timeline ruler with its own length written on it. Click one and that stretch is cut out of every unlocked track at once, and the rest of the timeline closes up behind it. Remove all takes the lot.

Finding the quiet stretches without knowing the language

The detection runs on the peak envelope that the timeline already builds to draw the waveform. It reads loudness. It does not know what a word is, it never sees a transcript, and it does not care which language you are speaking.

That sounds like a compromise and it is the opposite. A recording whose only audio is the system mix, with no microphone track at all, still gets chips. So does one in a language your machine has no speech model for at all. The feature that needs a transcript is the one further down this page, and it is limited exactly where you would expect.

One mouse click is louder than a whole sentence, for 50 ms

The first build of this was a threshold and a minimum length, and the owner of this project broke it by hand within an hour. This is the failure worth writing down.

The envelope stores a peak per bucket, roughly 20 buckets a second. A single mouse click inside a pause lights a whole 50 ms bucket, which splits that pause into 2 shorter runs, and each half is then too short to be offered as a silence. Somebody clicking around a dashboard without narrating it gets nothing offered at all.

The numbers off that recording say why the fix is not a louder threshold. In the stretches that held only clicks, 1 to 5 buckets out of every 20 came in above the threshold, with a mean of 0.03. In the stretches of real speech, 20 buckets out of 20 were above it, with a floor of 0.10. Length separates those 2 cases and loudness does not.

So a run above the threshold is only counted as talking once it has lasted a minimum time, which is the Ignore short sounds slider in that section, 0.15 s by default. That number was swept rather than picked: at 0.10 the clicky pause still came out in 3 pieces, at 0.15 everything the owner pointed at was found as one range, and at 0.25 it started offering a 0.74 s gap in the middle of a sentence as if it were a pause. Raising the loudness threshold instead, to 0.10, ate quiet speech immediately.

The number on the threshold slider is not decibels

The envelope has a 0.45 root curve applied to it before anything sees it, so the default threshold of 0.06 is a raw amplitude near 0.002, around 54 dB down. Reasoning about the slider as if it were linear will give you the wrong answer every time. Move it a little and look at the chips.

What clicking a chip does to the other tracks

The cut goes through every unlocked track: video, system audio, microphone, camera. Locked tracks are skipped entirely, and that is not a convenience. A removal that cut every track except a locked one would leave the timeline out of step with itself, sound against picture, for the whole rest of the recording.

Remove all works from the last range backwards to the first, so that each range still to go keeps the position it was measured at, and the whole lot commits as one undo entry. Press Ctrl+Z twice after using it and the second press does nothing, which is the behaviour you want and is easy to get wrong.

On that 28.61 second recording, at the defaults, the first build offered 4 fragments totalling 8.11 s and missed the opening entirely. The build that shipped offers 3 ranges totalling 15.47 s, which is the head, the whole middle pause including its clicks, and the tail. Remove all of them and 28.61 seconds becomes 13.3.

The one range it will never offer

A run of silence covering the whole recording is refused. That is the muted microphone case, and offering it would be offering to delete the recording. Everything else is fair game, head and tail included, which is a change from the first build: it held those back on the theory that people trim their own ends, and they do not. It is the first thing anybody wants gone.

Taking out the ums needs the transcript

Different job. Silence is a stretch with nothing in it. An um is a sound with a shape, and the only thing on your machine that can tell one from a word is the speech engine described in captions without uploading the recording. Generate a transcript first, or there is nothing to find.

Press Find and the panel lists what it found as one row per word, not one row per occurrence. "Um 14", "So 6". You tick the row, and every occurrence of that word goes, with each occurrence marked in amber on the transcript above so you can see exactly what is about to be cut before you cut it. A phrase spanning several words, "you know what i mean", counts as one occurrence and one row.

The words come in 3 grades, and the grade only decides where the scoring starts. A sound that is not a word in any sentence, um, uh, erm, starts high enough to be ticked on the list alone. Padding made of real words, "you know", "i mean", needs a pause around it before it ticks. A word that is filler only when it stands alone, like, so, well, actually, needs about half a second of hesitation either side, which is the difference between "so I opened the file" and "so, I opened the file".

The pause has to be measured off the audio, backwards

The obvious way to measure the pause around a word is to subtract its end time from the next word's start time. That gives zero. Every time.

Whisper times the onset of a word, not its end, so inside a caption line each word ends exactly where the next one begins, and the silence a speaker actually left is sitting inside what the timing calls the word. The pause is read off the audio envelope instead, walking backwards from the onset, and summed over half a second either side rather than measured as the quiet run touching the boundary, because the engine's onsets are tens of milliseconds out in both directions and the single bucket before an onset is as likely to hold the word's own first consonant as the silence in front of it. The first version, which walked back and stopped at the first loud bucket, returned a pause of zero for every word in a 7 minute recording, including one that had 3.8 seconds of silence in front of it.

"Quiet" is read off the recording too, from the 10th and 80th percentiles of its own envelope, measured only between the first word and the last. Measuring across the whole file breaks on a recording that is mostly silence: on one fixture where speech was 7 percent of the file, the 85th percentile was itself silence, and nothing was ever quiet enough to count.

A folding rule that ate the word for look

To match a word against the list, both are folded: punctuation dropped, case dropped, the regional variants of the same letter brought together, and a run of the same letter collapsed.

Collapsing a doubled letter is what a spellchecker would do and it was wrong here. In one of the languages we test with, it folded the word for "look" down to a form that then matched the word for "between", and another meaning "they see", in an ordinary sentence about customers. 3 words, 2 of them perfectly normal, folded onto one. That language doubles a letter to mean something and never triples one, so only runs of 3 or more collapse now, and every spelling of a hum is listed out by hand instead.

2 words in that language are deliberately kept off the list for the same reason: one of them means "also" and is among the commonest words there is, the other means "good".

What it finds on real speech

On a 7 minute 17 second recording of somebody talking unscripted, 609 words, it found 3 candidates and ticked 1: a filler opening a sentence after 3.8 seconds of silence. The 2 it left unticked were correct to leave. On a 22 second English recording with real hesitations in it, it found 10 and ticked 7, and none of the ticks were wrong. It missed 3 real fillers, including a sentence-opening "So,".

That asymmetry is on purpose. A miss costs you one tap in the panel. A false positive costs you a video with a word cut out of it.

The list is not what limits this

Whisper was trained on clean transcripts and it writes clean transcripts. Of the owner's spoken audio, 8.5 percent of the time, 19.7 seconds out of 230.7, carries no word at all in the transcript. The speaker was making a sound. The engine declined to write it down.

We tried telling the engine to keep them, feeding it a prompt written in hesitant speech before the run. Over the same 7 minutes it produced 613 words against the plain run's 609 and the filler counts were identical. The model is not choosing how to spell these sounds. It is choosing not to write them at all, and a prompt does not reach that decision. So 609 words of casual speech yielding 3 candidates is the transcript's ceiling, not the word list's.

Try it on a recording you already regret

Open one, generate a transcript, and press Find before you cut anything by hand. Read the amber marks in the transcript rather than the row counts: the rows tell you what it thinks, the marks tell you where. Then do the silence pass second, because removing words changes where the pauses are.

If it ticks something it should not have, that is the report worth sending to the feedback board, with the language you were speaking. A false positive in a language we do not speak is the one thing we cannot find on our own machines.