There is a common dilemma called the “cocktail party problem.” Perhaps you have never heard of it, but you have probably experienced it. It’s the challenge of separating conversation voices from distracting ambient noise. While it’s often annoying, researchers say it can also be a mentally taxing situation, worsened by hearing impairment.

Researchers at the University of Washington are working to address the widespread issue with smart headphones designed to proactively isolate the user’s conversation partners in noisy environments.

Their prototype is regular, off-the-shelf hardware that leverages artificial intelligence to quickly identify conversation partners using two to four seconds of audio.

AI Headphones Anticipate Conversation Flow

Two AI models power the system. The technology activates when the user starts talking, which prompts one of the models to track participants by performing what researchers call a “who spoke when” analysis. The rhythmic nature of human conversation helps form the foundation of the technology.

“Existing approaches to identifying who the wearer is listening to predominantly involve electrodes implanted in the brain to track attention,” the study’s senior author, Shyam Gollakot, said. “Our insight is that when we’re conversing with a specific group of people, our speech naturally follows a turn-taking rhythm.”

He added, “And we can train AI to predict and track those rhythms using only audio, without the need for implanting electrodes.”

Results of the tracking are then forwarded to a second model. Once the second model receives the results, it isolates the identified voices and plays the “cleaned-up” audio for the user without lag. When researchers tested the technology with 11 participants, the “proactive hearing assistants” demonstrated significant efficacy.

Researchers say that the group rated the headphone’s filtered audio more than twice as favorably as the baseline.

“Everything we’ve done previously requires the user to manually select a specific speaker or a distance within which to listen, which is not great for user experience,” said the study’s lead author, Guilin Hu. “What we’ve demonstrated is a technology that’s proactive — something that infers human intent noninvasively and automatically.”

Future Implications

Looking ahead, researchers believe this technology could greatly benefit users of hearing aids, earbuds, and smart glasses by automatically filtering their surrounding soundscapes. Researchers say the current prototype performed surprisingly well in complex scenarios. However, they noted that dynamic conversations with frequent overlap, long monologues, or languages beyond English, Mandarin, and Japanese require additional refinement.

Eventually, the research team expects the system to be miniaturized and run on a small chip, which would allow it to be integrated directly into hearing aids or earbuds.