The crack of a bat accompanied by a roar from the crowd can capture a baseball fan’s attention, even if they miss seeing the home run. Humans may be visual creatures, but we also perceive the world around us by listening to it.
To improve artificial perception, MIT Professor Paris Smaragdis SM ’97, PhD ’01 seeks to make machines that can hear and understand sound even better than humans.
A machine with super-human hearing could rapidly pinpoint the location of a traffic accident by listening for squealing tires and crunching metal, or identify when an industrial drill might break while it is boring through dense bedrock to dig a geothermal well.
“I like to say that I do the same thing the computer vision folks do, but instead I do it for hearing,” says Smaragdis, a professor with joint appointments in the Music and Theatre Arts program of the School of Humanities, Arts, and Social Sciences, and the Department of Electrical Engineering and Computer Science in the School of Engineering and the MIT Schwarzman College of Computing.
His research group explores areas like machine learning for sound source separation, smart devices that can augment the audio surroundings of humans, and better tools to help people understand, manipulate, analyze, and create music.
“Having had a musical background, I’ve been interested in examining these ideas. What does it take to make machines understand sound? There is a huge domain of applications for machines that can listen,” says Smaragdis, who joined MIT in 2025.
A musical prelude
Smaragdis, who grew up in Greece, began playing classical guitar at a young age. He enjoyed playing traditional pieces but reached a turning point in his musical journey when his father brought home a small synthesizer.
“That little synthesizer had a function that allowed you to make your own sounds, and I became completely captivated by the possibilities,” he says.
Synthesizers were taking a more prominent place in pop music during the 1980s, adding to his fascination with the technology.
As he advanced through school, Smaragdis wanted to study music made with synthesizers. But this was not an educational path offered by the more traditional European conservatories at the time.
Instead, he enrolled at Berklee College of Music in Boston, into a program where he could explore the use of computers and synthesizers to shape and manipulate sound.
“With computers and synthesizers, I could make new sounds that have never been heard before,” he says.
At the same time, Smaragdis became immersed in jazz fusion. He declared bass as his major at Berklee since it would allow him more opportunities to play with jazz ensembles.
“I found the bass to be sonically more interesting than the guitar. It was a more unexplored domain, and I’ve always gravitated toward unexplored domains,” he says.
That interest in the unexplored carried over into his coursework. He and mentor Richard Boulanger, a professor of electronic production and design, created new tools to help people use computers to write, produce, and enhance music.
Through his studies at Berklee, Smaragdis embarked on a collaborative project with Professor Barry Vercoe (who passed away in 2025) at the MIT Media Lab to create software synthesizers for music creation. Building on that project, Smaragdis pursued graduate studies in Vercoe’s group, where he completed his master’s degree and PhD, followed by a postdoc.
“In the Media Lab, I got exposed more to the computer part of computer music. That’s where my attention shifted a bit. Now I really wanted to make computers that could understand sound,” he says.
Tuning in to unexplored domains
At that time, many students in the Media Lab were using signal processing techniques to help machines handle sound. Smaragdis chose a less explored approach — machine learning — to separate himself from the pack.
He found that training machine-learning models to understand sound offered advantages that couldn’t be attained with traditional signal processing techniques.
“Over time, it became really clear to me that we need to teach computers how to listen. We can’t just tell them how to listen. If anything, we don’t even know how we humans listen,” he says. “We don’t have a vocabulary to explain what it means to listen, or how to tell the difference between a cat’s meow and the sound of a violin.”
After finishing his postdoc, Smaragdis began a career at Mitsubishi Electric Research Labs, where he worked on artificial intelligence systems that could scan video soundtracks and understand what is happening on screen.
“What if you record an entire soccer game, but you only want to watch the highlights? In computer vision, it is very difficult to identify what a highlight is. But on the other hand, if you only listen to the sounds, then when the crowd goes crazy you would know that something interesting is happening,” he says.
That system was eventually incorporated into electronic devices, including a personal video recording system Mitsubishi was developing at the time, and into traffic monitoring systems and cell phones.
Smaragdis later joined Adobe, where he created the audio research team that built user-friendly tools to help artists, musicians, and producers manipulate sound without needing a lot of expertise.
“As we did that work, I kept coming up with deeper technical problems that needed to be resolved. I jumped into academia to address that itch,” he says.
Smaragdis joined the faculty at the University of Illinois at Urbana-Champaign, where he and his group incorporated the latest machine-learning techniques into audio-processing systems.
One of the technologies they developed utilized neural networks for separating individual sounds from audio mixtures. That project caught the attention of the team working on the 2021 documentary mini-series “The Beatles: Get Back.”
The production team enlisted Smaragdis to help clean up some archival Beatles recordings.
“That was a very different way of audio processing than what I was doing in academia. Now, I had to do something that sounded good but didn’t necessarily have to look good on paper. We didn’t have a formula we could optimize,” he says.
Every recording involved different challenges. For instance, in one clip John Lennon was playing music while Paul McCartney was horsing around, reading headlines from newspapers and making fun of George Harrison.
“We were trying to come up with a technological solution that could help us do everything, as opposed to addressing each problem on its own,” he says.
His contributions helped the team win an Emmy Award for sound editing for a nonfiction or reality program, in 2022.
A concerted effort
After 15 years at the University of Illinois, Smaragdis decided to join the faculty at MIT to tackle the thorny, interdisciplinary problems that were emerging in the field.
His group now focuses on two main areas. For one, they develop systems that make computers more useful tools for musicians.
One project in this area is enabling machines to understand how to listen to music, so they can provide constructive feedback to musicians.
“We want to enhance the toolbox a musician has. We don’t want to use computers to replace musicians,” he says.
On the other side, his group focuses on the fundamental tools scientists and artists alike can use to process, analyze, and manipulate audio.
For instance, his team has recently been exploring algorithms that allow sophisticated signal processing where traditional frames of reference, such as sampling rates, frame sizes, or context-windows, are not fixed — or even defined.
That implies different computational structures that are yet unexplored, despite their potential for more efficient processing, Smaragdis says.
“That requires rethinking a lot of the basics, but once you have that ability, you can start getting much more flexible and more powerful systems,” he adds.
They also explore ways to incorporate the latest techniques, like large language models, into speech recognition and audio enhancement.
“AI is an immensely powerful tool, but I don’t think we have found the ideal way to use it for sound and music yet,” he says.
He also enjoys mentoring students and helping them materialize their visions for research projects.
Since few of his students have experience using computation in sound, he emphasizes hands-on lessons so they see and hear how to transform equations into something tangible.
Smaragdis still feels a thrill from making music, whether with a computer or a more traditional instrument. Off campus, he tries to play jazz bass whenever he can find the time, and enjoys exploring the rich musical community of Boston with his wife and 15-year-old son.
Outside the musical realm, he plays and coaches underwater hockey — a sport where players wear diving masks and use sticks to push a heavy puck across the bottom of a pool toward their opponent’s goal. He coached the underwater hockey team at the University of Illinois, and is hoping to revive the MIT underwater hockey club soon and start hosting tournaments on campus.
While underwater hockey may be a lesser-known sport, it fits with Smaragdis’ lifelong interest in pursuing the underexplored.
“I’ve always like doing things that were different from what everyone else was doing. If you can create a new sound that no one has heard before, what does that mean about your ability to create new things? I still find these questions completely fascinating,” he says.
