New app uses AI to enable anyone to make musical mashups

Imagine if the Police’s “Every Little Thing She Does Is Magic” opened Michael Jackson’s “Beat It,” then Cardi B’s “Bodak Yellow” riff blended with Jackson. This mix isn’t an impossible fantasy, but a reality with Mixboard, a tablet application that lets users without musical or editing experience create the songs of their dreams.

The app comes out of the Georgia Institute of Technology’s Center for Music Technology, whose director, Professor Gil Weinberg, and his students have been developing a mashup tool for years. Now, technology has finally caught up with their original vision. With the help of artificial intelligence (AI), the app sources artists the user might not naturally pair and finds the best parts of the song for the mashup.

“I think everyone can become creative with music,” Weinberg said. “Maybe not everyone knows all the theory that is required to compose a piece of music on a note level, but almost all people like music. If we give them a very easy way to put songs in different places, they can create something unique.”

The researchers presented their work in the paper “Mixboard — A Co-Creative Mashup Application for Novices” at the International Conference of New Interfaces for Musical Expression (NIME 2023) in May.

Making Mashups

To make a mashup, users drag and drop up to four songs from either Spotify or the Mixboard music library into vocals, bass, chords, and drum segments for up to 32 bars. The experience is inherently visual and lets users follow along as the app mixes their song choices, rendering in mere seconds. When finished, users can download and share their unique track.

“I worked on pretty much all parts of the project, from web interface and back end to making a brand-new iOS app,” said Raghavasimhan Sankaranarayanan, a Ph.D. student in music technology. “I am passionate about building stuff that is robust and production ready.”

AI DJ

The AI enables users to focus on developing the song instead of worrying about technical aspects and music theory. The app uses open-source music information retrieval libraries to determine song tempo, keys, and music stems, which help break down the song into individual mixes. The algorithm can stretch the sound to match the tempo or transpose keys to ensure both songs are in the same key. The researchers also provide templates of basic song structures to help users create an appealing track.

“The more we focus on developing AI for creative purposes, the more it can be inspiring and seen as a unique input humans couldn’t provide,” Weinberg said. “Together, with your own creativity and the AI creative input, the mashup can be something unique and new that you wouldn’t create only by yourself.”

Mixing in User Feedback

To ensure Mixboard was functional, the researchers conducted user surveys, led by Human-Computer Interaction master’s student Tommy Ottolin. Their studies evaluated the web interface with 45 subjects between the ages 18 and 27, excluding anyone with more than a year of music mixing or composition experience. Participants were given 30 minutes to test Mixboard. Next, they completed a semi-structured interview with the researchers followed by a 20-question survey using a five-point Likert scale asking about things like how much creative control users felt they had or how easy the app was to use. 

“Observing people interact with our Mixboard evokes a range of emotions in me, including pride, nervousness, and excitement,” said Qinying Lei, a music technology master’s graduate. “I’m so happy to see users become captivated by Mixboard and surprised by the unique music they produce.”

Mixboard’s Future

While the researchers continue to refine the app’s automation and improve iOS functionality, they also see much bigger potential than just a lab experiment. They hope to commercialize the app but are limited less by technology and more by licensing constraints. Weinberg has consulted with Kobi Abayomi, former senior vice president for Data Science at Warner Music Group, now head of Science for Gumbel Demand Acceleration and an Ivan Allen College of Liberal Arts advisory board member.

“Music rights holders should see apps like Mixboard as marketing opportunities, not licensing plays,” he said. “Getting people to have an increased affinity for music, or specifically your type of music, is what forward-thinking music marketing is all about.”

Ultimately, though, Mixboard is about democratizing access to music production and showing that anyone can be creative.

“Listening to the generated mashups getting better and better with each modification to our algorithm was amazing,” said Nitin Hugar, a music technology master’s graduate. “It was also fun to try and mash up the most improbable songs: an Indian song with African beats and some jazz chords was definitely not what I had thought would sound good together, but I was proven wrong many times. This project made me appreciate how music from different regions of the world are interconnected.”

#####

The Georgia Institute of Technology, or Georgia Tech, is one of the top public research universities in the U.S., developing leaders who advance technology and improve the human condition. The Institute offers business, computing, design, engineering, liberal arts, and sciences degrees. Its more than 45,000 undergraduate and graduate students, representing 50 states and more than 148 countries, study at the main campus in Atlanta, at campuses in France and China, and through distance and online learning. As a leading technological university, Georgia Tech is an engine of economic development for Georgia, the Southeast, and the nation, conducting more than $1 billion in research annually for government, industry, and society.