Andy M. Sarroff

Amelioration of Human Aging

This topic was investigated during my research assistantship for Dr. Ye Wang in the Sound and Music Computing Lab at the School of Computing in the National University of Singapore. In particular, I explored ways in which the health of elderly people may be benefitted by using music computation, information retrieval, large-scale database searching and crawling, and human biometry.

Emotion Classification

I worked as a research scientist at Sourcetone LLC, who develops Music Emotion Recognition (MER) technology. This work presents an interesting challenge, as emotional experience is highly personal and can vary greatly from song to song. Unlike most studies, their research categorizes emotion induction, rather than attribution.

Modeling of Spaciousness

(See a recent research proposal.)

Modeling human perception of spaciousness in recorded music is one of my primary research interests. Spaciousness, the impression of size, environment, source placement, and other key characteristics of music, is largely left out of music analysis tasks. Spaciousness is a multidimensional perceptual feature that transmits much important information in music. Fluctuations in spaciousness give us emotional cues, while virtual placement of sources through mixing techniques help us to imagine a soundstage, to name a few important information streams. Still, Music Information Retrieval researchers tend to ignore spatial cues in analysis. In fact, most of the time, only one channel of a stereophonic signal is used, or the signal is otherwise mono-summed. I feel that improved research into the perceived spatial information streams of music will yield rich rewards in overall performance for machine perception, music recommendation, and music control applications.

These were some of the motivating factors behind much of my graduate research and, ultimately, my Masters thesis, which was conducted in 3 stages:

• Development of objective measurements for spaciousness in stereophonic music.
• A two-tiered study of subjective perception of spaciousness.
• Modeling of spaciousness by exemplar-based machine learning methods.

A working model was developed, but there is still extensive research to be done. First, only three dimensions of spaciousness were examined:

• Width of the source distribution
• Extent of overall reverberation
• Extent of envelopment

Researchers have identified dozens of dimensions of perceived spaciousness. Predictive models would benefit from inclusion of additional dimensions. Secondly, they can be strengthened by using larger song databases, more human subjects, and experimentation with different learning algorithms. I hope to continue this research further.

Top-Down Parameterization and Control of Audio Signal

A logical conclusion to a robust model of spaciousness (or any other predictive model of human perception) is to give a user new parametric control over musical attributes. I would like to investigate new ways to allow music listeners and producers to control music in perceptually meaningful ways. Such controls could take the form of EQ-like knobs that manipulate dimensions of spaciousness, for example.

Machine Learning

I've been working with machine learning algorithms in a variety of ways, including using Support Vector Regression to nonlinearly map audio to subjective perception, auto-encoders for reduction of dimensionality, and supervised selection of robust feature sets. Upcoming tasks bring further experimentation with unsupervised clustering to reveal hidden structures for personalization and/or contextualization.

Education

Affiliations

• Bregman Musc and Auditory Research Studio (BMARS)
• New England Music Information Special Interest Group (NEMISIG)
• Audio Engineering Society
• Music and Audio Research Laboratory (MARL)

Sponsorship, Grants, Honors

Recording and Mixing

I produced music for several years in New York City. I began by working for Greene Street Recording. I worked briefly at Mission Sound Recording and Loho Studios in 2001, before moving to RPM Electronic Sound Studios, where I soon became Chief Engineer. Toward the beginning of 2004, I opened a production facility. Working under the name Woodshop Sound, I recorded, mixed, and mastered albums until 2007. (Note: The web site is inactive and very old!)

My professional experience in music continues to inform my work as a researcher. To this day, I am very interested in modeling the perceived attributes of production in recorded music.

Percussion

I have played drums in several bands, releasing two recordings. At Wesleyan University, I studied Samba percussion; South Indian Mridangam and vocal percussion (solkattu); Javanese gamelan; and vibraphone with Jay Hoggard.

Search By Groove

In this research project we investigate automatic methods to identify rhythmic patterns embedded within the multiple layers of sound in dance music, popular music, and recordings from non-Western regions of the World. Of particular interest is finding recurring rhythmic patterns in a large database of recordings. Sponsored by Google Inc., (Faculty Research Award) and the Neukom Institute for Computational Science (2011-2012 Graduate Fellowship).

Source Invariant Characterization of Reverb

It is well known that reverberation negatively impacts the performance of many signal analysis and decomposition tasks, such as blind source separation, speech recognition, and speaker diarization. We investigate methods for compactly describing a reverberant environment that may offer discriminative advantages to these tasks. Investigated while at Gracenote, Inc.

Automatic Playlist Generation

Automatic Playlist Generation (APG) is an algorithmic means for grouping and ordering music. APG systems are an integral part of automatic music recommendation systems; they generate sequences of songs and aim to maximize the listening time of their subscribers. It is therefore important that adjacent songs preserve controlled novelty in APG systems. This project investigated ways to encode expert knowledge of song transitions to generate playlists.

Study on the Perception of Spaciousness in Recorded Music

In order to facilitate a predictive model for perceived spaciousness, data was collected from human subjects. The study was launched in two phases—on the internet, and in a laboratory. It was programmed with html, javascript, php, flash, and mysql. Within approximately 2 weeks, over 1500 responses were collected. This data was used to confirm the consistency of perceived spaciousness across subjects and then used to train a predictive model. The web site, which continues to collect data, has attracted visitors from approximately 25 countries.

Wide-Volve

Wide-Volve is a graphical environment (programmed in Matlab) in which the lateral angle profile of the spectral components of an input file can be modified and/or matched to another song’s profile. This processor, using an azimuthal discrimination strategy, places spectral components of an input audio file into azimuthal bins. The absolute lateral angle is calculated per frequency bin using the sine-sine panning law. The user sets a power threshold above which signal spectra are plotted against absolute lateral angle. To boost perceptual relevancy, spectral components are plotted on a frequency axis warped to the Bark scale. A selector tool allows selection of frequency spectra. The spectra is scaled to the desired absolute lateral angle or, if a target file has been loaded, to the target file’s lateral angle profile. After previewing, a new audio file can be saved. A full description of the processor can be found in this paper.

Database Audio-Visual Browser

This java application was built to easily browse a large database of collected human responses to audio. It automatically populates itself by querying a data base and retrieving whichever tables and columns are desired. The bottom panel shows all of the songs in the database, their average ratings, and relative standard deviations in ratings. By clicking on a graphical representation of a song, a user can hear the audio file. The top panel allows the user to choose which responses to visualize; responses can be filtered by any table column and unique data field. While this browser was built to explore a very specific experiment database, it was designed to be easily generalized to other experiments and databases.

Mid/Side Processor

A Max patch which provides a convenient interface for scaling the FFT coefficients of the mid or side signal of a stereophonic signal. Although direct scaling of coefficients can lead to ringing, this processor can be used to generate quite interesting sounds from ordinary stereophonic audio.

Occupations

Internships

Research Assistantships

Teaching

Journal Articles

Sarroff, A.M. and Bello, J.P. Toward a Computational Model of Perceived Spaciousness in Recorded Music. The Jaurnal of the Audio Engineering Society, vol. 59, no. 7/8, pp. 498-513, 2011.

Conference Proceedings

Master's Thesis

Other