Software by CUSP Visiting Scholar Vincent Lostanlen featured in newly released record

Visiting Scholar Vincent Lostanlen’s research-creation collaboration with composer Florian Hecker is now available in a newly released album.

Vincent has been collaborating with Hecker since 2016 on various computer music pieces, including:

• Alte Oper, Frankfurt, 2016.
• Modulator (Scattering Transform). Museum of Modern Art, Frankfurth, 2016.
• Experimental Palimpsests. Lausanne Underground Film Festival, 2016.
• Live at BBC Maida Vale Studios, London, 2016.
• Halluzination, Perspektive, Synthese. Kunsthalle, Wien, 2017.

The new album (Lorenzo Senni. “The Shape of RemiXXXes to Come”. Warp, 2018) contains one track from Florian Hecker, “XAllegroX (Hecker Scattering.m Sequence).” The track is “a technical ambient mix essentially consisting of resynthesized material prominently featuring Vincent Lostanlen’s scattering.m toolbox for time-frequency scattering transforms. In an iterative process, over 50 runs, this resynthesis remix gradationally approximates the original starting from an indefinable mass.”

The “scattering.m toolbox” (pronounced “scattering dot M”) is a piece of software Vincent wrote during his years of PhD at École Normale Supérieure in Paris. His PhD dissertation is entitled “Convolutional operators in the time-frequency domain” and contains a description of the creative process. Vincent is still maintaining the software today, as a visiting scholar at NYU CUSP and NYU MARL.

Time-frequency scattering is a mathematical transformation of sound waves. Its core purpose is to mimick the way the human auditory system extracts information from its environment. In the context of improving the artificial intelligence of sounds, it has found succesful applications in automatic speech transcription as well as the recognition of urban sounds and musical sounds. 

Time-frequency scattering consists of a cascade of two separate operations, named layers. On one hand, the first layer imitates the functioning of the cochlea, as it decomposes acoustic frequencies from low to high. On the other hand, the second layer imitates the functioning of the primary auditory cortex, by analyzing the variations of the acoustic content through time between neighboring acoustic frequencies. Such variations of acoustic content are known as spectrotemporal modulations.

What distinguishes time-frequency scattering from previous generations of computational models of audition is that it does not restrict the time scale of spectrotemporal modulations to a single, predefined value; but instead, it measures them at many different time scales, ranging between one millisecond and several second.

The original idea of composer Florian Hecker was to transfer the application domain of time-frequency scattering from sound analysis to sound synthesis, while keeping the same, two-layer mathematical architecture. One major challenge of such project was to elaborate an algorithm to invert the analysis procedure. In 2015, Vincent Lostanlen solved the problem of inverting time-frequency scattering by connecting it to the theory of deep learning. He repurposed a method for training deep neural networks, named gradient backpropagation, to the deep cascade of time-frequency scattering layers.

Once gradient backpropagation was made available as part of his software library, it became possible, not only to compute the time-frequency scattering coefficients associated to any sound wave; but also, reciprocally, to retrieve the sound wave that would correspond to any user-defined vector of time-frequency scattering coefficients.

The modus operandi of Florian Hecker in his remix consists of two steps. First, he isolated the melodic loop of Lorenzo Senni’s “XAllegroX, and computed its scattering coefficients. Then, he ran the gradient backpropagation algorithm in order to retrieve the original melodic loop. Because gradient backpropagation is an iterative algorithm, it is unable to invert time-frequency scattering in a single step; instead, it starts with a random guess and adjusts its own output from one iteration to the next, thus progressively converging to the original.

As a consequence of this iterative process, it appears that gradient descent happens to be an artistically interesting way of modeling sound textures through time. Indeed, whereas the initial random guess is, with overwhelming probability, devoid of noticeable spectrotemporal modulations, time-frequency scattering progressively enhances these spectrotemporal modulations from one iteration to the next. Although, at the first iteration, the loop sounds hazy and static — or, a electronic musicians would call it, droney — it catches up in rhythmic content in subsequent iterations, thus producing a feeling of sonic rise that is fitting to the typical structuration of dance music.

The peculiarity of Hecker’s scattering.m remix is that this musical “rise” is produced, not over the well-known sonic attributes of frequency and amplitude (as is usually the case in EDM), but on a relatively novel, joint parameter of texture; that is, a notion of sonic complexity which consists of the organization of frequencies and amplitude through time. Therefore, the development of gradient backpropagation for time-frequency scattering over new avenues for musical creation with digital audio effects: in addition to remixing amplitude (by EQing) and frequency (by phase vocoder), it is now possible to remix texture itself, independently of amplitude and frequency.

Along the same lines of amplitude modulation (AM) and frequency modulation (FM), Lostanlen proposes to call this new musical transformation a meta-modulation (MM), because it operates over spectrotemporal modulations rather than on the direct acoustic content. The study of MM, both from mathematical and compositional standpoints, will be central to the collaboration between Lostanlen and Hecker in future years.

The vinyl record has been distributed widely throughout the US, Europe, and Japan. In its digital form, “The Shape of RemixXxes to Come” EP is available in digital format on Spotify, Apple Music, Youtube, Bleep, the iTunes Store, Amazon Music, and Google Play.

The remix was aired on the Camarilha dos Quatro weekly podcast. Mary Anne Hobbs, an English DJ and music journalist, shared another of the album songs on her BBC show “6 Music Recommends”. FACT also listed the record as one of the must-haves of the month.