Searching for audio by sketching mental images of sound : a brave new idea for audio retrieval in creative music production

Peter Knees, H.K.G. Andersen

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

9 Citations (Scopus)


We propose a new paradigmfor searching for sound by allow- ing users to graphically sketch their mental representation of sound as query. By conducting interviews with professional music producers and creators, we find that existing, text- based indexing and retrieval methods based on file names and tags to search for sound material in large collections (e.g., sample databases) do not reflect their mental concepts, which are often rooted in the visual domain and hence are far from their actual needs, work practices, and intuition. As a consequence, when creating new music on the basis of existing sounds, the process of finding these sounds is cum- bersome and breaks their work flow due to being forced to resort to browsing the collection. Prior work on organizing sound repositories aiming at bridging this conceptual gap between sound and vision builds upon psychological findings (often alluding to synaesthetic phenomena) or makes use of ad-hoc, technology-driven mappings. These methods fore- most aim at visualizing the contents of collections or individ- ual sounds and, again, facilitating browsing therein. For the purpose of indexing and querying, such methods have not been applied yet. We argue that the development of a search system that allows for visual queries to audio collections is desired by users and should inform and drive future research in audio retrieval. To explore this notion, we test the idea of a sketch interface with music producers in a semi-structured interview process by making use of a non-functional proto- type. Based on the outcomes of this study, we propose a conceptual software prototype for visually querying sound repositories using image manipulation metaphors.
Original languageEnglish
Title of host publicationICMR'16 :proceedings of the 2016 ACM International Conference on Multimedia Retrieval : June 6-9, 2016, New York, NY, USA
Place of PublicationNew York
PublisherAssociation for Computing Machinery, Inc
Number of pages8
ISBN (Print)978-1-4503-4359-6
Publication statusPublished - 2016
Externally publishedYes
Event2016 ACM on International Conference on Multimedia Retrieval (ICMR 2016) - New York, United States
Duration: 6 Jun 20169 Jun 2016


Conference2016 ACM on International Conference on Multimedia Retrieval (ICMR 2016)
Abbreviated titleICMR'16
Country/TerritoryUnited States
CityNew York
Internet address


  • audio retrieval
  • cross-domain retrieval
  • music production
  • retrieval by sketch


Dive into the research topics of 'Searching for audio by sketching mental images of sound : a brave new idea for audio retrieval in creative music production'. Together they form a unique fingerprint.

Cite this