Query interpretation : an application of semiotics in image retrieval

M.H.T. de Boer, P. Brandt, M. Sappelli, L.M. Daniele, K. Schutte, W. Kraaij

Research output: Contribution to journalArticleAcademicpeer-review


One of the challenges in the field of content-based image retrieval is to bridge the semantic gap that exists between the information extracted from visual data using classifiers, and the interpretation of this data made by the end users. The semantic gap is a cascade of 1) the transformation of image pixels into labelled objects and 2) the semantic distance between the label used to name the classifier and that what it refers to for the end-user. In this paper, we focus on the second part and specifically on (semantically) scalable solutions that are independent from domain-specific vocabularies. To this end, we propose a generic semantic reasoning approach that applies semiotics in its query interpretation. Semiotics is about how humans interpret signs, and we use its text analysis structures to guide the query expansion that we apply. We evaluated our approach using a general-purpose image search engine. In our experiments, we compared several semiotic structures to determine to what extent semiotic structures contribute to the semantic interpretation of user queries. From the results of the experiments we conclude that semiotic structures can contribute to a significantly higher semantic interpretation of user queries and significantly higher image retrieval performance, measured in quality and effectiveness and compared to a baseline with only synonym expansions.
Original languageEnglish
Article number12
Pages (from-to)435-449
Number of pages15
JournalInternational Journal on Advances in Software
Issue number3&4
Publication statusPublished - 30 Dec 2015


  • semantic interoperability
  • computational semiotics
  • reasoning
  • query expansion
  • semiotics
  • semantic web technologies
  • image retrieval
  • semantic reasoning
  • natural language queries


Dive into the research topics of 'Query interpretation : an application of semiotics in image retrieval'. Together they form a unique fingerprint.

Cite this