PCIR: Combining DHTs and peer clusters for efficient full-text P2P indexing

Odysseas Papapetrou, Wolf Siberski, Wolfgang Nejdl

Research output: Contribution to journalArticleAcademicpeer-review

13 Citations (Scopus)

Abstract

Distributed hash tables (DHTs) are very efficient for querying based on key lookups. However, building huge term indexes, as required for IR-style keyword search, poses a scalability challenge for plain DHTs. Due to the large sizes of document term vocabularies, peers joining the network cause huge amounts of key inserts and, consequently, a large number of index maintenance messages. Thus, the key to exploiting DHTs for distributed information retrieval is to reduce index maintenance costs. Various approaches in this direction have been pursued, including the use of hybrid infrastructures, or changing the granularity of the inverted index to peer level. We show that indexing costs can be significantly reduced further by letting peers form groups in a self-organized fashion. Instead of each individual peer submitting index information separately, all peers of a group cooperate to publish the index updates to the DHT in batches. Our evaluation shows that this approach reduces index maintenance cost by an order of magnitude, while still keeping a complete and correct term index for query processing.

Original languageEnglish
Pages (from-to)2019-2040
Number of pages22
JournalComputer Networks
Volume54
Issue number12
DOIs
Publication statusPublished - 26 Aug 2010
Externally publishedYes

Fingerprint

Costs
Query processing
Information retrieval
Joining
Scalability

Keywords

  • DHT
  • Hybrid topologies
  • P2P information retrieval

Cite this

Papapetrou, Odysseas ; Siberski, Wolf ; Nejdl, Wolfgang. / PCIR : Combining DHTs and peer clusters for efficient full-text P2P indexing. In: Computer Networks. 2010 ; Vol. 54, No. 12. pp. 2019-2040.
@article{af8c352f075a414b8b406dc9c8b86960,
title = "PCIR: Combining DHTs and peer clusters for efficient full-text P2P indexing",
abstract = "Distributed hash tables (DHTs) are very efficient for querying based on key lookups. However, building huge term indexes, as required for IR-style keyword search, poses a scalability challenge for plain DHTs. Due to the large sizes of document term vocabularies, peers joining the network cause huge amounts of key inserts and, consequently, a large number of index maintenance messages. Thus, the key to exploiting DHTs for distributed information retrieval is to reduce index maintenance costs. Various approaches in this direction have been pursued, including the use of hybrid infrastructures, or changing the granularity of the inverted index to peer level. We show that indexing costs can be significantly reduced further by letting peers form groups in a self-organized fashion. Instead of each individual peer submitting index information separately, all peers of a group cooperate to publish the index updates to the DHT in batches. Our evaluation shows that this approach reduces index maintenance cost by an order of magnitude, while still keeping a complete and correct term index for query processing.",
keywords = "DHT, Hybrid topologies, P2P information retrieval",
author = "Odysseas Papapetrou and Wolf Siberski and Wolfgang Nejdl",
year = "2010",
month = "8",
day = "26",
doi = "10.1016/j.comnet.2010.03.025",
language = "English",
volume = "54",
pages = "2019--2040",
journal = "Computer Networks",
issn = "1389-1286",
publisher = "Elsevier",
number = "12",

}

PCIR : Combining DHTs and peer clusters for efficient full-text P2P indexing. / Papapetrou, Odysseas; Siberski, Wolf; Nejdl, Wolfgang.

In: Computer Networks, Vol. 54, No. 12, 26.08.2010, p. 2019-2040.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - PCIR

T2 - Combining DHTs and peer clusters for efficient full-text P2P indexing

AU - Papapetrou, Odysseas

AU - Siberski, Wolf

AU - Nejdl, Wolfgang

PY - 2010/8/26

Y1 - 2010/8/26

N2 - Distributed hash tables (DHTs) are very efficient for querying based on key lookups. However, building huge term indexes, as required for IR-style keyword search, poses a scalability challenge for plain DHTs. Due to the large sizes of document term vocabularies, peers joining the network cause huge amounts of key inserts and, consequently, a large number of index maintenance messages. Thus, the key to exploiting DHTs for distributed information retrieval is to reduce index maintenance costs. Various approaches in this direction have been pursued, including the use of hybrid infrastructures, or changing the granularity of the inverted index to peer level. We show that indexing costs can be significantly reduced further by letting peers form groups in a self-organized fashion. Instead of each individual peer submitting index information separately, all peers of a group cooperate to publish the index updates to the DHT in batches. Our evaluation shows that this approach reduces index maintenance cost by an order of magnitude, while still keeping a complete and correct term index for query processing.

AB - Distributed hash tables (DHTs) are very efficient for querying based on key lookups. However, building huge term indexes, as required for IR-style keyword search, poses a scalability challenge for plain DHTs. Due to the large sizes of document term vocabularies, peers joining the network cause huge amounts of key inserts and, consequently, a large number of index maintenance messages. Thus, the key to exploiting DHTs for distributed information retrieval is to reduce index maintenance costs. Various approaches in this direction have been pursued, including the use of hybrid infrastructures, or changing the granularity of the inverted index to peer level. We show that indexing costs can be significantly reduced further by letting peers form groups in a self-organized fashion. Instead of each individual peer submitting index information separately, all peers of a group cooperate to publish the index updates to the DHT in batches. Our evaluation shows that this approach reduces index maintenance cost by an order of magnitude, while still keeping a complete and correct term index for query processing.

KW - DHT

KW - Hybrid topologies

KW - P2P information retrieval

UR - http://www.scopus.com/inward/record.url?scp=77955426847&partnerID=8YFLogxK

U2 - 10.1016/j.comnet.2010.03.025

DO - 10.1016/j.comnet.2010.03.025

M3 - Article

AN - SCOPUS:77955426847

VL - 54

SP - 2019

EP - 2040

JO - Computer Networks

JF - Computer Networks

SN - 1389-1286

IS - 12

ER -