EDBT/ICDT 2009 Joint Conference

Electronic Conference Proceedings

• Welcome
• Full Program
• ICDT Sessions
• ICDT Papers
• EDBT Sessions
  • Invited Talks
  • Research Track
    • System Architectures
    • Spatio-Temporal
    • Database Summarization
    • Query Processing
    • XML, XPath, XQuery
    • Graph Techniques
    • Privacy & Security
    • Data Models
    • Stream Processing
    • XML, XPath, XQuery
    • Database Summarization
    • Query Processing
    • Top-K Techniques
    • Graph Techniques
    • Data Mining
    • Heterogeneous & Distributed
    • System Architectures
    • Privacy & Security
    • Uncertainty
    • Workflow Techniques
    • Multi-Dimensional
    • Caching Techniques
    • Information Retrieval
    • Query Processing
    • Top-K Techniques
    • Potpourri
    • Provenance
    • Spatio-Temporal
    • Skylines
    • Transaction Processing
  • Industrial Track
  • Demo Track
  • Tutorials
• EDBT Papers
• Author Index
• Workshops

Scalable Stream Join Processing with Expensive Predicates: Workload Distribution and Adaptation by Time-Slicing

Authors

• Song Wang (HP Labs, USA)
• Elke Rundensteiner (Worcester Polytechnic Institute, USA)

Abstract

Multi-way stream joins with expensive join predicates lead to great challenge for real-time (or close to real-time) stream processing. Given the memory- and CPU-intensive nature of such stream join queries, scalable processing on a cluster must be employed. This paper proposes a novel scheme for distributed processing of generic multi-way joins with window constraints, called Pipelined State Partitioning (PSP). We target generic joins with arbitrarily join conditions, which are used in non-trivial stream applications such as image matching and biometric recognizing. The PSP scheme partitions the states into disjoint slices in the time domain, and then distributes the fine-grained states in the cluster, forming a virtual computation ring. Compared to replication-based distribution of non-equi-joins, PSP scheme is superior since: (1) zero state duplication and thus no repeated computations, (2) pipelined processing of every input tuple on multiple nodes to achieve low response time, and (3) cost-based adaptive workload distribution. We have implemented the proposed PSP schemes within the CAPE DSMS. Our experimental study demonstrates the significant performance improvements compared to the state-of-the-art generic distributed stream join algorithms.

Session

EDBT Research Session 9: Stream Processing (Wednesday, March 25, 11:00—12:30)