Efficient and Effective Multi-Modal Queries through Heterogeneous Network Embedding

No Thumbnail Available
File version
Author(s)
Duong, CT
Nguyen, TT
Yin, H
Weidlich, M
Mai, S
Aberer, K
Nguyen, QVH
Griffith University Author(s)
Primary Supervisor
Other Supervisors
Editor(s)
Date
2021
Size
File type(s)
Location
License
Abstract

The heterogeneity of today's Web sources requires information retrieval (IR) systems to handle multi-modal queries. Such queries define a user's information needs by different data modalities, such as keywords, hashtags, user profiles, and other media. Recent IR systems answer such a multi-modal query by considering it as a set of separate uni-modal queries. However, depending on the chosen operationalisation, such an approach is inefficient or ineffective. It either requires multiple passes over the data or leads to inaccuracies since the relations between data modalities are neglected in the relevance assessment. To mitigate these challenges, we present an IR system that has been designed to answer genuine multi-modal queries. It relies on a heterogeneous network embedding, so that features from diverse modalities can be incorporated when representing both, a query and the data over which it shall be evaluated. By embedding a query and the data in the same vector space, the relations across modalities are made explicit and exploited for more accurate query evaluation. At the same time, multi-modal queries are answered with a single pass over the data. An experimental evaluation using diverse real-world and synthetic datasets illustrates that our approach returns twice the amount of relevant information compared to baseline techniques, while scaling to large multi-modal databases.

Journal Title

IEEE Transactions on Knowledge and Data Engineering

Conference Title
Book Title
Edition
Volume
Issue
Thesis Type
Degree Program
School
Publisher link
Patent number
Funder(s)
Grant identifier(s)
Rights Statement
Rights Statement
Item Access Status
Note

This publication has been entered as an advanced online version in Griffith Research Online.

Access the data
Related item(s)
Subject

Information and computing sciences

Persistent link to this record
Citation

Duong, CT; Nguyen, TT; Yin, H; Weidlich, M; Mai, S; Aberer, K; Nguyen, QVH, Efficient and Effective Multi-Modal Queries through Heterogeneous Network Embedding, IEEE Transactions on Knowledge and Data Engineering, 2021

Collections