Skip to main content

Scalable Similarity Search

Objective

Similarity search is the task of identifying, in a collection of items, the ones that are “similar” to a given
query item. This task has a range of important applications (e.g. in information retrieval, pattern
recognition, statistics, and machine learning) where data sets are often big, high dimensional, and
possibly noisy. State-of-the-art methods for similarity search offer only weak guarantees when faced with
big data. Either the space overhead is excessive (1000s of times larger than the space for the data itself),
or the work needed to report the similar items may be comparable to the work needed to go through all
items (even if just a tiny fraction of the items are similar). As a result, many applications have to resort to
the use of ad-hoc solutions with only weak theoretical guarantees.

This proposal aims at strengthening the theoretical foundation of scalable similarity search, and
developing novel practical similarity search methods backed by theory. In particular we will:

- Leverage new types of embeddings that are kernelized, asymmetric, and complex-valued.

- Consider statistical models of noise in data, and design similarity search data structures whose
performance guarantees are phrased in statistical terms.

- Build a new theory of the communication complexity of distributed, dynamic similarity search,
emphasizing the communication bottleneck present in modern computing infrastructures.

The objective is to produce new methods for similarity search that are: 1) Provably robust, 2) scalable
to large and high-dimensional data sets, 3) substantially more resource efficient than current state-ofthe-
art solutions, and 4) able to provide statistical guarantees on query answers.

The study of similarity search has been an incubator for techniques (e.g. locality-sensitive hashing and
random projections) that have wide-ranging applications. The new techniques developed in this project
are likely to have significant impacts beyond similarity search.

Field of science

  • /natural sciences/mathematics/applied mathematics/statistics and probability
  • /natural sciences/computer and information sciences/artificial intelligence/pattern recognition
  • /natural sciences/computer and information sciences/data science/big data
  • /natural sciences/computer and information sciences/artificial intelligence/machine learning

Call for proposal

ERC-2013-CoG
See other projects for this call

Funding Scheme

ERC-CG - ERC Consolidator Grants

Host institution

IT-UNIVERSITETET I KOBENHAVN
Address
Rued Langgaardsvej 7
2300 Kobenhavn
Denmark
Activity type
Higher or Secondary Education Establishments
EU contribution
€ 1 889 711,73
Principal investigator
Rasmus Pagh (Dr.)
Administrative Contact
Georg Dam Steffensen (Mr.)

Beneficiaries (1)

IT-UNIVERSITETET I KOBENHAVN
Denmark
EU contribution
€ 1 889 711,73
Address
Rued Langgaardsvej 7
2300 Kobenhavn
Activity type
Higher or Secondary Education Establishments
Principal investigator
Rasmus Pagh (Dr.)
Administrative Contact
Georg Dam Steffensen (Mr.)