You may have heard about vector search, but what is it? Vector search engines — known as vector databases, semantic, or cosine search — find the nearest neighbors to a given (vectorized) query. The power of vector search is that it discovers similar documents that are not an exact textual match, as would be required by our inverted index example above; it instead uses vectors that describe some level of similarity.
The natural language processing (NLP) community has developed a technique called text embedding that encodes words and sentences as numeric vectors. These vector representations are designed to capture the linguistic content of the text, and they can be used to assess the similarity between a query and a document.
Some common use cases for vector search are:
- Answering questions
- Finding answers to previously answered questions, where the question asked is similar but not exactly the same in a textual form
- Making recommendations — for example, a music application finding similar songs based on your preferences
All of these use cases leverage vectors with tens of thousands of dimensions, providing comprehensive representation of the data for accurate similarity assessment and targeted recommendations.
Elasticsearch supports vector search via the dense_vector document type and its ability to run similarity searches between the vector in the document and the search term after it has been converted into a vector.
For those who want to delve a little deeper into generative AI, we also offer ESRE, the Elasticsearch Relevance Engine™, which is designed to power artificial intelligence-based search applications. ESRE gives developers a full suite of sophisticated retrieval algorithms and the ability to integrate with large language models.
Leave a Reply