k-NN Index

Method definitions are used when the underlying Approximate k-NN algorithm does not require training. For example, the following knn_vector field specifies that nmslib’s implementation of hnsw should be used for Approximate k-NN search. During indexing, nmslib will build the corresponding hnsw segment files.

Model id’s are used when the underlying Approximate k-NN algorithm requires a training step. As a prerequisite, the model has to be created with the Train API. The model contains the information needed to initialize the native library segment files.

  "type": "knn_vector",
  "model_id": "my-model"
}

However, if you intend to just use painless scripting or a k-NN score script, you only need to pass the dimension.

A method definition refers to the underlying configuration of the Approximate k-NN algorithm you want to use. Method definitions are used to either create a knn_vector field (when the method does not require training) or that can then be used to create a knn_vector field.

A method definition will always contain the name of the method, the space_type the method is built for, the engine (the native library) to use, and a map of parameters.

HNSW Parameters

Note — For nmslib, ef_search is set in the .

Note — For hnsw, “innerproduct” is not available when PQ is used.

HNSW Parameters

IVF Parameters

IVF training requirements

The IVF algorithm requires a training step. To create an index that uses IVF, you need to train a model with the Train API, passing the IVF method definition. IVF requires that, at a minimum, there should be nlist training data points, but it is . Training data can either the same data that is going to be ingested or a separate set of data.

Encoders can be used to reduce the memory footprint of a k-NN index at the expense of search accuracy. faiss has several different encoder types, but currently, the plugin only supports flat and pq encoding.

An example method definition that specifies an encoder may look something like this:

"method": {
    "name":"hnsw",
    "engine":"faiss",
        "encoder":{
            "name":"pq",
            "parameters":{
              "code_size": 8,
              "m": 8
            }
        }
    }
}

PQ Parameters

There are a lot of options to choose from when building your knn_vector field. To determine the correct methods and parameters to choose, you should first understand what requirements you have for your workload and what trade-offs you are willing to make. Factors to consider are (1) query latency, (2) query quality, (3) memory limits, (4) indexing latency.

If memory is not a concern, HNSW offers a very strong query latency/query quality tradeoff.

If you want to use less memory and index faster than HNSW, while maintaining similar query quality, you should evaluate IVF.

If memory is a concern, consider adding a PQ encoder to your HNSW or IVF index. Because PQ is a lossy encoding, query quality will drop.

Having a replica doubles the total number of vectors.

HNSW memory estimation

The memory required for HNSW is estimated to be 1.1 * (4 * dimension + 8 * M) bytes/vector.

As an example, assume you have a million vectors with a dimension of 256 and M of 16. The memory requirement can be estimated as follows:

IVF memory estimation

The memory required for IVF is estimated to be 1.1 * (((4 * dimension) * num_vectors) + (4 * nlist * d)) bytes.

As an example, assume you have a million vectors with a dimension of 256 and nlist of 128. The memory requirement can be estimated as follows:

1.1 * (((4 * 128) * 1000000) + (4 * 128 * 256))  ~= 563 MB

Additionally, the k-NN plugin introduces several index settings that can be used to configure the k-NN structure as well.

At the moment, several parameters defined in the settings are in the deprecation process. Those parameters should be set in the mapping instead of the index settings. Parameters set in the mapping will override the parameters set in the index settings. Setting the parameters in the mapping allows an index to have multiple knn_vector fields with different parameters.