Query

This interface functions as a composable query mechanism. This module is the backbone of the interaction between developers and the decentralized semantic network. Using the Discovery API, you can either search through your indexes with the help of either natural language based queries or the desired vector representation. The discovery protocol implements data model-agnostic vector search which defines a latent vector representation for embeddings retrieved from various language models suchs OpenAI Embeddings, LlamaIndex Embeddings, MistralAI Embeddings.

We define QUERY as the any natural language occurance ranging from keywords, phrase, question, documents etc. As examplified below:

Keyword: "quantum computing"

Phrase: "articles about quantum computing"

Question: "What are the latest trends in quantum computing?"

Document: "Trends on Quantum Computing\n\nIn recent years with the emergence ..."

Pipeline

We enable discovery in two main services as:

  1. Natural Language-based Search

  2. Vector-based Search

This enables to surf at your indexes with maximum efficiency. Our pipeline is shown below:

Our pipeline consistents of two main endpoints that utilizes vector or natural language based search enables to maximize surfing experience within our network for developers. By the help of latent embedding space, you can crawl through your index without depending onto only one language models for the indexing and searching.

We also support multiple knowledge storage for indexing your semantic document which enables to add additional filtering options depending on your metadata.

After vector search in our knowledge hubs, we implement a ranking function to provide the documents with maximum relevency to your query

Query Usage

After you authenticated into network, you can query via natural language via Index Network SDK.

// Search through y
const documents = await indexClient.query({
  indexIds: [ 
      "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo",
      "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo"
   ]
  query: "quantum computing",
  page: 1,
  limit: 10,
  filter: {
   indexItemUpdatedAt: {
     $gte: "2024-01-02"
   }
  },
  sort: "indexItemCreatedAt",
  order: 1
});

Vector Search Usage

Similar as to query using natural language, instead you can bring your own embedding to search to search indexes with our model-agnostic vectorstore.

// Search through y
const documents = await indexClient.query({
  indexIds: [ 
      "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo",
      "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo"
   ]
  embeddings: [ -0.0025, 1.2344, -0.531, ... ],
  model: "text-embedding-3-small"
  page: 1,
  limit: 10,
  filter: {
   indexItemUpdatedAt: {
     $gte: "2024-01-02"
   }
  },
  sort: "indexItemCreatedAt",
  order: 1
});

Response

The response will include search results with respect to your input query or vector. The each item response include similarity with indexed data and query with our ranking pipeline.

{
    "response": [
        { 
            "webPageItemId": "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo",
            "similarity": 0.8712
        },
        { 
            "webPageItemId": "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo",
            "similarity": 0.2533
        },
        { 
            "webPageItemId": "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo",
            "similarity": 0.1213
        },
        { 
            "webPageItemId": "kjzl6kcym7w8y7fjc89gmnkne7qpdz5ws5ryfji3i8dndjh2wxuii7z1anybovo",
            "similarity": 0.0122
        },        
    ]
}

Request Body Schema

The Body schema consists of several key properties:

PropertyTypeDescription

indexIds

array[indexId]

Array of indexes to be queried.

query

string

The query string or question to be answered.

filters

object

Filter object to add additional filters using metadata.

sort

string

Sorting column

page

number

Page number

skip

number

Skip offset

Additional Filtering

We also support where filtering for additional filtering indexes before search as below.

Using filters​

Index supports filtering queries by metadata and document contents. The where filter is used to filter by metadata, and the where_document filter is used to filter by document contents.

Filtering by metadata​

In order to filter on metadata, you must supply a where filter dictionary to the query. The dictionary must have the following structure:

{
    "metadata_field": {
        <Operator>: <Value>
    }
}

Filtering metadata supports the following operators:

  • $eq - equal to (string, int, float)

  • $ne - not equal to (string, int, float)

  • $gt - greater than (int, float)

  • $gte - greater than or equal to (int, float)

  • $lt - less than (int, float)

  • $lte - less than or equal to (int, float)

Using the $eq operator is equivalent to using the where filter.

{
    "metadata_field": "search_string"
}

# is equivalent to

{
    "metadata_field": {
        "$eq": "search_string"
    }
}

For further references, please check ChromaDB documentation

Endpoints

This section provides details about the available API endpoints.

Last updated