Ingesting documents

Load documents

PUT /api/v1/data/datasources/ProductManuals/documents
Content-Type: application/json

[
  {
    "id_": "manual-cam45-alignment-reset",
    "text": "To realign camera CAM-45, stop the line, open the maintenance bracket, adjust the lateral offset until the center marker matches, then retighten the bracket bolts.",
    "metadata": {
      "device_model": "CAM-45",
      "document_type": "manual",
      "section": "alignment_reset",
      "title": "CAM-45 Alignment Reset",
      "url": "https://docs.example.com/cam45/alignment",
      "page_number": 42
    }
  }
]

Field	Description
`id_`	Your stable ID for the document
`text`	The content of the document. For images, see Load images.
`metadata`	Optional attributes for filtering and for showing sources

The request is processed synchronously. When it returns, the documents are searchable.

Source metadata

When an agent cites a document, the source name and link are taken from these metadata keys:

Key	Used for
`title`	Name of the source
`url`	Link to the original document
`file_name` and `page_label`	Name of the source, shown as `{file_name} Page: {page_label}`, if `title` is missing

Without title, file_name and page_label, the url is shown as name. Without any of these keys, an internal ID is shown. Always set title and, if the original is reachable for users, url.

Load images

Images, for example diagrams or photos from a manual, are loaded with the same request. Send the image base64-encoded in image_resource instead of text:

[
  {
    "id_": "manual-cam45-wiring-diagram",
    "image_resource": {
      "data": "iVBORw0KGgoAAAANSUhEUgAA…",
      "mimetype": "image/png"
    },
    "metadata": {
      "device_model": "CAM-45",
      "title": "CAM-45 Wiring Diagram",
      "page_context": "Section 4.2 Electrical connection: connect the camera to the 24 V supply …"
    }
  }
]

Field	Description
`image_resource.data`	The image, base64-encoded
`image_resource.mimetype`	The image format, for example `image/png` or `image/jpeg`
`metadata.page_context`	Optional text that surrounds the image in the original document

During loading, a language model describes the image in text. This description is what the search finds. The model uses the metadata and page_context as context, so the more context you provide, the better the description. If you send your own text together with the image, the service uses your text and does not generate a description.

To show the original image, for example in your own chat interface, load it from the node:

GET /api/v1/data/datasources/{name}/nodes/{node_id}/image

The response contains the image with its content type. The request needs the same token and headers as all other requests.

What happens to each document

The service compares the id_ and a hash of the content with what is already stored:

Situation	Result
New `id_`	Loaded
Same `id_`, changed content	Updated
Same `id_`, same content	Skipped: `already_ingested_or_in_progress`
Different `id_`, same content as an existing document	Skipped: `duplicate_hash`

So you can safely send the same documents again. Unchanged documents are skipped and do not cost processing time.

Response

{
  "status": "partial_success",
  "accepted": ["manual-cam45-alignment-reset"],
  "skipped": [
    {
      "doc_id": "manual-cam45-copy",
      "doc_hash": "a3f5e9d2b1c4…",
      "reason": "duplicate_hash",
      "conflicting_doc_id": "manual-cam45-calibration"
    }
  ],
  "update_counts": { "ProductManuals": { "documents": 1, "nodes": 3 } }
}

HTTP status	`status`	Meaning
`201`	`created`	All documents loaded
`207`	`partial_success`	Some documents loaded, see `skipped`
`200`	`already_ingested_or_in_progress`	Nothing loaded, all documents were already there
`409`	`all_skipped`	Nothing loaded because of conflicts

List, read and delete documents

Operation	Request
List documents	`GET /api/v1/data/datasources/{name}/documents`
Read one document	`GET /api/v1/data/datasources/{name}/documents/{doc_id}`
Delete a document	`DELETE /api/v1/data/datasources/{name}/documents/{doc_id}`

Deleting a document also removes all its nodes from the search index.

The document list returns a map of content hash to document ID:

{
  "doc_hashes": { "a3f5e9d2b1c4…": "manual-cam45-alignment-reset" },
  "next_cursor": 100,
  "has_more": true
}

The list is paged. Use the query parameters limit (default 100, maximum 10,000) and cursor. Pass next_cursor from the previous response as cursor until has_more is false.

Use this list to find documents that no longer exist in your source system, and delete them.

Inspect nodes

Nodes are the searchable pieces of your documents. Inspect them to check how documents were split.

Operation	Request
List nodes	`GET /api/v1/data/datasources/{name}/nodes` (paged, `limit` up to 1,000)
Read one node	`GET /api/v1/data/datasources/{name}/nodes/{node_id}`
Find nodes by metadata	`POST /api/v1/data/datasources/{name}/filters?metadata_filters=…` with an RQL filter. Optional: `top_k` (default 10) and `sort`, for example `-created`.