Ingestion best practices

The quality of an agent’s answers depends on the quality of the documents it can search. These practices help agents find the right information.

Plan your datasources

• Group by topic and audience. Create one datasource per kind of content, for example ProductManuals and ServiceTickets. Agents can then search only what is relevant.
• Group by access. Access is controlled per datasource. Put documents that only some users may see into their own datasource.
• Describe them. Write a clear description. It helps you and others understand what belongs where.

Choose stable document IDs

The id_ decides whether a document is new or an update.

• Derive the ID from the source, for example the file path, the ticket number or the page ID in your source system.
• Never use random IDs or timestamps. Every load would create duplicates instead of updates.
• Keep the ID when the content changes. The document is then updated instead of added a second time.
• Use one ID per logical document. If you split a document yourself, for example by section, add the section to the ID: manual-cam45#alignment.

Design metadata

Metadata lets agents and applications narrow a search, and lets users see where an answer came from.

• Plan a schema per datasource and use the same keys and value formats for all documents.
• Use simple values: strings, numbers and booleans. Use ISO 8601 for dates, for example 2026-01-13.
• Add a source reference with title and url, so that citations show a readable name and link to the original. See Source metadata.
• Add the attributes users filter by, for example device model, plant, language or document type.
• Keep values consistent. CAM-45 and cam45 are different values for a filter.
• Do not put large text into metadata. Content belongs in text.

Prepare the text

• Send clean text. Remove navigation, headers, footers and repeated boilerplate before loading.
• Keep the structure. Headings and lists in Markdown help the search to understand context.
• Convert structured data. For JSON records, use the json_to_markdown or json_to_yaml transformation, so that the search works on readable text.
• One topic per document works better than one large file with many topics.

Splitting

By default, long documents are split into pieces of about 1,024 tokens with a small overlap. Each piece becomes a node and is searched on its own.

Set do_not_split: true on the datasource if each document is already a small, self-contained unit, for example a ticket, an FAQ entry or a product record. The whole document then stays together in search results.

Keep datasources up to date

• Load changes regularly. Sending unchanged documents again is cheap: they are skipped.
• Delete what no longer exists. Compare the document list with your source system and delete documents that were removed there.
• Send moderate batches. Requests are processed synchronously. Send documents in batches rather than all at once, and check the skipped list of every response.
• Test with real questions. After loading, run typical user questions through the search API and check whether the right documents are found.