Website Crawling

Open Data Sources -> Website to add or update the website your agent learns from.

Choose the right starting URL

A root URL such as https://example.com/ can discover the whole site. A nested URL such as https://example.com/help/ stays under that path, so it is a useful way to keep an agent focused on a help centre or product area.

Set a page limit that is large enough for the useful part of the site, not simply the largest value available:

• The normal starting limit is 100 pages.
• Setup discovery can inspect up to 300 candidate pages when making recommendations.
• The largest value accepted by the UI is 5000 pages.

Larger crawls take longer and can pull in archives, faceted navigation, and duplicate pages. Start small, review the result, then increase the limit if useful pages were missed.

Follow the crawl

After you save, the Website page reports progress through states such as:

• queued
• crawling
• indexing
• complete
• error

Complete means processing finished; it does not mean every discovered URL was indexed. Redirects, unsafe URLs, pages without usable text, errors, and pages beyond the limit may be skipped.

If the crawl reaches its limit, raise the limit only after checking whether the missing section is useful. For very large sites, a focused path is usually better than indexing everything.

Review the result

Inspect the page list before testing answers:

• Keep definitive product, service, policy, pricing, contact, and help pages active.
• Disable thin pages, old announcements, search results, tag archives, duplicate locales, and content outside the agent’s job.
• Open suspicious pages in the inspector to confirm that the extracted text is readable and complete.

Keep it current

Run a new crawl after meaningful website changes. For an urgent policy change, update or disable the old source immediately; a direct FAQ can provide a concise answer while the website is being corrected and re-indexed.