Websites
Last updated
Last updated
This knowledge can be used in the in the AI Agent builder
The Add Website feature in EasyBeam allows you to add knowledge from a website to your AI agent. This feature includes configurable encoding strategies and an optional web crawler to retrieve structured information efficiently.
To add website knowledge:
Open the Knowledge page (the book on the left nav)
Open the Add Website modal.
Enter the URL of the website you want to process.
Configure the encoding strategy and optional crawling settings.
Click Add Website to submit.
EasyBeam offers three encoding strategies to optimize how website content is chunked and stored for retrieval:
Precise
256
64
Technical documentation, code
Balanced
512
128
General content like articles and blogs
Context-Heavy
1024
256
Long-form narrative content, complex explanations
Each strategy balances chunk size and overlap to ensure effective retrieval when querying the AI.
The Website Crawler allows you to fetch multiple pages from a website by following links within a defined scope.
Toggle the Website Crawler (Beta) switch to enable crawling.
Configure additional crawling options.
Crawl Depth
Limits how many levels of links are followed (currently 1-2).
Max Pages to Crawl
Defines the number of pages the crawler will retrieve (min: 10, max: 500).
Use JavaScript
Enables JavaScript rendering for dynamically loaded content.
Crawl Scope
Defines which links the crawler will follow.
Same Domain
Only follows links on the same domain (e.g., example.com).
Same Root Domain
Follows links across subdomains (e.g., *.example.com).
All Links
Follows any external link (use with caution).
You can specify a URL pattern (using a regular expression) to limit the pages the crawler follows. Example: