NTENT Crawlers

The term "Crawler" is a generic term for any program that automatically discovers and downloads content from websites on the internet.

NTENT crawlers are proprietary programs owned and operated by NTENT. The information gathered from web sites by NTENT is indexed and made available via public-facing web sites.

List of crawlers used by NTENT:CrawlerTable

 NTENT crawlers honor robots.txt directives and partner publisher agreements. Our crawlers are intended to be polite and unobtrusive netizens. You can control access to NTENTbot using directives specified by the Robot Exclusion Protocol (see robots.txt page).

You may contact the NTENT crawler team at any time by sending an email to crawler@ntent.com.


The Content Team