Open topic with navigation

# UseSitemapFromRobots

Some `robots.txt` files include the location of the sitemap(s) for the site. This parameter specifies whether the connector should look for sitemap URLs in the `robots.txt` file.

You can only set this parameter if you have configured a single start URL with the Url parameter.

If the connector does not find the URL of a sitemap, it crawls the site by following hyperlinks from the start URL.

Type: | Boolean
Default: | False
Required: | No
Configuration Section: | TaskName or FetchTasks or Default
Example: | `UseSitemapFromRobots=TRUE`
See Also: | Url

© 2018 EntIT Software LLC, a Micro Focus company | Send documentation feedback to Micro Focus Close Send documentation feedback To open the configured email client on this computer, open an email window. Otherwise, copy the information below to a web mail client, and send this email to swpdl.idoldocsfeedback@microfocus.com. We appreciate your feedback! Product: Topic Title: Help Topic: Feedback:

_FT_HTML5_bannerTitle.htm