Handling content dynamically generated via JavaScript


If your website uses JavaScript to generate the bulk of the content (also known as client-side rendering), our crawler will attempt to process the dynamically generated content on the web page (if such a need is automatically detected or the JavaScript processing option is enabled in the settings), however many JavaScript algorithms may not be processed.
JavaScript Processing Limitations:
  • Please note that our crawler does not load or process JavaScript code from any external sources whose hostname (domain) is different from the website's, such as scripts loaded from CDN or API servers, including website subdomains.
  • Our crawler does not process content that is dynamically generated by any specific user action, such as scrolling a page or clicking on an element.
  • Keep in mind that the MySitemapGenerator crawler only crawls links that are an HTML <a> tag with an "href" attribute, which also applies to content that is dynamically generated using JavaScript. Our algorithms do not recognize or process any other elements or events that function as links but are not a corresponding HTML <a> tag. This means that any other website navigation formats will not be processed, and therefore the content they point to will not be processed either.