• BrikoX@lemmy.zip
      link
      fedilink
      English
      arrow-up
      4
      arrow-down
      1
      ·
      1 year ago

      They don’t pretent to be googlebot, they use their own crawler they just don’t share the name they use for it, so sites can’t exclude it with robots.txt. They just scrape the same sites that googlebot does, so if the site is excluded by googlebot they also skip it.