skilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 1 year agoX updates its Terms to prohibit crawling/scraping of its datastackdiary.comexternal-linkmessage-square50fedilinkarrow-up1361arrow-down122file-textcross-posted to: hakendran@feddit.de
arrow-up1339arrow-down1external-linkX updates its Terms to prohibit crawling/scraping of its datastackdiary.comskilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 1 year agomessage-square50fedilinkfile-textcross-posted to: hakendran@feddit.de
minus-squaredingleberry@discuss.tchncs.delinkfedilinkEnglisharrow-up15arrow-down2·1 year agoJust update robot.txt coward!
minus-squareIwasondigg@lemmy.onelinkfedilinkEnglisharrow-up5·1 year agoTook a look at their robots.txt, it appears to block all bots except Google.
Just update robot.txt coward!
Took a look at their robots.txt, it appears to block all bots except Google.