web scraping - How can a webpage be made such that they can not be scraped by bots? -

-

this question has developed off answer here.

my question therefore steps can 1 take wend off standard scrapers?

in addition previous mentions of robots.txt, robots meta tag, , using more javascript, 1 of sure methods know of put restricted content behind user login. limit purpose-built bots. add strong captcha (like recaptcha) user login , purpose-built bots blocked too.

if site looking verify identity of client (ie: including whether it's bot), that's user-logins for. :)

user login's can disabled if strange activity detected.


Comments

Post a Comment

Popular posts from this blog

How do you change vbulletin's home page to the forums instead of the default (CMS or Activity Stream)? -

-
i have created website using vbulletin have installed cms , blog modules in in admin default setting forums when enter homepage url redirecting me http://www.demo.com/content/ content cms page please help for vbulletin versions < 4.2. need modify forum index page located in top level of vbulletin directory. based on url in question it's file: http://www.demo.com/index.php here contents of file, can choose whether cms or forum default home page. /** * can choose default script here. uncomment appropriate line * set default script. note: uncomment 1 of these, must * add // comment out script(s) not want use * default script. * * can choose default script if not plan move * file root of website. */ /** * use cms default script: */ require('content.php'); /** * use forum default script: */ // require('forum.php'); for vbulletin versions >= 4.2. there's new "navigation manager". you'll find control settin...
Read more

c++ - Troubles when compiling phash program -

-
here (probably) simple problem: i'm trying use perceptual hashing library phash ubuntu 11.10. had ffmpeg installed, way i've done: sudo apt-get install libphash0 sudo apt-get install libphash0-dev then tried compile program: #include <iostream> #include <phash.h> using namespace std; int main() { ulong64 myhash=0; ph_dct_imagehash("test.jpg", myhash); cout<<myhash<<endl; } when compiling, prints out: undefined reference `ph_dct_imagehash' any suggestion? should do? in advance! matteo monti you forgot link phash library, linker cannot find function. try adding -l phash gcc command line (or update makefile). if doesn't work, maybe you'll need specify library path (the location of *.a file) using -l "/usr/lib/"
Read more