Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I would think CL using a proxy that silently blocks client ips that interact with the service suspiciously may be cheaper than sending out lawyer letters and more immediately effective. Just prefer technical solutions to legal threatening I guess..


For fun and as a learning experience, I built a simple CL scrapping web site. I set it up on a small aws instance about 6 months ago and after playing around with and making modifications/updates to it over the course of a month or two, I largely forgot about it. After seeing this thread, it dawned on me that I had not received an e-mail update on any of my search alerts for a while. I ssh'ed into the server and got a 403 with 'wget craigslist.com', so my guess is that they are doing some sort of blocking. Looking at the logs, the block probably started about a month ago.


Modifying the user agent to mimic a normal browser might solve the problem.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: