Pholcidae

Pholcidae is a lightweight, small web crawler ("spider", hence the name), which crawls publicly accessible pages and following links. Pholcidae does read, cache and follow directions in any existent robots.txt.

Feel free to email me if you think Pholcidae is doing something it shouldn't!