Crawler4j

Open-source multi-threaded web crawler for Java. Simple interface; configurable depth, politeness, SSL, proxy, resumable crawling. Maven artifact edu.uci.ics:crawler4j.