Ex-Crawler

Configurable Java crawler with distributed/volunteer features. MySQL, MSSQL or PostgreSQL storage; plugin interfaces, socket server, webfrontend search engine; graphical client for distributed crawling.