JWCrawler(Java-based Web Crawler)

"JWCrawler" is a website management tool. It can search for all the html files and image files under certain website.

This program is built by me, you can download it HERE.

1. This program contains two files: spider.java, and HtmlFilter.java

2. How to compile and run it.

In command window, type:

   		javac jwcrawler.java
   		java jwcrawler
   		

You can change the seed URL text field at the top left corner of the window, or you can just leave it.

Click "Get Pages" button, wait a while. The results will be shown in the bottom left box.

Click an URL in this box, the detailed information about this URL will be shown in bottom right box.

If you want to save the results, click the "Save" button in right top corner, choose an html file, then click ok. You can open the saved file using web browser.

3. There are some options:

• Include Image: If this box is checked, the spider will search for html files and Image files.
• Inside Site: If this box is checked, the spider will not go out of the seed URL.
• MaxPages: The spider will stop after get this number of results.

A snap shot of the program is: