Monday, 18 October 2021

WebsiteSpider - Free website crawling with analysis functionality

seoBOXX WebsiteSpider Produkt Logo

Completly free spider or crawler for websites you will find many on the World Wide Web but very few of them provide both, a basic analysis functionality as well as a clear presentation of the data. The seoBOXX WebsiteSpider offers you completely free very fast and efficient spider-/crawl functionality and determines a lot of information of your domain in a very short time. Here both, the basic data of the domain and the data of each URL of the selected domain are analyzed and the data obtained is stored in a database file. So also the subsequent loading of the database and the evaluation or the generation of a report is possible at any time without reanalysing the domain.

The MTWS-CoreEngine

The problems of crawling or spidering lies firstly in the crawl speed and the other in the detail of the data obtained and their storage and optical presentation. The basis of the site is the WebsiteSpider is the MTWS-CoreEngine (MultiThreadedWebsiteSpider), the basic component of the WebsiteSpider and most of the other products. The MTWS-CoreEngine uses multiple threads to download the URLs source, analyses it, stores the collected data in a database and fills the URL list of the domain with the new found URLs. This fundamental component works very quickly and efficiently with its optimized source code and includes several algorithms for error correction for the accesses to the web servers, which include more and more mechanisms to quickly and consecutive regard multiple hits from an IP address as an attack.

Why using the seoBOXX WebsiteSpider?

Not only the fast and efficient MTWS-CoreEngine but also the visual presentation and reporting functionality of the WebsiteSpider makes him a first-class SEO tool for the onpage area. He works quickly and efficiently, and cost you nothing. A free SEO Tool with this functionality has no equal on the market. Since the MTWS-CoreEngine is also the basis for most other products from our company it is also used for the WebsiteSpider. Take a look at the detailed overview of the functionality and features of the WebsiteSpider in the detailled feature list.

The analysis procedure

Even during the analysis you receive a constantly updated view of the associated URL's as well as the HTTP status and the mime type of the respective URL. For better differentiation are analyzed URLs in green, being processed in blue and faulty shown in red. The analysis process takes multithreaded, which means that depending on the setting not only a URL to the other but several URL's are analyzed in parallel. This will increase the analysis speed by a multiple.

WebsiteSpider active analysis view

Furthermore you have the possibility to adjust the analysis parameters, the multithreading parameters and the HTTP access parameters in the settings to the last detail to meet your needs.

HTTP settings Threading settings Proxy settings UserAgent settings

The result views for the domain and URL data

The WebsiteSpider has different results views. On the one hand there is the detail view of the domain analysis. Here you will find all the information of the server, the hoster and website analysis statistics. Additional informations are GEO IP information of the server, the robots.txt and the humans.txt. The registration information of the domain are also available via reverse DNS lookup but remain, as all the data displayed, in the analysis database.

General domain data Domain statistics H tag distribution GEO IP data of the server location Reverse DNS LookUp robots.txt file humans.txt file FavIcon Mobile Touch Icon

Furthermore, there is a detailed view for each analysed URL. Here you will find all information and analysis results of this special URL. For better and faster overview the results have been provided with additional charts and graphics. From this view you can directly laod the source code of the URL into the integrated SourceCodeEditor. When loading the source code into the integrated SourceCodeEditor you can choose between the current source code from the web or the analysed sourcecode from the database. Take a look at the detailed overview of the functionality and features of the WebsiteSpider in the detailled feature list.

Link data Extended link data Link distribution META keywords Refering URLs Linked URLs H tag distribution META tags Redirects Connection protocol Link source code Image data

The generated reports (SEO, Domain, Url)

The WebsiteSpider generates three different report types. The contents of the reports are not variable in this free version. The website spider generates, just as the WebsiteAnalyser, the following reports/report types:

  • 1. SEO analysis report
    The SEO Analysis Report includes a complete summary of the domain data and any URL data. Here you will find statistical analyzes with associated diagrams for easy illustration of the analyzed data, as well as lists of missing URLs, URLs without keywords, URLs without title tag or broken links - a list of URLs which responsed the HTTP request with a http status code error - and much more. So you create an analysis report with a few clicks that helps you identify quickly and clearly the missing URLs of your website and much more.
  • 2. Domain analysis report
    The domain analysis report in particular contains information relating to the domain itself. Also a small statistical summary of the analysis results is added to the report. The domain analysis includes the domain age, the server, the hosting, WHOIS information and much more.
  • 3. URL analysis report
    The URL analysis report generates an overview of all data of the selected URL. Those basic information are the PageRank, HTTP status, keywords, title tag, META tags and H tags. Although the list of referring URLs (Parent) and the linked URLs (Child) is included in this report.

The integrated SEO Tools for image downloading

The absolute highlight of the built-in tools is the image and file downloader. He works as an assistant and guides you through the entire process. First, you can select the MIME Type of the data you want to download automatically. After making all the settings, the selected files are downloaded from the analyzed URLs into a directory of your choice, sorted in subfolders by MIME-Type. With this tool it is possible to download all images of a domain quick and easy into a folder. The WebsiteSpider only gives you the opportunety to download image files while the WebsiteAnalyser will provide full data support for any MIME-Type.

Image- and file downloader - Welcome Image- and file downloader - MIME type selection Image- and file downloader - Settings Image- and file downloader - Download progress Image- and file downloader - Summary with LOG

Conclusion and future

Through the large number of features, the visual presentation of the results and the integrated SEO Tool's the Webmaster and SEO is given a powerful product for the field of OnPage analysis to the hand, which will constantly being expanded and updated.


Feature request / Support request

Do you miss a feature or you miss a function? Do you have anything positive or negative? Let us know and we will see if we will integrate this feature in an updated version. Take advantage of our support form for feature requests.

Error report / Support request

You have found an error in the WebsiteSpider that occurred not just once and is reproducible? You want to send us this report so that we can eliminate the error in an update? Take advantage of our support form for error reports.