Introduction

Completly free spider or crawler for websites you will find many on the World Wide Web but very few of them provide both, a basic analysis functionality as well as a clear presentation of the data. The seoBOXX WebsiteSpider offers you completely free very fast and efficient spider-/crawl functionality and determines a lot of information of your domain in a very short time. Here both, the basic data of the domain and the data of each URL of the selected domain are analyzed and the data obtained is stored in a database file. The efficient MTWS-CoreEngine gives the WebsiteSpider the opportunety to run multiple analyzes in parallel and to produce the required analysis results quickly and efficiently.

The MTWS-CoreEngine

The problems of crawling or spidering lies firstly in the crawl speed and the other in the detail of the data obtained and their storage and optical presentation. The basis of the site is the WebsiteSpider is the MTWS-CoreEngine (MultiThreadedWebsiteSpider), the basic component of the WebsiteSpider and most of the other products. The MTWS-CoreEngine uses multiple threads to download the URLs source, analyses it, stores the collected data in a database and fills the URL list of the domain with the new found URLs. This fundamental component works very quickly and efficiently with its optimized source code and includes several algorithms for error correction for the accesses to the web servers, which include more and more mechanisms to quickly and consecutive regard multiple hits from an IP address as an attack.

The generated reports (SEO, Domain, Url)

The WebsiteSpider generates three different report types. The contents of the reports are not variable in this free version. The website spider generates, just as the WebsiteAnalyser, the following reports/report types:

• 1. SEO analysis report

  The SEO Analysis Report includes a complete summary of the domain data and any URL data. Here you will find statistical analyzes with associated diagrams for easy illustration of the analyzed data, as well as lists of missing URLs, URLs without keywords, URLs without title tag or broken links - a list of URLs which responsed the HTTP request with a http status code error - and much more. So you create an analysis report with a few clicks that helps you identify quickly and clearly the missing URLs of your website and much more.

• 2. Domain analysis report

  The domain analysis report in particular contains information relating to the domain itself. Also a small statistical summary of the analysis results is added to the report. The domain analysis includes the domain age, the server, the hosting, WHOIS information and much more.

• 3. URL analysis report

  The URL analysis report generates an overview of all data of the selected URL. Those basic information are the PageRank, HTTP status, keywords, title tag, META tags and H tags. Although the list of referring URLs (Parent) and the linked URLs (Child) is included in this report.