Log file analysis is one of the less commonly practised, but most data rich techniques available to SEO practitioners, allowing for the harvesting and review of important and otherwise hidden information that may present problems for search engine bots
What is log file analysis?
From every interaction, your server records: the timestamp, the requesting server’s IP, the URL requested, the status code returned and more besides. This table of catalogued interactions is referred to as either a ‘log file’ or ‘event log’ and, using this data, log file analysis seeks to identify (using tools or manually) such data as the relationship between search engine bot frequency and page performance, possible status errors, overlooked/uncrawled content, performance/time metrics and much more.
What Click Consult’s Log File Analysis entails
Using a minimum of 15 days of raw data obtained from all the web servers for your domain, with no filtering or modifications applied, and employing a mixture of manual, spreadsheet analysis and software analysis, the Click Consult team will identify the following:
- Which pages can or cannot be crawled, whether (for larger, eCommerce sites) the crawl budget is being used efficiently.
- Response codes encountered by the search engines during their crawl – for example: 3xx, 4xx, and 5xx response codes which can be indicative of issues with site content and structure.
- Identify crawl shortcomings that may indicate underlying site-wide issues – such as hierarchy, or internal link structure.
- Identify sections prioritised by search engines, and might consider the most important, which will then lead into a further ongoing strategy.
What a log file analysis can achieve
While there can be no conclusive answer applied generically, log file analysis allows experienced SEO professionals to uncover issues with information architecture, site structure and usability that no SaaS product can currently identify in full – though we do use the best available tools that can cover some of these bases.
Through thorough analysis, it is possible to improve visibility, usability and a host of related metrics that can suffer if issues are not identified and resolved – such as conversions, traffic, pages per sessions and more.