Once one of the only methods to track interactions with your site, log file analysis is not practiced as often as it once was – partially due to the technical proficiency it requires. With every visit to your site, your server records: the timestamp, the requesting server’s IP, the URL requested, the status code returned and more besides.

These interactions are tabulated and referred to as either: a ‘log file’, or ‘event log’, and represent an extremely data rich resource for analysis – both using tools and manually. Log file analysis allows us to see, amongst other things, the relationship between search engine bot frequency and page performance, the existence of status errors, overlooked or uncrawled content, performance and much more.

Using a minimum of 15 days of raw data – which is obtained from all of the web servers currently active for your domain – with no filtering or modifications applied, we employ a mixture of manual, spreadsheet, and software analysis to pick apart and piece back together a cohesive picture of how your website is performing at a fundamental level. There are an awful lot of SaaS tools available to SEOs today, but while they offer tremendous insight, there are none at present that covers absolutely all of the bases. For that reason, we believe that it still pays to make sure that this treasure trove of information is mined as part of developing the best SEO strategies. This analysis allows us to determine:

Which pages can or cannot be crawled, and whether (for large eCommerce sites) the crawl budget is being used efficiently.

Which response codes are encountered by the search engine spiders (user agents) during their crawls of your site – for example: 3xx, 4xx, and 5xx response codes can all be indicative of issues with site content and structure (if appearing unexpectedly, irregularly or too often).

If there are crawl shortcomings which could indicate underlying site-wide issues – such as problems with site hierarchy, or internal linking.

Whether there are sections of your site prioritised by search engines, and which they may consider the most important – this can lead to either an effort to reprioritise to the ‘right’ areas or the development of these sections as part of your ongoing strategy.

Log file analysis represents one of a number of strings to the modern SEO and should be judged as such. As it offers a wide scope for research and analysis, log file analysis can contribute to improving the visibility and usefulness of a website and, therefore, forms part of a toolkit that SEOs have been using to help brands earn ROI from their digital marketing activities.

In addition, as this is no longer a universally practiced technique, the chances are that a site with a routinely analysed log file can be sure that they are catching issues that their competitors are almost certain to miss, making it more likely that your brand will be better prepared for a battle in the SERPs.

click consult's seo team hard at work

While other agencies may have lost the knowledge of log file analysis to the mists of time, not only have we retained it, but we have continued to build upon it and create spreadsheets specifically to facilitate the analysis.

Our experts are passionate about analysis and we’re delighted to let them do what they love, helping our clients to succeed in search through a data driven approach to research, analysis and implementation – even at this most fundamental level.

Organic search, commonly referred to as search engine optimisation (SEO), employs a combination of creative and technical skills to improve the visibility of a website for commercially important key terms. In doing so, the aim is to increase the number of consumers exposed to a brand at vital points in the consumer journey, to increase traffic to the website and, therefore, increase consumer awareness and the number of commercial actions (purchases, downloads, contact form submissions etc).

Our team grows businesses organically

Enter a few details, and you can speak to our award-winning team of organic search experts.

Our latest insights

Our SEO videos

Facebook Twitter Instagram Linkedin Youtube