Indexation & Formatting in SERPs

One problem that seems to plague certain organisations is how their pages appear in the search engine results pages (SERPs). Whether it’s as simple as a page’s meta description being overwritten or as serious as a page appearing in the SERPs that shouldn’t be, there are a few things that can be done to reassure website owners that things like this won’t happen.

Don’t Rely on txt

Whilst txt is something that should always be used and set up correctly it’s not something that should be relied on to prevent indexation. For example, it’s a common misconception that using “Disallow: page”  will prevent certain pages from being indexed when, in reality, it just means that robots won’t crawl this.

It does prevent robots from crawling this page or directory but unfortunately it doesn’t mean that it won’t index the page. Instead, it won’t read any of the content on the page and, if there are links to that page, it’ll be indexed as a blank page, without any description and usually with a generic title such as “Submit URL”. Below is an example of Digg’s misuse of txt:

Solution: Use Meta Tags Instead!

Fortunately, there’s a really easy way of preventing pages from being indexed altogether – the “robots” meta tag!

Meta tags aren’t something that most people will think of being useful for anything now-a-days, but they’re far more useful than just writing an enticing meta description, they can also be used to tell search engine robots what to do. Below is an example of a robots meta tag in action:

<meta name="robots" content="noindex, follow" />

What the above is telling the robot is that it shouldn’t index the page (noindex), but it should follow links on the page (follow). This is particularly beneficial and is recommended on most pages that have valuable links but are too sensitive to be indexed because it still allows link juice to come from this page and distribute page rank effectively.

How to Use Robots Meta Tag

Below is a description of the different Robots Meta Tag arguments:

• noindex: This means the page is not indexed
• nofollow: This means all links on the page become nofollowed (an effective alternative to rel=nofollow on anchors).
• follow: This means all links on the page become followed.
• noarchive: This means the page is not cached by Google, particularly useful for regularly updated posts pages.
• noodp: This stops the description and titles tags being overwritten (only works on home page).
• nosnippet: This prevents google from generating a description based on the on-page text (Google only).

By using a combination of the above, it is possible to prevent search engines from indexing pages that it shouldn’t. For a quick fix on those pages that you don’t want indexed, I’d recommend using the following code:

<meta name="robots" content="noindex, follow" />

Download the Robots Meta Tag Cheat Sheet!

As a surprise, we’ve brought the cheat sheet download forward to this week! Download the Robots Meta Tag Cheat Sheet here.

Written by Jason John Mills