Search Engine Workshops UK

Archive for the ‘spiders’ Category

There is a tendency for websites to get ever more complicated driven by better functionality, better design etc, but this is often at the expense of the site’s search engine friendliness. So have you looked at your site through Google’s eyes recently?

Googlebot (Google’s spider that visits your site and collects all the information on it that Google needs to match a page with a search query), is a simple fellow and can’t cope with images and other fancy stuff.

A quick and easy way of getting a good idea of what Googlebot sees is to

Turn off images

Turn off JavaScript

Turn of CSS

Disable Flash

then see what your page looks like.

Using Firefox

To turn off images and JavaScript

Go to

Tools → Options → Content → untick Load Images Automatically and Enable JavaScript boxes.

To disable CSS

Go to

View → Page Style and tick No Style.

Now just refresh the page.

Using Internet Explorer

I tried to do the same thing using Internet Explorer and my advice is to download Firefox if you haven’t already, it will be quicker. But if you really do want to try the same thing in IE, then

To turn off images

Go to

Tools → Internet Options → Advanced, then scroll down to MultiMedia and untick Show Pictures

To disable JavaScript

Go to

Tools → Internet Options → Security → Custom Level, scroll down to scripting and click disable javaScript.

Now restart IE.

Microsoft removed the ability to disable CSS in version 5 so the only way is to edit the registry files – not recommended!! – just use Firefox.

Using Firefox

To disable Flash

Disabling Flash is rather different. You can download a Firefox extension, FlashBlock.

This will then show a place holder wherever there is a Flash element on the page which, when clicked, will enable the Flash element and the page will then appear as the designer intended. FlashBlock is intended to block Flash generated ads and once downloaded you will need to click the place holder every time you load a page that contains a Flash element. This can be annoying when using Google Analytics, for example, since GA relies heavily on Flash.

Now what does your web page look like?

With images, javaScript and CSS turned off and Flash disabled, then what does your page look like?

If you can’t make head nor tail of the page, then probably Google can’t either, and if you can’t follow the links then you have real problems, because neither can Google and linking, including internal linking is key to search ranking success!

Also bear in mind that Google looks at a page by directly reading the code and not through a browser, so even disabled images, javaScript, CSS and Flash only gives a indication of what Googlebot sees – but it can be quite revealing though.

Search engine spiders don’t like to trip up on broken links and it is surprising how easy it is for them to creep in – particularly as sites grow or if there are several people responsible for updating content.

Webmaster Tools is a very useful tool and will certainly alert a website owner to major issues. However there are other tools that will do a much more comprehensive job of checking a site’s links. One such is Xenu’s Link Sleuth, this can be downloaded free. It goes through the site in a very similar way to a search engine spider and reports back

• broken links
• redirects
• local broken links – ie anchor links
• orphan links

It will also provide a list of URLs within the site and generate a sitemap, though any automated sitemap needs a lot of tweaking to be of any real value.

Broken link report

Xenu will go through a site and report all sorts of broken links including missing image links, missing javaScript files, style sheets etc which of course is valuable information for more than just keeping spiders happy. It will also check links to external sites so if the URL of a site your site links to has changed, this will be picked up. Again valuable information since you don’t want either your visitors or the spiders to meet a 404 when sent off to find a useful bit of information.

The local link report picks up errors on anchor links – these again are very easy to creep onto a page – change a paragraph and it is easy to forget an anchor link to or from the original.

Redirects

The redirect report shows just how technical a web page has become! Even quite a simple site is likely to have a myriad of redirects within the code. Many of these are not a problem – if you have any Flash on the site for example there will be a redirect that comes into operation if the visitor cannot open Flash. But it is worth just checking to see if there is anything that looks alarming and then to get it checked out with the IT guys.

Xenu and ecommerce sites

Xenu is a free tool and a good one but it does need to be used with caution especially on dynamically generated sites. Ecommerce sites are the most likely to cause problems. Dynamically generated sites can generate dynamic URLs and these can lead a spider to go round and round in circles – this used to quite a problem for search engines and ecommerce sites sometimes had difficulty in being spidered. Xenu can get caught in this sort of loop and this will cause a major headache for the site’s server and may well cause it to crash. So if you are running an ecommerce site, then your ecommerce package will probably include a link checker which shouldn’t have any detrimental effect on the system, so use this. It’s a bit like running Google Analytics on a site like Amazon, GA is a fantastic tool but there comes a point when a free tool is just not up to the job!