Part 3: Ensuring your website is found by search engine spiders
There are three main ways that search spiders can find your site. The first is if you go to the search engine, find the ‘submit your URL’ page, and enter it. The second is to have other web sites link to you, as these links provide a ‘bridge’ that spiders can cross to find your web site. The third is called ‘pinging’ and this is a new function that tends to be restricted to blogs. When a blog or web site built using blog technology updates it’s content, it sends an electronic message to search engines to tell them to visit (this site is a blog and does this). All of these methods of being found are valid, but search engines tend to prefer the latter two. More on why later.
How long does it take to be indexed?
This is a the proverbial piece of string question. Sometimes Google will crawl your whole web site and include it in it’s index the same day you launch it (happened to me last week), sometimes it will take weeks. Sometimes it will only crawl part of your web site immediately, and then crawl the rest of it weeks later. You get the point. Be patient, and keep building links into your web site, and this will improve your chances of being indexed.
Incidentally, if you want to see which of your pages are in Google’s index you can use the site command in Google. Go to the Google search box and type in site:www.yourdomainname.com (replacing yourdomainname.com with your actual domain of course!) and google will list all the pages from your web site in it’s index. Here is a site: query for my online vitamin e-commerce web site which lists all of the pages in the Google index. Alternatively for more information on your web site, you can use the info command info:www.yourdomainname.com and this will show you additional information Google has about your site, such as the number of inbound links. I’ve found this tool to be a bit buggy in that it usually fails to list all the pages, and often ignores scores of inbound links, but it’s interesting as a basic tool. There is a more involved tool from Google called Google Webmaster Tools which gives you great information about how your site is doing in Google’s index, and it even goes so far as to tell you which search queries you’re appearing against, and which search queries people are clicking on to get to you. To use this you will need to have a Google account (ie gmail) and a basic command of html or ftp as you have to ‘verify’ to Google that you actually own the web site before it will start to give you the indepth information, but I’d recommend signing up and giving it a go here: http://www.google.com/webmasters/tools/
Yahoo has it’s own Site Explorer tool, which actually is much more useful and provides more comprehensive results, and can be found here: https://siteexplorer.search.yahoo.com/
Now that we’ve considered how to be found by search spiders, and how to check whether all your pages are in the search engine index, we’ll take a look at getting your pages to rank highly in the search engine results in part 4.
Go on... click it... you know you want to!
One Comment
i am a webmaster myself and i always seek information on the net on how to increase website traffice and make a good website’”*