Search engines work a lot like old-fashioned library card catalog systems when it comes to scanning the estimated 50 billion-plus pages that comprise the Internet. Say you're searching for a book on how to write a business plan at your local library -- maybe in the days before card catalogs were computerized. First, you'd look in the "B" drawer of the catalog system, flipping through index cards until "business plans" appeared in a card's subject line. Then you'd jot down the shelf location of the book you wanted and search the stacks until you found your selection.
Internet search engines are based on a similar concept. Type "business plans" into a search engine and it will scan Web pages for matches, and then list them according to relevancy.
This is because popular search engines such as Google, Yahoo, and MSN don't search the World Wide Web directly. Instead, they scan proprietary databases that rank Web pages according to specific criteria established by each search engine company. The challenge for business owners is to make sure their sites meet the requirements for inclusion in these databases.
To classify the vast virtual library that is the Internet, search engine companies use software programs called "spiders."
The spiders aren't smart enough to type a URL into a browser window or use a search engines themselves to find material. To add new pages to a database, the spiders periodically crawl through links embedded in sites they've already cataloged, looking for fresh links.
The spiders then hand off the new pages to another software program, which scans them for key words and indexes them accordingly. As a result, when an entrepreneur searches for "business plans," the database can instantly serve up hundreds of thousands of results, ranked according to the search engine's relevance criteria.
Search engine relevance
To get your website listed at the top of the search results list, you need to know the nuts and bolts of how the different search engines work.
Link Quality: One way that the three largest search engines -- Google, Yahoo! Search, and MSN Search -- rank sites is according to the number and quality of links pointing to them. Link quality is determined by the importance of the other Web pages pointing to the site, i.e., Google deems pages that it's already given a high index rating to as more important than lower-ranking pages or pages that don't appear in its database at all.
Location, Location, Location: In addition to the frequency of keywords on a page, the location of those words is very important to securing higher rankings on most major search engines, according to Search Engine Watch, a website that analyses the search industry. That means putting the keywords near the top of the page, in headlines or maybe the first few paragraphs, stand the best chance of boosting your site in the rankings.
Other Tricks: Different search engines have other subtle differences that may impact how your company's Web pages are ranked. Some search companies index more individual Web pages and some fewer, according to Search Engine Watch. That's why you get different results when you plug in the same search on different search engines. There are also search engines that have gotten wise to the tricks of the trade and now penalize pages or exclude them from an index if they find out that a keyword has been repeated hundreds of times on a page to propel the page up in the rankings.