This FAQ was last updated on January 15, 2011
This is an unofficial document and the views and opinions expressed are those of the author alone.
This document is intended to provide an overview of how to design and promote a Web site so that it obtains good coverage in search engines (like Google) and directories (like Yahoo).
It is part of the Search Engine marketing pages that have been maintained for some years at sofer.com.
[to be added]
[to be added]
[to be added]
[to be added]
The document TITLE has the most significant effect on the visibility of a page. Not only will it have a direct effect on the ranking of a page, but it is the text that will be presented in search results. Make it short and relevant. Use words that reflect the theme of the page (i.e. the search phrase) and that will make your title stand out. Avoid prefacing your title with the words "Welcome to..." and resist the temptation to place your company name in the title of every page. Aim for the editorial brevity of a newspaper headline.
A meta-tag carries information about a document and is placed in the HEAD of the HTML page. An example of a meta-tag is the DESCRIPTION meta-tag; here's an example:
<meta name="description" content="Frequently asked questions about designing Web pages that are likely to have high search engine visibility">
Use the DESCRIPTION meta-tag, because by default this is what will accompany your entry in the results of most search engines. Make it short - about 150 characters or so - to ensure that it does not get truncated in search results, and make it interesting. Reiterate the TITLE and expand on the search phrase.
Use the KEYWORDS meta-tag. Not all seach engines use them, because of the potential for abuse (i.e. the stuffing with text largely unrelated to the content of the page), but they are strongly recommended. They should be used to reiterate the words used in the title and description, including synonyms, plurals, regional variations and even common mis-spellings. In general: use all lower case (because most searches do); don't use commas (they are unnecessary); and don't repeat words more than once or twice (because some search engines penalise excessive repetition).
The over-enthusiastic use of search terms in your TITLE, DESCRIPTION and KEYWORDS tags is self-defeating and is likely to dilute the rank of your site; it is usually better for a page to appear near the top of one search query than for it to appear lower down on several.
For this reason it is generally not recommended to target more than a single search phrase or, at most, two or three closely related phrases in the same page.
The first content that appears on every page should be a heading that reiterates the text of the TITLE. This should be followed by a repeat of the DESCRIPTION, and then the text that describes the main theme of the page. For all search engines, and particularly those that ignore meta-tags, the use of relevant text is essential. If you keep your pages on-topic, mindless repetition of keywords should not be necessary, although it is important that the key search words appear frequently in your text, particularly in the first couple of hundred words, and particularly in permutations that echo likely search phrases.
Make sure that navigation links and other text do not appear above your main text in the HTML source of your pages. If pages are designed with supplementary text in the left margin, then the rowspan tag should be used to move the main text to the top of the page:
<table border=1> <tr><td>[empty cell]</td> <td rowspan=2>main text (cell rowspan=2)</td></tr> <tr><td>extra text</td></tr> </table>
produces,
| [empty cell] | main text (cell rowspan=2) |
| extra text |
When linking to a page on your site, use the TITLE of the page as the link text. For search engines that measure the relevance of links, this is likely to help reinforce the relevance of the page to the search terms used in the TITLE.
Try to avoid representing any of your link text as images. If you insist on it, then use ALT tags, so that (at least some) engines can identify the text.
Some search engines distinguish between singular and plural versions of a word, while some don't, so, particular care needs to be taken where both versions are likely to be used in search terms.
If both singular and plural versions of words are to be targeted, then there are two options: either create two different pages that target the words separately; or, include both versions in the same page, which means including them both in the title and description of the page and using them both in your page copy.
Be very careful when using frames. Some indexing agents will ignore your frameset and go straight to any text that has been placed outside the frameset. It is important to ensure that relevent text and links are placed here. These will need to be maintained and updated in tandem with the visible content on a site. Remember, even if you do manage to get your pages indexed, visitors to your site are likely to be referred to an orphaned frame lacking the navigational structure carefully constructed for it.
Taking a broader view, expert opinion on Web design is broadly of the opinion that frames are a bad idea, in most circumstances (see Why Frames Suck (Most of the Time)).
Text contained in a Flash presentation will not be indexed by search engines. Flash should not be used for pages that you want to appear in search engines.
Also, avoid excessive use of JavaScript. It makes a page harder to maintain and harder for search engines to decipher. Put any JavaScript functions in a separate file and link it from the page header
By making your page source cleaner, it may also be marginally preferable to use a cascading style sheet instead of font tags to determine the way your text looks (see Effective Use of Style Sheets). It will probably make a significant difference to site maintenance, too.
By a wide and apparently increasing margin, Yahoo and Google are likely to be the most important in terms of generating traffic to your site.
Before submitting your site to any directories, you should be satisfied that the TITLE and DESCRIPTION of your home page reflect your most important search terms and that they convey the right impression of your site. This should be the text that is submitted.
Even though early in 2002 the $299 charge was made into a recurring annual fee, an entry in Yahoo is still likely to be a sensible use of even a modest marketing budget. For advise on how to submit to Yahoo, first read the Yahoo guidelines, then check out How to Get Listed in Yahoo!.
You should also add an entry to dmoz, the Open Directory Project. Dmoz is now used by several search services, including AltaVista, Hotbot, Lycos and Google, and it's free. Re-submit your site if doesn't appear on dmoz within a month.
You may also want to consider LookSmart).
If you've already submitted a different description to Yahoo, tough luck. It is very unlikely that you will be able to persuade them to change your listing.
You should submit your home page to the major search engines. This should include, as a minimum: AltaVista, Excite, Fast, Google, HotBot, Lycos, and Northern Light.
In addition to submission, you should also create a link to your new site from a page on an existing site that is already indexed. In many cases, this will be a more effective method of getting a site indexed than submission.
Manual submission is recommended. Increasingly, automated submission will not work. A very high percentage of submitted pages received by search engines come from automated services. As a result, the engines are developing strategies for preventing automated submissions and are also becoming less reliant on submissions, in general; instead, links from existing pages are of increasing importance. So, use automated submission services with care and, even if you submit manually, don't expect amazing results.
Some search engines, including Inktomi, Fast and AltaVista, offer paid inclusion services that guarantee regular indexing of content.
Without using paid inclusion services, sites can usually expect to be re-visited once every several weeks or so. Hence, for content that is subject to regular updates, paid inclusion offers a distinct advantage. For sites with relatively static content, the advantages may be more marginal.
For the obsessively inclined, paid services offer the opportunity for fast feedback on page tweaks; instead of waiting several weeks to see the effect of changes in page design or copy, the effects are likely to be observable within a day or so.
The cost of registering the home page of a site is modest enough that for most sites it is likely to be good value for money, particularly if the site is not yet indexed. It may also make sense to pay for pages offering time-dependent services (such as special deals or news).
Inktomi was the first to offer a paid inclusion service. Payment is made on an annual basis starting at around $30 per page. If you manage several domains or are likely to want to experiment with updating different pages, then the Inktomi Search/submit services offered by Network Solutions and Trellian offer best value for money (as of October 2001).
Fast PartnerSite has a banded pricing structure with 1,000 pages costing $100 per month.
A single page on the AltaVista Express Inclusion service costs $39 for six months.
A pay-per-click service will be good value for money if the cost per click does not exceed the value of those referrals. So, the value of the service will depend largely on the type of business you are in and the bidding level for the search phrases you are targeting.
Overture is unlikely to generate a large proportion of your traffic, but on more obscure phrases this can cost as little as 5¢ per visitor. This compares pretty favourably to costs of up to $10 per visitor for the average banner ad campaign.
If you think paying for traffic is a waste of money and your energy would be better spent learning the secrets of the search engine wizards, then just look at the bidding on the phrase "search engine positioning". Last time I looked, it was approaching $5 per click. If these companies knew a more cost-effective way of generating traffic, one supposes they would be using it.
Several services are listed under the Pay-Per-Click category in the Open Directory.
Google's AdWords service delivers simple text-only advertising based on keyword searches on the Google search engine.
Early in 2002 Google introduced a pay-per-click version of their service which is likely to supercede the earlier service based on page impressions.
With their not insignificant traffic figures, Google is likely to offer as good a destination for your online advertising budget as anywhere, although the usual caveats about assessing the value of each site referral apply.
The overall traffic levels on industry- and country-specific sites is likely to be orders of magnitude less than on the major engines and directories, but they are likely to be more targeted.
If there is a charge for inclusion in a directory, then, as with Overture, this cost should be balanced against the traffic generated.
The first places to look for search engine, directory and guide sites are Yahoo, Dmoz, and LookSmart. You may well find a category that lists topic guides for your industry. If you do, then check every site in the category for possible inclusion of your site.
Identify sites that offer complementary services which are likely to be of interest to users of your site; link to them and invite them to reciprocate.
In particular, do this with popular sites and do it with those sites that appear near the top of searches that use the keyword phrases that you are targeting.
If your site contains news, you might want to consider a syndication service like moreover or isyndicate.
Each search engine employs different algorithms for determining page position and these algorithms are subject to change over time. However, there are two broad generalizations that can be made about search engine positioning:
One: word frequency in a page is important, particularly in the page title and near the top of the page.
Two: the number of links to a page is important, particularly from sites that are in some way recognised as being authoritative and relevant.
Of course, the question of what it is that makes a site authoritative or relevant, from the point of view of a search engine, is an interesting one. different search engines address this issue in different ways.
Other criteria that may be significant, include:
PageRank has been defined by the google architects in The Anatomy of a Large-Scale Hypertextual Web Search Engine as follows:
We assume page A has pages T1...Tn which point to it (i.e., are citations). The parameter d is a damping factor which can be set between 0 and 1. We usually set d to 0.85. Also C(A) is defined as the number of links going out of page A. The PageRank of a page A is given as follows:
PR(A) = (1-d) + d (PR(T1)/C(T1) + ... + PR(Tn)/C(Tn))
Note that the PageRanks form a probability distribution over web pages, so the sum of all web pages' PageRanks will be one.
In other words, "a page can have a high PageRank if there are many pages that point to it, or if there are some pages that point to it and have a high PageRank".
Also, not only should links be sought from pages that have a high PageRank, but pages with fewer outbound links are likely to contribute proportionately more PageRank to pages that they link to.
PageRank is also affected by the size of the "damping factor", which the authors go on to describe:
"...PageRank can be thought of as a model of user behavior. We assume there is a "random surfer" who is given a web page at random and keeps clicking on links, never hitting "back" but eventually gets bored and starts on another random page. The probability that the random surfer visits a page is its PageRank. And, the d damping factor is the probability at each page the "random surfer" will get bored and request another random page. One important variation is to only add the damping factor d to a single page, or a group of pages. This allows for personalization and can make it nearly impossible to deliberately mislead the system in order to get a higher ranking."
In The PageRank Citation Ranking: Bringing Order to the Web, the Google architects also note:
"We have performed most experiments with a [damping factor] that is uniform over all web pages...although this technique has been quite successful, there is an important problem with it. Some Web pages with many related links receive an overly high ranking. Examples of these include copyright warnings, disclaimers, and highly interlinked mailing list archives.
"A compromise between the two extremes of uniform [damping factor] and single page [damping factor] is to let [the damping factor apply from] all the root level pages of all web servers. Notice this will allow some manipulation of PageRanks. Someone who wished to manipulate this system could simply create a large number of root level servers all pointing at a particular site."
Although the exact details of the algorithms employed by Google are not in the public domain and are going to be subject to change over time, it is reasonable to deduce from the above that, other things being equal, links from home pages are likely to carry more weight than links from elsewhere on a site.
Aside from the PageRank algorithm, Google also indexes anchor text (i.e. the text of a link), text proximity, and visual presentation details such as font size of words.
All this has several implications: Internal links on a site contribute to PageRank, as well as links from other sites; a link from a home page is likely to contribute more to PageRank than a link from other pages; the more links on a page, the less contribution each link will make to PageRank; and the link text is significant.
To sum up, the Google way of site optimization may be summarized as follows:
The phenomenon is described in some detail in an article in Microcontent News. In summary, it is the boosting of the position of a page in Google search results by linking to it with anchor text that is the same as the search term being targeted. This can have the effect of moving a page to the top of search results even when the search term does not appear in the text of the page.
There is a comprehensive summary of how Alta Vista works.
Probably the most relevant page for the purposes of site optimization is on link popularity.
AltaVista values:
One criterion that makes AltaVista distinctive is that it gives additional value to pages that act as hubs.
It also reiterates the likely advantage of separating important content into separate domains (and even placing content in the home directory).
An AltaVista optimized site is likely to include hub pages for each significant group of search terms, linked from the home page.
Search engines are increasingly using algorithms that attempt to identify the main themes of not just a page, but an entire site. This means that the rank of any one page can be affected by the content that is on other pages on the site.
A sensible response to this is to identify a core set of keywords across a site (or a section within a site) and ensure that those keywords are used in every page.
A new site can be set up as:
Some search engines almost certainly do differentiate between these three variations in distinguishing one site from another. Published documentation from both Google and AltaVista indicate that domains (and possibly sub-domains) are given more weight than sub-directories.
Moreover, separate domains continue to offer a marginal benefit in directory submission, where a site with a new domain is less likely to be rejected than a site created on an existing (and already submitted) domain.
[to be added]
[to be added]
[to be added]
[to be added]
[to be added]