Technology of the Month

If you maintain a website, you may have wondered at times why you are unable to find them through searches or that they are way below in the result set. The reasons for this are very complex. Different search engines employ different algorithms for finding the websites that best match the search term. Many of them do not reveal the exact algorithm because, website owners can then easily manipulate their sites to satisfy the algorithm to bubble their sites up.

Web search engines crawl the web using “spiders” at some unspecified intervals, start at the home page of a website and traverse a set of pages that are linked from there. Then they traverse links in each of those subsequent pages until they run out of links. When they examine the content of each of these pages, they perform “indexing” (a process that helps the search proceed faster) based on the algorithms that are proprietary. You can read more about how a Web Spider or Crawler works at Wikipedia.

The website owner can also use a facility that is honored by all search engines to protect a portion or the entire website from being indexed. You can read about how to do this at the Web Robots Page .

As you can imagine, trying to do this on literally millions of pages takes a very long time and this is why some pages may not be indexed for some period of time. Here, we provide some simple tips on how you can improve the searchability of your websites.

The common theme that will emerge from below is that as a content provider, you need to anticipate how the visitors to your website are likely to search for information contained in your web pages. As a first step, you can use your own search experiences in preparing your pages. So, avoiding terminology that is understood only by a small section of the readers is very important. For example, very Wesleyan-specific terms and highly technical lingo should be avoided if you want to attract a wide range of audience to your site.

A Short and Descriptive Title

It is important first to have a title for the document. Secondly, it should be short and describe the content really well. When you go to your web page, all the browsers show the title at the top of the browser window. If it is blank, you know you don’t have a title. All of the HTML editors such as Frontpage or GoLive provide a way for you to manage the page titles (if you view them as HTML, the title is enclosed between <title> and </title> tags).

We also encourage you to use [Wesleyan University] as a part of your pages that are hosted on Wesleyan servers.

If you do a Google search, the very first line you see is the title of the page. Make sure that the title describes the content of the page well. If not, refine the title until you are satisfied. Please remember that your changes will not affect the search results immediately.

Short Web Pages with vigorous content are better

There are two schools of thought on how to structure content – one long page that contains all relevant content, with navigational aids at the top (using name tags); or individual pages that address a specific topic.

In the former case, all content is accessible with just scrolling down a page; printing will require one to print the entire page; maintaining can be clumsy because you now have one long document to work with; loading time for such a document is longer and the reader may not be patient to wait;

In the second case, pages address specific topics, so they are concise; they require additional work in terms of structuring navigation; they tend to require more mouse clicks to move around; however, they load fast; printing can be easy if you are interested in specific topics, but if you want the entire set, it is harder;

It turns out that from the search perspective, the recommendation is to have short pages that are to the point – terse and vigorous. Obviously this is very dependent on what you are trying to do and may not be applicable to all instances. For example, if you have a long report or a published paper, then the approach to break them into shorter pages is not appropriate. However, having the abstract separately, with a link to the entire document would help readers find the content faster.

Linking is important

First of all, in order for your page to be indexed by a search engine, it needs to be linked from other pages. For example, at Wesleyan, there are departmental pages and individual websites that are available to faculty and staff. When you prepare a new website, make sure that it is linked to the appropriate page(s) rather than making it standalone.

In addition, the more linked your pages are form other sites, the better are the chances that it will be placed appropriately in the search results. Since linking is key to the search results, it is also important that you link other relevant institutional websites in your web pages. For example, if you find that a faculty colleague’s research and publication pages are relevant to the topic that you are writing about, you should consider linking those from your site. It is this collective linking that will benefit the entire community in terms of search and drawing more visitors to our web.

Finally, use descriptive texts as hyperlinks. In other words, rather than saying things like “click here”, say “click here to visit the Wesleyan University homepage”.

Use Structural HTML elements in organizing your pages

Many of us tend to avoid basic organization of web pages based on sectioning or even using the paragraph tags. From search perspectives, having some of the basic HTML structural elements such as section headings, text marked as strong (instead of simply bold), bulleted lists do make a difference and the text at the top of the page seems to make a lot more difference than the text at the tail end of the page. So, try to use these features whenever appropriate. Most HTML editors provide these features. For example, Section headings are the HTML equivalents of <h1> (Section) or <h2> (Subsection) type of tags which are rendered differently. Each of the suggestions in this document, for example, are coded with Subsection tags.

Hopefully, you will take some time to revise your website and apply the suggestions provided here. I would like to thank Adrian Cooke from University Communications for prompting us to write about this topic as well as provide pointers.