Images from Juanico Environmental Consultants Ltd and US Dept of Energy
Did you know that while we are suffering from a glut of information -‘infobesity’ – with a typical search in Google yeilding thousands or millions of results, Google only actually searches one fifth of the available information on the internet?
This is called the surface web, and the most commonly used search engines trawl this area for information. The remaining four-fifths or 80% of the web is referred to as the deep web, the hidden web or the invisible web. It is made up of information locked away in password-protected databases, white papers, and grey literature.
According to the Australian Law Postgraduate Network, “The term grey literature refers to research that is either unpublished or has been published in non-commercial form. Examples of grey literature include:
- government reports
- policy statements and issues papers
- conference proceedings
- research reports
- market reports
- working papers.
Professional associations, academics, pressure groups and research institutes are only some of the sources of grey literature. Much grey literature is of high quality, although grey literature has generally not passed through the process of peer review…. Grey literature is often the best source of up-to-date research in specific areas. Another benefit of grey literature is that it is often written in an accessible style, providing a clear, concise introduction to difficult or complex topics.”
White papers, on the other hand, are “detailed, sometimes highly researched, documents intended to offer a much fuller picture of the capabilities of a product or company. Unlike an advertisement or press release, white papers are normally not promotional (though certainly some are) but rather, through strong writing and hopefully good research, these documents attempt to establish a level of credibility for a company and its products or services. Since many white papers are grounded in research these often contain good information, especially in terms of results of customer surveys, sales trends, and industry forecasts.” (KnowThis.com)
The last major part of the deep web, and the one most likely to impact on school students, is information contained in databases, both free and subscription. One of the best reasons to promote these databases to your students (generally the content in these is made up of journal articles, research papers, theses, multimedia and news archives) is because a person has checked these sites for reliability and included them in the database – as opposed to a robotic search engine which cannot discern if a site is relevant or not. To see if there is a free database available on a particular subject, type it and the word ‘database’ into Google. If there is one, Google will usually find it.
As well as looking for databases, there are many specialised search engines or websites to help you access the deep web. For academic research, Infomine from the University of California and ipl2 are two of the better web sites you can use, but also try these suggestions below:
Magazine & Journal Databases
This page is a guide to journal databases which are free on the web. Many subscription-only databases are also available through libraries, so contact your local library for details.
Discover over 70,000+ searchable databases and specialty search engines.
A comprehensive listing of dynamic searchable databases. Find databases with highly relevant documents that cannot be crawled or indexed by surface web search engines.
Search the Deep Net : Turbo 10 sends your query to over 800 specialist search engines.
Search for digital resources held across hundreds of university repositories.
Tools to Help You Use the Hidden Web
- CiteSeer: Scientific Literature Digital Library
- IncyWincy: The Invisible Web Search Engine
- Librarians’s Index to the Internet
Information from The Hidden Web Workshop
Invisible Web Research Tools
- Librarians Index to the Internet (look for Databases)
- Digital Librarian
- Library of Congress Online Catalog
- Union Institute A-Z Database List
- New York Public Library Databases Online
- SearchSystems Public Records Directories
- Turbo 10
- IncyWincy Search
- Gary Price’s Direct Search
- MedNets Medical Database Gateway
- MedBioWord: Science & Medical Journals & Databases
- Argus Clearinghouse
For more interesting reading see Michael Bergman’s article, The Deep Web: Surfacing Hidden Value and Invisible Web: What is it, how to find it, and its inherent ambiguity from UC Berkeley.