Escolar Documentos
Profissional Documentos
Cultura Documentos
by
Dax R. Norman
National Security Agency
PGIP Class 0001
19 December 2001
The views expressed in this paper are those of the author and
do not reflect the official policy or position of the
Department of Defense or the U.S. Government.
ACKNOWLEDGEMENTS
daughter, who for two years worked and played one man short of a full team,
I would also like to thank LTC (ret) Karl Prinslow, at the time, a
contractor employed by the U.S. Army Foreign Military Studies Office, for his
Thank you must also go to my Thesis Chairman, Dr. Alex Cummins and
Thesis Reader Robyn Winder for their conscientious support of the Joint
ii
CONTENTS
List of Graphics……………………………………………………………………………..…. v
Chapter
Page
Validity Matters, 2
Credibility Counts More, 3
The Challenge of Credible Sources, 4
Assumptions, 6
A Unique Study, 6
Review of Thesis, 8
2. LITERATURE REVIEW……………………………………………………………...10
Range of Thought, 10
Every Man’s Printing Press, 17
Information Gaps, 18
3. METHODOLOGY……………………………………………………………………….19
4. FINDINGS…………………………………………………………………………...…..34
5. CONCLUSIONS……………………………………………………………..………….62
iii
Appendices
Bibliography………………………………………………………….………...…………..106
iv
LIST OF GRAPHICS
Tables
Page
…....54
…55
…57
57
….59
….60
…..61
v
10. Benchmark Web Site Evaluation Work Sheet, Spot…………………………
…66
69
…71
…73
76
101
Graph
…..61
vi
ABSTRACT
There is little argument today that open sources and the World-Wide-
Web have a role to play in intelligence, but little has been written about
collect open source intelligence from the Web; the ever increasing cost of
from every Web site of intelligence interest. Due to the enormous size of the
upon set of criteria for evaluating Web sites of potential intelligence value.
Credible Web sites can be identified. However, without these criteria, and a
method to share the results, hundreds of analysts will repeatedly find the
same Web sites of dubious credibility as other analysts; they will attempt to
evaluate the sites’ usefulness and credibility by many widely different
standards, and share their results with only a few close coworkers. The
quality of these Web site evaluations will vary widely based on the subject of
for their opinions of those criteria. From this survey the author developed a
weighted list of credibility criteria and a methodology that both the subject-
matter expert and the subject-matter novice will find useful. With these
the IC can evaluate Web sites within their area of expertise and share that
This thesis identifies valid criteria for evaluating the credibility of open
Web sites; identifies the target level of credibility for all intelligence sources;
offers a Web site evaluation worksheet; and compares the credibility of open
and although subject-matter experts are the best evaluators, any analyst can
evaluate a Web site when he does not have a subject-matter expert to assist
him.
CHAPTER 1
the Web contains every type of print, audio, and visual data from every type
Information analysts must sort the useful information from the junk.
However, what is useless for one person may be just right for someone else.
credible Web sites from untrustworthy, or non-credible Web sites. This thesis
used a survey structured to answer several key issues and the research
question: how to identify credible sources on the Web. The hypothesis was
that credible Web sites can be confidently identified by evaluating the Web
1
VALIDITY MATTERS
This thesis will also show that most analysts do not attempt to identify
credible sources, but evaluate the validity of the data in the sources. There
simultaneously relevant and meaningful. Validity can also refer to the proper
tool.2 This thesis uses validity as an attribute of data that is verifiably correct.
Validity is what the analyst means when he asks, is this data correct?
information rather than the source, and alone does not measure believability,
examine the data for consistency, verify it with other sources, or verify that it
1
G. & C. Merriam Co., Webster’s New Collegiate Dictionary (Springfield, MA:
G. & G. Merriam Co., 1975), under “Valid.” Cited hereafter as Webster’s.
2
Jum C. Nunnally, Psychometric Theory (New York: McGraw-Hill Book
Company, 1967), 75.
2
report, because organization focus changes, analysts change jobs, and there
This thesis will provide a tool for the general analysts to evaluate Web
Experts will also be able to use the recommended criteria and credibility
scale to evaluate Web sites in a consistent manner that other people will
or strategy, or politics very well for every data element included in his
subject, they rely on sources that they trust to provide valid data. This trust
they are not expert, or that are too complex for any one person to
3
See Appendix A, Web Site Evaluation Worksheet, for the relative credibility
scale, benchmark Web site evaluation worksheets, and a blank evaluation
worksheet.
4
Webster’s, under “Credible.”
3
understand. Because it is impractical for analysts to validate every data
sources. In the area of Open Source Intelligence (OSINT), this is even more
disciplines, and the multitude of unclassified open sources.5 The source must
be judged credible before the data can be judged valid. Of course this can
credible source than a valid data element. For example, it would be better to
know where to find a foreign leader’s official travel schedule, than to know
where the leader will travel next. This is true because this credible source
can tell one where the next trip will be, any changes to his next trip, and the
less important that every data element the source provides is validated.
information available to anyone for a fee or for free. OSINF becomes open
analysts are more likely to use the sources most accessible to them. The
5
Joint Chiefs of Staff, Joint Pub 1-02, Department of Defense Dictionary of
Military and Associated Terms, URL:
<http://www.dtic.mil/doctrine/jel/doddict/data/f/02542.html>, accessed 13 February
2000. Cited hereafter as Joint Pub 1-02. This thesis uses intelligence disciplines, such
as OSINT, as defined in Joint Pub 1-02.
4
Web has the potential to put a worldwide library on the desk of every analyst.
single search of the Web in seconds that would take a librarian a career to
complete. This is because the librarians know which sources are credible
the Web. How to identify credible sources on the Web is the challenge of this
thesis.
credible sources, and index them for everyone to use. However, even in such
research question that this thesis will answer is how to identify credible
sources on the Web. The focus is on Web sites because library science and
judge. With the help of expert Web searchers from industry, defense, and
5
are established which provide analysts with the tools and vocabulary to
ASSUMPTIONS
This thesis does make some assumptions. The first two are that open
credibility is relative to its intended use and user. For example, a CNN
but not sufficiently credible for basic intelligence for which the analyst has
some time to conduct research, or when the product will become the
A UNIQUE STUDY
Although other studies establish criteria for evaluating Web sites, such
Internet, I have not found a study that focuses on establishing the credibility
6
of Web sites.6 Cooke’s work is an excellent guide to evaluating the overall
quality of many types of Web sites. The closest Joint Military Intelligence
focuses on the accessibility and use of open source, not the credibility of
sources. Although Reva Basch’s Secrets of the Super Net Searchers includes
the question of credibility, it is less formal than this study and asks the
Super Net Searchers does not focus on any one issue, but asks many
questions of the industry experts. However, many criteria from Basch’s book
were included in the thesis survey used for this study. This thesis surveyed
establish common criteria for evaluating the credibility of Web sites.9 The
REVIEW OF THESIS
6
Alison Cooke, Authoritative Guide to Evaluating Information on the Internet
(New York: Neal-Schuman Publishers, Inc., 1999).
7
Major Robert M. Simmons, USA, Open Source Intelligence: An Examination
of Its Exploitation in the Defense Intelligence Community, MSSI Thesis (Washington,
DC: Joint Military Intelligence College, August 1995.)
8
Reva Basch, Secrets of the Super Net Searchers (Wilton, CT : Pemberton
Press, 1996).
9
E-mail Survey, “Joint Military Intelligence College Thesis Survey: Credibility
Criteria for Web Sites,” conducted by the author, July-August 2001. Hereafter cited
as Survey.
7
The research for this thesis began with a literature review, found in
Chapter two. From the literature several authors were selected who either
However, the literature also revealed tangent issues that influence how or
expert Web searchers for judging the credibility of Web sites. Those criteria
were included in the thesis survey, which was the primary research tool used
by the author.
intelligence analysts, coding the survey results and entering the data into the
the research questions and the key issues. The recommended credibility
criteria were determined by identifying the criteria that analysts most often
determining the relative weights for each criterion and a relative credibility
how the target source-credibility level was determined for most intelligence
products.
8
The results of the survey calculations are shown in the findings
organized to answer the research question and each key issue, which in short
sources; and the research questions of evaluation criteria, and needed level
of credibility,
The conclusions are in Chapter five, and include analysis of the survey
results. The thesis concludes that credible Web sites can be identified,
evaluated, and shared with other analysts. Known weaknesses in the survey
are mentioned in the findings and conclusions chapters. Chapter six also
used; the competed evaluation worksheets for the benchmarked Web sites;
9
CHAPTER 2
LITERATURE REVIEW
RANGE OF THOUGHT
Community Open Source Program Office.10 Most experts agree that OSINF
not been significant attention paid to the issue of identifying credible Web
expert opinions, and research projects from universities, just to name some
and industry. Therefore, the literature reviewed for this study included
in this study include: Robert David Steele of Open Source Solutions Inc.; Dr.
Wyn Bowen of Kings College, London, writing for Jane’s Intelligence Review;
Washington, D.C.; Reva Basch, author of Secrets of the Super Net Searchers;
10
Director of Central Intelligence, Director of Central Intelligence Directive
2/12 (Washington, D.C.: n.p., 1 March 1994). Hereafter cited as DCID 2/12.
11
Basch, 110.
10
and Allison Cooke, author of Authoritative Guide to Evaluating Information on
All these points of view agree that there is more data available now
than an analyst can manage unaided. Their approach is what differs. Steele
and Bowen would expand the Intelligence Community, which is not going to
happen without a long, and gradual culture change. Clift sees a need for
better automated tools for data retrieval, including an on-line index of open
sources .12 Cooke and Basch offer solutions for today: evaluate sources based
on criteria similar to those used for traditional print media. This thesis will
demonstrate that the ideas of each of these authors combined with the
unique knowledge and access. Steele would have analysts consult open
sources first, including subject experts in industry and academia, and then
12
A. Denis Clift, Clift Notes: Intelligence and the Nation’s Security
(Washington, D.C.: Joint Military Intelligence College, 1999), 51-57.
11
has proposed his own plan for intelligence in the 21st Century, called
Century.13 Steele sees a great need to expand the access that analysts have
several new groups, including scholars and business people, which constitute
the Virtual IC.15 It is these sources that Steele sees as the gold mine of
expand access to OSINF, primarily secondary sources, which are derived from
experts should be the people to evaluate Web sites, which is unique in this
sources, not the source of first resort as Steele suggests. Bowen, who is a
13
Robert D. Steele, Intelligence and Counterintelligence: Proposed Program
for the 21st Century, URL: <http://www.oss.net/OSS21>, accessed 5 January 2000.
Cited hereafter as Steele, Intelligence.
14
Steele, Intelligence, under “Introduction.”
15
Steele, Intelligence, under “Part III” Figure 18.
16
Steele, Intelligence, under “Part III.”
12
professor at Kings College, London, and writes for Jane’s Intelligence Review,
value found in the public domain. Bowen thinks that the role of OSINF is to
suggests. Bowen thinks the experts’ role should be to identify the useful
sources to keep and collect, (not specific data) and the worthless sources to
ignore. In his view, experts would also serve to evaluate sources for
17
Dr. Wyn Bowen, “Intelligence: A Valuable National Security Resource,”
Jane’s Intelligence Review, 1 November 1999, Dow Jones Interactive, “Publications
Library,” “All Publications,” Search Terms “Open Source Intelligence,” URL: <
http://djinteractive.com>, accessed on 4 March 2000.
18
Bowen, under “Technical Sources.”
19
Bowen, under “Conclusion.”
13
A. Denis Clift, President of the Joint Military Intelligence
College
Clift is also concerned about information overload, and sees a need for
D.C. His views are his own and do not represent that of the U.S.
intelligence. He also served as Editor for the United States Naval Institute
Clift gives a short explanation of the open source programs available today to
analysts, but does not indicate how accessible the information is. I observed
lines of analysts waiting to use Internet terminals in the JMIC library in 1999
Community (IC) that OSINF will only be used to its highest potential when it is
on the analyst’s desk. The work lost walking to a terminal down the hall or in
the next building is not worth the effort to analysts unfamiliar with the
sources, or inundated with other sources at their finger tips. Clift writes that
OSINF plays an important role in intelligence, and states that the IC already
20
Clift, 51-57.
14
(CIRC) of the National Air Intelligence Center and the Defense Scientific and
have within formation overload and the need for better-automated selection
tools.22 However, the technology Clift wants is not yet intelligent enough to
Basch does not address the Intelligence Community, but does address
the issue of how to select trustworthy Web sites. Basch, as well as Cooke,
takes the most practical approach to finding credible information in the flood
interviewing 35 of the best Internet searchers. In 1996, she was the news
editor for ONLINE, DATABASE, and ONLINE USER magazines and had been an
online researcher for about 21 years. Since then, she has published a series
of Super Searchers books. For Secrets of the Super Net Searchers she
21
Clift, 54.
22
Clift, 56.
15
affecting online researchers and included the following, which relate to Web
23
site credibility:
• How are the quality and reliability of unfamiliar Web sites judged?
credible. Susan Feldman stated in Super Net Searchers that a “Web site
your own devices to analyze and evaluate.”24 However, Mary Ellen Bates, also
conference in Virginia on 10 May 2001 that she does not rely on personal
Cooke also does not address the Intelligence Community, but does
address the issue of how to select trustworthy Web sites. Cooke also
23
Basch, 3.
24
Basch, 31.
25
Mary Ellen Bates, Presentation to WebSearch University Conference in
Reston, VA, 10 September 2001.
16
recommends using evaluation criteria similar to that used for print media,
author’s implicit thesis is that although there is much useless, outdated, and
can be found and the quality can be assessed.26 Like Clift and Bowen, Cooke
carefully evaluate Web sites using criteria similar to criteria used to evaluate
print media.
media. These criteria include the reputation of the publisher and author,
criteria work well when the number of publishers in a particular field are
quantifiable and their past work can be located and reviewed. However,
26
Alison Cooke, Authoritative.
27
Jan Alexander and Marsha Tate, “The Web as a Research Tool: Evaluation
Techniques,” Wolfgram Memorial Library, Widener University, Chester, PA, URL:
<http://www.science.widener.edu/~withers.evalout.htm,> accessed 13 March 2001.
17
benefit of peer or editorial review, or regard for brand name reputation.
Among the millions of Web pages available to the public today are many of
INFORMATION GAPS
Basch, Cooke or Alexander, the issue of credibility still remains. How does a
subject-matter novice know which sources he can believe? The other issue is
that of relativity. Is a Web site that is credible enough for a high school term
18
CHAPTER 3
METHODOLOGY
organized by key issues and research questions. The key issues are
answered. The key issues include: how is open source information relevant
analysts biased toward popular source titles; are foreign sites in English less
credible sources on the Web, it was necessary to separate the question into
two parts. The first part of the research question was what criteria can be
use to identify credible Web sites. The second part of the research question
was how credible should any intelligence source be. The methodology relies
on logic, and statistics, and is somewhat complex due to the many steps
Even before the survey could be developed, the basic question needed
19
literature review provided several views on the role of open sources in
intelligence. The opinions of Steele and Clift offered convincing reasons that
intelligence must include open source information. The reasons for using
SURVEY DEVELOPMENT
sources on the Web, this thesis needed to answer several key issues
question. Two research methods were used to answer the key issues and
online DIA course material, Lexis-Nexus, Dow Jones Interactive, the NSA
Library, and academic Web pages. This literature review uncovered some
answers to the key issues and provided the majority of the concepts tested
were made and the second draft was given to Professor Jerry P. Miller,
20
survey questions to maintain Likert-type scales for the responses. Likert
options that are understood by most populations and can be used to measure
The second draft was also sent to LTC (ret) Karl Prinslow, project
150 military reservists who work via telecommuting to collect and acquire
recipients were able to display the survey on their computers, and would be
anyone who was able to receive the E-mailed survey to respond to it without
special software.
separate surveys were distributed by E-mail. In the coding and analysis, the
two surveys were treated as one survey, with some questions not applicable
several questions at the end, which would not apply to industry or academia,
who have an interest in open source intelligence (OSINT). The exact number
21
of IC analysts cannot be determined because it was sent to a mail-list, which
often changes. This method had the effect of randomizing the population
analysts. Four of these 18 E-mailed the survey to 238 others, for a total of
356 IC analysts. This chain of events was evident from the E-mail headings
and some respondents informed the author who forwarded the survey to
to participate in the E-mail Industry Survey. The Industry Survey was then E-
Defense analysts E-mailed the survey to about 120 other defense analysts. A
total of about 179 analysts are known to have received the Industry Survey.
Together, the two surveys reached about 535 analysts who have an interest
The survey was structured to answer several key issues and the
28
Appendices B and C include a copy of the E-mailed surveys.
22
8a through 8r. Question 8 listed the criteria most often mentioned by
8. How much credibility does each of the following factors add to the
total credibility of a Web site? Use the following scale:
first asked to list the criteria they currently use; they were later asked to
29
Survey, questions 6 and 8.
23
evaluate the list of criteria in questions 8a through 8r. If the survey
population had been asked about specific criteria (question 8) before being
asked what criteria they actually use (question 6), they may have been
influenced to include the listed criteria from question 8 as criteria that they
analysts revealed that there were criteria that analysts would use only after
they were told of them. Discussions with analysts prior to the survey
development had also revealed that many analyst do not know how they
determine what is a credible source, and that many analysts may only
about the difference between data validity and source credibility. The
most-often chosen) for each criterion in survey questions 8a through 8r, and
of variance would indicate little agreement among the analysts. Only criteria
included in the recommended criteria list. This means analysts most often
24
believe (mode) that the satisfaction of any one of these recommended
Then the arithmetic mean (average) credibility was calculated for each
value. The relative value is how much more important, on average, analysts
think one criterion is than another criterion. The assumption here is that
such attributes are cumulative, and the more recommended criteria a site
these criteria were calculated, and those criteria that were suggested by 50
scale in the survey, they were arbitrarily assigned the average relative value
any criteria not included in question 8, but also did not significantly affect the
25
Step 4. From question 6, added to the list of recommended criteria,
those criteria not already on the recommended list, and that had a mean
occurrence of 50 percent or greater (at least half the analysts listed the
criteria).
Step 6. List all the recommended criteria and their individual mean
credibility as their relative values. 30
The criteria’s relative value can then be used to evaluate a Web site.
When evaluating a Web site for credibility, the relative values can be
summed for the criteria that the evaluated site satisfies. The site credibility
score can be compared to other known credible sites listed latter in this
Even after an analyst has calculated the credibility score of a Web site,
9. How credible must an intelligence source be to use its data in the following
intelligence products? Use includes when you would use qualifiers such as
"possible survived". Choose the required level of credibility for each type of
intelligence.
30
See Table 1 in the findings chapter for the list of recommended criteria and
their relative values.
31
See Appendix A for the benchmarked Web site evaluation worksheets.
32
Survey, questions 9a – 9f.
26
Scale:
___7) No Opinion
___6) 100 percent Credible
___5) 75 percent Credible
___4) 50 percent Credible
___3) 25 percent Credible
___2) 10 percent Credible
___1) 0 percent Credible
Analyst were the asked to choose the required level of credibility for:
was calculated and is the product-credibility levels, which are shown in Table
converted into a score so that analysts can simple add the results of an
for sources that analysts use for a particular intelligence product. When a
potential Web site is evaluated, the analyst calculates the credibility score of
levels in Table 2. The sum of the evaluated Web site should be at least equal
the table. The source-credibility level of each intelligence product type was
Web site’s score which would equal the product-credibility level that was
27
Web site evaluation, which also demonstrates how the product-credibility
Example:
the credibility of a Web site. This process is based on the theory that the
analysts are the best criteria for evaluating Web sites. The weight or relative
value of each criterion is based on the average score given the criterion by
28
KEY ISSUE: OFFICIAL CREDIBILITY CRITERIA
official criteria are for evaluating the credibility of sources. However, after
were aware of such a policy. It reasoned that if analysts were not aware of
such a policy, its existence was irrelevant. The survey results of this question
Community. The lack of such criteria may call into question the consistency
5. Does your organization have official criteria that you are told to use
for determining the credibility of any source? "Any source" means published,
proprietary, and classified sources. Choose all that apply:
33
Survey, question 5.
29
Discussions with analysts and the literature review indicated that well-
known publication titles are perceived as more credible than obscure titles,
even though the analysts may have never seen the well-known titles.
titles. This key issue was answered by comparing the well-known titles in
7. How credible are the following information sources given only their
titles? Choose one from the following scale:
Well-known Titles:
a. NY Times
b. Washington Post
c. Harvard.edu Web site
j. NationalGeographic.com Web site
k. JanesDefenseWeekly.com Web site
l. InformationWeek.com Web site
m. DowJonesInteractive.com Web site
Obscure Titles:
d. RussianArmy.ru, Web site in Russian
e. RussianArmy.ru Web site in English
f. IsraelIndependentNews.is Web site in Hebrew
g. IsraelIndependentNews.is Web site in English
h. FrenchIndependentNews.fr Web site in French
i. FrenchIndependentNews.fr Web site in English
34
Survey, questions 7a – 7l..
30
However, there was a problem with how this question was structured
and the findings may not be valid. Judging from the comments in the
surveys, it was evident that analysts were not able to make credibility
judgments for many sources based on titles alone either because they had
alone.35
and comparing 7h to 7i. The validity of these questions was preserved by not
including any real publications or Web site titles, which the analysts may be
familiar with.36
7. How credible are the following information sources given only their
titles? Choose one from the following scale:
35
Survey, questions 7a – 7l.
36
Survey, questions 7d – 7i.
31
e. RussianArmy.ru Web site in English
unclassified sources and less often, how do classified sources compare to one
another. This is a comparison that is likely to change over time. One JMIC
of favor as access success improves for one source or another. These issues
were only included in the IC Survey and most analysts answered as though
7. How credible are the following information sources given only their
titles? Choose one from the following scale:
37
Survey, questions 7n – 7s.
32
7r. SIGINT reporting
7s. MASINT
range for all sources included in question 7, and compared them to each
ETHICS
PRIVACY:
You do not need to include your name; however, if you choose to
include your name, it will only be used by me to contact you if I need
more information regarding your comments. I will not quote you
directly unless you indicate in Questions 3 and 4 that I may do so.
Otherwise, only me and my Thesis Chairman, Professor Alex Cummins
… will have access to respondent names. Any record of the names in
association with the responses will be destroyed after the research is
completed, except those names included in the thesis with
permission.39
38
See Table 7, and Table 8 in the findings chapter.
39
Survey, Privacy.
33
CHAPTER 4
FINDINGS
review that could answer the research question and the key issues. Then the
results of the survey are described , followed by how these results answered
the research question and the key issues. The survey determined what
showed that analysts see only a small difference in the credibility of open
40
Survey, question 6.
41
Survey, questions 7a through 7s.
42
See Table 8 in findings chapter for comparison of classified and
unclassified source credibility.
34
Just as useful as the credibility criteria is the credibility scale
Web site. The survey results also determined a target level of credibility for
credible and non-credible Web sites validated the criteria and demonstrated
debate. Steele suggests that analysts should reference OSINF first, and then
collection to fill the intelligence gaps.44 This approach would acquire data
from the least expensive sources first. Steele calls for 5 percent of the
Open sources include what is already publicly known about a subject, and
35
report, and should be considered before any classified collection is
people at risk for information that may have been found in a foreign Web
site, foreign newspaper, or company brochure. These open sources can also
more expensive than open sources, open sources should always be the first
too low. Therefore, OSINF affects the cost of intelligence, the timely access
time and money spent by countless analysts’ attempting to sort the useful
observed that every analyst who makes use of Web sites for open source
intelligence must rediscover which sites are useful and credible, even though
an expert at another agency or just down the hall may have already
evaluated the site. Also, when a Web site is recommended by one analyst to
another analyst, there is no consistent way to evaluate the Web site and
46
Bowen, under “Collection Strategy.”
36
methodology to evaluate Web sites and consistently communicate that
of expert credibility criteria for Web sites, which were incorporated into the
Basch that one should evaluate Net sources the same as one would print
47
Survey, questions 8a – 8r.
48
Basch, 9.
49
Basch, 9.
50
Basch, 9.
37
• Bias. The researcher must understand the source’s bias.51
• Expert opinion.
subject experts.55
51
Basch, 9, 15.
52
Basch, 31.
53
Basch, 48.
54
Basch, 9.
55
Basch, 31.
56
Basch, 31.
57
Basch, 31.
38
• Subject area Web pages created by subject librarians are a good
• Origin.
corroboration.61
58
Basch, 139.
59
Basch, 40, 110.
60
Basch, 9.
61
Basch, 16.
62
Basch, 9, 96.
63
Basch, 132.
39
• Format. Is the source professionally formatted, indicating attention
much easier than print publishing once was. For this reason, I would not
give Web site format the same weight as print media format.
• Association.
government?65 Although I think industry is often the best source for some
credible sources.
• Reputation.
academic Web sites. However, Web site evaluators need to verify that an
academic Web page represents the institution and not just a student.68
64
Basch, 132.
65
Basch, 31.
66
Basch, 49.
67
Basch, 31.
68
Basch, 132, 224.
40
• Know which publishers, universities, or companies are well
• Attribution.
• Does the source clearly identify its self and its purpose? 71
• Indications of the source include the text of the Web site, the name
of the Web server in the URL, and the directory name in the URL, which
• I would also recommend viewing the Web site’s HTML source code
for revision dates, and statements of attribution not shown in the Web
site’s body.
• Motivation.
69
Basch, 110, 137.
70
Basch, 32.
71
Basch, 16.
72
Basch, 140.
73
Basch, 140.
74
Basch, 77.
41
• The presence of a counter on a Web site indicates the author cares
that people know that other people like his site enough to visit it.75
popularity is how many and which other Web sites include links to the
should indicate his own level of knowledge in the topic area. This
intelligence products.
contributed to the thesis survey question 8, which asked how much does
Internet included three areas: what is high quality information, how to find it,
and how to evaluate it. Each of these areas contributed to the development
75
Basch, 132.
76
Basch, 133.
42
of relevant questions in the thesis survey. On the topic of high-quality
information, Cooke explains that some of the most common problems with
• information overload
• outdated material
do not even include, databases, news services, and FTP sites. The citation
academic publishers, people publish because they can, not because they
amount of time to sort through. A useless site can have all the gloss, format,
explains that of forty WWW medical sites evaluated, only four included the
this level of inaccuracy is possible because Web authors are their own editor
77
Cooke, 89.
78
Cooke, 12.
79
Cooke, 62.
43
and publisher, allowing no opportunity for critical review which most scholars
Methods for finding data on the Web are unique to the Web and online
of:
• search engines
Cooke explains that search engines such as Excite and Lycos (or
relevance ranking, and are not useful for finding nor evaluating sources for
quality. They are also generally limited to Web sites and index every page on
every site, further multiplying the number of results per query.80 I have
observed that some search engines such as Google have resolved this
multiple indexing of a single site by displaying only the first indexed page,
Cooke also writes that subject catalogs and directories such as Yahoo
and Galaxy are more useful because site authors write the site descriptions;
catalog experts choose the hierarchy category to place the site; and only
sites are indexed, not every page. However, these sites are still very large,
and because the indexing is done by people rather than machines, as is the
80
Cooke, Chapter 2.
44
case with search engines, Web site directories are not revisited as often and
Cooke also wrote that rating and reviewing services use different,
usually unpublished criteria for rating the best sites. These include
are even better yet for finding high-quality sources because a person other
than the author has reviewed the site based on some criteria. However,
for content or accuracy, and the evaluators are not subject-matter experts.83
Cooke believes that the best place to find high-quality sources is from
methods used in libraries. They are often subject-matter specific and site
for several internet source types. The criteria can be used for overall
evaluation of Web sites, not specifically for credibility as this thesis does.
Cooke’s criteria are based on surveys of hundreds of internet users, and were
81
Cook, Chapter 2.
82
Cook, Chapter 2.
83
Cooke, Chapter 2.
84
Cooke, 92.
45
validated by professional librarians. The unique evaluation criteria for each
criteria, included:
• databases
• FTP archives
• FAQs
authority and reputation of the institution within its field, as well as the date
the page was last updated.85 Criteria for a subject-based Web site include
the purpose of the site, comprehensiveness, and whether the page includes
electronic journals and magazines include the site’s authority and reputation
as well as whether the site has been referenced by a known reputable journal
85
Cooke, 90.
86
Cooke, 97.
46
that filters its own articles for accuracy.87 These criteria were included in the
The primary purpose of the thesis survey was to identify criteria for
through 8r. Those criteria, which analysts most often gave a credibility value
respondents. The first two were not recommended by experts, but were
calculated for each recommended criteria from question 8. The mean then
87
Cooke, 98.
47
The criteria recommended in survey question 6 were then listed, and
criteria from question 8, those criteria from question 6 that were not already
greater (at least half the analysts listed the criterion). Surprisingly, there
open survey question number 6. The criteria that were mentioned most
question 6.
the relative values of each criterion, and which criteria were chosen for
recommendation.89
88
See Table 15. Survey Question 6: Personal Criteria Analysts Currently Use
to Determine Credibility.
89
Survey, questions 8a – 8r.
48
8a. Recommended by 66 0 4.94 5 Yes
subject-matter expert in the
topic of the Web page.
8b. Recommended by a 65 1 3.65 4 Yes
generalist.
8c. Listed by an Internet 63 3 3.56 4 Yes
subject guide that evaluates
Web sites.
8d. Listed in a search engine 64 2 2.39 1 No
such as AltaVista
8e. Listed in a Web directory 62 4 2.65 2 No
organized by people, such as
Yahoo.
8f. Content is perceived 64 2 3.78 5 Yes
current.
8g. Content is perceived
accurate. 63 3 4.56 5 Yes
8h. A peer or editor reviewed 65 1 4.52 5 Yes
the content.
8i. Content's bias is obvious. 65 1 3.06 4 Yes
8j. Author is reputable. 64 2 4.64 5 Yes
8k. Author is associated with 65 1 4.42 5 Yes
a reputable organization.
8l. Publisher or Web host is 65 1 4.02 5 Yes
reputable.
8m. Content can be 65 1 5.17 5 Yes
corroborated with other
sources
8n. Other Web sites link to, or 65 1 3.68 5(b) Yes
give credit to the evaluated
site
8o. Server or domain is 65 1 3.45 4 Yes
copyrighted or trademark
name, like IMB.com.
8p. Statement of attribution. 64 2 3.78 5 Yes
8q. Professional appearance 65 1 2.86 4 Yes
of Web site.
8r. Professional writing style 64 2 3.16 3 No
of Web page.
(a) (a) Table Explanatory Notes. Mode Values: 1=0 percent, 2=10 percent, 3=25
percent, 4=50 percent, 5=75 percent, 6=100 percent credible. Mode is the
most-often chosen score respondents gave each criterion. Only modes of 50
percent credible and higher are recommended. The Mean is the average score
respondents gave each criterion. The Mean is assigned to each recommended
criteria as their relative values which are latter summed when evaluating a Web
site.
(b) (b) Multiple modes exist. The smallest value is shown
49
recommended criteria to known credible, and known non-credible Web sites,
establish the high-end of the relative credibility scale. The relative values of
each criterion that the site satisfied were then summed for the site’s relative
credibility score. Then the average of the three credible Web sites was
than it was to find known non-credible Web sites to evaluate. This was
because it did not seem useful to benchmark a Web site so obviously non-
credible that no analysts would consider using it, negating the need for an
evaluation at all. Due to this difficulty, only one non-credible Web site was
KoreanNewsSite was selected because the author had evaluated this site for
a previous research paper and had found it non-credible, and yet a challenge
to evaluate. The challenge to evaluating it came from its mix of very credible
publisher. The key points that made the publisher’s articles non-credible
50
The figures below represent the relative credibility scale and how these
Web site should rate a relative credibility score of about 46.75, and a non-
51
Benchmark Credible Web sites Evaluated Score
Spot Image Corporation, www.spot.com 43.19
International Telecommunications Union, www.itu.int 48.24
NY Times On the Web, nytimes.com 48.82
Average Score 46.75
analysts must also know what the target or required level of credibility is for
7) No Opinion
6) 100 percent Credible
5) 75 percent Credible
4) 50 percent Credible
3) 25 percent Credible
2) 10 percent Credible
1) 0 percent Credible
90
Survey, questions 9a – 9f.
52
9f. Warning, an alert to take action
credibility level for six types of intelligence products. The mode was
calculated for survey questions 9a – 9f. The mode is the most-often chosen
required level of source credibility. The statistics indicate that most analysts
percent credible.91 This was a surprise because the author expected to see a
requiring the greatest level of credibility. This presumption was based on the
belief that analysts require less information about an imminent threat than
impact of ignoring the least threat is so much greater than ignoring the most
products should be 75 percent credible. If the most credible Web sites have a
91
See Table 2.
53
Table 2. Questions 9a-f. Required Level of Source Credibility for
Intelligence Products.92
Number of Required Credibility
Cases
Valid Missing Mode Range
(b) percent percent
9a. Research, special topic 35 31 50 0-100
summaries percent(a percent
)
9b. Current, day-to-day 35 31 75 0-100
developments percent percent
8c. Estimative, identifies trends or 35 31 75 0-100
forecasts opportunities or threats percent percent
9d. Operational, tailored, focused, to 35 31 75 0-100
support a military, intelligence, or percent percent
diplomatic activity
9e. Scientific or technical, in-depth, 35 31 75 0-100
focused assessments of trends or percent percent
capabilities
9f. Warning, an alert to take action 35 31 75 0-100
percent percent
Required-credibility level for all 75
Intelligence Product Sources percent
(a) Multiple modes exist. The smallest value is shown. Just as many
respondents chose 75 percent.
(b) Missing responses are primarily because non Intelligence Community
personnel were not asked these questions in the survey. Mode is based on
valid responses.
you are told to use for determining the credibility of any source? "Any source"
54
here is that only criteria formally sanctioned by the organization are likely to
analysts are either not aware of official credibility criteria or do not think such
criteria exist in their organization for unclassified sources, and 70.4 percent
Many analysts commented that they rely on their own or other expert
opinions to determine source credibility, and official criteria are not needed.
In many cases this may be true; however, in a large organization there are
many levels of expertise, and without criteria and standards overall reporting
takes on the credibility of the least qualified analyst. Without such criteria,
demonstrate that the vast majority of analysts are not aware of source
55
Table 3. Question 5. Part 1, Official Criteria for Unclassified
Sources.94
Number Percen Valid Cumulative
of Cases t Percen Percent
t
No, I don't know the 49 74.2 75.4 75.4
official criteria for
Unclassified sources
No, I don't know of 7 10.6 10.8 86.2
official criteria for
Unclassified sources.
Yes, I know the official 9 13.6 13.8 100.0
criteria for Unclassified.
sources
Total 65 98.5 100.0
Missing 1 1.5
Total 66 100.0
94
Survey, question 5.
95
Survey, question 5.
56
SURVEY FINDINGS, OBJECTIVITY AND FOREIGN LANGUAGE SOURCES
credible than obscure sources, even if the analysts had never observed the
However, because many analysts had first hand knowledge of many of the
well-known sources, their evaluations were biased, and could not be made on
that the well-known sources are credible, on a scale of 1-to-7 that equates to
is interesting to note that many analysts, who chose undecided for both the
well-known and obscure sources, explained that they were unable to decide
indication that analysts do not assume that a source is credible because they
have heard of it, but never seen it. No analysts rated any open source as
Certainly True, and only one source, JanesDefenseWeekly.com Web site, was
96
Survey, question 7.
97
See Table 5 and 6.
57
Also, there was no significant difference in the rating given to native-
language Web sites versus foreign Enlish-language Web sites.98 This was an
issue because in discussions with analysts before the survey, some analysts
said that they had observed a difference in the content of the native-
issue, it is apparently not one many analysts have observed.100 Question 7a-
m asked:101
7. How credible are the following information sources given only their
titles? Choose one from the following scale:
58
Q7j. 61 5 5.114 Credible .8583
NationalGeographic.com 8
Web Site
Q7k. 64 2 5.343 Strongly .8207
JanesDefenseWeekly.com 8 Credible
Web Site
Q7l. InformationWeek.com 61 5 4.704 Credible .7152
Web Site 9
Q7m. 61 5 4.950 Credible .7622
DowJonesInteractive.com 8
Web Site
Overall Credibility 5.02 Credible
89
103
Survey, questions 7d, 7e, 7f, 7 g, 7h, 7i.
59
In any discussion of open source credibility within the Intelligence
classified sources. This issue affects the relevance of the research question
credibility, then the IC should focus more on the cheaper source, which are
sources and classified sources would affect the cost of intelligence, which
rating given to all classified sources was 5.0811 on a scale of 1-to-7, which
equates to credible; however, the most-often chosen (mode) rating was 4.0,
which is undecided.104 As the table below demonstrates, the mean and the
mode disagree.
104
See Table 7.
105
Survey, questions 7n – 7s.
60
comments.
Q7q. IMINT without 35 31 4.9429 4.00 1.0831
national analysts
annotations or
comments.
Q7r. SIGINT reports 35 31 5.5429 5.00 .8521
Q7s. MASINT 33 33 5.0303 4.00 1.0150
Overall Credibility 5.0811 4.00
Credibl Undecide
e d
sources, which were rated undecided (4.55) on average. However, the most-
often chosen rating (mode) of both classified and unclassified sources was
undecided (4.0).
106
Survey, questions 7a – 7s.
61
comments.”107 As Chart 1 below shows, there is little consensus on the
credibility of IMINT without annotations, and the chart lacks the expected bell
16
14
12
10
6
Number of Cases
0 N = 35.00
3.0 4.0 5.0 6.0 7.0
107
Survey, question 7q.
108
Survey, question 7q.
62
CHAPTER 5
CONCLUSIONS
Web. The author hypothesized that this could be done by analysts who are
not expert in the subject of the Web site, by applying criteria identified by
credible sites scored high and the one non-credible site scored very low, even
these benchmark Web sites then functioned as the high and low ends of a
relative-credibility scale. Other evaluated Web sites’ scores will likely fall
between the high and low ends of the credibility scale. The position of the
evaluated site on the scale then puts the site’s level of credibility in context.
The criteria and scale were useful on their own, but intelligence
analysts need to know how credible is enough; therefore, the thesis survey
also asked analysts to rate how credible a source should be to allow its use in
several types of intelligence products. The result was that sources for all
63
then 36.06 on a linear scale of 7.46 (Non-credible) to 46.75 (Very Credible).
The fact that the known credible Web sites scored high and the known poor
site scored very low validated the recommended credibility criteria and their
relative credibility scores (weights). It is also of interest that the top four
the criteria suggested by experts in the literature review, and scored high
criteria, weights, and scale, which any analyst may now use to evaluate Web
These criteria are not without a weakness. Some criteria require a modest
will still make the better evaluators.109 Therefore, all evaluators should
include their own credentials on the evaluation sheets they share to maintain
evaluated sites to which any OSIS user could contribute. Such an index
that such a intelligence catalog would save the all-source analysts many
109
Bowen, under “Collection Strategy.”
64
hours of research, multiply the knowledge of experts, add to the credibility of
open sources in intelligence products, and reduce the need for classified
research when the data is available from an obscure but credible Web site.
There were also several issues that may have affected the objectivity
of the survey results and the relative importance of some criteria. These
were identified in the thesis as key issues. The key issue of open source’s
toward well-known source titles, such a bias was not evident in the survey
results. Also, analysts did not generally believe that the English-language
version of foreign Web sites were less credible than the native-language
about the classified sources, although analysts were given the opportunity to
choose no opinion, which few chose. The final conclusion of this thesis is that
any analysts can discern credible Web sites from the non-credible by using
matter experts, any analysts can evaluate a Web site using standard criteria,
65
which were recommended by expert researchers and approved by a broad
selection of analysts.
the primary all-source intelligence agencies, establish an OSIS Web site that
agreed to the criteria included here or other criteria, the rest of the
Alternatively, this open source index could be divided by subject area, and
for their subject area alone. Volunteer subject guides have already been
used on the internet by Yahoo and other online companies. However, it is not
often clear how they evaluate Web sites. This IC index of Web sites would
66
APPENDIX A
67
Table 10. Benchmark Web Site Evaluation Work Sheet, Spot.
Site Name, Address: Spot Image Corporation, http://www.spot.com
Evaluator Name , Expertise : Dax Norman, Intelligence Analyst,
Telecommunications, and Government
Criteria Comments Mean Satisfie
Score d
Criteria
?
8a. Recommended by Dr. Bowen, 4.94 YES
Recommended lecturer in War Studies at King’s
by subject- College London, article for
matter expert Jane’s Intelligence Review,
in the topic of 11/01/1999, “Open-source Intel:
the Web page. A Valuable National Security
Resource”.
8b. 3.65 NO
Recommended
by a
generalist.
8c. Listed by 3.56
an Internet
subject guide
that evaluates
Web sites.
8f. Content is 3.78 YES
perceived
current.
8g. Content is 4.56 YES
perceived
accurate.
8h. A peer or 4.52 NO
editor
reviewed the
content.
8i. Content's Biased toward accurate data. 3.06 YES
bias is obvious
8j. Author is Source rather than author is 4.64 YES
reputable. SPOT, CNES, the French space
agency and other satellite
companies.
8k. Author is Sources, rather than author are 4.42 YES
associated commercial satellite companies
with a who sell their products through
reputable the reputable organization,
organization. SPOT Imagery Corp. A search of
groups.google.com located
many favorable articles about
the company including a news
release from the newsgroups,
sci.space.news by
68
69
Table 11. Benchmark Web Site Evaluation Work Sheet, ITU.
Site Name, Address: International Telecommunications Union,
http://www.itu.int
Evaluator Name , Expertise : Dax Norman, Intelligence Analyst,
Telecommunications, and Government
Criteria Comments M Satisfied
ean Criteria?
Scor
e
8a. Recommended Used by the U.S. 4.94 YES
by subject matter Government.
expert in the topic of
the Web page.
8b. Recommended 3.65 NO
by a generalist.
8c. Listed by an 3.56
Internet subject
guide that evaluates
Web sites.
8f. Content is 3.78 YES
perceived current.
8g. Content is 4.56 YES
perceived accurate.
8h. A peer or editor Journal editors are listed. 4.52 YES
reviewed the
content.
8i. Content's bias is If there is a bias it is 3.06 YES
obvious. toward making the
telecommunications
market look better than it
is.
8j. Author is Authors are not always 4.64 NO
reputable. given, and little
information other than
ITU press releases can be
found about them, but
sources of contributing
data is always provided,
and includes reputable
sources such as
Vodaphone Group.
70
Telephone and Telegraph.
This reviewer recognizes
that analysis found in
these sources may be
more positive than
reality.
71
Table 12. Benchmark Web Site Evaluation Work Sheet, NY
Times.
Site Name, Address: NY Times On the Web, http://nytimes.com
Evaluator Name , Expertise : Dax Norman, Intelligence Analyst,
Telecommunications, and Government
Criteria Comments M Satisfied
ean Criteria?
Scor
e
8a. Recommended 4.94 YES
by subject-matter
expert in the topic of
the Web page.
8b. Recommended 3.65 NO
by a generalist.
8c. Listed by an 3.56
Internet subject
guide that evaluates
Web sites.
8f. Content is 3.78 YES
perceived current.
8g. Content is Yes, most of the time. 4.56 YES
perceived accurate. Errors are corrected
quickly online and given
fare space.
72
link to, or give credit pages were located that
t the evaluated Web link to this Web site.
site.
8o. Server or domain 3.45 YES
is
copyrighted or
trademark
name, like
IMB.com.®
8p. Statement of 3.78 YES
attribution.
8q. Professional 2.86 YES
appearance of Web
site.
TOTAL 48.8
2
73
Table 13. Benchmark Web Site Evaluation Work Sheet, Korea.
Site Name, Address: KoreanNewsSite
Evaluator Name , Expertise : Dax Norman, Intelligence Analyst,
Telecommunications, and Government
Criteria Comments M Satisfied
ean Criteria?
Scor
e
8a. Recommended 4.94 NO
by subject matter
expert in the topic
of the Web page.
8b. Recommended 3.65 NO
by a generalist.
8c. Listed by an 3.56
Internet subject
guide that
evaluates
Web sites.
8f. Content is Linked articles are one to 3.78 YES
perceived current. two weeks old but are still
relevant. However,
KoreanNewsSite’s
personal articles are not
dated.
74
8j. Author is A search of the Google 4.64 NO
reputable. newsgroups found many
discussion threads I the
soc.culture.Korean
newsgroup, which used
KoreanNewsSite as a
standard comparison for
poor, biased reporting. 111
75
authors, and himself.
KoreanNewsSite includes a
page which he claims to
describes himself;
however, there is some
doubt in the newsgroups
that his biography is true
because of inconsistencies
in dates and his claimed
age.
76
Table 14. Blank Web Site Evaluation Work Sheet.
Site Name, Address:
Evaluator Name , Expertise :
Scale: 46.75 = Very Credible, 7.46 = Non-credible
Target Score for Intelligence Sources: 35.06
Criteria Comments Mean Satisfied
Score Criteria?
1. Recommended by 4.94
subject-matter expert in
the topic of the Web
page.
2. Recommended by a 3.65
generalist.
3. Listed by an Internet 3.56
subject guide that
evaluates Web sites.
4. Content is perceived 3.78
current.
5. Content is perceived 4.56
accurate.
6. A peer or editor 4.52
reviewed the content.
7. Content's bias is 3.06
obvious.
8. Author is reputable. 4.64
9. Author is associated 4.42
with a reputable
organization.
10. Publisher or Web host 4.02
is reputable.
11. Content can be 5.17
corroborated with other
sources.
12. Other Web sites link 3.68
to, or give credit to the
evaluated Web site.
13. Server or domain is 3.45
copyrighted or
trademark name, like
IBM.com.®
14. Statement of 3.78
attribution.
15. Professional 2.86
appearance of Web site.
TOTAL
(Copyright: Dax R. Norman, 2001. Unlimited use is allowed with this statement
included.)
77
APPENDIX B
thesis.
WHY:
I am conducting research for my Masters Thesis at the Joint Military
Intelligence College. To do this I need assistance from many other analysts
who use the Web professionally to find information. I understand that your
time is valuable and I sincerely appreciate your assistance. I hope to
demonstrate through this research which criteria is most widely accepted for
measuring the credibility of Web sites. Also, your comments will help me to
categorize the survey responses, and credibility criteria used by different
industries or professions regarding Web sites used by professionals across
various industries.
PRIVACY:
You do not need to include your name; however, if you choose to include your
name, it will only be used by me to contact you if I need more information
regarding your comments. I will not quote you directly unless you indicate in
Questions 3 and 4 that I may do so. Otherwise, only me and my Thesis
Chairman, Professor Alex Cummins (410-854-4605) will have access to
respondent names. Any record of the names in association with the
responses will be destroyed after the research is completed, except those
names included in the thesis with permission.
INSTRUCTIONS:
78
Instructions for responding to this research survey are included on page 2 of
the survey. My preferred method is to type an X next to your answers and E-
mail this survey back to dnorman@carr.org. This is most easily done by
clicking on "Reply" in your E-mail tool. Please use the comments area after
each question if you feel that you need to explain an answer. However, the
"Comment" should not be used as a response choice. Please don't conduct
any research or attempt to locate the Web sites mentions here.
Respectfully,
Dax R. Norman
Joint Military Intelligence College, Cohort 10
xyz-xyz-xyz work
xyz-xyz-xyz fax
dnorman@xyz.xyz home E-mail
------------------------------------------------Page 1 Below
Research Conducted by
Dax R. Norman
JMIC, Cohort 10
2 July 2001
The views expressed in this paper are those of the author and respondents,
and do not reflect the official policy or positions of the U.S. Government.
------------------------------------------------Page 2 Below
or
Dax R. Norman, JMIC Cohort 10
1234 Blank Street RD
79
Blankville USA, 12345
or
XYZ-XYZ-XYZX Office Phone
XYZ-XYZ-XYZ Commercial Fax, Attn: Dax Norman
Your Name:
Industry Segment:
Phone Numbers:
E-mail:
Profession (What kind of work do you do?):
Date:
------------------------------------------------Page 3 Below
INSTRUCTIONS: Place an X next to your answers.
Note: The phrase, "any source" means published, proprietary, and classified.
80
Comments:
4. May I include your name and your responses in publicly published articles
as a follow on to this thesis? Choose one:
___a. Responses Only
___b. Name and Responses
___c. Neither
Comments:
5. Does your employer have official criteria that you are told to use for
determining the credibility of any source? "Any source" means published,
proprietary, and classified sources. Choose one:
___a. Yes, I know the official criteria for evaluating information sources.
___b. No, I don't know of official criteria for evaluating information
sources.
___c.. No, I don't know the official criteria for evaluating information
sources.
Comments:
6. List up to five criteria that you use to determine the credibility of any
information source.
a.
b.
c.
d.
e.
------------------------------------------------Page 4 Below
7. How credible are the following information sources given only their titles?
Choose one from the following scale:
A. NY Times
___7) = Certainly True
___6) = Strongly Credible
___5) = Credible
___4) = Undecided
___3) = Non-credible
___2) = Strongly Non-credible
___1) = Certainly False
Optional Comments or Why This Choice:
81
B. Washington Post
___7) = Certainly True
___6) = Strongly Credible
___5) = Credible
___4) = Undecided
___3) = Non-credible
___2) = Strongly Non-credible
___1) = Certainly False
Optional Comments or Why This Choice:
82
G. IsraelIndependentNews.is Web site in English
___7) = Certainly True
___6) = Strongly Credible
___5) = Credible
___4) = Undecided
___3) = Non-credible
___2) = Strongly Non-credible
___1) = Certainly False
Optional Comments or Why This Choice:
83
L. InformationWeek.com Web site
___7) = Certainly True
___6) = Strongly Credible
___5) = Credible
___4) = Undecided
___3) = Non-credible
___2) = Strongly Non-credible
___1) = Certainly False
Optional Comments or Why This Choice:
------------------------------------------------Page 5 Below
8. How much credibility does each of the following factors add the total
credibility of a Web site? Use the following scale:
B. Recommended by a generalist.
___6) 100 percent Credibility
___5) 75 percent Credibility
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
84
Optional Comments or Why This Choice:
85
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
Optional Comments or Why This Choice:
J. Author is reputable.
___6) 100 percent Credibility
___5) 75 percent Credibility
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
Optional Comments or Why This Choice:
86
N. Other Web sites link to or give credit to the evaluated site.
___6) 100 percent Credibility
___5) 75 percent Credibility
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
Optional Comments or Why This Choice:
END OF SURVEY
THANK YOU FOR TAKING THE TIME TO ASSIST ME WITH THIS PROJECT.
87
Dax R. Norman, JMIC Cohort 10
1234 Blank Street RD
Blankville USA, 12345
XYZ-XYZ-XYZX office
XYZ-XYZ-XYZX home
dnorman@xyz.xyz <mailto:dnorman@xyz.xyz>
88
APPENDIX C
thesis. Also, internal contact phone numbers, and E-mail address were
removed.
WHY:
I am conducting research for my Masters Thesis at the Joint Military
Intelligence College. To do this I need assistance from many other analysts
who use the Web professionally to find information. I understand that your
time is valuable and I sincerely appreciate your assistance. I hope to
demonstrate through this research which criteria is most widely accepted for
measuring the credibility of Web sites. Also, your comments will help me to
categorize the survey responses, and credibility criteria used by different
industries or professions regarding Web sites used by professionals across
various industries.
PRIVACY:
You do not need to include your name; however, if you choose to include your
name, it will only be used by me to contact you if I need more information
regarding your comments. I will not quote you directly unless you indicate in
Questions 3 and 4 that I may do so. Otherwise, only me and my Thesis
Chairman, Professor Alex Cummins (410-854-4605) will have access to
respondent names. Any record of the names in association with the
responses will be destroyed after the research is completed, except those
names included in the thesis with permission.
89
INSTRUCTIONS:
Instructions for responding to this research survey are included on page 2 of
the survey. My preferred method is to type an X next to your answers and E-
mail this survey back to dnorman@carr.org. This is most easily done by
clicking on "Reply" in your E-mail tool. Please use the comments area after
each question if you feel that you need to explain an answer. However, the
"Comment" should not be used as a response choice. Please don't conduct
any research or attempt to locate the Web sites mentions here.
Please share this survey with any other analysts or researchers that you think
may be interested.
Respectfully,
R. Norman
Joint Military Intelligence College, Cohort 10
XYZ-XYZ-XYZX fax
XYZ-XYZ-XYZX other fax
dnorman@xyz.xyz home E-mail
------------------------------------------------Page 1 Below
Research Conducted by
Dax R. Norman
JMIC, Cohort 10
8 July 2001
The views expressed in this paper are those of the author and respondents,
and do not reflect the official policy or positions of the U.S. Government.
------------------------------------------------Page 2 Below
90
or
Dax R. Norman, JMIC Cohort 10
1234 Blank Street RD
Blankville USA, 12345
or
Your Name:
Industry Segment/Organization:
Phone Numbers:
E-mail:
Profession (What kind of work do you do?):
Date:
Note: The phrase "any source" in this survey means both classified and
unclassified sources.
------------------------------------------------Page 3 Below
INSTRUCTIONS: Place an X next to your answers.
2. Do you use the Web for work related research? Choose one:
___a. Daily
___b. Weekly
___c. Monthly
___d. Rarely
___e. Never
91
Comments:
4. May I include your name and your responses in publicly published articles
as a follow on to this thesis? Choose one:
___a. Responses Only
___b. Name and Responses
___c. Neither
Comments:
5. Does your organization have official criteria that you are told to use for
determining the credibility of any source? "Any source" means published,
proprietary, and classified sources. Choose one:
___a. Yes, I know the official criteria for evaluating UNCLASSIFIED
information sources.
___b. No, I don't know of official criteria for evaluating UNCLASSIFIED
information sources.
___c.. No, I don't know the official criteria for evaluating UNCLASSIFIED
information sources.
Choose one:
___d. Yes, I know the official criteria for evaluating CLASSIFIED
information sources.
___e. No, I don't know of official criteria for evaluating CLASSIFIED
information sources.
___f.. No, I don't know the official criteria for evaluating CLASSIFIED
information sources.
Comments:
6. List up to five criteria that you use to determine the credibility of any
information source.
a.
b.
c.
d.
e.
ATTENTION PLEASE. Complete this page before reading the next page. The do
not return to this page.
------------------------------------------------Page 4 Below
92
7. How credible are the following information sources given only their titles?
Choose one from the following scale:
A. NY Times
___7) = Certainly True
___6) = Strongly Credible
___5) = Credible
___4) = Undecided
___3) = Non-credible
___2) = Strongly Non-credible
___1) = Certainly False
Optional Comments or Why This Choice:
B. Washington Post
___7) = Certainly True
___6) = Strongly Credible
___5) = Credible
___4) = Undecided
___3) = Non-credible
___2) = Strongly Non-credible
___1) = Certainly False
Optional Comments or Why This Choice:
93
Optional Comments or Why This Choice:
94
Optional Comments or Why This Choice:
------------------------------------------------Page 5 Below
8. How much credibility does each of the following factors add the total
credibility of a Web site? Use the following scale:
95
___6) 100 percent Credibility
___5) 75 percent Credibility
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
B. Recommended by a generalist.
___6) 100 percent Credibility
___5) 75 percent Credibility
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
Optional Comments or Why This Choice:
96
___1) 0 percent Credibility
Optional Comments or Why This Choice:
J. Author is reputable.
___6) 100 percent Credibility
___5) 75 percent Credibility
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
Optional Comments or Why This Choice:
97
___5) 75 percent Credibility
___4) 50 percent Credibility
___3) 25 percent Credibility
___2) 10 percent Credibility
___1) 0 percent Credibility
Optional Comments or Why This Choice:
98
___1) 0 percent Credibility
Optional Comments or Why This Choice:
9. How credible must an Intelligence source be to use its data in the following
intelligence products? Use includes when you would use qualifiers such as
"possible survived". Choose the required level of credibility for each type of
intelligence.
Scale:
___7) No Opinion
___6) 100 percent Credible
___5) 75 percent Credible
___4) 50 percent Credible
___3) 25 percent Credible
___2) 10 percent Credible
___1) 0 percent Credible
Optional Comments or Why This Choice:
99
___4) 50 percent Credible
___3) 25 percent Credible
___2) 10 percent Credible
___1) 0 percent Credible
Optional Comments or Why This Choice:
END OF SURVEY
THANK YOU FOR TAKING THE TIME TO ASSIST ME WITH THIS PROJECT.
100
101
APPENDIX D
102
Table 15. Survey Question 6: Credibility Criteria Analysts Currently
Use.
Criteria: Coun Percen Valid Cumulativ
t t Percent e Percent
recommended by a trusted 2 .7 .7 54.4
source
past performance 2 .7 .7 55.2
official Web site 2 .7 .7 55.9
involved in current mission, 2 .7 .7 56.6
policy, or planning
level of detail 2 .7 .7 57.3
domain name extension 2 .7 .7 58.0
reasonable the source knows the 2 .7 .7 58.7
information
circumstances 2 .7 .7 59.4
amount of technical information 2 .7 .7 60.1
reputation of site 2 .7 .7 60.9
ePharmaceuticals 1 .4 .4 61.2
stated criteria for
inclusion of information 1 .4 .4 61.6
Dow Jones Newswire 1 .4 .4 61.9
FDA 1 .4 .4 62.3
PubMED 1 .4 .4 62.6
JAMA 1 .4 .4 63.0
reasonable with other credible 1 .4 .4 63.3
information
name of organization providing 1 .4 .4 63.7
information
interest 1 .4 .4 64.1
verifiable references 1 .4 .4 64.4
consistent with other information 1 .4 .4 64.8
corroborated by trusted source 1 .4 .4 65.1
fact based 1 .4 .4 65.5
several reports in different 1 .4 .4 65.8
languages
how other publications use the 1 .4 .4 66.2
source
resources 1 .4 .4 66.5
source available with IP access 1 .4 .4 66.9
via DoD intranet
reputation of publisher 1 .4 .4 67.3
classified corroborated by 1 .4 .4 67.6
unclassified
own knowledge 1 .4 .4 68.0
publication source 1 .4 .4 68.3
history of reliability 1 .4 .4 68.7
background supplied with 1 .4 .4 69.0
Website
103
Table 15. Survey Question 6: Credibility Criteria Analysts Currently
Use.
Criteria: Coun Percen Valid Cumulativ
t t Percent e Percent
long established 1 .4 .4 69.4
source is identified 1 .4 .4 69.8
site owner apparent 1 .4 .4 70.1
personal experience with the 1 .4 .4 70.5
source
utility 1 .4 .4 70.8
sources non-internet work 1 .4 .4 71.2
reputable 1 .4 .4 71.5
established print publications 1 .4 .4 71.9
publication reputation 1 .4 .4 72.2
past publications 1 .4 .4 72.6
author locatable 1 .4 .4 73.0
past validity 1 .4 .4 73.3
mainstream source 1 .4 .4 73.7
topic area audio and video 1 .4 .4 74.0
several reports on following dates 1 .4 .4 74.4
second opinion of site 1 .4 .4 74.7
reliability of author 1 .4 .4 75.1
willingness make corrections 1 .4 .4 75.4
published industry journal 1 .4 .4 75.8
industry analysts 1 .4 .4 76.2
financial analysts 1 .4 .4 76.5
domain names 1 .4 .4 76.9
proper attribution and dates 1 .4 .4 77.2
past accuracy 1 .4 .4 77.6
quality of information 1 .4 .4 77.9
proximity to origin 1 .4 .4 78.3
includes citations 1 .4 .4 78.6
past use of source 1 .4 .4 79.0
experience 1 .4 .4 79.4
publisher 1 .4 .4 79.7
media source 1 .4 .4 80.1
authoritative quotes 1 .4 .4 80.4
reasonable current 1 .4 .4 80.8
consistent with confidential 1 .4 .4 81.1
information
source past credibility 1 .4 .4 81.5
IP address 1 .4 .4 81.9
ISP 1 .4 .4 82.2
direct or indirect collection 1 .4 .4 82.6
associated with credible 1 .4 .4 82.9
organization
several reports in geographic 1 .4 .4 83.3
area newspapers
104
Table 15. Survey Question 6: Credibility Criteria Analysts Currently
Use.
Criteria: Coun Percen Valid Cumulativ
t t Percent e Percent
sources used 1 .4 .4 83.6
knowledge about source 1 .4 .4 84.0
information flow 1 .4 .4 84.3
cited by other analysts 1 .4 .4 84.7
source name recognizable 1 .4 .4 85.1
outstanding organization 1 .4 .4 85.4
is this their profession 1 .4 .4 85.8
author hobbyist or a professor 1 .4 .4 86.1
reliable past reporting 1 .4 .4 86.5
second party evaluation 1 .4 .4 86.8
witting or not 1 .4 .4 87.2
collection conditions 1 .4 .4 87.5
source background 1 .4 .4 87.9
relevancy 1 .4 .4 88.3
consistency 1 .4 .4 88.6
association with government 1 .4 .4 89.0
scholarly journals 1 .4 .4 89.3
government publications 1 .4 .4 89.7
educational reference source 1 .4 .4 90.0
stability of information 1 .4 .4 90.4
copyrighted material 1 .4 .4 90.7
attribution provided 1 .4 .4 91.1
reputable url 1 .4 .4 91.5
site owner corporate or 1 .4 .4 91.8
government
professional language 1 .4 .4 92.2
professional appearance 1 .4 .4 92.5
lists sources 1 .4 .4 92.9
personal contacts 1 .4 .4 93.2
industry consultants 1 .4 .4 93.6
veracity 1 .4 .4 94.0
relation to government 1 .4 .4 94.3
external consistency 1 .4 .4 94.7
trusted source 1 .4 .4 95.0
reasonableness 1 .4 .4 95.4
source past behavior 1 .4 .4 95.7
past use of deception 1 .4 .4 96.1
collection method 1 .4 .4 96.4
past reliability of source 1 .4 .4 96.8
subject 1 .4 .4 97.2
size of site 1 .4 .4 97.5
how facts are divulged 1 .4 .4 97.9
proximity to original source 1 .4 .4 98.2
sources job 1 .4 .4 98.6
105
Table 15. Survey Question 6: Credibility Criteria Analysts Currently
Use.
Criteria: Coun Percen Valid Cumulativ
t t Percent e Percent
first hand knowledge 1 .4 .4 98.9
experience with source 1 .4 .4 99.3
published by an organization 1 .4 .4 99.6
personal experience with subject 1 .4 .4 100.0
Total 281 100.0 100.0
106
BIBLIOGRAPHY
Alexander, Jan and Marsha Tate. “The Web as a Research Tool: Evaluation
Techniques.” Wolfgram Memorial Library, Widener University. Chester,
PA. URL: <http://www.science.widener.edu/~withers.evalout.htm. >
Accessed
13 March 2001.
Clift, A. Denis. Clift Notes: Intelligence and the Nation’s Security. Washington,
D.C.: Joint Military Intelligence College, 1999.
107
Nunnally, Jum C. Psychometric Theory. New York: McGraw-Hill Book
Company, 1967.
108
ANNEX 1.
SURVEY RESULTS
Questions that did not apply to respondents because the question only
applied to intelligence community analysts are represented by “999”. The
word “blank”, or “0” represents questions, which could have been answered
but were not.
109
RESPONCE CATEGORY SEGMENT PROFESSI Q1 Q2 Q3 Q4 Q5 Q5PART2
1 3 College Educator a a a a b 999
2 1 Intell IAnalyst b c a a b 999
3 2 Business Research a a b b b 999
4 2 InfoTech Executiv b b b b a 999
5 2 Research CompSci a b a a b 999
6 1 Defense Info Res a a b a a 999
8 1 Defense IAnalyst a b b a b 999
9 2 Health Research a a a a a 999
10 1 Defense IAnalyst a a b b b 999
11 3 College Educator b a a a b 999
12 1 Defense IAnalyst b b b a c 999
13 2 Finance CPA a a b a b 999
14 3 College Educator a b b b b 999
15 2 InfoTech blank a a a a b 999
16 2 Business Research b a b b c 999
17 1 Trade Linguist a a b b b 999
18 3 College Educator a b a a b 999
19 1 Defense IAnalyst a a b b b 999
20 1 Defense IAnalyst a a b b b 999
21 1 Defense CompSci a a b a b 999
22 1 Defense IAnalyst b a a a a 999
23 1 Defense IAnalyst b a b b b 999
24 2 Medicine Libraria a a a a a 999
25 2 Law Lawyer a a a a b 999
26 1 Defense IAnalyst a a b a a 999
27 1 Defense IAnalyst b c a a a 999
28 1 Defense IAnalyst a a b a b 999
29 1 Defense IAnalyst a a b b b 999
30 1 Defense Training a a b b c 999
31 1 Defense IAnalyst b a b b a 999
32 1 Intell IAnalyst a b b a b blank
33 1 Intell IAnalyst b b a a b e
34 1 Intell IAnalyst b b a a b e
35 1 Intell IAnalyst a b a a b d
36 1 Intell IAnalyst b a a a b e
37 1 Intell IAnalyst b d b b blank d
39 1 Intell IAnalyst a a a a b blank
40 1 Comms Research a a a a b e
41 1 Intell IAnalyst a a b a b e
42 1 Intell IAnalyst b a a a b e
43 1 Intell IAnalyst a a b b b e
45 1 Defense IAnalyst a a b b b blank
46 2 Intell IAnalyst a a b b b e
47 1 Defense Library b a a a b e
48 1 Defense IAnalyst b b b a b e
49 1 Intell IAnalyst b a b a c f
50 1 Intell IAnalyst b a a a b blank
51 1 Intell IAnalyst a a a a b e
52 1 Intell Library b a b b b d
53 1 Intell IAnalyst a a b b b d
54 1 Intell IAnalyst b d a a c f
55 1 Intell IAnalyst a a b a a d
56 1 Intell IAnalyst c d a a b blank
57 1 Intell IAnalyst b c b b b d
58 1 Intell IAnalyst a c a a b blank
59 1 Intell IAnalyst a a b a b d
60 1 Intell IAnalyst a d a a c f
62 1 Intell Physics a a a a b e
63 1 Defense IAnalyst a a b a b blank
64 1 Defense IAnalyst c d b b b e
65 1 Intell IAnalyst b c b b b blank
66 1 Intell IAnalyst b b a a b e
Q6
verifiable references, data verifiable, author expert in subject, author associated with recognized organization.
Author, fact based, credibility of publisher
resources, bias, interest, currency, verifiable
published industry journal, industry analysts, financial analysts, industry cosultants, personal contacts
lists references or sources, professional appearance, profossional language, referrenced by trusted source, recommen
author, publisher, original source, auhor's expertise
owner of site, presentation, content, coroborated, professional presentation
JAMA, PubMED, FDA, ePharmaceuticals, Dow Jones Newswire
own knowledge, author's credentials, cited by other authoritative sources, accuracy, presentation, corroborative
timeliness, authority of author, affiliation of author, original source, expert recommendation
several reports in different languages, several reports in target area newspapers, several reports on followig dates
locate author, past publications, pulication reputation, established print publications, site investors, corroborati
Site suppliers and funders, professionalism, content, bias, reputation
is ita a kown and trusted source and author, trusted recommendation, perceived bias, site ownership and sponsorship
source is identified, corroboration, accuracy, clarity, current
well known and respected, long established, background supplied with website, history of reliability, balanced and u
publication source, reputation of vendor
personal experience with the source, personal experience with subject, profesional appearance and utility, sources n
Corroboration, ownership, history of validity
Established news service, corroboration, secondary evaluation of primary site, mainstream source,
credentials, corroboration, common sense, reliability, climate surrounding activities
reputable news service, reliability of author, author's knowledge of subject, corroborative, willingness make correc
stated criteria for inclusion of information, author's authority, corroborative, stability of information, appropria
reputation of publication or site, corporate or government owner, reputation or credentials of author, cited by othe
Educational reference source, government publications, established news services, scholarly journals
association with government, reliability, veracity, motives, experience of source
national origin, relation to government, bias, internal consistency, external consistency
author, nationality of author, feasibility of information, date of information
presentation, published by an organization rather than individual, content, age of information, experience with sour
original source with first hand knowledge, involved in current mission, policy, or planning, level of involvement cu
how close to the original source, past reliability, reputation of source, likelyhod source would know the informatio
name of organization providing information, data of material, size of the site
data repeated in other sources, does it fit with other credible information, who is the information from (govt, medi
Author and sponsor of site, subject, amount of technical information, date of information
ability t verify information against other sources, past reliability of source, does unclassified information suppor
collection system, past use of deception, supportive information, source identity
easonableness, consistency with other information, trusted source
reliability from experience, accuracy by judgement of factual possibility, consistency, relavency
author's bias, age of information, corrobation with other sources
background of the source, past credibility, motive, corroboration, timeliness
corroborative, agreement with personal views expressed in private, environmental influences impacting source, check
past performance, authority of source, does the information correlate with known information
domain extension, author, media source, publishing house, experience
past use of source, source cited by a trusted source, source includes citations, known bias6
reputation, closeness to original source, corroboration, internal consistency, bias
does the source have access to the information provided, past reporting, bias, is it a source
reputation of source, specificity of source information, corroboration, quality of information
past accuracy, bias, general reliability on intelligence vs press, access of reporter to source information
known and reputable source, corroboration by trusted source, appropriateness of source for the informatin sought
past reliability, corroborative, known bias, target audeance, age of information, proper attribution and dates
blank
domain names, IP address, ISP, coroborative
scientific information including specifications, official company web sites
direct or indirect intelligence collection, collection conditions,who is original source, witting or not, motives, s
domain name extensions, already evaluated in other media, verifiable, information provided already know to be true,
reliability of past reporting, depth of information provided, reputation of source
blank
source name recognizable, reputain, commonly used source cited by other analysts or writers, source is reporting in
bia, bona fides, originality of information, corroboraton with independent sources, flow of information, generation
freshness of information, source's access to the information, motivation for providing the information, level of exp
reputation of author, what is known about the source, personal knowledge about the source, how other publications u
publisher or author of information, sources used, datga of the information, or published date, compare to other sour
Q7A Q7B Q7C Q7D Q7E Q7F Q7G Q7H Q7I Q7J Q7K Q7L
55 4 4 4 4 4 4 4 4 4 5
55 5 blank blank blank blank blank blank 6 6 blank
55 5 4 4 4 4 4 4 5 5 5
55 5 4 4 4 4 4 4 5 5 5
55 6 5 5 5 5 5 5 6 5 5
55 4 3 3 blank blank blank blank 7 6 5
66 6 4 4 4 5 4 4 6 5 5
77 5 4 4 4 4 4 4 6 4 5
66 6 blank 5 blank 4 5 5 6 6 5
55 4 blank blank blank blank blank blank 5 5 4
45 5 4 blank blank 4 blank 4 4 4 4
76 6 4 4 4 4 4 4 6 6 6
56 5 2 2 2 2 3 3 5 5 4
66 5 4 4 4 4 4 4 5 5 5
55 blank blank blank blank blank blank blank blank 5 5
55 6 4 4 4 4 4 4 5 5 4
54 5 4 2 4 5 4 5 5 6 5
66 6 4 4 3 3 5 5 7 6 5
55 4 3 2 4 4 4 4 6 7 5
#NULL! 5 5 1 1 4 4 3 3 4 6 5
55 4 4 5 5 5 5 3 5 4 4
65 6 5 5 6 5 6 5 4 6 6
55 5 4 4 blank 4 4 4 4 4 5
66 4 4 4 5 4 5 5 6 6 4
55 6 4 4 4 4 4 4 6 6 4
55 4 4 3 5 5 5 5 5 6 5
44 4 4 4 4 4 4 4 5 6 4
55 4 4 4 4 4 4 4 4 6 4
33 2 2 2 4 4 4 4 5 6 4
53 5 4 4 4 4 4 4 5 6 4
6 blank 6 3 3 4 4 5 5 blank 6 4
66 6 4 4 4 4 4 4 6 6 5
66 6 4 4 4 4 4 4 6 6 blank
55 4 4 4 4 4 4 4 5 5 5
54 5 4 4 blank blank blank blank 5 5 blank
35 4 4 2 4 5 4 4 5 5 4
66 4 4 4 4 4 4 4 6 6 6
65 4 4 4 5 5 4 5 5 7 6
65 4 4 4 4 4 4 4 6 6 4
55 4 4 3 4 4 4 4 5 5 4
44 4 4 4 4 4 4 4 4 4 4
64 4 4 5 4 5 4 3 6 7 7
54 4 4 4 4 4 4 4 4 5 5
66 6 4 4 4 4 4 4 6 6 5
66 blank blank blank blank blank blank blank blank 5 000
67 5 2 1 4 4 3 3 6 6 5
66 5 4 4 4 4 4 4 5 6 5
55 4 3 3 4 4 4 4 5 6 5
66 5 5 4 4 4 4 4 6 6 5
55 4 4 3 4 4 4 4 4 5 5
55 5 4 4 4 4 4 4 5 5 5
54 5 4 4 4 4 4 4 4 4 4
44 4 5 3 4 3 5 5 5 5 5
44 4 4 4 4 4 4 4 5 5 4
55 5 4 4 4 4 4 4 3 4 4
55 5 4 4 4 4 4 4 5 6 6
55 4 4 4 4 4 4 4 4 4 4
66 5 4 5 4 5 4 5 6 5 5
55 6 4 4 4 4 blank 4 4 5 4
55 5 4 4 4 4 4 4 5 5 4
#NULL! 5 blank blank blank blank blank blank blank blank blank blank
44 5 4 4 4 4 4 4 4 4 4
Q7M Q7N Q7O Q7P Q7Q Q7R Q7S Q8A Q8B Q8C Q8D Q8E
6 999 999 999 999 999 999 5 4 1 1 2
blank 999 999 999 999 999 999 6 6 5 1 2
5 999 999 999 999 999 999 3 2 3 1 2
5 999 999 999 999 999 999 5 3 4 5 5
5 999 999 999 999 999 999 5 5 4 1 1
6 999 999 999 999 999 999 5 4 4 4 4
4 999 999 999 999 99 999 5 4 3 2 2
7 999 999 999 999 999 999 5 4 5 4 4
5 999 999 999 999 999 999 5 5 4 3 3
5 999 999 999 999 999 999 5 3 4 1 1
4 999 999 999 999 999 999 4 3 #NULL! #NULL! 4
6 999 999 999 999 999 999 6 5 4 2 3
5 999 999 999 999 999 999 5 4 3 2 2
4 999 999 999 999 999 999 5 3 4 1 3
nu 999 999 999 999 999 999 4 0 0 0 0
5 999 999 999 999 999 999 5 4 5 5 5
5 999 999 999 999 999 999 5 4 5 3 3
5 999 999 999 999 999 999 5 5 4 3 3
4 999 999 999 999 999 999 4 2 2 2 #NULL!
4 999 999 999 999 999 999 6 1 3 3 3
5 999 999 999 999 99 99 5 5 4 4 5
5 999 999 999 999 999 999 6 5 1 1 1
5 999 999 999 999 999 999 6 4 4 4 5
6 999 999 999 999 999 999 5 5 3 2 2
4 999 999 999 999 999 999 5 3 4 3 3
5 999 999 999 999 999 999 5 4 3 3 3
4 999 999 999 999 999 999 3 1 1 1 1
4 999 999 999 999 999 999 5 4 4 4 4
4 999 999 999 999 999 999 5 5 5 4 4
5 999 999 999 999 999 999 5 2 1 1 1
6 6 5 5 6 6 5 4 3 0 1 1
6 5 5 6 5 5 6 6 1 3 1 3
5 4 5 6 4 6 4 5 4 5 1 3
5 4 5 5 4 4 4 4 2 2 1 1
5 4 5 5 4 6 blank 5 1 3 1 2
4 4 6 6 5 6 6 5 3 3 2 3
6 4 6 5 5 6 6 5 4 5 3 3
6 4 5 7 7 7 7 5 5 3 1 3
5 3 blank 6 6 6 5 5 3 3 1 2
4 3 5 5 6 5 5 4 3 3 2 2
blank blank 4 4 4 4 4 3 1 1 1 1
6 5 6 5 5 6 7 5 5 4 3 0
5 6 6 6 6 6 4 5 3 3 4 2
5 4 5 6 5 5 5 6 5 3 2 2
000 4 5 6 6 6 6 4 3 3 3 3
5 4 6 6 4 5 5 5 4 4 3 3
5 4 6 7 6 7 4 5 4 5 2 2
5 4 6 6 4 6 6 5 4 4 2 2
6 4 6 5 4 5 4 6 5 5 4 4
5 4 5 5 4 4 6 5 4 5 4 5
4 3 5 4 3 5 4 5 3 4 3 3
5 4 6 6 6 5 4 5 4 4 3 2
5 4 5 6 5 5 5 4 3 3 1 1
4 4 5 6 4 5 6 5 3 2 1 1
4 3 5 6 4 7 6 5 2 3 1 2
6 4 5 4 4 5 4 5 4 3 1 2
4 5 6 6 7 7 4 6 4 5 3 2
5 4 5 6 7 5 5 5 4 3 3 3
4 4 6 5 4 5 4 5 4 4 3 0
5 4 5 5 4 5 5 5 5 5 4 4
blank 5 6 7 6 6 blank 6 5 4 2 2
5 3 5 6 6 5 5 4 2 2 1 1
Q8F Q8G Q8H Q8I Q8J Q8K Q8L Q8M Q8N Q8O Q8P Q8Q
3 4 4 4 5 4 4 5 5 4 5 4
4 4 6 5 6 6 5 6 4 2 3 3
4 6 4 3 4 3 3 4 2 4 3 1
4 4 3 4 4 5 5 4 4 4 3 3
5 5 5 5 5 5 5 6 5 4 5 4
4 5 4 2 5 5 4 5 4 4 5 4
3 4 2 2 4 2 2 5 2 1 3 2
4 4 6 1 5 5 5 5 5 3 4 4
4 5 5 4 5 5 5 5 4 3 3 4
4 4 5 4 4 4 1 5 2 2 4 2
6 6 6 4 #NULL! 5 3 6 5 5 4 5
2 4 5 2 5 5 5 6 3 4 4 4
3 4 5 2 5 5 5 5 4 4 3 3
3 5 5 1 5 5 4 6 4 3 3 2
0 0 0 0 0 0 0 0 0 0 0 0
5 5 5 4 5 5 5 6 5 5 4 4
4 5 5 1 5 5 5 6 5 4 4 4
4 5 6 6 5 5 4 6 5 4 6 4
4 4 3 2 5 5 3 5 3 4 0 2
1 1 3 1 6 6 6 6 3 3 2 1
5 5 5 3 5 4 4 5 4 4 4 5
2 5 5 6 5 5 5 6 1 1 1 1
6 6 6 5 5 5 5 5 5 4 5 4
4 5 5 4 5 4 4 5 4 5 4 3
5 5 5 5 5 5 5 6 5 4 4 3
4 4 5 4 5 4 4 5 4 3 5 3
2 4 3 1 3 2 1 3 1 1 1 1
5 4 5 2 5 5 5 6 5 5 5 4
5 5 5 4 5 5 5 6 5 5 5 6
3 5 5 5 5 5 5 5 4 4 5 1
0 6 1 2 5 5 5 6 3 4 3 4
3 5 5 2 5 5 4 6 2 4 5 4
5 5 5 2 5 4 5 5 5 5 5 4
5 5 5 5 6 5 3 6 4 3 4 3
4 2 3 1 4 3 3 5 1 3 3 3
5 5 3 1 2 3 3 4 2 2 2 3
5 5 5 2 5 5 3 6 4 1 4 1
3 5 5 2 5 5 4 6 2 4 5 4
3 4 4 2 4 5 2 5 5 1 4 1
5 6 4 2 4 4 3 5 4 2 3 1
1 0 3 2 2 2 1 3 1 1 1 1
5 0 5 2 5 5 5 5 5 5 5 4
5 5 5 4 5 3 4 5 3 4 3 3
3 4 5 4 4 4 4 5 2 4 3 3
4 5 4 3 3 4 2 5 4 2 3 2
4 4 5 5 5 5 5 6 5 4 4 3
3 5 5 1 5 5 3 5 3 3 3 1
2 5 5 4 5 5 5 5 4 3 3 1
5 5 5 4 5 5 5 6 5 5 5 4
5 5 5 4 5 5 5 5 5 5 5 5
4 4 3 4 4 4 4 4 4 4 5 3
5 5 5 4 5 3 4 5 3 4 3 3
3 3 3 2 4 3 3 5 4 6 5 1
2 5 5 1 3 3 2 5 2 2 2 2
1 4 5 4 6 4 5 6 2 2 2 2
3 4 5 3 5 4 3 5 5 2 4 4
5 4 3 4 4 3 3 3 3 4 5 2
3 3 4 3 4 5 4 5 4 5 5 2
3 5 5 4 5 5 5 5 4 4 3 3
5 5 5 3 5 5 5 5 5 5 5 4
4 4 5 4 5 5 4 6 4 2 4 1
3 4 4 3 4 4 4 5 3 1 2 1
Q8R Q9A Q9B Q9C Q9D Q9E Q9F
2 999 999 999 999 999 999
3 999 999 999 999 999 999
1 999 999 999 999 999 999
3 999 999 999 999 999 999
5 999 999 999 999 999 999
4 999 999 999 999 999 999
3 999 999 999 999 999 999
4 999 999 999 999 999 999
4 999 999 999 999 999 999
2 999 999 999 999 999 999
4 999 999 999 999 999 999
4 999 999 999 999 999 999
3 999 999 999 999 999 999
3 999 999 999 999 999 999
0 999 999 999 999 999 999
5 999 999 999 999 999 999
4 999 999 999 999 999 999
5 999 999 999 999 999 999
3 999 999 999 999 999 999
3 999 999 999 999 999 999
4 999 999 999 999 999 999
1 999 999 999 999 999 999
4 999 999 999 999 999 999
4 999 999 999 999 999 999
2 999 999 999 999 999 999
3 999 999 999 999 999 999
0 999 999 999 999 999 999
5 999 999 999 999 999 999
6 999 999 999 999 999 999
3 999 999 999 999 999 999
4 4 5 5 5 4 5
4 5 4 5 3 5 2
4 3 3 5 5 5 5
4 6 4 6 5 6 4
3 4 4 4 4 5 5
3 4 4 5 5 4 4
1 5 5 4 5 3 6
4 6 5 5 6 6 6
2 5 3 5 4 7 2
1 4 3 4 5 5 6
7 5 5 5 5 5 1
4 5 7 4 5 5 5
1 1 1 1 3 1 1
3 4 4 4 5 5 7
3 4 4 4 5 4 5
3 6 5 5 5 5 4
2 4 5 4 5 7 5
2 5 5 4 4 3 5
4 7 5 5 6 5 6
5 4 3 3 3 3 2
3 6 6 6 6 6 6
1 3 3 2 3 1 4
1 7 7 7 7 4 7
2 5 3 4 5 5 3
2 7 7 7 7 7 7
3 4 4 3 5 7 4
2 6 6 6 6 6 6
3 4 5 5 6 5 5
3 5 5 5 5 5 4
5 5 4 4 5 4 4
2 5 5 5 5 5 5
1 4 2 5 1 5 2