Information Systems

Department
College of Computing
Sciences
New Jersey Institute
of Technology
Research Paper Review - Structure and
Format Guidelines
Michael Bieber
This set of guidelines was originally designed for a course, so it
includes grading guidelines, which most readers can ignore.
Notes:
1. Follow the guidelines here on structuring and formatting your review.
2. Include each category in a separate section or subsection. Ensure that you discuss
each category! (If a category does not apply, then actually write in that
(sub)section that the category does not apply.) We will be looking for each
category when grading.
3. Reference other articles you've read, if appropriate. Just as in the articles you've
read, include a reference marker where you refer to each in your review, and
provide full bibliographic references in a bibliography section at the end of your
report.
4. Your final report should be 4 full pages or more (if measured in Times New
Roman 10pt single spaced), which is equivalent to 7-10 pages in most Web
browsers. This does not include your bibliography, citation analysis, diagrams or
large gaps between parts of your report. There is no maximum length.
5. Do not copy any part of the article into your review. If you want to use more than
3-4 of the author's words, then use quotation marks, and add a page number from
the article (if available) when you cite this quotation.
6. Proof-read your review for grammar and clarity as well as spelling. Spell checkers
do not fix grammar. You should be sure that you have caught all mistakes and
written clearly. Most word processors have a grammar checker - use it!
7. For the printed copy of your review, turn in just a plain stapled black & white
copy of your review. Do not put your review in a folder. Avoid fancy covers,
colors or fonts. We want to judge your content, not visual presentation.
8. Please reread the note on plagiarism posted in the syllabus and on the course Web
site.

Additional References to Outside Materials

In sections 6-14, utilize at least 3 citations to articles outside those on
the class reading list. Use these articles to back up your analysis, to
describe or address issues, implications or recommendations for
further research or development.

Grading Notes
We have included the grading criteria in these guidelines.
Unless otherwise noted, the categories are:
great: fully discussed, explained well and well-justified
ok: partial discussion; not explained well or well-justified
poor: barely discussed
zero: not discussed

Length Grading
We take 5 points off for each full page missing (i.e., less than 4 full
pages) and 3 points off for every half page missing. Note that large
gaps between sections will not count towards the full page length.

Editing Grading
Has the review been prooofread for both spelling and grammar? If not
up to 3 points off for poor grammar and up to 3 points off for poor
spelling.

Preparing for Your Review

The purpose of research articles is to present new or refine conceptual
ideas, or to present new evidence for conceptual ideas. A research
article review results from critically examining a research article. You
will have to read your article several times to understand it fully
enough to review properly.
Often, comparing your article to others will help you determine its
quality.
Also, think about the article and its research ideas in terms of each of
the different week's concepts and frameworks we study in class. Do
the ideas in the article fit all the categories of appropriate class
concepts, models and frameworks? This often will help you see things

the authors missed, think about things the authors write in a new light,
or see that the authors indeed covered a topic thoroughly.

Outline of the Article Review

Please include the following categories in your article review.
1. Full Bibliographic Reference (-3 if missing)
2. Introduction: Objectives, Article Domain, Audience, Journal and
Conceptual/Emprical Classification (8)
3. Very Brief Summary (4)
4. Results (8)
5. Class Readings (4)
6. Contributions (8)
7. Foundation (4)
8. Synthesis with Class Materials (12 +8 extra credit)
9. Analysis & Additional Analysis (4 +8 extra credit)
10. General Critique (10 + 6 extra credit)
11. Further Critique of a Conceptual Article -or- (12)
Further Critique of an Empirical Article (12+2 extra credit)
12. Issues (listed by the author) (6+3 extra credit)
13. Issues (in your opinion) (6+6 extra credit)
14. Impact (9)
15. Questions (6)
16. Annotated Bibliography (-5 if missing)
17. Citation Analysis Appendix (6)
In addition, you can lose points if your review is too short or poorly
edited. See the Grading Notes section above.

1. Full Bibliographic Reference

State the full bibliographic reference for the article you are reviewing
(authors, title, journal name, volume, issue, year, page numbers, etc.)
Important: this is not the bibliography listed at the end of the article,
rather the citation of the article itself!
Grading: -3 if missing

2. Introduction: Objectives, Article

Domain, Audience, Journal and
Conceptual/Emprical Classification

Note: For the on-line reviews done in some class sections, this
category may be broken up into several separate subcategories. For
the written review, please discuss all of these subcategories together
as follows.
Paragraph 1: State the objectives (goals or purpose) of the article.
What is the article's domain (topic area)?
Paragraph 2:
Audience: State the article's intended audience. At what level is it written, and
what general background should the reader have; what general background
materials should the reader be familiar with to understand the article?
Appropriate Journal?: Why is the journal appropriate (or inappropriate) for this
article? (Check the mission statement or purpose of the journal itself from its
cover or its Web site.)
Paragraph 3: State whether the article is "conceptual" or "empirical",
and why you believe it is conceptual or empirical. Empirical articles
and conceptual articles have a similar objective: to substantiate an
argument proposed by the author. While a conceptual article supports
such an argument based on logical and persuasive reasoning, an
empirical article offers empirical evidence to support the argument.
Empirical articles offer substantial, detailed evidence which the authors
analyze using statistical methods. Empirical articles must include
hypotheses (or propositions), detailed research results, and (statistical)
analyses of this empirical evidence. Empirical research includes
experiments, surveys, questionnaires, field studies, etc, and to limited
degree, case studies. Conceptual articles may refer to such empirical
evidence, but do not provide the detailed analysis of that evidence. Of
course, both types of articles can use real life examples to back up
their points. Just because an article provides examples, does not
necessarily mean that it is empirical. (The lesson to take home is not to
consider a conceptual article to be an empirical one just because it
provides some summarized or some unanalyzed data.)
Grading: Objectives: great - 3; ok - 2; poor - 1
Grading: Audience/Journal Appropriateness: great - 3; ok - 2; poor - 1
Grading: Conceptual vs. empirical: great - 2; ok/poor - 1

3. Very Brief Summary

For our article reviews, we do not want you to spend much space
summarizing the article. Instead we are more interested in your
analysis of the article.
Thus, in this section, summarize the article only very briefly (2-3
paragraphs).
If possible, use the IS research paradigm as the format of your
summary, but remaining very brief:

Paragraph 1: what is the problem or opportunity being addressed

Paragraph 2: which solution is proposed (the solution could be a new model or a
theory that explains the problem)
Paragraph 3: what evidence is put forth that this solution is appropriate (If this is
an empirical article, be sure to briefly describe what kind of empirical study was
done as part of the evidence)

Grading: great - 4; ok - 2; poor - 1

4. Results
Very briefly summarize the important points (observations,
conclusions, findings) and "take home messages" in the article.
Please do not repeat lists of items in the articles - just summarize the
essence of these if you feel they are necessary to include.
Grading: great - 8; ok - 5; poor - 2

5. Class Readings
1. Does this article directly cite any of the class readings, i.e., does any
class reading appear explicitly in its bibliography or reference section?
If not, state this explicitly. If so, clearly describe how the authors use
the cited article. How does the article you are reviewing relate to
and/or build upon the class article it cites?
If this article does not cite any class readings then just state this. (If
you do not state this explicitly, you will not receive credit for this
section.) Do not discuss any other readings, such as other readings on
the same topic or by the same author. Save any discussions of similar
articles for your synthesis section below.
2. Do any of the class readings cite your article (besides the textbook)?
If so, clearly describe how.
If no class readings cite your article, then write in your review "No class
readings cite this article." (If you do not state this explicitly, you will
not receive credit for this section.)
Be sure to add all references you cite to the bibliography.
Grading: great - 4; ok - 2; poor - 1 {If none, then score 4 by default if
this has been stated explicitly.}

6. Contributions
An article makes a "contribution" by adding to the knowledge of
researchers in a research field. An article can make a contribution to
the research field in many ways. Does it provide a new way to look at a
problem? Does it bring together or "synthesize" several concepts (or

frameworks, models, etc.) together in an insightful way that has not

been done before? Does it provide new solutions? Does it provide new
results? Does it identify new issues? Does it provide a comprehensive
survey or review of a domain? Does it provide new insights?
Also, is it salient (relevant and current) to a particular scientific issue or
managerial problem? Are the issues addressed introduced in a way
that their relevance to practice is evident? Would answers to the
questions raised in the article likely to be useful to researchers and
managers?
Note: Do not discuss the contributions of the technologies the article
describes, but rather the contributions of the article itself!
The article's contributions should be original. To the best of your
knowledge, are they? Are the article's take-home messages new?
Describe each contribution clearly in a separate paragraph or bullet
point. Discuss why the contribution is important.
Alternatively, if you believe the article makes no contributions, explain
why clearly.
Grading: great - 8; ok - 5; poor - 2

7. Foundation
Good research often is built upon theories and frameworks that other
researchers have developed. Sometimes articles will be substantially
based upon this prior work, and refer back to it in some detail. (Not all
research articles will do this.)
Which theoretical foundations does this article and research build on, if
any? In what ways? Include references/citations of the foundation work.
(You can determine this in part from the works the article cites.)
Note, however, that most works cited are not core foundational work,
but rather just support certain aspects of the article. Similarly, do not
confuse a general discussion of related topics as foundational work.
If the article does not build upon key pieces of prior research, then
write in your review "This article does not build upon any foundation
research." (If you do not state this explicitly, you will not receive credit
for this section.)
Grading: great - 4; ok - 3; poor -1 {If none, then score 4 by default if
this has been stated explicitly}

8. Synthesis with Class Materials

The synthesis section should be at least one full page.
Synthesis means analyzing a particular topic by comparing and
contrasting it with, and thinking about it from the viewpoint of, the
class materials from across the semester. These materials include the
articles, models, frameworks, guidelines and other concepts we've

covered. (Of course, only certain materials will be relevant for any
given article.)
Note: You have to do this synthesis! You need to relate this article to
other things we have studied, so by definition you will not find this
analysis in the article itself!
Discuss the article's research ideas and results in terms of any relevant
materials covered in class or which you have found in the readings. You
can also check the concepts in the "to know" link on the "quick links"
portion of the course Web site. Cite these readings explicitly, including
their source in the bibliography and a bibliographic marker in the text
(e.g., [Turoff et al., 1999]).
You also could analyze the approach the author took to the article's
analysis and discussion. Discuss the article's approach and results in
terms of one or more of the frameworks, etc., from the text or
readings, or any you find elsewhere. For example, if the authors
discuss any type of information system, you could use Alter's WCA
analysis to examine how they approached that information system. Try
to do this for all the models and frameworks, etc., which apply to your
article.
As part of this analysis, reference other articles you've read, when
appropriate. Compare the approach, results and contribution with all
articles about similar topics or with a similar approach. For example, if
your article develops a new framework, compare it with
Bandyopadyhah's framework criteria (and vice versa - whoever does
Bandyopadyhah's article could test his criteria on frameworks from the
other readings). Include any articles you cite in the bibliography and
use bibliographic markers in the text.
For all of these, do your synthesis comparison in as much depth as you
can!
Grading: four items up to 20 points total (12 points plus 8 points extra
credit) - for each item: great - 5 ok - 2; poor - 1
Great: discussed deeply and relating the article in detail with the
synthesized models and frameworks.
OK: the synthesized information is only discussed in general

9. Analysis
Note: Many people assume this category is the same as "General
Critique". It is not. General Critique is a different category from this,
and follows below.
What has changed since the article was written? How do it's lessons,
ideas and theories still apply? To what extent has its issues been
resolved?
Grading: great - 4; ok - 2; poor - 1

Additional Analysis
Optionally, try applying the article's models, frameworks and
guidelines, etc. yourself. Do you find them useful?
In addition, you may optionally add your own additional analysis in a
separate subsection. (Do not repeat the author's analysis in the paper you could summarize this as part of the results section.)
Grading: this section is extra credit only: great - 8; ok - 5; poor - 2

10. General Critique

In this section you should state your opinions of how well (or poorly)
the authors did their research and presented the research results in the
article. Your critique can contain both positive and negative comments.
Justify and explain in detail each of your critique points in a separate
paragraph of at least 4-5 sentences.
The following are suggestions only:
Does it build upon the appropriate foundation (i.e., upon appropriate prior
research)?
Did the authors choose the correct approach, and then execute it properly?
How confident are you in the article's results, and why?
Are its ideas really new, or do the authors simply repackage old ideas and perhaps
give them a new name?
Do the authors discuss everything they promise in the article's introduction and
outline?
What are the article's shortcomings (faults) and limitations (boundaries)? Did it
discuss all of the important aspects and issues in its domain (topic area)?
In what way should the article have made a contribution, but then did not?
Do the authors make appropriate comparisons to similar events, cases or
occurrences?
How complete and thorough a job did the authors do? Do the authors include an
adequate discussion, analysis and conclusions? Did they justify everything
adequately? Did they provide enough background information for the intended
audience to understand it? For you to understand it?
Were there adequate and appropriate examples and illustrations?
For full credit, ask yourself these questions when justifying your
critique points:
why/why not?
how?
what distinguishes the differences/different approaches, and in what ways?
Grading: four items up to 16 points total (10 points plus 6 points extra
credit) - for each item: great - 4; ok - 2; poor - 1

11c. Further Critique of a Conceptual

Article
*** only for conceptual articles

{adapted from guidelines from Dr. Dan Robey, Georgia State

University}
A critique of a conceptual article examines the logic of the arguments
made by the authors. Both strengths and weaknesses should be
identified in a critique. Explain and justify each of your critique points
in at least 3-4 sentences. Give examples whenever possible.
To the best of your abilities, discuss each of the following categories in
a separate paragraph:
1. LOGICAL CONSISTENCY: Do any parts of the article or research
contradict or invalidate other parts? If so, have the authors
acknowledged and explained this adequately?
2. COHERENCE: Does the article make sense? Did the authors
approach this article (and this research) sensibly? Does the article
develop an argument that follows a coherent line of reasoning? Are the
boundaries of the argument reasonably well defined? Does the
argument anticipate most, if not all, rival arguments?
Does the article flow in a logical sequence? Do later parts build
logically upon earlier parts?
3. SUBSTANCE: Does the article provide an argument or a line of
reasoning that offers insight into important issues, or does it merely
summarize previous studies in a shallow way that does not reflect
depth of analysis? Does the article provide ways (a model, framework,
guidelines, etc.) to guide future thinking about the issue(s) the author
is addressing?
4. FOCUS: Is there a clear audience that the authors address? Was the
article written at the appropriate level for this audience?
Grading: for each: great - 3 ok - 2; poor - 1

11e. Further Critique of an Empirical

Article
*** only for empirical articles

{adapted from guidelines from Dr. Dan Robey, Georgia State

University}
A critique of an empirical article examines the strength of the empirical
evidence supporting the author's argument. Both strengths and
weaknesses should be identified in a critique. Explain and justify each
of your critique points in at least 3-4 sentences.
To the best of your abilities, discuss each of the following categories in
a separate paragraph:

1. CLARITY: Is the article's purpose and argument clear? Do the

researchers clearly develop a major research question, proposition, or
hypothesis that is to be evaluated in the empirical study and discussed
in this article? If the study is exploratory (preliminary), is sufficient
justification for an exploratory strategy given?
2. THEORETICAL GROUNDING: Is the researcher's argument grounded
in more basic theory? Is it clear whether the structure of the empirical
study (i.e., what they do) was derived from theory, or just made up? In
theory-building articles, is the need for new theory adequately
established?
3. DESIGN OF RESEARCH INVESTIGATION: Is it clear exactly how the
empirical study was carried out? Is the design of the research approach
(field study, experiments, questionnaires, etc. - both contents and how
they will be used) adequate to address the common threats to internal
and external validity? Have appropriate controls been established, and
is the selection of research sites justified? Are the hypotheses and
experiments, etc., significant?
4. MEASUREMENT: Empirical studies can have quantitative
measurements (i.e., numeric results) and qualitative or subjective
measurements. Are the measures used adequately described (i.e.,
what is measured in the study and how)? Are data on the reliability and
validity of these measures reported? Does the article feel anecdotal or
solidly supported with evidence? For example, in case or field studies,
are the results well documented? Is it clear who the subjects were, and
with whom interviews were carried out? Were important results crosschecked, i.e., determined across a range of subjects or just gotten from
one or two subjects?
5. ANALYSIS: Is the analysis of empirical data conducted properly? Do
the data conform to the requirements of any statistical tests used? Are
qualitative data adequately described and presented?
6. DISCUSSION AND CONCLUSIONS: In discussing the results of the
empirical study, do the authors remain true to the actual findings of
the study? Are the claims made in the conclusion of the article actually
supported by the empirical data? If the study is exploratory, do the
authors offer research questions or hypotheses for future research?
7. BIASES: Do the biases of the authors affect the design of the
research or the interpretation of the results? Are the authors aware of
potential biases and the affect on the study?
Grading: for each: great - 2 ok/poor - 1

12. Issues (listed by the author)

What open questions or issues has the author stated remain
unresolved? Discuss each in a separate paragraph of 5-10 sentences.
Each issue's paragraph should take the following format:

what is the issue?

why do you believe this is an important issue?
in what way is it unresolved
suggestions for resolving it - if you give your own suggestions (instead of or in
addition to the authors', then precede each with "I would propose ..." If it has been
resolved since the article was written, then state how it was resolved.

Note: If you have any critiques in this section, they most likely belong
in the General Critique section instead.
Grading: 3 items up to 9 points total (6 points plus 3 points extra
credit) - for each item: great - 3; ok - 2; poor - 1

13. Issues (in your opinion)

List several open questions or issues which remain unresolved in your
opinion? For example, what possible future research questions could
arise from this article? Discuss each in a separate paragraph of 5-10
sentences. Each issue's paragraph should take the following format:
what is the issue?
why do you believe this is an important issue?
in what way is it unresolved
suggestions for resolving it
Note: If you have any critiques in this section, they most likely belong
in the General Critique section instead.
Grading: 4 items up to 12 points total (6 points plus 6 points extra
credit) - for each item: great - 3; ok - 2; poor - 1

14. Impact
To determine how much impact this article has had, do a citation
analysis. Discuss what this citation analysis shows, and why; don't just
list the citations! (See the Citation Analysis Guidelines (.doc) and
Handout (.pdf) posted on the course Web site.)
If the article has no citations, then write in your review "I found no
citations in the Science Citation Index, the Social Sciences Citation
Index or on the Internet." Then clearly explain why you believe there
were no citations at all. If you found citations in some indexes or on the
Internet but not the others, then explain this as well.
Include your citation lists in an appendix to your review (see below for
details).
Grading - impact discussion: great - 3; ok - 2; poor - 1

15. Questions

List three insightful questions of your own, arising from this article. Do
not ask definitions, but rather questions that really make one think.
Grading: 3 questions, up to 6 points total - for each question: great/ok 2; poor - 1

16. Annotated Bibliography

For every item you have cited in your report, you need a full reference
and an annotation explaining it. This includes references to any class
materials, as well as the three additional citations utilized in sections 614.
1. List the full bibliographic references (authors, title, journal name, volume, issue,
year, page numbers, etc.) for anything you have cited in your review.
IMPORTANT: This is NOT the bibliography listed at the end of the article. It is
the bibliographic references for any readings you yourself referred to inside your
review.
2. Write 2-4 sentences describing the article.
3. Write 2-3 sentences describing why you cited it.
Also, be sure that you have included a bibliographic marker to each
(such as [Bieber & Smith, 2001]) in the text of your review.
Grading: -5 if missing references; -3 if you mention the authors
explicitly in your text and put the references in this bibliography
section, but forget to explicitly place citation markers in your text.

17. Citation Analysis Appendix

There is a separate page on the course Web site describing citation
analysis.
This appendix will have three sections:
the citations you found in the Science Citation Index
the citations you found in the Social Sciences Citation Index
the citations you found through a thorough Web search on the Internet
If the article has no citations for any of these three, then write in that
section "I found no citations in the [Science Citation Index or the Social
Sciences Citation Index or on the Internet]."
Note, if your article has more than 20 citations, you only need to
include a selection of them:
State how many citations each index has and the Web search found
List 1-2 citations for each year in which the article has been cited. Try to include
citations from several different journals spread over your selection ? Include a
citation analysis to see who has cited it and how.

Grading - citation analysis: 2 points for each of the regular indexes,

and 2 points for the Internet search; if the article is from 1998 or later
then: 1 point for each of the regular indexes, 4 points for the Internet
search.