132

IEEE TRANSACTIONS ON EDUCATION, VOL. 43, NO. 2, MAY 2000

Data Warehousing: A Tool for the Outcomes

Assessment Process
Joanne Ingham

AbstractTo meet EC-2000 assessment criteria, information

from all university constituents needs to be routinely collected
and tracked longitudinally. Typically, one finds that even when
the information exits, it is located across several decentralized
data bases run by different units. Additionally, many universities
are either running older mainframe legacy systems or are in
the process of making the transition to newer systems. These
conditions make it difficult to collect information regarding
students and their performance throughout their undergraduate
career and after. Data warehousing is a system of organizing
institutional data that can support the assessment process. This
paper describes one universitys experience with the development
and management of a data warehouse through the support of the
National Science Foundation Gateway Coalition.
Index TermsAssessment, outcomes assessment.

A review of the literature reveals that data warehousing is becoming an increasingly popular way to store and retrieve data,
primarily in business settings, and more recently in colleges
and universities [2]. Information about this new application has
been reported in conference proceedings [3][5] and in journals
[6][8]. In addition, several data warehouse organizations provide information and white papers on their Web sites.1
This case study will provide a description of the process used
to build and maintain a data warehouse within the Office of Institutional Assessment. As regional and professional accreditation bodies and state education departments adopt requirements
for documentation of continuous improvement, the benefits of
the data warehouse will become evident over time.
II. THE DEVELOPMENT PROCESSCYCLE 1

I. INTRODUCTION

OLYTECHNIC University is a small, private institution

and a member of the National Science Foundation (NSF)
Gateway Coalition. Programs are offered in engineering (chemical, civil, computer, electrical, mechanical), applied sciences
(chemistry, environmental science, information management,
math, physics, computer science), management, liberal studies,
humanities, and social science. Three campuses serve approximately 1700 undergraduates and 1300 graduate students.
The university has been using the Integrated Student Information System (ISIS) as a main source of data, as well as
the record-keeping system. Departmental records are typically
maintained in local data bases, spreadsheets, and computer
and paper files. Additional information is stored on university
UNIX servers, including e-mail, course information, and course
and departmental distribution lists. Currently, the university
is converting its system into an integrated enterprise system
(Peoplesoft). Given the diversity of the existing systems and
the relative difficulty of retrieving information to respond to
the anticipated expectations of EC-2000 and the outcomes
assessment process, establishing a data warehouse was crucial.
Table I describes the comparative benefits of data warehousing.
A data warehouse literally warehouses information about an
organization in a secured computing systems environment to
allow multiple users to extract meaningful, consistent, and accurate data for decision making [1]. A well-designed data warehouse supports an assessment environment in which human energy is focused on the continuous improvement process and not
the mechanical collection of data.
Manuscript received August 1999; revised November 29, 1999.
The author is with Polytechnic University, Brooklyn, NY 11201 USA (e-mail:
jingham@poly.edu).
Publisher Item Identifier S 0018-9359(00)04310-7.

With support from the NSF Gateway Coalition, the process of

building the data warehouse began in the summer of 1997. The
goal of this project was to store institutional data from all appropriate areas of the university in a relational data-base format to
enable timely, accurate analysis for tracking and analyzing patterns of change. The first cycle of development entailed building
the data warehouse The data were drawn from the university
legacy system, departmental data bases, and external university
data, as depicted in Fig. 1. There were four distinct steps, which,
when completed, resulted in the birth of the data warehouse.
Step 1-Recognition and Support for a Centralized
Process: In response to the assessment expectations of
the NSF Gateway Coalition, as well as EC-2000, the effort
was made to establish the Office of Institutional Assessment
and to support the creation of a longitudinal tracking system
to follow students academic careers. In August 1997, a
director and project leader were hired. All necessary computer equipment and software (Access, SQL, Visual Basic,
ASP, COBOL, C-shell Script) were purchased, installed, and
configured by October. In this instance, the data warehouse
was developed and operational within a six-month period.
The warehouse was populated with ten years of institutional
data from the legacy system. For larger institutions or those
whose systems complexities differ, the time required for this
phase may vary.
Step 2Identify Available University Data: The second
step involved determining which other institutional data existed, who managed it, and how the data were formatted. To
accomplish this, meetings were held with each department
director to identify the nature and format of data they routinely
1See the Data Warehousing Knowledge Center at http://www.datawarehousing.org and The Business Intelligence and Data Warehousing Glossary at
http://www.sdgcomputing.com/glossary.htm.

00189359/00$10.00 2000 IEEE

INGHAM: DATA WAREHOUSING

133

TABLE I
COMPARATIVE BENEFITS OF USING A DATA WAREHOUSE VERSUS A LEGACY SYSTEM

Fig. 1.

Building a data warehouse.

collected and stored. While the array of data available were

substantial, it seemed that each department stored data in
different formats. Integrating the potpourri of data files into
a consistent format presented a real challenge for the project
leader. For example, separate relational data-base management
systems existed for offices whose format was determined by
outside funding agencies, the alumni functions and data were
outsourced, and the academic departments used a variety of
software and, often, paper reports. Data from state reports and
the Cooperative Institutional Research Program and the results
of national assessments were processed by researchers at other
institutions.
As a result of requesting data from each department, a number
of critical issues became apparent. These issues arose, in part,
due to a lack of awareness about the revised accreditation expectations. There was a recognized need to make public, and institutionalize, the outcomes assessment process at the institution.
The benefits of a university statement on assessment were rec-

ognized. Such a statement was prepared and is now included in

the new university catalog. Assessment activities had to become
formally recognized as a routine part of university practice.
Second, several directors were reluctant to share their files
with others. They expressed concern about maintaining the confidentiality of information collected for federally funded programs. To address this concern, the vice president and dean of
engineering and applied sciences sent formal release letters clarifying the fundamental purpose of the request and insuring that
only aggregate data would be reported. Last, meetings were conducted with the directors and data-base managers to determine
the scope of the current data and, most important, to identify the
quality or purity of the data available. Unless meticulous care is
taken to import accurate and clean data, the fundamental value
of the warehouse is compromised.
Step 3Data Integration: After all available data had been
identified and examined, they then had to be integrated into the
warehouse. For research purposes and longitudinal tracking for
the Gateway Coalition, a decision was made to include ten years
of history data. The history data were migrated from the university legacy system. These data included the 21-day census data
files, the freshmen cohort files, graduation files, and registrar
files containing grades. Files for the alumni were requested by
program and years out as needed.
The actual collection, examination, and storage of information in the warehouse involved several activities. All files had
to be checked for data quality and to verify that the data were
valid. All duplicate and/or dead data needed to be eliminated
before migration could take place. In addition, data consistency
was checked repeatedly by cross-referencing data from different
sources.
Step 4Developing a Process to Feed the Data Warehouse: Ideally, the data warehouse should be built with the
capacity to accommodate assessment data as they are collected.
A decision was made to move, when possible, away from
paper and pencil data collection strategies and move toward
electronic, Web-based methods. Testing the usefulness of
several different strategies was accomplished by piloting three
Gateway Coalition assessment tools on a small scale. Table II
describes the comparative benefits of a Web-based process.

134

IEEE TRANSACTIONS ON EDUCATION, VOL. 43, NO. 2, MAY 2000

TABLE II
COMPARATIVE BENEFITS OF PAPER VERSUS WEB-BASED ASSESSMENT

Among the first Gateway assessment surveys prepared was a

faculty survey to assess the degree to which faculty participation
in the coalitions activities fostered instructional innovation in
the classroom. The completed paper forms were collected and
the data were entered manually. This process was very labor
intensive and took three weeks to distribute, collect, and enter.
The first pilot effort with a Web-based evaluation involved
the freshman design course. The survey data were collected and
analyzed electronically. Given that the evaluation form existed,
populating the survey was the most time-consuming part. The
Web evaluation form was available to students over a period of
three weeks, and the results were available immediately.
The next pilot involved the mechanical engineering alumni
survey. This survey was prepared and made available as both a
Web-based and paper version A letter with the Web address and
a paper survey were mailed home, and the alumni could select
which method to use. Both approaches were employed to better
determine the relative response rates with an alumni population.
A Web-based civil engineering course-level assessment tool
was tested with nine sophomore- through senior-level courses.
We were, at this point, very confident in the mechanics and security of the technology. Based on further review of the results
of the pilot experiences, an institutional decision was made to
go ahead with a Web-based assessment approach and to pilot
several more assessment tools.

on cohort persistence and graduation rates for gender and ethnicity were run. Last, a series of beta-tests confirmed that the
Web-based survey process was functional and secure.
The startup phase required personnel and equipment. Cycle
1 activities were completed over the course of six months by
one full-time professional working exclusively on this project.
A total of 960 hours were involved. As Cycle 2 was planned,
a decision was made to add a student assistant at 20 hours per
week to handle the Web-based survey activities. As the warehouse became functional, requests for data were, as expected,
numerous. The total spent on new computers, software and peripherals, and eventually a dedicated server came to $15 000.
An additional $35 000 in salaries brought the project costs to
$50 000.
Along with these accomplishments, however, several problems were encountered that necessitated modifications during
Cycle 2. Specifically, as the data warehouse grew, we found
that the data-base software performed slowly. In fact, the functionality and security of the software products were inadequate
for our expanding needs. As Web-based assessment procedures
were applied, the Web connection and speed were not sufficient for our needs. Parallel to this, the university Web team
made a decision to return to a UNIX system utilizing Oracle
for data-base activities.

IV. CYCLE 2SOLVING PROBLEMS

III. EVALUATION OF CYCLE 1
After several months, a number of substantial accomplishments were achieved. An Office of Institutional Assessment was
established and the staff was selected. The space was prepared
and the equipment was set up and functional. Institutional data
for a ten-year period were collected, cleaned, and integrated into
the warehouse. Having run several tests to confirm the accuracy of information generated and the integrity of the system,
the technology was considered operational. The first reports

Experience with the warehouse capabilities and a full appreciation of the scope of the project prompted the decision to move
into a UNIX environment for speed, security, and availability
of institutional support and consistency. Having also made the
decision to adopt a Web-based assessment process, additional
issues had to be considered.
First, additional hardware was purchased. To support the Web
activities, a dedicated server was added to our system. A Sun
Solaris 26 Ultra 10 server and Sun DDS2 backup tape system

INGHAM: DATA WAREHOUSING

were purchased The software selected included Apache Web

Server, Perl 5m and MySQL relational data base with built in
security features. The MySQL data base will allow for a table
up to 6 GB, which we estimate to be adequate for a few decades
of institutional data. Concerns about sufficient room would not
be an issue for some time to come.
With this setup, the Apache could handle up to 50 concurrent users. This feature is more than adequate for an organization with an anticipated enrollment of 5000 graduate and undergraduate students. Based on the first cycle results, new features
such as data integrity checking were also introduced. Results
are now processed before they are submitted or inserted into the
data warehouse. As the warehouse continued to grow, other features were introduced to allow for searches by two, three, or four
letters of the last name to minimize typing.
V. WEB-BASED PROCESS MODIFICATIONS
When the preparation began for university-wide Web-based
course evaluations, a number of critical steps were taken. It was
at this juncture that the Gateway Coalition assessment activities
became integral to the technology.
Working with the dean, department heads, faculty and the
Strategic Planning Team, the universitys mission statement,
program- and course-level goals, and objectives were prepared.
For each undergraduate course, the goals and objectives were
linked with the competencies identified by the Gateway Coalition and EC-2000. A course evaluation form was developed incorporating two component parts. The first part, developed by
the Gateway Coalition, was used to assess specific competencies and skills linked to EC-2000 Criterion 3. The second part
was a locally prepared course evaluation form developed by the
faculty.
Based on lessons learned through beta-tests, a course evaluation process was established that allows each undergraduate
student to complete course evaluations on a Web site2 only for
courses they were registered for that semester. We learned that
the Web site should be available approximately three weeks
prior to the final exam period. Second, the Web address should
be easy to remember and accessible from off-campus locations.
The design of the system insured that students were limited
to evaluating only courses they were registered in, and only
one time. The evaluations were easy to complete and could be
completed at one sitting or several, as desired. Students were
also able to submit written comments. The abundance and
richness of written commentary provided in pilots were delightfully surprising. A few sentences, as well as paragraphs of
thoughtful commentary, were submitted. The comments ranged
from positive to negative. The feedback was often filled with
praise for specific aspects of the course and instructor. Critical
comments were, for the most part, constructive. The student
responses were analyzed and prepared for dissemination using
Access. This feedback then became part of the data warehouse.
The final report was made available to the campus community
on the university intranet.

2See

www2.poly.edu.

135

VI. WAREHOUSE AS A RESOURCE

As faculty and administrators learned more about the new
expectations of the NSF Gateway Coalition, ABET, CSAB, and
Middle States and the importance of outcomes assessment, the
volume of requests increased. Additionally, as information on
retention and graduation rates was disseminated, the value of the
assessment process and capabilities of the longitudinal tracking
system were slowly recognized.
The benefits to the faculty, department heads, and administrators were several. Access to timely, consistent, and thorough
data as needed was faster and more convenient. The administration of alumni surveys by the assessment office allowed the faculty to look at the process and results and not spend a lot of energy on the administration and collection of information. The establishment of routine procedures for user-friendly course-level
assessment meant that the course evaluation process would be
handled on an institutional rather than department level. The
warehouse is also a resource to support the development and implementation of assessment plans for grants. Last, the university
now has the ability to support institutional research using the
longitudinal tracking system.
VII. CONCLUSION
The data warehouse currently manages 100 000 student
records covering a ten-year span, as well as 30 000 course
records over a similar period of time. Data on freshman cohort
retention across any number of variables, as well as similar
graduation history, are available from 1988. Additional data
have been integrated into the system including alumni data,
student placement test scores, and co-op information. Further,
compared to statically processed information, the new data
warehouse has the capability of on-line dynamic results report
generation.
As benchmark data are gathered and the outcomes assessment process is refined, the data warehouse will expand exponentially. The end result is the availability of superior means of
documenting and analyzing continuous improvement on an institutional, program, and course level.
ACKNOWLEDGMENT
The author wishes to thank the members of the NSF Gateway
Coalition for their support with this project. She also wishes to
thank Polytechnic University, Dr. W. R. McShane, Dr. R. Roess,
and A. Polevoy.
REFERENCES
[1] R. Bargain and H. Edelstein, Building, Using, and Managing the Data
Warehouse. Upper Saddle River, NJ: Prentice-Hall, 1997.
[2] J. Frost, M. Dalyrmple, and M. Wang, Focus for institutional researchers: Developing and using a student decision support system, in
Proc. 1998 AIR Annu. Forum, Minneapolis, MN, 1998, pp. 115.
[3] L. J. Mignerey, A data warehouseThe best buy for the money, in
Proc. CAUSE Annu. Conf., Orlando, FL, 1994, p. VI-5-1.
[4] J. D. Porter and J. J. Rome, A data warehouse: Two years later . . .
lessons learned, in Proc. CAUSE Annu. Conf., Orlando, FL, 1994, p.
VI-6-1.
[5] A. Polevoy and J. Ingham, Data warehousing: A tool for facilitating
assessment, in Proc. 29th ASEE/IEEE Frontiers in Education Conf.,
San Juan, Puerto Rico, 1999, p. 11b1-7.

136

[6] C. R. Thomas, Information architecture: The data warehouse foundation, CAUSE/EFFECT, vol. 60, no. 2, p. 3133/3840, Summer 1997.
[7] M. Singleton, Developing and marketing a client/server-based data
warehouse, CAUSE/EFFECT, vol. 16, no. 3, pp. 4752, Fall 1993.
[8] M. Bosworth, Rolling out a data warehouse at UMass: A simple start
to a complex task, CAUSE/EFFECT, vol. 18, no. 1, pp. 4045, Spring
1995.

IEEE TRANSACTIONS ON EDUCATION, VOL. 43, NO. 2, MAY 2000

Joanne M. Ingham was born in Rome, NY, on January 20, 1952. She received
the B.S. degree in biology, secondary education, from the State University of
New York, Oswego, the M.S. degree in counseling from Long Island University,
Brookville Center, NY, and the Ed.D. degree in curriculum and instruction from
St. Johns University, Jamaica, NY.
She has been an Administrator at Polytechnic University, Brooklyn, NY, since
1994 and is currently the Director of Institutional Assessment and Retention in
the Office of Academic Affairs. She has been active in research in the fields
of outcomes assessment and learning styles. She recently completed a research
project with a Fulbright scholar from Mexico comparing the learning styles and
measures of creative performance of American and Mexican engineering students. She taught undergraduate and graduate courses in education at St. Johns
University; Queens College, Queens, NY; and Adelphi University, Garden City,
NY. She is an international Consultant in learning styles.