Digital Preservation SDSC

SDSC Digital Preservation Project with NARA
Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.npaci.edu/DICE/
NationalPartnershipforAdvancedComputationalInfrastructure
SanDiegoSupercomputerCenter
Data and Knowledge Systems Group

Staff
Reagan Moore Ilkai Altintas Chaitan Baru Sheau Yen Chen Charles Cowart Amarnath Gupta George Kremenek M. Kulrul Bertram Ludscher Richard Marciano A. Memon XuFei Qian Roman Olshanowsky Arcot Rajasekar Abe Singer Michael Wan Ilya Zaslavsky Bing Zhu
Graduate Students
A. Bagchi S. Bansal A. Behere R. Bharath S. Bharath L. Sui N. Cotofana D. Le J. Trang L. Yin +/- NN
Undergraduate Interns
Topics
Digital preservation approach Levels of abstraction Application to NARA collections
Persistent Archive Approach

Preservation of authentic documents Create archivable form for digital entity Define context by assembling a collection Create archivable form for collection Manage persistent archive Support self-instantiating archive Support discovery and presentation
NationalPartnershipforAdvancedComputationalInfrastructure SanDiegoSupercomputerCenter
E R A : A r ch iv a l C o m p o n en t s C o n c ep t
S t o r a g e R e s o u r c e B r o k e r / E x t e n s i b l e M e ta - d a t a C A T a l o g
A c c e s s io n i n g
T ap es
ERA Concept model

A rc h i v a l R ep o s i t o r y
C o ll e c t i o n
G r i d S e c u r i t y I n f r a s t r u c tu r e
R ef e r en c e W o r k b en c h
W o r k b en c h
A c c e s s io n
Q u ery
C o ll e c t i o n D is k s
V e r if y R e b u il d
C o ll e c t i o n
W ra p & C o n t a i n e r iz e
In te rn et
D es c r ib e
M e ta d a t a
P r e se n t
M e d i a ti o n o f
R e c o r ds S c h e d u le s
I n f o r m a t io n u s i n g
X M L
O rd e r F u lf i ll m e n t S y s te m
A r c h iv a l R e s e a r c h C a t a l o g
Fundamental Challenge Technology Evolution

Data is a sequence of bits Presentation applications are needed to display a digital entity, based upon a data model Applications issue I/O calls to operating systems Operating systems send commands to storage and display systems
Presentation of Digital Objects

Application
Operating System
Storage System
Display System
Digital Object
Technology Management - Emulation

Old Application
Wrap Application
New Operating System
New Storage System
New Display System
Digital Object
Technology Management SDSC

New Application
New Operating System

Wrap Storage System Wrap Display System
Old Storage System

Migrate Encoding Format
Old Display System
Digital Object
Specifying Levels of Abstraction

Technology management becomes simpler if the persistent archive infrastructure operates on abstractions, rather than an explicit physical implementation of a resource Need abstractions for
Digital objects Repositories
Managing Distributed Storage

Separate the organization of digital objects from their physical storage
Logical Name Space to manage attributes about the digital objects Data handling system to manage interactions with remote storage systems
Create storage abstraction layer Storage Resource Broker (SRB) provides data management system
Information ManagementLogical Name Space

Set of attributes to describe digital entities that are registered into the logical name space
SRB metadata - Unix file system semantics Provenance metadata - Dublin Core Resource metadata - User access control lists Discipline metadata - User defined attributes
Each digital entity may have unique attributes

SDSC Storage Resource Broker & Meta-data Catalog Levels of Abstraction

Application
C, C++, Linux Libraries I/O Unix Shell Java, NT Browsers DLL / Prolog Python Predicate Web WSDL Clients
Consistency Management / Authorization-Authentication
Logical Name Latency Space Management

Catalog Abstraction Databases DB2, Oracle, Sybase
Data Transport
Metadata Transport
Prime Server
Storage Abstraction Archives File Systems Databases

HPSS, ADSM, HRM UniTree, DMF Unix, NT, Mac OSX DB2, Oracle, Postgres
Servers
Types of Digital Entity Abstractions

Logical representation
What does the digital entity represent? What is the associated meaning?
Physical representation
What is the physical structure of the digital entity?
Levels of Abstraction for Bits

Abstraction for Digital Entity Logical: I-nodes Physical:
Track / Sector
Digital Entity
Bit Stream Physical:

File System (NFS/AFS/NTFS)
Abstraction for Repository Repository
Logical: File Name
Disk
Levels of Abstraction for Data

Abstraction for Digital Entity Logical: Data Model
(units, semantics)
Physical:
Encoding Format (syntax, structure)
Digital Entity
Files
Logical: Name Space
Physical:
Data Handling System -SRB/MCAT
File System, Archive

Information Management
Abstraction layer for interacting with information repositories
Manage the schema and physical table structures of a database Extensible schema User defined attributes
Extensible Metadata CATalog (EMCAT) manages collections mySRB.html interface supports dynamic collection creation
Levels of Abstraction for Information

Abstraction for Digital Entity Logical: Collection Schema Physical: XML Syntax
Digital Entity
Metadata Attributes Logical: Database Schema
Physical: EMCAT/CWM
Database
Knowledge Management - Characterizing Properties of Collections

Characterization of relationships between attributes
Semantic / logical - cross-walks Procedural / temporal - records management Structural / spatial - GIS
Characterization of knowledge repository operations Mapping from collection attributes to discipline concepts Mapping from knowledge relationships to rules for application in inference engines
Levels of Abstraction for Knowledge

Abstraction for Digital Entity Logical: Relationship Schema Physical:
ER/UML/XMI/ RDF syntax
Digital Entity
Concept Space
(ontology instance)
Logical:
Knowledge Repository Schema
Physical:
Model-based Mediation System
Knowledge Repository
Persistent Archives
Storage system abstraction
Logical name space and data manipulations
Information repository abstraction

Logical schema and physical table structure
Knowledge repository abstraction

Topic maps and inference rules
Digital object abstraction

Data model and encoding format
NARA Prototype
Demonstrate ability to ingest, archive, recreate, query, and present a digital object from a 1 million record E-mail collection (RFC1036)
2.5 GB of data 6 required fields 13 optional fields User defined fields (over 1000)
Determine resources required to scale size of collection

XML DTD for E-mail
<!ELEME NT rfc1036_mesg (headers, body)> <!ELEME NT headers (required_headers, optional_headers, other_headers)> <!ELEME NT body #PCDATA>
<!ELEME NT required_headers (From, Date, Newsgroups, Subject, Me ssage-ID, Path)> <!ELEME NT optional_headers (Folloup-To?, Expires?, Reply-To?, Sender?, References?, Control?, Distribution?, Keywords?, Summary?, Approved Lines?, Xref?, Organization?)> <!ELEME NT other_headers other+>  <!ELEME NT From #PCDATA> <!ELEME NT Date #PCDATA> <!ELEME NT Newsgroups #PCDATA> <!ELEME NT Subject #PCDATA> <!ELEME NT Message-ID #PCDATA> <!ELEME NT Path #PCDATA> <!ATTLIST From <!ATTLIST Date <!ATTLIST Newsgroups <!ATTLIST Subject <!ATTLIST Message-ID seqno seqno seqno seqno seqno CDATA #REQUIRED> CDATA #REQUIRED> CDATA #REQUIRED> CDATA #REQUIRED> CDATA #REQUIRED>
<!ATTLIST P ath
seqno CDATA #R EQUIRED>
Formatted Message Using XML DTD
Web-based Interface for Accessing the E-mail Collection
Automation of Ingestion Process

Application of an Accessioning Template
Defines the concepts, policies or acceptance of the collection
Creation of attributes that represent the accessioning template concepts Analysis of attributes for anomalies and implied inherent knowledge
Information Generation Processes

Create occurrence index
(Occurrence, attribute, value) This is needed to be able to recreate original form of digital object
Analyze completeness of information

Inverse index of attribute values Identifies unexpected values - consistency
Analyze closure of collection

Are additional concepts needed to represent inverse index value ranges?
Ingestion Processes for Collection
Aggregation of original objects into containers for storage
Data Organization
Data Storage

Migration of objects into a standard representation
Information Generation Attribute Selection Attribute Tagging
Data Organization
Collection Storage

Accession Template Closure Concept/Attribute Attribute Inverse Indexing
Knowledge Generation Attribute Selection Attribute Tagging
Information Generation Occurrence Tagging
View Management Data Organization

Collection Storage
Persistent Collection
Define context for archiving data -annotate information content Create archivable form - standard encoding format Archive information content along with data Test closure of the collection - all digital objects that can be discovered in the collection are members of the collection Test completeness of the collection - inherent relationships within the collection can be cast in terms of attributes generated from the annotated information.
Differentiate between inherent knowledge and anomalies / artifacts
Self-Instantiating Archive
Archive the processes that are used to control the ingestion process
Conversion to archivable form Annotation of information content
When accessing the collection, retrieve the processes and the original digital objects
Apply the processing steps to re-create the information content Query the result to discover desired digital objects
A self-instantiating archive is a virtual data grid

Differentiating between Data, Information, and Knowledge

Data
Digital object Objects are streams of bits
Information
Any tagged data, which is treated as an attribute. Attributes may be tagged data within the digital object, or tagged data that is associated with the digital object
Knowledge
Relationships between attributes Relationships can be procedural/temporal, structural/spatial, logical/semantic, functional
Types of Knowledge Relationships

Logical / semantic
Digital Library cross-walks
Temporal / procedural
Workflow systems
Spatial / structural
GIS systems
Functional / algorithmic
Scientific feature analysis
Knowledge Based Data Grids

Ingest Services Management
XTM DTD
Access Services
Knowledge
Relationships Between Concepts
Knowledge Repository for Rules
Rules - KQL SDLIP
Knowledge or Topic-Based Query / Browse
XML DTD
(Model-based Access) Information Repository Attribute- based Query
Information
Attributes Semantics
(Data Handling System - SRB) Grids Data Fields Containers Folders

MCAT/HDF
Storage (Replicas, Persistent IDs)
Feature-based Query
Further Information
http://www.npaci.edu/DICE

Digital Preservation SDSC

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Digital Preservation SDSC

Enviado por

Direitos autorais:

Formatos disponíveis

SDSC Digital Preservation Project with NARA

Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.npaci.edu/DICE/

Data and Knowledge Systems Group

Persistent Archive Approach

ERA Concept model

Fundamental Challenge Technology Evolution

Presentation of Digital Objects

Technology Management - Emulation

New Operating System

New Storage System

New Display System

Technology Management SDSC

New Operating System

Old Storage System

Old Display System

Specifying Levels of Abstraction

Managing Distributed Storage

Information ManagementLogical Name Space

Each digital entity may have unique attributes

SDSC Storage Resource Broker & Meta-data Catalog Levels of Abstraction

Consistency Management / Authorization-Authentication

Logical Name Latency Space Management

Storage Abstraction Archives File Systems Databases

Types of Digital Entity Abstractions

Levels of Abstraction for Bits

Bit Stream Physical:

Abstraction for Repository Repository

Logical: File Name

Levels of Abstraction for Data

Abstraction for Repository Repository

Logical: Name Space

File System, Archive

Levels of Abstraction for Information

Metadata Attributes Logical: Database Schema

Abstraction for Repository Repository

Knowledge Management - Characterizing Properties of Collections

Levels of Abstraction for Knowledge

Abstraction for Repository Repository

Information repository abstraction

Knowledge repository abstraction

Digital object abstraction

Determine resources required to scale size of collection

XML DTD for E-mail

seqno CDATA #R EQUIRED>

Formatted Message Using XML DTD

Web-based Interface for Accessing the E-mail Collection

Automation of Ingestion Process

Information Generation Processes

Analyze completeness of information

Analyze closure of collection

Ingestion Processes for Collection

Aggregation of original objects into containers for storage

Ingestion Processes for Collection

Ingestion Processes for Collection

Knowledge Generation Attribute Selection Attribute Tagging

Information Generation Occurrence Tagging

View Management Data Organization

A self-instantiating archive is a virtual data grid

Differentiating between Data, Information, and Knowledge

Types of Knowledge Relationships

Knowledge Based Data Grids

Relationships Between Concepts

Knowledge Repository for Rules

Rules - KQL SDLIP