Escolar Documentos
Profissional Documentos
Cultura Documentos
Using a Namespace
FASTFIND LINKS Document Organization Product Version Getting Help Table of Contents
MK-99ARC023-04
Copyright 20092011 Hitachi Data Systems Corporation, ALL RIGHTS RESERVED No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or stored in a database or retrieval system for any purpose without the express written permission of Hitachi Data Systems Corporation (hereinafter referred to as Hitachi Data Systems). Hitachi Data Systems reserves the right to make changes to this document at any time without notice and assumes no responsibility for its use. This document contains the most current information available at the time of publication. When new and/or revised information becomes available, this entire document will be updated and distributed to all registered users. Some of the features described in this document may not be currently available. Refer to the most recent product announcement or contact your local Hitachi Data Systems sales office for information about feature and product availability. Notice: Hitachi Data Systems products and services can be ordered only under the terms and conditions of the applicable Hitachi Data Systems agreement(s). The use of Hitachi Data Systems products is governed by the terms of your agreement(s) with Hitachi Data Systems. By using this software, you agree that you are responsible for: a) Acquiring the relevant consents as may be required under local privacy laws or otherwise from employees and other individuals to access relevant data; and b) Ensuring that data continues to be held, retrieved, deleted, or otherwise processed in accordance with relevant laws. Hitachi is a registered trademark of Hitachi, Ltd. in the United States and other countries. Hitachi Data Systems is a registered trademark and service mark of Hitachi, Ltd. in the United States and other countries. All other trademarks, service marks, and company names are properties of their respective owners.
Contents
Preface........................................................................................................ ix
Intended audience . . . . Product version . . . . . . Document organization . Syntax notation . . . . . . Related documents. . . . Getting help. . . . . . . . . Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..... ..... ..... ..... ..... ..... ..... . . . . . . . .. .. .. .. .. .. .. . . . . . . . . . . . . . . . . . . . . . .... .... .... .... .... .... .... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix . .ix ..x . .xi . .xi . xiv . xiv
Understanding objects........................................................................2-1
Object naming considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Contents Using a Namespace
iii
Cryptographic hash value . . . . . . . . . . . . . . . . . . . . Object ingest time and change time . . . . . . . . . . . . . Data protection level. . . . . . . . . . . . . . . . . . . . . . . . Retention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Object retention settings . . . . . . . . . . . . . . . . . . Deleting objects and versions under retention . . . Holding objects . . . . . . . . . . . . . . . . . . . . . . . . Retention classes . . . . . . . . . . . . . . . . . . . . . . . Returned retention information . . . . . . . . . . . . . Specifying retention settings . . . . . . . . . . . . . . . Specifying a date and time . . . . . . . . . . . . . . Specifying an offset . . . . . . . . . . . . . . . . . . . Specifying a retention class . . . . . . . . . . . . . . Shredding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Versioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Creating versions . . . . . . . . . . . . . . . . . . . . . . . Retrieving and listing versions . . . . . . . . . . . . . . Deleting objects with versioning enabled . . . . . . Purging objects . . . . . . . . . . . . . . . . . . . . . . . . Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Custom metadata . . . . . . . . . . . . . . . . . . . . . . . . . .
...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ...... ......
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . 2-2 . . . . . . . . 2-3 . . . . . . . . 2-3 . . . . . . . . 2-3 . . . . . . . . 2-5 . . . . . . . . 2-5 . . . . . . . . 2-6 . . . . . . . . 2-6 . . . . . . . . 2-8 . . . . . . . . 2-8 . . . . . . . .2-10 . . . . . . . .2-11 . . . . . . . .2-13 . . . . . . . .2-13 . . . . . . . .2-14 . . . . . . . .2-14 . . . . . . . .2-15 . . . . . . . .2-16 . . . . . . . .2-16 . . . . . . . .2-17 . . . . . . . .2-17
Accessing a namespace.....................................................................3-1
URLs for HTTP access to the namespace URL formats. . . . . . . . . . . . . . . . . . Enabling URLs with hostnames . . . . Using an IP address in the URL . . . . URL considerations . . . . . . . . . . . . . Authenticating namespace access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... ... ... ... ... ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2 3-2 3-4 3-5 3-6 3-8
iv
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
vi
Glossary Index
vii
viii
Preface
This book is your guide to working with Hitachi Content Platform (HCP) namespaces. It introduces HCP concepts and describes how HCP represents content using familiar data structures. It includes instructions for accessing a namespace and explains how to store, view, retrieve, and delete objects in a namespace, as well as how to change object metadata such as retention and shred settings. It also contains usage considerations to help you work more effectively with namespaces. This book does not discuss using the default namespace. For information on that namespace, see Using the Default Namespace. Note: Throughout this book, the word Unix is used to represent all UNIXlike operating systems (such as UNIX itself or Linux).
Intended audience
This book is intended for people who need to know how to store, retrieve, and otherwise manipulate data and metadata in an HCP namespace. It addresses both those who are writing applications to access one or more namespaces and those who are accessing a namespace directly through a command-line interface or GUI (such as Windows Explorer). If you are writing applications, this book assumes you have programming experience.
Product version
This book applies to release 4.1 of HCP.
ix
Document organization
Document organization
This book contains eleven chapters and one appendix.
Chapter/Appendix
Chapter 1, Introduction to Hitachi Content Platform
Description
Contains an introduction to HCP concepts and explains how namespace access works; also outlines what you can and cannot do with the data and metadata in a namespace Describes objects and their properties, including naming considerations and the metadata HCP stores for each object in a namespace Provides instructions for using HTTP to access a namespace Describes the HTTP commands that let you access and manage objects Describes the HTTP commands that let you access and manage directories Describes the HTTP commands that let you access and manage object metadata Describes the HTTP commands that let you access and manage custom metadata Describes the HTTP API that lets you retrieve metadata for objects that meet criteria that you specify Describes the HTTP commands that get information about namespace configuration and statistics Describes the browser interface that you can use to access and manage namespace content Contains considerations for accessing and using a namespace Contains a reference to all HTTP commands, response codes, and headers Contains the implementation of Java classes used in examples elsewhere in the book
Chapter 3, Accessing a namespace Chapter 4, Working with objects and versions Chapter 5, Working with directories Chapter 6, Working with system metadata Chapter 7, Working with custom metadata Chapter 8, Using the HCP metadata query API Chapter 9, Retrieving namespace information Chapter 10, Using the Namespace Browser Chapter 11, General usage considerations Appendix A, HTTP reference Appendix B, Java classes for examples
Tip: If you are new to HCP, be sure to read the first two chapters of this book before writing HCP applications or accessing namespaces directly.
Syntax notation
Syntax notation
The table below describes the conventions used for the syntax of commands, expressions, URLs, and object names in this book.
Notation boldface Meaning
Type exactly as it appears in the syntax (if the context is case insensitive, you can vary the case of the letters you type) Replace with a value of the indicated type Vertical bar Choose one of the elements on either side of the bar, but not both Square brackets Include none, one, or more of the elements between the brackets Parentheses Include exactly one of the elements between the parentheses Replace with a directory path with no file or object name This book shows: hold=true|false You enter: hold=true or: hold=false This book shows: http[s]://namespace-name.
Example
This book shows: https://namespace-name.tenant-
name.hcp-name.domain-name/rest
You enter: https://finance.europe.hcp.example.com/rest
italics |
[ ]
tenant-name.hcp-name.domain-name/proc
You enter: http://finance.europe.hcp.example.com/proc or: https://finance.europe.hcp.example.com/proc This book shows: (+|-)hhmm You enter: +0500 or: -0500 This book shows: rest/directory-path You enter: rest/corporate/employees
( )
-path
Related documents
The following documents contain additional information about Hitachi Content Platform:
xi
Related documents
Using the Default Namespace This book describes the file system
HCP uses to present the contents of the default namespace. It provides instructions for accessing the namespace by using the HCP-supported protocols for the purpose of storing, retrieving, and deleting objects, as well as changing object metadata such as retention and permissions.
xii
Related documents
Using HCP Data Migrator This book contains the information you
need to install and use the HCP Data Migrator (HCP-DM) utility distributed with HCP. This utility enables you to copy data between local file systems, HCP namespaces, and earlier HCAP archives. It also supports bulk delete operations. The book describes both the interactive window-based interface and the set of command-line tools included in HCP-DM.
Using the HCP Client Tools This book contains the information you
need to install and use the set of client command-line tools distributed with HCP. These tools enable you to find files and to copy and move files to and from namespaces. The book contains many examples that show command-line details and the overall workflow. Note: For most purposes, the HCP client tools have been superseded by HCP Data Migrator. However, they have some features, such as finding files, that are not available in HCP-DM.
xiii
Getting help
Getting help
The Hitachi Data Systems customer support staff is available 24 hours a day, seven days a week. If you need technical support, please call:
United States: (800) 446-0744 Outside the United States: (858) 547-4526
Note: If you purchased HCP from a third party, please contact your authorized service provider.
Comments
Please send us your comments on this document: hcp.documentation.feedback@hds.com Include the document title, number, and revision, and refer to specific sections and paragraphs whenever possible. Thank you! (All comments become the property of Hitachi Data Systems.)
xiv
1
Introduction to Hitachi Content Platform
Hitachi Content Platform (HCP) is a distributed storage system designed to support large, growing repositories of fixed-content data. HCP stores objects that include both data and metadata that describes the data. It distributes these objects across the storage space but still presents them as files in a standard directory structure. HCP provides access to stored objects through the HTTP protocol, as well as through user interfaces such as the Namespace Browser and Search Console. This chapter introduces basic HCP concepts and includes information on what you can do with an HCP namespace.
11
Object-based storage
HCP stores objects in the repository. Each object permanently associates data HCP receives (for example, a file, an image, or a database) with information about that data, called metadata. An object encapsulates:
12
Namespaces provide a mechanism for separating the data stored for different applications, business units, or customers. For example, you could have one namespace for accounts receivable and another for accounts payable. Namespaces also enable operations to work against selected subsets of repository objects. For example, you could perform a query that targets the accounts receivable and accounts payable namespaces but not the employees namespace. Namespaces are owned and managed by administrative entities called tenants. A tenant typically corresponds to an actual organization such as a company or a division or department within a company. A tenant can also correspond to an individual person.
Namespace access
HCP provides several techniques for accessing and managing data in the namespace. These include:
REST interface Metadata query API Namespace Browser Search Console HCP Data Migrator HCP client tools REST interface
Clients use an HTTP-based REST interface to access the namespace. Using this interface, you can perform actions such as adding objects to the namespace, viewing and retrieving objects, changing object metadata, and deleting objects. You can access the namespace programmatically with applications, interactively with a command-line tool, or through a GUI interface.
13
The figure below shows the relationship between original data, objects in a namespace, and the HTTP access protocol.
Client HTTP
Namespace
Object
Original data
Namespace Browser
The HCP Namespace Browser lets you manage namespace content and view information about namespaces. With the Browser, you can:
List, view, and retrieve objects and versions of objects Create empty directories Store and delete objects Display namespace information, including:
14
Retention classes that you can use in a namespace Permissions for namespace access Statistics about a namespace
Search Console
The HCP Search Console is an easy-to-use web application that lets you search for and manage objects based on specified criteria. For example, you can search for objects stored before a certain date or larger than a specified size and then delete them or prevent them from being deleted. The Search Console works with either of two implementations, which must be enabled at the HCP system level: The HDDS search facility This facility interacts with Hitachi Data Discovery Suite (HDDS), which performs searches and returns results to the HCP Search Console. HDDS is a separate product from HCP. The HCP search facility This facility is integrated with HCP and works internally to perform searches and return results to the Search Console.
Only one of the search facilities can be enable at any given time. If neither one is enabled, HCP does not support using the Search Console to search namespaces. The system associated with the enabled search facility is called the active search system. The active search system (that is, HDDS or HCP) maintains an index of data objects in each search-enabled namespace. The index is based on object content and metadata. The active search system uses the index for fast retrieval of search results. When objects are added to or removed from the namespace or when object metadata changes, the active search system automatically updates the index to keep it current. For information on using the Search Console, see Searching Namespaces. Note: Not all namespaces support search. To find out whether a namespace is search enabled, see your namespace administrator.
15
Copy objects, files, and directories between local file systems, HCP
namespaces, and earlier HCAP archives
View the content of objects and files, including the content of old
versions of objects
Rename files and directories on the local file system View object, file, and directory properties Create empty directories Add, replace, or delete custom metadata for objects
HCP-DM has both a graphical user interface (GUI) and a command-line interface (CLI). For information on using HCP-DM, see Using HCP Data Migrator.
16
You can also have HCP compress and decompress metadata query API
requests and responses. For more information on this, see Request HTTP elements on page 5-5.
17
Operations
Retrieve objects and system metadata. Check for object existence. Check for and retrieve custom metadata. Add objects. Create directories. Set and change system and custom metadata. Delete objects, empty directories, and custom metadata. Delete objects and their historical versions. Delete or purge objects regardless of retention. Place objects on hold. Search for objects. For information on this, see Searching Namespaces and Chapter 8, Using the HCP metadata query API.
Some operations require multiple permissions. For example, to place an object on hold, you need to have both write and privileged permissions. Similarly, to perform a privileged purge, you need delete, privileged, and purge permissions. Permissions are set at two levels:
18
Operations
Replication
Replication is the process of keeping selected tenants and namespaces in two HCP systems in sync with each other. Basically, this entails copying object creations, deletions, and metadata changes from one system to the other. HCP also replicates the tenant and namespace configuration, data access accounts, and retention classes. The HCP system in which the objects are initially created is called the primary system. The second system is called the replica. Replication has several purposes, including:
If the primary system suffers irreparable damage, the replica can serve
as a source for disaster recovery.
Note: Replication is an add-on feature to HCP. Not all systems include it.
Operations
You use familiar commands and tools to perform operations on a namespace. Some operations relate to specific types of metadata. For more information on this metadata, see Chapter 2, Understanding objects. Operations that store or retrieve data can optionally transmit the data in gzip-compressed format. For more information on this, see the individual commands used for those operations.
Operation restrictions
The operations you can perform are subject to these restrictions:
19
Operations
Supported operations
You can perform these operations on a namespace:
Write data to the namespace If versioning is enabled, store new versions of existing objects Override default metadata when storing an object Create an empty directory in the namespace Check for object existence View the content of an object View object metadata Delete an object Delete an empty directory Set retention for an object that has none Extend the retention period for an object Set or change a retention class for an object Hold or release an object Enable shredding an object Change the index setting for an object Add, replace, or delete custom metadata for an object
110
Operations
Add or retrieve object data and custom metadata in a single operation Check for and read custom metadata List retention classes available in the namespace List namespace permissions for the user List the namespace statistics List the namespaces accessible to you Use the HCP metadata query API to get information about objects that
meet specified criteria in one or more namespaces
Prohibited operations
HCP never lets you:
Modify the fixed-content portion of an object. Delete an object thats under retention if you do not have the privileged
permission or if the namespace is configured to prevent this operation.
Delete a directory that contains one or more objects. Shorten an explicitly set retention period.
111
Operations
112
2
Understanding objects
Objects in a namespace have a variety of properties, such as a retention period and an index setting. These properties are defined for each object by the object metadata. You can view all the metadata for objects in a namespace and modify some of it. A namespace can be configured to allow objects to have multiple versions. Each version is an independent copy of the object that includes both data and metadata. A new version is created each time you save the object, but not when you modify metadata or custom metadata. A version can also be a special entity that represents a deleted object. Several namespace properties affect objects. For example, the namespace can be configured to automatically delete objects after their retention periods expire. The namespace configuration also specifies defaults for several metadata values. When you add an object to the namespace, the object inherits these values unless you override them. This chapter provides detailed information about objects and their properties, including object names, metadata you can change, custom metadata, and object versions. For a complete list of object metadata, see Checking the existence of an object or version on page 4-12.
21
Object names are case sensitive. Names can include nonprinting characters, such as spaces and line breaks. All characters are valid except the NULL character (ASCII 0 (zero)) and the forward slash (ASCII 47 (/), which is the separator character in directory paths).
The maximum length for the combined directory path and name of an
object, including separators, is 4,095 bytes.
22
The object was closed after being added to the namespace. Any metadata, including custom metadata, was changed. The object was recovered from a replica. The HCP search facility tried but could not index the object. When this
happens, the facility sets the change time for the object to two weeks in the future, at which time it tries again to index it. If an object has not changed since ingestion, the ingest time and change time may not be identical. This is because, the ingest time is set when HCP opens the object for write and the change time is set when HCP closes the object after ingestion is complete.
Retention
Objects have a retention property, which determines how long the object must remain in the namespace. This can range from allowing the object to be deleted at any time to preventing the object from ever being deleted. While an object cannot be deleted due to retention, it is said to be under retention.
23
Retention
Retention settings Each object in a namespace has a retention setting. The namespace is configured with a default retention setting. When you add an object to the namespace, the object inherits the namespace retention setting unless you explicitly override it. You can change the retention setting for an existing object at a later time. Note: If the default retention setting is in the past, new objects that would otherwise get that setting are added with a setting of Deletion Allowed (0). For information on this setting, see Object retention settings on page 2-5. Retention periods The retention period for an object is the length of time the object must remain in the namespace. A retention period can be a specific length of time, infinite time, or no time, in which case the object can be deleted at any time. When the retention period for an object expires, the object becomes deletable. Normally, if you try to delete an object thats under retention, HCP prevents you from doing so. For the exception, see Deleting objects and versions under retention on page 2-5. Retention classes A retention class is a named retention setting. Retention classes let you manage object retention consistently. For more information on retention classes, see Retention classes on page 2-6. Automatic deletion A namespace can be configured to automatically delete objects after their retention periods expire. For an object to be deleted automatically:
24
Retention
Text value
Deletion Allowed Deletion Prohibited
Meaning
The object can be deleted at any time. The object cannot be deleted, except by a privileged delete, and the retention setting cannot be changed. The object does not yet have a retention setting. An object that has this value cannot be deleted, except by a privileged delete. You can change this retention setting to any other setting. The object cannot be deleted, except by a privileged delete, until the specified date and time.
-2
Initial Unspecified
A datetime value in the format described in Specifying a date and time on page 2-10
To provide maximum flexibility, HCP supports several different formats for specifying and retrieving retention settings. The valid formats depend on whether you are retrieving information about an object, specifying a setting, or using a retention class. For information on:
The values that are returned when you get information about an object,
see Returned retention information on page 2-8
The formats that you can use to set a retention value, see Specifying
retention settings on page 2-8
25
Retention
Holding objects
If your data access account has the privileged permission, you can place an object on hold. An object that is on hold cannot be deleted, even by a privileged delete operation. Also, you cannot store new versions of an object that is on hold. Holding objects is particularly useful when the objects are needed for legal discovery. While an object is on hold, you cannot change its retention setting. You can, however, change its shred setting. If the namespace is configured to allow changes to custom metadata for objects under retention, you can also change its custom metadata. Tip: You can use the HCP Search Console to place multiple objects on hold at the same time. For information on permissions, see Data access permissions on page 1-8
Retention classes
A retention class is a named retention setting. For example, a retention class named HlthReg-107 could have a duration of 21 years. All objects assigned to that class could then not be deleted for 21 years after theyre added to the namespace. A retention class can specify:
A duration after object creation 0 (Deletion Allowed) -1 (Deletion Prohibited) -2 (Initial Unspecified)
26
Retention
The format can omit parts with zero values. For example, this value specifies a 21 year retention period:
+21y
You can use retention classes to consistently manage data that must conform to a specific retention rule. For example, if local law requires that medical records be kept for a specific number of years, you can use a retention class to enforce that requirement. Namespace administrators create retention classes. Each namespace has its own independent set of retention classes. When creating a class, the administrator specifies the class name, the retention setting, and whether HCP can automatically delete objects in the class when the retention period expires. Note: Automatic deletion must be enabled for the namespace for objects under retention to be automatically deleted. For more information on automatic deletion, see Automatic deletion on page 2-4. A namespace can be configured to allow administrative users to increase or decrease class retention times and delete classes. Otherwise, the retention time for a class can only be increased and classes cannot be deleted. In all cases, any change to a retention class affects the retention period of objects that have been assigned to that class. You can assign a retention class to an object when you save the object or at a later time. For detailed information on the rules for assigning retention classes, see Specifying retention settings on page 2-8. For information on getting an XML listing of retention classes, see Listing retention classes on page 9-5. For information on viewing the retention class list in the Namespace Browser, see Viewing retention classes on page 10-12.
27
Retention
Value
X-HCP-RetentionString
One of: The retention period end time expressed as a datetime value in the format described in Specifying a date and time on page 2-10 One of the special values Deletion Allowed, Deletion Prohibited, or Initial Unspecified
X-HCP-RetentionClass
The retention class of the object or an empty string if the object is not assigned to a retention class. The information is returned in this format:
(retention-class-name, retention-classduration)
If the retention class is deleted, the retention-classduration value is undefined. X-HCP-RetentionHold A value of true or false specifying whether the object is on hold.
Note: If a class is deleted, the retention periods for objects that were assigned to the class do not change, and the objects are retained until the time specified in the X-HCP-Retention and X-HCP-RetentionString values.
28
Retention
If youre changing the setting for an existing object that has an explicit
datetime retention setting, you can change the setting to lengthen the retention period but not to shorten it. The table below describes the values you can use to specify the retention setting for an object and contains rules for specifying these values. These values are not case sensitive.
Value
0 (zero) or Deletion Allowed
Description
Allows the object to be deleted at any time. You can assign this value to an object only when you add it to the namespace or when its retention setting is -2. The value -0 is equivalent to 0 (zero).
-1 or Deletion Prohibited
Prevents the object from being deleted and its retention setting from being changed. You can assign this value to an object that is not in a retention class at any time. If an object is assigned to a retention class and that class is then deleted, the retention setting for that object changes to -1.
-2 or Initial Unspecified
Specifies that the object does not yet have a retention setting. You can assign this value to an object when you add the object to the namespace. You can also directly change the retention setting for an object from 0 to -2. You can change -2 to any other retention setting.
datetime
Prevents the object from being You can assign this value to an specified date and time is later object. You cannot assign it to setting is -1.
deleted until the specified date and time. object that is not in a retention class if the than the current retention setting for the an object for which the current retention
For a description of the datetime format, see Specifying a date and time below.
29
Retention
(Continued)
Value
offset
Description
Specifies a period for which to retain the object. You can assign this value to an object that is not in a retention class at any time, except when its current retention setting is -1. HCP determines the retention setting date and time from the offset and the time the object was added to the namespace, the current retention setting, or the current time. For a description of the offset format, see Specifying an offset on page 2-11.
C+retention_class_name
Assigns the object to a retention class. You can assign this value to an object if any one of these is true: The original retention period for the object has expired. The original retention period for the object has not expired, and the class results in a retention period that is longer than the current retention period. The retention setting for the object is 0 or -2. The retention setting for the object is -1, and the class has a retention setting of -1. The object is in a retention class, and the duration of the new class is not shorter than the duration of the original class.
For more information on retention classes, see Retention classes on page 2-6.
For more information on the retention settings listed in the first four rows of this table, see Object retention settings on page 2-5.
The calendar date that corresponds to 1450137600 is Tuesday, December 15, 2015, at 00:00:00 EST.
210
Retention Z represents the offset from UTC and is specified as: (+|-)hhmm
For example, 2015-11-16T14:27:20-0500 represents the start of the 20th second into 2:27 PM, November 16, 2015, EST. If you specify certain forms of invalid dates, HCP automatically adjusts the retention setting to make a real date. For example, if you specify 201011-33, which is three days past the end of November, HCP changes it to 2010-12-03.
Specifying an offset
You can set retention by specifying an offset from:
The current time The time at which the object was added to the namespace The current retention setting for the object
Because you can only extend a retention period, the offset must be a positive value. Offset syntax To use an offset as a retention setting, specify a standard expression that conforms to this syntax:
^([RAN])?([+-]\d+y)?([+-]\d+M)?([+-]\d+w)?([+-]\d+d)?([+-]\d+h)?([+-]\d+m)?([+-]\d+s)?
Description
211
Retention
(Continued)
Character N* \d+ y M w d h m s
*
Description
The current time An integer in the range 0 (zero) through 9,999 Years Months Weeks Days Hours Minutes Seconds
R, A, and N are mutually exclusive. If you dont include any of them, the default is R.
The time measurements included in an expression must go from the largest unit to the smallest (that is, in the order in which they appear in the syntax). Tip: When you add an object to the namespace, R, A, and N are equivalent; that is, they all represent the current date and time. Because A and N are more intuitively meaningful, you should use either one of them instead of R for these purposes. Offset examples Here are some examples of using an offset to extend a retention period:
This value sets the retention value to 100 years past the time when the
object was stored:
A+100y
This value sets the end of the retention period to 20 days minus five
hours past the current date and time:
N+20d-5h
This value extends the current retention period by two years and one
day:
R+2y+1d
212
Shredding
For example:
C+SEC17a
Shredding
Shredding, also called secure deletion, is the process of deleting an object and overwriting the places where its copies were stored in such a way that none of its data or metadata can be reconstructed. Every object has a shred setting that determines whether it will be shredded when its deleted. The shred setting values are:
213
Versioning
Versioning
Versioning is the capability of the namespace to store multiple versions of objects. If the namespace supports versioning, you can save a new version of an object, and the previous version will be kept. All objects, including those created when versioning is not enabled, have version IDs. Version IDs are integers. Each time a new version of an existing object is created, it is assigned an ID that is greater than the previous version of the object, but the numbers may not be consecutive. Directories do not normally have versions. One exception exists: if a directory contains or contained one or more versions of an object and also contained, at some time, a subdirectory with the same name as the object, the deleted subdirectory has a version ID. For more information on this, see Listing object versions on page 4-16. If versioning is enabled for the namespace, pruning may also be enabled. Pruning is the automatic deletion of old versions after a specific length of time. HCP does not prune versions of objects that are on hold. Retention settings apply to individual object versions. If you specify a retention value for an object by using an offset or retention class, the retention period is relative to the time the current version was stored, not when the first version was stored. Hold settings apply to all versions of an object; if you put an object on hold, all historic versions are also put on hold. Each version of an object has its own retention, shred, and index settings and its own custom metadata. While the index settings of historic versions can vary, only the current version is returned by a search. Administrators can enable, disable, and reenable versioning without losing version information. However, while versioning is disabled, you cannot list or access historic versions. If versioning is later reenabled, you can again list, retrieve, and purge any preexisting versions (until HCP prunes them because of their age).
Creating versions
If versioning is enabled, you can store new versions of an existing object by storing the object using the same URL you used to create the object originally. HCP retains the previous version until it is automatically deleted or you explicitly purge the object.
214
Versioning
Updates to the system metadata or custom metadata for an object do not create a new version of the object. When HCP creates a new version of an object, the previous version becomes a historic version. The historic version:
Can be accessed by using the version ID that was assigned when you
first stored it. The new version:
Gets a version ID that is greater than the ID of the previous version Inherits the system and custom metadata from the previous version,
unless you specify new metadata when you add the new version to the namespace Note: You cannot create new versions of an object that is under retention or on hold or that has a retention setting of -2 (Initial Unspecified). For more information about creating versions, see Adding an object or version of an object on page 4-2.
215
Versioning
Retains a copy of the deleted object as a historic version. Creates a special version, called a deleted version, as the current
version to indicate that the object has been deleted. This version has a version ID but does not include any object data or metadata. After you delete an object, any attempt to get the object without specifying the version ID of a historic version results in a 404 (Not Found) error. HCP keeps all historic versions of a deleted object until they are pruned or the object is purged. If shredding is enabled on a version, HCP shreds the version when it is automatically deleted. Because HCP keeps deleted objects as historic versions, you can retrieve an accidentally deleted object if HCP has not yet pruned it. To do so, request the object, specifying the version ID of the version you want. You can restore the deleted object by storing the retrieved version using the original URL. When you list the versions of an object that has been deleted and restored, the returned XML for the deleted version identifies the version state as deleted, contains the date and time the object was deleted, and contains the version ID for the deleted version. The remaining XML contains information about the object that you deleted. For an example of a version list that includes a deleted version, see Example: Listing the versions of an object on page 4-19. For more information on deleting objects, see Deleting an object and using privileged delete on page 4-38.
Purging objects
You cannot delete specific historic versions of an object, but if your data access account has the purge permission, you can purge the object to delete all its versions. If the current version of an object is under retention, you cannot purge the object. However, if your data access account has both the purge and privileged permissions, you can perform a privileged purge to delete all versions of the object. For more information on purge and privileged purge operations, see Purging an object and using privileged purge on page 4-43.
216
Indexing
Indexing
Each object has an index setting that is either true or false. The setting is present regardless of how HCP and the namespace are configured. When building the search index, HCP uses this setting to determine whether to include the object in the index. Additionally, metadata query API requests can use this setting as a search criterion, and third-party applications can use this setting for their own purposes. Overriding default index settings The namespace is configured with a default index setting. When you add an object to the namespace, the object inherits the namespace index setting unless you explicitly override it. For more information on overriding default index settings, see Specifying metadata on object creation on page 6-2. Changing index settings You can enable or disable indexing for an existing object. When you disable indexing, HCP removes the object from the search index, so searches do not find the object. When you enable indexing, HCP adds the object to the index. For more information on changing index settings, see Modifying object metadata on page 6-5.
Custom metadata
Custom metadata is user-supplied descriptive information about an object. Custom metadata is stored as a unit. You can add, replace, or delete it in its entirety. You cannot modify it in place. Custom metadata is typically specified using XML, but this is not required. The namespace configuration determines whether HCP checks that custom metadata is well-formed XML. While checking is enabled, if you try to store custom metadata that is not well-formed XML, HCP rejects it. You can use a single HTTP request to store or retrieve object data and custom metadata together. The namespace configuration determines what you can do with custom metadata for objects that are under retention. The namespace can be set to:
217
Custom metadata
Allow only the addition of new custom metadata for objects under
retention and disallow replacement or deletion of existing custom metadata
Storing custom metadata, see Storing custom metadata on page 7-2 Using a single request to store object data and custom metadata, see
Example 4: Storing object or version data with custom metadata (Unix) on page 4-9
218
3
Accessing a namespace
You use the HTTP protocol to access a namespace. The namespace configuration determines whether you need to use HTTP with SSL (HTTPS). To access a namespace, you can write applications that use any standard HTTP client library, or you can use a command-line tool, such as cURL, that supports HTTP. You can also use a web browser to access the namespace. Using the HTTP protocol, you can store, view, retrieve, and delete objects. You can specify certain metadata when you store new objects. You can also change the metadata for existing objects. You can add, replace, and delete custom metadata and get information about the namespace. Additionally, you can get information on objects that match query criteria in one or more namespaces. HCP is compliant with HTTP/1.1, as specified by RFC 2616. This chapter provides general information on using HTTP to access HCP namespaces and describes how to access an individual namespace. For information on URLs for the metadata query API, which can retrieve information about the contents of multiple namespaces, see Request URL on page 8-4. The examples in this book use cURL and Python with PycURL, a Python interface that uses the libcurl library. cURL and PycURL are both freely available open-source software. You can download them from http:// curl.haxx.se. Note: As of version 7.12.1 of PycURL, the PUT method has been deprecated and replaced with UPLOAD. The Python examples in this book show PUT but work equally well with UPLOAD. For a condensed reference of the HTTP methods you use and responses you get when accessing a namespace, see Appendix A, HTTP reference.
31
URL formats
The following sections show the URL formats you can use for namespace access. These formats all use the hostname to identify the HCP system and namespace. As an alternative, you can use an IP address for the system, as described in Using an IP address in the URL on page 3-5. Notes:
The URL formats and examples that follow show https. If the
namespace configuration does not require SSL security for the HTTP protocol, you can specify http instead of https in your URLs.
For example:
https://finance.europe.hcp.example.com
In this example:
32
To access a specific object or directory in the namespace, append the directory or object path to the namespace URL and the interface identifier rest. The format for this is:
https://namespace-name.tenant-name.hcp-name.domain-name/rest /directory-path[/object-name]
Directory and object names are case sensitive. You cannot tell from a URL whether it identifies an object or a directory. URLs for namespace information You may have access to multiple namespaces owned by a given tenant. To retrieve information about these namespaces, use this format for the URL:
https://namespace-name.tenant-name.hcp-name.domain-name/proc
In the URL, proc must be all-lowercase. A URL used to retrieve information about multiple namespaces owned by a tenant must start with the name of one of the namespaces. The response contains information about all the namespaces you can access, not just the one specified in the URL. Heres a URL that can be used to list all accessible namespaces owned by the europe tenant, including the finance namespace:
https://finance.europe.hcp.example.com/proc
33
For more information on retrieving namespace information, see Chapter 9, Retrieving namespace information.
The HCP system is configured for DNS. Check with your namespace
administrator as to whether this is true.
The client hosts file contains mappings from the hostname identifiers for
each namespace to HCP system IP addresses. Note: For information on considerations for using hostnames and IP addresses, see Namespace access by IP address on page 11-2. Using a hosts file All operating systems have hosts files that contain mappings from hostnames to IP addresses. If your HCP configuration does not support DNS, you can use this file to enable access to the namespace by hostname. The location of the hosts file differs among operating systems.:
34
On Mac OS X: /private/etc/host
The hosts file contains lines consisting of an IP address and one or more hostnames, separated by white space. One entry can map multiple hostnames to an IP address, or each mapping can have a separate entry. Each line can have a trailing comment, or comments can be put on separate lines. Comments start with a number sign (#). Blank lines are ignored. The hosts file must map each hostname that you use to an HCP system IP address. The hostnames must be fully qualified. For example, if you need to access the contents of two namespaces owned by the tenant named europe, you need to have two hostname entries, one for each namespace. Hostname mapping considerations An HCP system has multiple IP addresses. You can map each hostname to more than one of these IP addresses in the hosts file. The way multiple mappings are used depends on the client platform. For information on how your client handles multiple mappings in a hosts file, see your client documentation. Note: If any of the HCP system IP addresses are unavailable, timeouts may occur when using a hosts file for system access. Sample hosts file entries The following lines show the portion of a hosts file that maps the hostnames for two namespaces, which are owned by a single tenant, to two IP addresses, 192.168.130.13 and 192.168.14. The example has only a single hostname per entry to make the file contents easier to read.
# europe tenant data access 192.168.130.13 finance.europe.hcp.example.com# finance namespace 192.168.130.14 finance.europe.hcp.example.com# finance namespace 192.168.130.13 support.europe.hcp.example.com# support namespace 192.168.130.14 support.europe.hcp.example.com# support namespace
35
If you use an IP address in the URL, you provide the hostname in an HTTP Host header. In curl commands, you use the -H option and specify a Host header with the namespace, tenant, HCP system, and domain names, in this format:
-H "Host: namespace-name.tenant-name.hcp-name.domain-name"
In the preceding syntax, the left and right brackets ([]) and parentheses (()) are required and do not indicate an optional component. This example shows a Python script that uses an IP address:
import pycurl import os filehandle = open("wind.jpg", 'rb') curl = pycurl.Curl() curl.setopt(pycurl.URL, "https://192.168.125.125/rest/images/wind.jpg") curl.setopt(pycurl.COOKIE, "hcp-ns-auth= bXl1c2Vy:3f3c6784e97531774380db177774ac8d") curl.setopt(pycurl.SSL_VERIFYPEER, 0) curl.setopt(pycurl.SSL_VERIFYHOST, 0) curl.setopt(pycurl.HTTPHEADER, ["Host: finance.europe.hcp.example.com"]) curl.setopt(pycurl.PUT, 1) curl.setopt(pycurl.INFILESIZE, os.path.getsize("wind.jpg")) curl.setopt(pycurl.READFUNCTION, filehandle.read) curl.perform() print curl.getinfo(pycurl.RESPONSE_CODE) curl.close() filehandle.close()
URL considerations
The following considerations apply to specifying URLs in HTTP requests against a namespace. For considerations that apply specifically to naming new objects, see Object naming considerations on page 2-2.
36
URL length For all HTTP methods except POST, the portion of a URL after rest, excluding any appended parameters, is limited to 4,095 bytes. If an HTTP request includes a URL that violates that limit, HCP returns a status code of 414. URL character case All elements except for the hostname are case sensitive. Object names with non-ASCII, nonprintable characters When you store an object with non-ASCII, nonprintable characters in its name, those characters in the name are percent-encoded in the name displayed back. Regardless of how the name is displayed, the object is stored with its original name, and you can access it either by its original name or by the name with the percent-encoded characters. Non-UTF-8-encoded characters in directory listings When you view a directory listing in a web browser, non-UTF-8-encoded characters in object names are percent-encoded. Percent-encoding for special characters Some characters have special meaning when used in a URL and may be interpreted incorrectly when used for other purposes. To avoid ambiguity, percent-encode the special characters listed in the table below.
Character
Space Tab New line Carriage return + % # ? & %20 %09 %0A %0D %2B %25 %23 %3F %26
Percent-encoded value
Percent-encoded values are not case sensitive. Note: Do not percent-encode metadata parameters appended to URLs. For information on these parameters, see Specifying metadata on object creation on page 6-2.
37
Quotation marks with URLs in command lines When using a command-line tool to access the namespace through HTTP, you work in a Unix, Mac OS X, or Windows shell. Some characters in the commands you enter may have special meaning to the shell. For example, the ampersand (&) used in URLs to join multiple metadata parameters also often indicates that a process should be put in the background. To avoid the possibility of the Unix, Mac OS X, or Windows shell misinterpreting special characters in a URL, always enclose the entire URL in double quotation marks.
Note: HCP does not require passwords to be hashed. For security reasons, never use a clear-text password unless you are using HTTP with SSL. If you do not provide valid credentials, HCP responds with a 302 (Found) error message and Location header with a URL. If you get this error, do not use the URL. Instead, use correct credentials in the hcp-ns-auth cookie The user authentication cookie for a data access account named myuser and a password value of p2Ss#0rd looks like this:
hcp-ns-auth=bXl1c2Vy:6ecaf581f6879c9a14ca6b76ff2a6b15
The GNU core utilities provide base64 and md5sum commands that can generate the required values. With these commands, a line such as this can create the required cookie:
echo hcp-ns-auth=`echo -n user-name | base64`:`echo -n password | md5sum` | awk '{print $1}'
38
The character before echo, before and after the colon, and following md5sum is a backtick (or grave accent). The echo command -n option prevents the command from appending a newline to its output. This is required to ensure correct Base64 and MD5 values. For more information about the GNU core utilities, see http:// www.gnu.org/software/coreutils/. Other tools to generate Base64-encoded text and MD5 hash values are available for download on the web. For security reasons, do not use interactive public web-based tools to generate these values.
39
310
4
Working with objects and versions
HCP lets you perform operations on individual objects. Also, if the namespace youre using is configured to allow versioning, HCP can store multiple versions of an object. You can use this technique, for example, to preserve multiple states of a file that changes over time. Note: In this chapter, the term object refers to an object that does not have versions or to the current version of an object with multiple versions. The term historic version refers to any version of an object except the current one. You can perform these operations on objects and object versions:
Check whether an object or version exists List the versions of an object Retrieve all or part of an object or version (with or without custom
metadata)
Delete an object or the current version of an object Purge all versions of an object
This chapter describes how to perform these operations, including how to delete and purge objects that are under retention. You can also manage the metadata and custom metadata for an object. For more information on this, see Chapter 6, Working with system metadata, and Chapter 7, Working with custom metadata.
Working with objects and versions Using a Namespace
41
An hcp-ns-auth authorization cookie A URL specifying the location in which to store the object A body containing the fixed-content data to be stored in the namespace
Request contents sending data in compressed format You can send object data in compressed format and have HCP decompress it before storing it. To do this, in addition to specifying the request elements listed above:
Use gzip to compress the content before sending it. Include a Content-Encoding request header with a value of gzip. Use a chunked transfer encoding.
42
Request contents adding object data and custom metadata together If youre using a single request to add object data and custom metadata, the PUT request must specify these HTTP elements:
An hcp-ns-auth authorization cookie An X-HCP-Size header specifying the size, in bytes, of the data object A URL specifying the location in which to store the object A type URL query parameter with a value of whole-object A body containing the fixed-content data to be stored, followed by the
custom metadata, with no delimiter between them When you store an object with its custom metadata in a single operation, the object data must always precede the custom metadata. This differs from the behavior when you retrieve an object and its custom metadata, where you can tell HCP to return the results in either order. You can send the body in gzip-compressed format and have HCP decompress the data before storing it. To do this, make sure that the request includes the elements described in Request contents sending data in compressed format on page 4-2. Access permission To add an object or version of an object to the namespace, you need write permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
201
Meaning
Created
Description
HCP successfully stored the object. If versioning is enabled and an object with the same name already exists, HCP created a new version of the object.
43
Code
400
Meaning
Bad Request One of:
Description
The request has a Content-Encoding header that specifies gzip, but the data is not in gzip-compressed format. The request has a type=whole-object query parameter, and either: The request does not have a X-HCP-Size header. The X-HCP-Size header value is greater than the content length.
HCP has custom metadata XML checking enabled, and the request includes custom metadata that is not wellformed XML. If the request that causes this error contains both object data and custom metadata, HCP creates an empty object before it returns the error. To resolve this issue, you can either: Fix the custom metadata and retry the request. Add the object again without any custom metadata, thereby replacing the empty object. You can then fix the metadata at a later time and add it in a separate request.
The URL in the request is not well-formed. The request contains an unsupported parameter or an invalid value for a parameter.
If more information about the error is available, the HTTP response headers include the HCP-specific X-HCPErrorMessage header.
44
Code
403
Meaning
Forbidden One of:
Description
The authenticated user doesnt have write permission. For a request to add a version of an existing object, the object is under retention. The namespace does not exist. The access method (HTTP or HTTPS) is disabled.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 409 Conflict One of: HCP could not add the object to the namespace because it already exists and versioning is not enabled. HCP could not add a new version of an existing object because another version is currently being added.
One of: Not enough space is available to store the object. Try the request again after objects or versions are deleted from the namespace or the namespace capacity is increased. The request is trying to store an object that is larger than two TB. HCP cannot store objects that are larger than two TB. The request is trying to store custom metadata that is larger than one GB. HCP cannot store custom metadata that is larger than one GB.
415
The request has a Content-Encoding header with a value other than gzip.
45
Request-specific response headers The table below describes request-specific response headers returned by a successful request. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
Header
X-HCP-Hash
Description
The cryptographic hash algorithm HCP uses and the cryptographic hash value of the stored object or version, in this format: X-HCP-Hash: hash-algorithm hash-value You can use the returned hash value to verify that the stored data is the same as the data you sent. To do so, compare this value with a hash value that you generate from the original data.
X-HCP-CustomMetadata Hash
Returned only if the request contains both data and custom metadata. The cryptographic hash algorithm HCP uses and the cryptographic hash value of the stored custom metadata, in this format: X-HCP-CustomMetadataHash: hash-algorithm hash-value You can use the returned hash value to verify that the stored custom metadata is the same as the metadata you sent. To do so, compare this value with a hash value that you generate from the original metadata.
X-HCP-VersionId
Example 1: Storing an object or object version Heres a sample HTTP PUT request that stores a new object or new version of a file named wind.jpg in the images directory.
46
Request headers
PUT /rest/images/wind2.jpg HTTP/1.1 Host: /finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 238985
Response headers
HTTP/1.1 201 Created Location: /rest/images/wind2.jpg X-HCP-VersionId: 79885459513089 X-HCP-Hash: SHA-256 E830B86212A66A792A79D58BB185EE63A4FADA76BB8A1... X-HCP-Time: 1259584200 Content-Length: 0
Example 2: Sending object data in compressed format (Unix) Heres a Unix command line that uses the gzip utility to compress the wind.jpg file and then pipes the compressed output to a curl command. The curl command makes an HTTP PUT request that sends the data and informs HCP that the data is compressed.
47
Request headers
PUT /rest/images/wind.jpg HTTP/1.1 Host: /finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 124863 Transfer-Encoding: chunked Content-Encoding: gzip Expect: 100-continue
Response headers
HTTP/1.1 100 Continue HTTP/1.1 201 Created Location: /rest/images/wind.jpg X-HCP-VersionId: 79885459513089 X-HCP-Hash: SHA-256 E830B86212A66A792A79D58BB185EE63A4FADA76BB8A1... X-HCP-Time: 1259584200 Content-Length: 0
Example 3: Sending object data in compressed format (Java) Heres the partial implementation of a Java class named HTTPCompression. The implementation shows the WriteToHCP method, which stores an object (or version) in an HCP namespace. The method compresses the data before sending it and uses the Content-Encoding header to tell HCP that the data is compressed. The WriteToHCP method uses the GZIPCompressedInputStream helper class. For an implementation of this class, see GZIPCompressedInputStream class on page B-2.
import org.apache.http.client.methods.HttpPut; import org.apache.http.HttpResponse; import com.hds.hcp.examples.GZIPCompressedInputStream; class HTTPCompression { . . . void WriteToHCP() throws Exception { /* * Set up the PUT request. * * This method assumes that the HTTP client has already been * initialized. */ HttpPut httpRequest = new HttpPut(sHCPFilePath);
48
// Indicate that the content encoding is gzip. httpRequest.setHeader("Content-Encoding", "gzip"); // Open an input stream to the file that will be sent to HCP. // This file will be processed by the GZIPCompressedInputStream to // produce gzip-compressed content when read by the Apache HTTP client. GZIPCompressedInputStream compressedInputFile = new GZIPCompressedInputStream(new FileInputStream( sBaseFileName + ".toHCP")); // Point the HttpRequest to the input stream. httpRequest.setEntity(new InputStreamEntity(compressedInputFile, -1)); // Create the HCP authentication cookie and put in cookie store. httpRequest.setHeader("Cookie", NS_AUTH_TOKEN + "=" + sEncodedUserName + ":" + sEncodedPassword); /* * Now execute the PUT request. */ HttpResponse httpResponse = mHttpClient.execute(httpRequest); /* * Process the HTTP response. */ // If the return code is anything but in the 200 range indicating // success, throw an exception. if (2 != (int)(httpResponse.getStatusLine().getStatusCode() / 100)) { throw new Exception("Unexpected HTTP status code: " + httpResponse.getStatusLine().getStatusCode() + " (" + httpResponse.getStatusLine().getReasonPhrase() + ")"); } } . . . }
Example 4: Storing object or version data with custom metadata (Unix) Heres a Unix command line that uses an HTTP PUT request to store the object data and custom metadata for a file named wind.jpg. The request stores the object in the images directory. The cat command appends the contents of the wind-custom-metadata.xml file to the contents of the wind.jpg file. The result is piped to a curl command that sends the data to HCP.
49
Request headers
PUT /rest/images/wind2.jpg HTTP/1.1 Host: /finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d X-HCP-Size: 237423 Content-Length: 238985
Response headers
HTTP/1.1 201 Created Location: /rest/images/wind2.jpg X-HCP-VersionId: 79885459513089 X-HCP-Hash: SHA-256 E830B86212A66A792A79D58BB185EE63A4FADA76BB8A1... X-HCP-CustomMetadataHash: SHA-256 86212A6692A79D5B185EE63A4DA76BBC... X-HCP-Time: 1259584200 Content-Length: 0
Example 5: Storing object or version data with custom metadata (Java) Heres the partial implementation of a Java class named WholeIO. The implementation shows the WholeWriteToHCP method, which uses a single HTTP PUT request to store data and custom metadata for an object (or version). The WholeWriteToHCP method uses the WholeIOInputStream helper class. For an implementation of this class, see WholeIOInputStream class on page B-8.
import org.apache.http.client.methods.HttpPut; import org.apache.http.HttpResponse; import com.hds.hcp.examples.WholeIOInputStream; class WholeIO { . . . void WholeWriteToHCP() throws Exception {
410
/* * Set up the PUT request to store both object data and custom * metadata. * * This method assumes that the HTTP client has already been * initialized. */ HttpPut httpRequest = new HttpPut(sHCPFilePath + "?type=whole-object"); FileInputStream dataFile = new FileInputStream(sBaseFileName); // Put the size of the object data into the X-HCP-Size header. httpRequest.setHeader("X-HCP-Size", String.valueOf(dataFile.available())); // Point the HttpRequest to the input stream with the object data // followed by the custom metadata. httpRequest.setEntity( new InputStreamEntity( new WholeIOInputStream( new FileInputStream(sBaseFileName), new FileInputStream(sBaseFileName + ".cm")), -1)); // Create an HCP authentication cookie and put in the cookie store. httpRequest.setHeader("Cookie", NS_AUTH_TOKEN + "=" + sEncodedUserName + ":" + sEncodedPassword); /* * Now execute the PUT request. */ HttpResponse httpResponse = mHttpClient.execute(httpRequest); // If the return code is anything but in the 200 range indicating // success, throw an exception. if (2 != (int)(httpResponse.getStatusLine().getStatusCode() / 100)) { throw new Exception("Unexpected HTTP status code: " + httpResponse.getStatusLine().getStatusCode() + " (" + httpResponse.getStatusLine().getReasonPhrase() + ")"); } } . . . }
411
Meaning
Description
HCP found the specified object or version and returned its metadata. This code is also returned if the URL specified a directory, not an object. The requested version is a historic deleted version. HCP could not find an object, version, or directory at the specified URL. The specified object or version does not exist, or the HEAD request specified the current version of an object that has been deleted.
204 404
412
Request-specific response headers The table below describes request-specific response headers returned for this operation. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
Header
X-HCP-Custom-Metadata X-HCP-DPL X-HCP-Hash
Description
A true or false value indicating whether the object has custom metadata. The data protection level. The cryptographic hash algorithm HCP uses, along with the cryptographic hash value stored for the object, in this format: X-HCP-Hash: hash-algorithm hash-value You can use the returned hash value to verify that the stored data is the same as the data you sent. To do so, compare this value with a hash value that you generate from the original data.
A true or false value indicating whether the object is marked for indexing. The time when HCP stored the object, in seconds since January 1, 1970, at 00:00:00 UTC. A true or false value indicating whether the object, has been replicated. The value is true only if the current version of the object and all metadata, including any custom metadata, has been replicated. The end of the retention period for the object, in seconds since January 1, 1970, at 00:00:00 UTC. This value can also be 0, -1, or -2. The name of the retention class to which the object belongs. This value is an empty string if the object is not in a retention class. A true or false value indicating whether the object is on hold.
X-HCP-Retention
X-HCP-RetentionClass
X-HCP-RetentionHold
413
Header
X-HCP-RetentionString
Description
The end of the retention period, in this format:
yyyy-MM-ddThh:mm:ssZ
In this format, Z is the offset between local time and UTC. For example, 2015-11-16T14:27:20-0500 represents the start of the 20th second into 2:27 PM, November 16, 2015, EST. The header value can also be Deletion Allowed, Deletion Prohibited, or Initial Unspecified. X-HCP-Shred X-HCP-Size X-HCP-Type X-HCP-VersionId A true or false value indicating whether HCP will shred the object after it is deleted. The size of the object, in bytes. The entity type. This is always object for objects and versions of objects. The version ID of the object.
Example: Checking the existence of an object Heres a sample HTTP HEAD request that checks the existence of an object named wind.jpg in the images directory.
414
Request headers
HEAD /rest/images/wind.jpg HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 Content-Type: image/jpeg Content-Length: 238985 X-HCP-Type: object X-HCP-Size: 238985 X-HCP-Hash: SHA-256 36728BA190BC4C377FE4C1A57AEF9B6AFDA98720422960... X-HCP-VersionId: 80111794065921 X-HCP-IngestTime: 1251746782 X-HCP-RetentionClass: (SEC17a, A+7y) X-HCP-RetentionString: 2016-08-31T15:26:22-0400 X-HCP-Retention: 1472671582 X-HCP-RetentionHold: false X-HCP-Shred: false X-HCP-DPL: 1 X-HCP-Index: false X-HCP-Custom-Metadata: false X-HCP-Replicated: false
415
An hcp-ns-auth authorization cookie. The URL of the object. The version=list URL query parameter. This parameter is not case
sensitive. Request contents requesting deleted versions By default, the list does not include deleted versions (that is, the marker versions that indicate when an object was deleted); it shows only versions that were ingested and have not been pruned. To get a listing that includes deleted versions, specify this query parameter:
deleted=true
You can also specify deleted=false, which results in the default behavior. Access permission To list object versions you need read permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 404 OK Not Found
Meaning
Description
HCP successfully returned the version list. HCP could not find the specified object.
416
Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Response body The body of the HTTP response to a GET request to list object versions contains an XML document listing the versions, in this format:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/ns-versions.xsl"?> <versions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/ns-versions.xsd" path="object-path" utf8Path="object-path" parentDir="parent-directory-path" utf8ParentDir="parent-directory-path" deleted="true|false" showDeleted="true|false" namespaceName="namespace-name" utf8NamespaceName="utf-8-encoded-namespace-name"> <entry urlname="object-name" utf8Name="object-name" type="object" size="size-in-bytes" hashScheme="hash-algorithm" hash="hash-value" retention="retention-seconds-after-1/1/1970" retentionString="retention-datetime-value" retentionClass="retention-class-name" hold="true|false" shred="true|false" dpl="dpl" index="true|false" customMetadata="true|false" replicated="true|false" state="created|deleted" version="version-id" ingestTime="ingested-seconds-after-1/1/1970" ingestTimeString="ingested-datetime" changeTimeMilliseconds="change-milliseconds-after-1/1/1970.uniqueinteger" changeTimeString="yyyy-MM-ddThh:mm:ssZ" /> . . . </versions>
417
The deleted attribute of the versions element indicates whether the object is currently deleted. The value of the showDeleted attribute indicates whether the list includes deleted versions (true) or only the versions that were ingested (false). The state attribute of the entry for each individual version specifies whether the version contains data or is a marker indicating that the object was deleted (that is, a deleted version). The XML entry for a deleted version differs from the entry created when the deleted object was ingested in these ways:
The state value is deleted. The version value is different. The ingestTime and changeTimeMilliseconds values are the date and
time the object was deleted. If the directory that contains the object contained at some time a subdirectory with the same name as the object, the results of a version list request that includes deleted versions shows deleted versions for both the object and the directory. For example, assume you do the following: 1. Create a maintenance subdirectory of the departments directory. 2. Delete the maintenance directory. 3. Save a maintenance object in the departments directory. In this case, a version listing for the maintenance entry that includes deleted versions shows these items:
An entry, with a version ID, for the deleted maintenance directory Entries for all versions of the
version entries for the object
maintenance object, including any deleted
418
Example: Listing the versions of an object Heres a sample HTTP GET request that saves an XML listing of the versions of an object named earth.jpg located in the images directory. The object has three versions: the original version that was ingested, a deleted version, and a version that was ingested after the original version was deleted. The request asks for all versions, including deleted versions.
Request headers
GET /rest/images/earth.jpg?version=list HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
419
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 Content-Type: text/xml Content-Length: 2080
Response body
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/ns-versions.xsl"?> <versions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/ns-versions.xsd" path="/rest/images/obsolete/earth.jpg" utf8path="/rest/images/obsolete/earth.jpg" parentDir="/rest/images/obsolete"> utf8ParentDir="/rest/images/obsolete"> deleted="false" showDeleted="true" namespaceName="finance" utf8NamespaceName="finance"> <entry urlName="earth.jpg" utf8Name="earth.jpg" type="object" size="1204765" hashScheme="SHA-256" hash="42C605FBFFCD7CEAC36BE62F294374F94503D1DC1793736EF..." retention="0" retentionString="Deletion Allowed" retentionClass="" hold="false" shred="false" dpl="2" index="true" customMetadata="false" state="created" version="80232488492929" replicated="false" ingestTime="1258469462" ingestTimeString="11/17/2009 9:51AM" changeTimeMilliseconds="1258469513362.00" changeTimeString="2009-11-17T09:51:53-0400" /> <entry urlName="earth.jpg" utf8Name="earth.jpg" type="object" size="1200857" hashScheme="SHA-256"
420
Listing object versions hash="36728BA190BC4C377FE4C1A57AEF9B6AFDA98720422960B19..." retention="0" retentionString="Deletion Allowed" retentionClass="" hold="false" shred="false" dpl="2" index="true" customMetadata="false" replicated="false" state="deleted" version="80232488876481" ingestTime="1258405200" ingestTimeString="11/16/2009 4:00PM" changeTimeMilliseconds="1258405200259.00" changeTimeString="2009-11-16T:16:00-0400" /> <entry urlName="earth.jpg" utf8Name="earth.jpg" type="object" size="1200857" hashScheme="SHA-256" hash="36728BA190BC4C377FE4C1A57AEF9B6AFDA98720422960B19..." retention="0" retentionString="Deletion Allowed" retentionClass="" replicated="false" hold="false" shred="false" dpl="2" index="true" customMetadata="false" replicated="false" state="created" version="80232489767169" ingestTime="1258392981" ingestTimeString="11/16/2009 12:36PM" changeTimeMilliseconds="1258392981147.00" changeTimeString="2009-11-16T:12:16-0400" /> </versions>
421
422
true The custom metadata should precede the object data. false The object data should precede the custom metadata.
The default is false. You can also retrieve the data in gzip-compressed format by specifying an Accept-Encoding header containing the value gzip or *. The header can specify additional compression algorithms, but HCP uses gzip only. Request contents requesting a partial object To retrieve only part of the object or version data, in addition to the elements described in Request contents and, optionally, Request contents requesting data in compressed format on page 4-22, specify an HTTP Range request header with the range of bytes of the object data to retrieve. The first byte of the data is in position 0 (zero), so a range of 1-5 specifies the second through sixth bytes of the object, not the first through fifth. These rules apply to the Range header:
If you omit the Range header, HCP returns the complete object data. If you specify a valid range, HCP returns the requested amount of data
with a status code of 206.
If you specify an invalid range, HCP ignores it and returns the complete
object data, with a status code of 416.
You cannot request partial object data and custom metadata in the
same request. If the request includes a Range header and a type=whole-object query parameter, HCP returns an HTTP 400 error response.
423
The table below shows the ways in which you can specify a byte range.
Range specification
start-positionend-position
Description
Bytes in start-position through end-position, inclusive. If end-position is greater than the size of the data, HCP returns the bytes from start-position through the end of the data. Bytes in start-position through the end of the object data. Bytes in the offset-fromend position, counted back from the last position in the object data, through the end of the object data.
Example
Five hundred bytes beginning with the two hundred first: 200-699
start-position
All the bytes beginning with the seventy-sixth and continuing through the end of the object: 75The last 25 bytes of the object: -25
offset-from-end
Access permission To retrieve an object or version, you need read permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 OK
Meaning
Description
HCP successfully retrieved a complete object or version. This code is also returned if the URL specified a valid directory path and HCP returned a directory listing.
204 206
The requested version is a historic deleted version. HCP successfully retrieved the requested byte range.
424
Code
400
Meaning
Bad Request
Description
The request was not valid. These are some, but not all, of the possible reasons: The request has both a type=whole-object query parameter and a Range request header. The URL in the request is not well-formed. The namespace does not exist. The request contains an unsupported parameter or an invalid value for a parameter.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 404 Not Found One of: HCP could not find an object, version, or directory at the specified URL. The request was for the current version of a deleted object.
The request has an Accept-Encoding header that does not include gzip or specify *. One of: The specified start position is greater than the size of the requested data. The size of the specified range is 0 (zero).
Request-specific response headers The table below describes request-specific response headers for this operation. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
Header
Content-Encoding
Description
Returned only if HCP compresses the data before returning it. Always gzip.
425
Header
Content-Length
Description
The length, in bytes, of the returned data. The header has these characteristics: If you requested that the response be compressed, this is the compressed size of the returned data. If you requested uncompressed object data without custom metadata, the value is the same as the value of the X-HCP-Size header. If you requested uncompressed partial content, the value is the size of the returned part and is equal to the difference between the start-position and end-position values in the Content-Range header. If you requested uncompressed object data and custom metadata, the value is the sum of the size of the object data (the X-HCP-Size header) and the size of the custom metadata.
If the returned data is large, HCP may send a chunked response, which does not include this header. Content-Range Returned only when getting partial contents. The byte range of the returned object data, in this format:
start-positionend-position/total-length
total-length is the object size and is the same as the value of the X-HCP-Size header. Content-Type The type of content: If you requested all or part of the object data only, this is the Internet media type of the object data, such as text/plain or image/jpg. If you requested the object data and custom metadata together, this value is always application/octet-stream.
X-HCP-ContentLength
Returned only if HCP compresses the data before returning it. The uncompressed length of the returned data.
X-HCP-CustomMetadata
A true or false value indicating whether the object has custom metadata.
426
Header
X-HCP-CustomMetadata ContentType
Description
Returned only if the request asked for the object data and custom metadata. Always text/xml.
X-HCP-CustomMetadata First
Returned only if the request asked for the object data and custom metadata. One of: true The custom metadata precedes the object data false The object data precedes the custom metadata
X-HCP-DataContentType
Returned only if the request asked for the object data and custom metadata. The object Internet media type, such as text/plain or image/jpg.
X-HCP-DPL X-HCP-Hash
The data protection level. The cryptographic hash algorithm HCP uses and the cryptographic hash value of the stored object or version, in this format: X-HCP-Hash: hash-algorithm hash-value You can use the returned hash value to verify that the stored data is the same as the data you sent. To do so, compare this value with a hash value that you generate from the original data.
A true or false value indicating whether the object is marked for indexing. The time when HCP stored the object, in seconds since January 1, 1970, at 00:00:00 UTC. A true or false value indicating whether the object has been replicated. The value is true only if the current version of the object and all metadata, including any custom metadata, has been replicated. The end of the retention period for the object, in seconds since January 1, 1970, at 00:00:00 UTC. This value can also be 0 (deletion allowed), -1 (deletion prohibited), or -2 (initial unspecified). The name of the retention class assigned to the object. This value is an empty string if the object is not in a retention class.
X-HCP-Retention
X-HCP-RetentionClass
427
Header
X-HCP-RetentionHold X-HCP-RetentionString
Description
A true or false value indicating whether the object is on hold. The end of the retention period, in this format:
Response body The body of the HTTP response contains the requested object or range of data. If the request asks for an object and its custom metadata, the body contains the object and custom metadata in the requested order, with no delimiter between them. If the object has no custom metadata, HCP returns only the object. In this case, the values of the Content-Length and X-HCP-Size headers are identical and the X-HCP-Custom-Metadata header value is false. If the request includes an Accept-Type header containing gzip or with the value *, HCP compresses the body using the gzip encoding.
428
Example 1: Retrieving all object data Heres a sample HTTP GET request that retrieves an object named wind.jpg in the images directory.
Tip: If a GET request unexpectedly returns a zero-length file, use the -i parameter with curl to return the response headers in the target file. These headers may provide helpful information for diagnosing the problem.
Request headers
GET /rest/images/wind.jpg HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
429
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 Content-Type: image/jpeg Content-Length: 2938985 X-HCP-Type: object X-HCP-Size: 238985 X-HCP-Hash: SHA-256 36728BA190BC4C377FE4C1A57AEF9B6AFDA98720422960... X-HCP-VersionId: 80205544854849 X-HCP-IngestTime: 1258469614 X-HCP-RetentionClass: X-HCP-RetentionString: Deletion Allowed X-HCP-Retention: 0 X-HCP-RetentionHold: false X-HCP-Shred: false X-HCP-DPL: 2 X-HCP-Index: false X-HCP-Custom-Metadata: false X-HCP-Replicated: false
Response body
The contents of the wind.jpg object
Example 2: Retrieving partial object data Heres a sample HTTP GET request that retrieves the first 10,000 bytes of an object named status27.txt in the status directory.
430
Request headers
GET /rest/status/status27.txt HTTP/1.1 Range: bytes=0-9999 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 206 Partial Content X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 Content-Type: text/plain Content-Range: bytes 0-9999/38985 Content-Length: 10000 X-HCP-Type: object X-HCP-Size: 38985 X-HCP-Hash: SHA-256 36728BA190BC4C377FE4C1A57AEF9B6AFDA98720422960... X-HCP-VersionId: 80232488876481 X-HCP-IngestTime: 1258469462 X-HCP-RetentionClass: X-HCP-RetentionString: Deletion Allowed X-HCP-Retention: 0 X-HCP-RetentionHold: false X-HCP-Shred: false X-HCP-DPL: 2 X-HCP-Index: false X-HCP-Custom-Metadata: false X-HCP-Replicated: false
431
Response body
The first 10,000 bytes of the contents of status27.txt
Example 3: Retrieving a historic version of an object Heres a sample HTTP GET request that retrieves version 80232998058817 of an object named wind.jpg in the images directory.
Request headers
GET /rest/images/wind.jpg?version=80232998058817 HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
432
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 Content-Type: image/jpeg Content-Length: 238985 X-HCP-Type: object X-HCP-Size: 238985 X-HCP-Hash: SHA-256 36728BA190BC4C377FE4C1A57AEF9B6AFDA98720422960... X-HCP-VersionId: 80232998058817 X-HCP-IngestTime: 1258469614 X-HCP-RetentionClass: X-HCP-RetentionString: Deletion Allowed X-HCP-Retention: 0 X-HCP-RetentionHold: false X-HCP-Shred: false X-HCP-DPL: 2 X-HCP-Index: false X-HCP-Custom-Metadata: false X-HCP-Replicated: false
Response body
The contents of the requested version of the object.
Example 4: Retrieving object data in compressed format (command line) Heres a sample curl command that tells HCP to compress the wind.jpg object before sending it to the client and then decompresses the returned content.
433
Request headers
GET /rest/images/wind.jpg HTTP/1.1 Host: finance.europe.hcp.example.com Accept-Encoding: deflate, gzip Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 Content-Encoding: gzip Content-Type: image/jpeg X-HCP-Type: object X-HCP-Size: 238985 X-HCP-Hash: SHA-256 36728BA190BC4C377FE4C1A57AEF9B6AFDA98720422960... X-HCP-VersionId: 80205544854849 X-HCP-IngestTime: 1258469614 X-HCP-RetentionClass: X-HCP-RetentionString: Deletion Allowed X-HCP-Retention: 0 X-HCP-RetentionHold: false X-HCP-Shred: false X-HCP-DPL: 2 X-HCP-Index: false X-HCP-Custom-Metadata: false X-HCP-Replicated: false
Response body
The contents of the wind.jpg object in gzip-compressed format.
434
Example 5: Retrieving object data in compressed format (Java) Heres the partial implementation of a Java class named HTTPCompression. The implementation shows the ReadFromHCP method, which retrieves an object from an HCP namespace. It uses the Accept-Encoding header to tell HCP to compress the object before returning it and then decompresses the results.
import org.apache.http.client.methods.HttpGet; import org.apache.http.HttpResponse; import java.util.zip.GZIPInputStream; class HTTPCompression { . . . void ReadFromHCP() throws Exception { /* * Set up the GET request. * * This method assumes that the HTTP client has already been * initialized. */ HttpGet httpRequest = new HttpGet(sHCPFilePath); // Indicate that you want HCP to compress the returned data with gzip. httpRequest.setHeader("Accept-Encoding", "gzip"); // Create Authentication Cookie and put in Cookie Store. httpRequest.setHeader("Cookie", NS_AUTH_TOKEN + "=" + sEncodedUserName + ":" + sEncodedPassword); /* * Now execute the GET request. */ HttpResponse httpResponse = mHttpClient.execute(httpRequest); /* * Process the HTTP Response. */ // If the return code is anything but in the 200 range indicating // success, throw an exception. if (2 != (int)(httpResponse.getStatusLine().getStatusCode() / 100)) { throw new Exception("Unexpected HTTP status code: " + httpResponse.getStatusLine().getStatusCode() + " (" + httpResponse.getStatusLine().getReasonPhrase() + ")"); }
435
/* * Write the decompressed file to disk. */ FileOutputStream outputFile = new FileOutputStream( sBaseFileName + ".fromHCP"); // Build the string that contains the response body for return to the // caller. GZIPInputStream bodyISR = new GZIPInputStream(httpResponse.getEntity().getContent()); byte partialRead[] = new byte[1024]; int readSize = 0; while (-1 != (readSize = bodyISR.read(partialRead))) { outputFile.write(partialRead, 0, readSize); } } . . . }
Example 6: Retrieving object data and custom metadata together (Java) Heres the partial implementation of a Java class named WholeIO. The implementation shows the WholeReadFromHCP method, which retrieves an object and its custom metadata in a single data stream, splits the object from the custom metadata, and stores each in a separate file. The WholeReadFromHCP method uses the WholeIOOutputStream helper class. For an implementation of this class, see WholeIOOutputStream class on page B-9.
import org.apache.http.client.methods.HttpGet; import org.apache.http.HttpResponse; import com.hds.hcp.examples.WholeIOOutputStream; class WholeIO { . . . void WholeReadFromHCP() throws Exception { /* * Set up the GET request and specifying the whole-object I/O. * * This method assumes that the HTTP client has already been * initialized. */ HttpGet httpRequest = new HttpGet(sHCPFilePath + "?type=whole-object");
436
// Create Authentication Cookie and put in Cookie Store. httpRequest.setHeader("Cookie", NS_AUTH_TOKEN + "=" + sEncodedUserName + ":" + sEncodedPassword); // Request the custom metadata before the object data. // This can be useful if the application examines the custom metadata // to set the context for the data that will follow. httpRequest.setHeader("X-HCP-CustomMetadataFirst", "true"); /* * Now execute the GET request. */ HttpResponse httpResponse = mHttpClient.execute(httpRequest); // If the return code is anything but in the 200 range indicating // success, throw an exception. if (2 != (int)(httpResponse.getStatusLine().getStatusCode() / 100)) { throw new Exception("Unexpected HTTP status code: " + httpResponse.getStatusLine().getStatusCode() + " (" + httpResponse.getStatusLine().getReasonPhrase() + ")"); } /* * Determine whether data or custom metadata is first. */ Boolean cmFirst = new Boolean(httpResponse.getFirstHeader( "X-HCP-CustomMetadataFirst").getValue()); /* * Determine the size of the first part based on whether the object * data or custom metadata is first. */ // Assume object data is first. int firstPartSize = Integer.valueOf(httpResponse.getFirstHeader("X-HCP-Size").getValue()); // If custom metadata is first, do the math. if (cmFirst) { // subtract the data size from the content length returned. firstPartSize = Integer.valueOf( httpResponse.getFirstHeader("Content-Length").getValue()) - firstPartSize; } /* * Split and write the file to disk. */ WholeIOOutputStream outputCreator= new WholeIOOutputStream( new FileOutputStream(sBaseFileName + ".fromHCP"), new FileOutputStream(sBaseFileName + ".fromHCP.cm"), cmFirst);
437
. . .
}
438
Request using privileged delete To perform a privileged delete, the request must specify both of these parameters:
privileged=true reason=reason-text
reason-text must be from one through 1,024 characters long and can contain any valid UTF-8 characters, including white space.
You can specify the parameters in either of these ways:
439
Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 OK
Meaning
Description
HCP successfully deleted the object. This code is also returned if the URL specified a directory and HCP deleted it. One of: The object is under retention. The authenticated user doesnt have permission to perform the requested operation. The namespace does not exist. The access method (HTTP or HTTPS) is disabled.
403
Forbidden
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 404 409 Not Found Conflict The URL did not specify an object or directory in the namespace. HCP could not delete the specified object because it is currently being written to the namespace.
Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Example 1: Deleting an object Heres a sample HTTP DELETE request that deletes the object named wind.jpg from the images directory in the namespace.
440
Request headers
DELETE /rest/images/wind.jpg HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 Content-Length: 0
Example 2: Performing a privileged delete Heres a sample HTTP DELETE request that deletes the object named wind.jpg, which is under retention, from the images directory in the namespace.
441
Request headers
DELETE /rest/images/wind.jpg HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 65 Content-Type: application/x-www-form-urlencoded
Request body
Contains the privileged and reason parameters as form-encoded data.
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1258469462 X-HCP-SoftwareVersion: 4.1.0.123 Content-Length: 0
442
An hcp-ns-auth authorization cookie The URL of the object or directory A purge=true parameter
Request using privileged purge To perform a privileged purge, the request must specify these additional parameters:
privileged=true reason=reason-text
reason-text must be from one through 1,024 characters long and can contain any valid UTF-8 characters, including white space.
You can specify the parameters in either of these ways:
443
Meaning
Description
HCP successfully purged all versions of the object. One of: The object is under retention and you do not have the privileged permission. You dont have permission to perform the requested operation. The namespace does not exist. The access method (HTTP or HTTPS) is disabled.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header.
444
Code
404 409
Meaning
Not Found Conflict
Description
HCP could not find the specified object. HCP could not purge the specified object because it is currently being written to the namespace.
Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Example 1: Purging an object Heres a sample HTTP DELETE request that purges all versions of the object named wind.jpg from the images directory. This example uses a URL query string to pass the operation parameters.
Request headers
DELETE /rest/images/WindVersion.jpg?purge=true HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1258469462 Content-Length: 0
445
Example 2: Performing a privileged purge Heres a sample HTTP DELETE request that purges the object named wind.jpg, which is under retention, from the images directory. This example uses form-encoded data to pass the operation parameters and percentencodes the reason because it contains an ampersand.
Request headers
DELETE /rest/images/wind.jpg HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 90 Content-Type: application/x-www-form-urlencoded
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1258469462 Content-Length: 0
446
5
Working with directories
HCP lets you perform these operations on directories:
Create an empty directory Check the existence of a directory List the contents of a directory Delete an empty directory
This chapter describes these procedures.
51
An hcp-ns-auth authorization cookie A URL specifying the location in which to put the object A type URL query parameter with a value of directory.
Access permission To add a directory to a namespace, you need write permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
201 409
Meaning
Created Conflict
Description
HCP successfully created the directory. HCP could not create the directory in the namespace because it already exists.
Request-specific response headers The directory creation operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Example: Adding a directory Heres a sample HTTP PUT request that creates a new obsolete directory in the images directory.
52
Request headers
PUT /rest/images/obsolete?type=directory HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 201 Created Location: /rest/images/obsolete Content-Length: 0
53
Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 404 OK Not Found
Meaning
Description
HCP found a directory or object at the specified URL. HCP could not find a directory or object at the specified URL.
Request-specific response headers The table below describes the request-specific response header for this operation. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
Header
X-HCP-Type
Description
The type of the entity. This is always directory for directories. A value of object indicates that the requested entity is an object.
Example: Checking the existence of a directory Heres a sample HTTP HEAD request that checks the existence of a directory named images.
54
Request headers
HEAD /rest/images HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 X-HCP-Type: directory Content-Length: 0
55
Request contents listing deleted directories and objects By default, the returned list does not include deleted directories or objects. If the namespace supports versioning you can include deleted objects and directories in the list. To do this, add the following query parameter to the URL:
deleted=true
You can also specify deleted=false, which results in the default behavior. Also specify the deleted=true parameter to list the contents of a deleted directory (which can contain only deleted objects and directories). Access permission To list the contents of a directory, you need read permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 OK
Meaning
Description
HCP retrieved the directory listing. This code is also returned if the URL specified an object and HCP retrieved it. HCP could not find a directory or object at the specified URL.
404
Not Found
Request-specific response headers The table below describes the request-specific response header for this operation. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
Header
X-HCP-Type
Description
The type of the entity. The header value is always directory for directories.
Response body The body of the HTTP response consists of XML that lists the contents of the requested directory, including metadata for the objects the directory contains. It lists only the immediate directory contents, not the contents of any subdirectories.
56
57
Notes:
Request headers
GET /rest/images HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
58
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 X-HCP-SoftwareVersion: 4.1.0.123 X-HCP-Type: directory Content-Type: text/xml Content-Length: 577
Response body
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/ns-directory.xsl"?> <directory xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/ns-directory.xsd" path="/rest/images" utf8Path="/rest/images" parentDir="/rest"> utf8ParentDir="/rest" dirDeleted="false" showDeleted="true" namespaceName="finance" utf8NamespaceName="finance"> <entry urlName="fire.jpg" utf8Name="fire.jpg" type="object" size="19461" hashScheme="SHA-256" hash="42C605FBFFCD7CEAC36BE62F294374F94503D1DC1793736EF367..." retention="0" retentionString="Deletion Allowed" retentionClass="" ingestTime="1284580187" ingestTimeString="9/15/2010 3:49PM" hold="false" shred="false" dpl="2" index="true" customMetadata="false" version="80238375537921" replicated="false" changeTimeMilliseconds="1284580187533.00" changeTimeString="2010-09-15T15:49:47-0400" state="created" /> <entry urlName="obsolete" utf8Name="obsolete" type="directory"
59
Deleting an empty directory state="deleted" /> <entry urlName="wind.jpg" utf8Name="wind.jpg" type="object" size="238985" hashScheme="SHA-256" hash="E830B86212A66A792A79D58BB185EE63A4FADA76BB8A1C257C01..." retention="0" retentionString="Deletion Allowed" retentionClass="" ingestTime="1258469462" ingestTimeString="11/17/2009 9:51AM" hold="false" shred="false" dpl="2" index="true" customMetadata="false" version="80238376132993" replicated="false" changeTimeMilliseconds="1258469462598.00" changeTimeString="2009-11-17T09:51:57-0400" state="created" /> </directory>
Note that the obsolete directory was deleted. It does not appear in a directory listing if the URL does not include the deleted=true parameter.
510
Request-specific return codes The table below describes the return codes that have specific meanings for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 403 OK Forbidden
Meaning
Description
HCP successfully deleted the directory. One of: The directory specified for deletion is not empty. The authenticated user doesnt have permission to perform the requested operation. The namespace does not exist. The access method (HTTP or HTTPS) is disabled.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 404 409 Not Found Conflict HCP could not find the specified directory. HCP could not delete the specified directory because it is currently being written to the namespace.
Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Example: Deleting a directory Heres a sample HTTP DELETE request that deletes the directory named obsolete from the images directory in the namespace.
511
Request headers
DELETE /rest/images/obsolete HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 Content-Length: 00
512
6
Working with system metadata
You can perform these operations on the system metadata associated with objects:
Specify metadata when you create an object Modify the metadata for an existing object List the metadata for an object
This chapter describes how to specify and modify system metadata. For information on listing the metadata for an object, see Checking the existence of an object or version on page 4-12. For more information on metadata, see Chapter 2, Understanding objects. Note: When converting legacy applications to use the HTTP REST interface to access to an HCP namespace, be aware that the values used to specify certain metadata settings differ from those used for metadata settings in the default namespace and HCAP 2.x archives. For information on the default namespace, see Using the Default Namespace.
61
An hcp-ns-auth authorization cookie A URL specifying the location in which to put the object A body containing the object to be put in the namespace Any combination of zero or more of the URL query parameters
described in the table below
Parameter
hold index retention
Valid values
true or false true or false Any valid retention expression, as described in Specifying retention settings on page 2-8 true or false
Description
Places an object on hold or specifies that its not on hold Specifies whether the object should be indexed for search Specifies the retention setting for the object Specifies whether to shred the object after it is deleted
shred
Note: Each object has index metadata setting regardless of whether a search facility is enabled at the system level.
62
Access permission To add objects and versions to the namespace, you need write permission for the namespace. To specify the hold parameter, you also need the privileged permission. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
201 409 413
Meaning
Created Conflict File Too Large
Description
The request successfully added the specified object. HCP could not add the object to the namespace because it already exists and versioning is not enabled. One of: Not enough space is available to store the object. Try the request again after objects or versions are deleted from the namespace or the namespace capacity is increased. The request is trying to store an object that is larger than two TB. HCP cannot store objects larger than two TB.
Request-specific response header The table below describes the request-specific HTTP response header. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
Header
X-HCP-Hash
Description
The cryptographic hash algorithm HCP uses and the cryptographic hash value of the stored object, in this format: X-HCP-Hash: hash-algorithm hash-value You can use the returned hash value to verify that the stored data is the same as the data you sent. To do so, compare this value with a hash value that you generate from the original data.
63
Example: Storing a file and setting the retention value Heres a sample HTTP PUT request that stores a file named wind.jpg in the images directory and sets the retention value to the HlthReg-107 class.
Request headers
PUT /rest/images/wind.jpg?retention=C+HlthReg-107 HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 238985
Response headers
HTTP/1.1 201 Created Location: /rest/images/wind.jpg X-HCP-VersionId: 80238663473089 X-HCP-Hash: SHA-256 E830B86212A66A792A79D58BB185EE63A4FADA76BB8A1C... X-HCP-Time: 1258392981 Content-Length: 0
64
An hcp-ns-auth authorization cookie. A URL specifying the object for which you are changing the metadata. A body containing form-encoded metadata (application/x-www-formurlencoded content type). The content can consist of any combination of the parameters described in the table below.
Parameter
hold index retention
Valid values
true or false true or false Any valid retention expression, as described in Specifying retention settings on page 2-8 true or false
Description
Places an object on hold or specifies that its not on hold Specifies whether the object should be indexed for search Specifies the retention setting for the object Specifies whether to shred the object after it is deleted
shred
Request limitations These limitations affect the valid requests and their results:
You cannot change the hold and retention settings in the same request. You cannot change the shred setting from true to false.
Access permission To modify object metadata, you need write permission for the namespace. To specify a hold parameter, you also need the privileged permission.
65
Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 400 OK Bad Request
Meaning
Description
The request successfully updated the metadata. One of: The request is trying to change the retention setting from a retention class to an explicit setting, such as a datetime value. The request is trying to change the retention setting and the retention hold setting at the same time. The request is trying to change the retention setting for an object on hold. The request is trying to change the shred value from true to false. The URL in the request is not well-formed. The request contains an unsupported parameter or an invalid value for a parameter.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 404 Not Found HCP could not find the specified object.
Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Example: Changing multiple metadata values for an existing object Heres a sample HTTP POST request that makes these changes to the metadata for the wind.jpg file:
Increases retention by one year from the current value Turns on shredding on delete Turns off indexing
66
Request headers
POST /rest/images/wind.jpg HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 37 Content-Type: application/x-www-form-urlencoded
Response headers
HTTP/1.1 200 OK Content-Length: 0
67
68
7
Working with custom metadata
You can perform these operations on custom metadata:
Store or replace the custom metadata for an object Check the existence of custom metadata for an object or version Retrieve the custom metadata for an object or version Delete the custom metadata for an object
This chapter describes the procedures for the operations listed above.
71
An hcp-ns-auth authorization cookie A URL specifying the object for which you are setting the custom
metadata
Use gzip to compress the content before sending it. Include a Content-Encoding request header with a value of gzip. Use a chunked transfer encoding.
Access permission To store custom metadata, you need write permission for the namespace.
72
Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
201 400
Meaning
Created Bad Request
Description
HCP successfully stored the custom metadata. One of: The namespace is configured with custom metadata XML checking enabled, and the request includes custom metadata that is not well-formed XML. The request has a Content-Encoding header that specifies gzip, but the custom metadata is not in gzipcompressed format. The URL in the request is not well-formed. The request contains an unsupported parameter or an invalid value for a parameter.
If more information about the error is available, the HTTP response headers include the HCP-specific X-HCPErrorMessage header. 403 Forbidden One of: The authenticated user doesnt have write permission. The object is under retention and the namespace does not allow adding or replacing custom metadata for objects under retention. The namespace does not exist. The access method (HTTP or HTTPS) is disabled.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 404 413 Not Found File Too Large HCP could not find the object for which you are storing the custom metadata. The request is trying to store custom metadata that is larger than one GB. HCP cannot store custom metadata that is larger than one GB. The request has a Content-Encoding header with a value other than gzip.
415
73
Request-specific response headers The table below describes the request-specific response header for this operation. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
Header
X-HCP-Hash
Description
The cryptographic hash algorithm HCP uses and the cryptographic hash value of the stored custom metadata, in this format: X-HCP-Hash: hash-algorithm hash-value You can use the returned hash value to verify that the stored data is the same as the data you sent. To do so, compare this value with a hash value that you generate from the original data.
Usage consideration The PUT of custom metadata may fail with a 400 (Bad Request) error in either of these cases:
The XML has a large number of different elements and attributes. In this case, try restructuring the XML to have fewer different elements and attributes. For example, try concatenating multiple element values, such as the different parts of an address, to create a new value for a single element. If you cannot restructure the XML to prevent failures, ask your namespace administrator about reconfiguring the namespace to prevent HCP from validating custom metadata XML.
74
Example: Storing custom metadata for an object Heres a sample HTTP PUT request that stores the custom metadata defined in the wind.custom-metadata.xml file in for an existing object named wind.jpg.
Request headers
PUT /rest/images/wind.jpg?type=custom-metadata HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 317
Response headers
HTTP/1.1 201 Created X-HCP-Hash: SHA-256 20BA1FDC958D8519D11A4CC2D6D65EC64DD12466E456... Location: /rest/images/wind.jpg X-HCP-Time: 1259584200 Content-Length: 0
75
An hcp-ns-auth authorization cookie The URL of the object The type=custom-metadata URL query parameter
If an object has multiple versions, specify this optional URL query parameter to check whether a specific version of the object has metadata:
version=version-id
If you omit the version parameter, HCP checks the current version of the object. Access permission To check for the existence of custom metadata, you need read permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 204 404 OK No Content Not Found
Meaning
Description
The specified object has custom metadata. The specified object does not have custom metadata. HCP could not find the specified object.
Request-specific response headers If HCP finds custom metadata for the specified object:
The Content-Type response header value is txt/xml. The Content-Length response header value is the length of the custom
metadata XML.
76
The HCP-specific response headers contain the same object metadata as the response to a HEAD request that checks the existence of an object. For example, the X-HCP-Hash header contains the hash value for the object, not the custom metadata. For details on these headers, see Checking the existence of an object or version on page 4-12. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Example: Checking the existence of custom metadata Heres a sample HTTP HEAD request that checks the existence of custom metadata for an object named wind.jpg in the images directory.
Request headers
HEAD /rest/images/wind.jpg?type=custom-metadata HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
77
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1258469462 X-HCP-SoftwareVersion: 4.1.0.123 Content-Type: text/xml Content-Length: 317 X-HCP-Type: object X-HCP-Size: 19461 X-HCP-Hash: SHA-256 42C605FBFFCD7CEAC36BE62F294374F94503D1DC1793736... X-HCP-VersionId: 80205544854849 X-HCP-IngestTime: 1258372270 X-HCP-RetentionClass: X-HCP-RetentionString: Deletion Allowed X-HCP-Retention: 0 X-HCP-RetentionHold: false X-HCP-Shred: false X-HCP-DPL: 2 X-HCP-Index: true X-HCP-Custom-Metadata: true X-HCP-Replicated: false
An hcp-ns-auth authorization cookie The URL of the object The type=custom-metadata URL query parameter
78
Request contents retrieving custom metadata for a specific version If an object has multiple versions, specify this optional URL query parameter to retrieve the custom metadata for a specific version of the object:
version=version-ID
If you omit the version parameter, HCP retrieves the custom metadata for the current version of the object. Request contents retrieving data in compressed format To request that HCP return custom metadata in gzip-compressed format, use an Accept-Encoding header containing the value gzip or *. The header can specify additional compression algorithms, but HCP uses gzip only. Access permission To retrieve custom metadata, you need read permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 204 404 406 OK No Content Not Found Not Acceptable
Meaning
Description
HCP successfully retrieved the custom metadata. The specified object does not have custom metadata. HCP could not find the specified object. The request has an Accept-Encoding header that does not include gzip or specify *.
Request-specific response headers The HCP-specific response headers contain the same object metadata as the response to a HEAD request that checks the existence of an object. For example, the X-HCP-Hash header contains the hash value for the object, not the custom metadata. For details on these headers, see Checking the existence of an object or version on page 4-12. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
79
If the request specifies a gzip-compressed response, the response headers have these characteristics:
HCP returns a Content-Encoding header with a value of gzip. HCP returns a X-HCP-ContentLength header with the length of the
returned custom metadata before it was compressed. If HCP can provide information about an invalid request, the response has an X-HCP-ErrorMessage header describing the error. Response body The body of the HTTP response contains the custom metadata. Example: Retrieving custom metadata for an object Heres a sample HTTP GET request that retrieves the custom metadata for an object named wind.jpg in the images directory.
Request headers
GET /rest/images/wind.jpg?type=custom-metadata HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
710
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1258469462 X-HCP-SoftwareVersion: 4.1.0.123 Content-Type: text/xml Content-Length: 317 X-HCP-Type: object X-HCP-Size: 19461 X-HCP-Hash: SHA-256 42C605FBFFCD7CEAC36BE62F294374F94503D1DC1793736... X-HCP-VersionId: 80205544854849 X-HCP-IngestTime: 1258372270 X-HCP-RetentionClass: X-HCP-RetentionString: Deletion Allowed X-HCP-Retention: 0 X-HCP-RetentionHold: false X-HCP-Shred: false X-HCP-DPL: 2 X-HCP-Index: true X-HCP-Custom-Metadata: true X-HCP-Replicated: false
Response body
<?xml version="1.0" ?> <weather> <location>Massachusetts</location> <date>20091130</date> <duration_secs>180</duration_secs> <temp_F> <temp_high>31</temp_high> <temp_low>31</temp_low> </temp_F> <velocity_mph> <velocity_high>22</velocity_high> <velocity_low>17</velocity_low> </velocity_mph> </weather>
711
Request contents The DELETE request must specify these HTTP elements:
An hcp-ns-auth authorization cookie The URL of the object The type=custom-metadata parameter
Access permission To delete custom metadata, you need delete permission for the namespace. Request-specific return codes The table below describes the return codes that have specific meaning for this request. For descriptions of all possible return codes, see HTTP return codes on page A-9.
Code
200 204 403 OK No Content Forbidden
Meaning
Description
HCP successfully deleted the custom metadata. The specified object does not have custom metadata. One of: The authenticated user doesnt have the delete permission. The object is under retention and the namespace does not allow deleting custom metadata for objects under retention. The access method (HTTP or HTTPS) is disabled.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 404 409 Not Found Conflict HCP could not find the specified object. HCP could not delete the custom metadata because it is currently being written to the namespace.
Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
712
Example: Deleting custom metadata of an object Heres a sample HTTP DELETE request that deletes the custom metadata for an object named wind.jpg located in the images directory.
Request headers
DELETE /rest/images/wind.jpg?type=custom-metadata HTTP/1.1 Host: finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-RequestId: E62EFD408DBB7B8F X-HCP-Time: 1259584200 Content-Length: 0
713
714
8
Using the HCP metadata query API
The HCP metadata query API lets you search HCP for objects that meet specific criteria and get back metadata for the matching objects. With this API, you can search not only for objects and versions currently in the repository but also for information about deleted or purged objects. This API is particularly useful for applications that need to track changes to namespaces. This chapter describes how to use the metadata query API to retrieve information about objects.
81
Namespace in which the object is stored Object change time Operations on the object Object index setting Directory that contains the object
A query can specify multiple namespaces and directories, a range of change times, and any combination of create, delete, and purge operations. The API accepts query entries in XML or JavaScript Object Notation (JSON) format and can return results in either format. For example, you could use XML to specify the entries and request that the response be in JSON. Note: This chapter uses entry to refer to an XML element and the equivalent JSON object and property for an XML attribute or the equivalent JSON name/value pair. When you use the query API, HCP returns a set of operation records each of which identifies an object and an operation on the object and contains additional metadata for the object and operation. For more information on operation records, see Operation records below. Because a large number of matching objects can result in a very large response, HCP lets you limit the number of operation records returned for a single request. You can retrieve data for all the matching objects by using multiple requests. This process is called using a paged query. For more information on paged queries, see Paged queries on page 8-4.
82
Operation records
HCP maintains records of object creation, deletion, and purge operations (also referred to as transactions). Deletion and purge information is helpful for applications, such as search applications, that must track changes to namespace contents. The HCP system configuration determines how long HCP keeps deletion or purge records. By default, HCP returns only basic information about the object and operation, including the operation type, object identification information, and the change time. For creation records, the change time is the time the object was last modified. For deletion and purge records, the change time identifies the time of the operation. If you specify a verbose entry with a value of true in a query, HCP returns complete system metadata for the object and operation, as listed in object entry on page 8-18. Records returned while versioning is enabled If versioning is enabled for a namespace, these rules determine which operation records HCP returns:
HCP returns creation records for versions of deleted objects until these
versions are pruned.
HCP returns deletion records until it removes them from the system, as
determined by the system.
HCP returns a single purge record for each purge operation. It does not
return any records for the individual purged objects. HCP returns purge records until it removes them from the system. Records returned while versioning is disabled If you create and then delete an object while versioning is not enabled, HCP does not return the creation record. It returns the deletion record until HCP removes that record from the system. If versioning was enabled at an earlier time but is no longer enabled, HCP continues to return records of all operations performed during that time according to the rules listed in Records returned while versioning is enabled above. If you delete any object while versioning is disabled, HCP does not return any creation records, regardless of when the object was created.
83
Request URL
Paged queries
In some cases, a query can result in a very large number of operation records, which can overload or reduce the efficiency of the client. You can prevent this by using paged queries, where you issue multiple requests that each retrieve a limited number of operation records. The client can process the records from each response before requesting additional data. To use a paged query:
For each request after the first, specify a lastResult entry containing the
values of the urlName, changeTimeMilliseconds, and version properties in the last record returned in response to the previous request.
Determine whether the response contains the final record of the query
result set by checking the value of the code property of the response status entry:
If the value is INCOMPLETE, more records remain. Request another page. If the value is COMPLETE, all records have been retrieved.
For an example of using a paged query, see Example 3: Using a paged query to retrieve a large number of records on page 8-26.
Request URL
The URL format in a metadata query API request depends on whether you use a DNS name or IP address to connect to the HCP system and on the namespace or namespaces you want to access.
84
Request URL
Unlike with requests to the /rest interface, do not specify a namespace in this URL. If the tenant has granted system-level users administrative access to itself, you can use a system-level user account for your query. In this case, use a URL with this format:
https://admin.hcp-name.domain-name/query
In this case, the request can include any namespaces owned by any tenants to which system-level users have administrative access. A query that does not specify any namespace returns information about all namespaces in all such tenants, including the default namespace. For this URL format, you need to use HTTP with SSL security (HTTPS).
The request Host header must specify the applicable hostname for either DNS name format; for example, europe.hcp.example.com or admin.hcp.example.com. For this URL format, you need to use HTTP with SSL security.
85
Request considerations
Request considerations
The following considerations apply to metadata query API requests:
If the metadata query uses HTTP with SSL security and the HCP system
uses a self-signed SSL server certificate, the request must include an instruction not to perform SSL certificate verification. With cURL, you do this by including the -k option in the request command line. In Python with PycURL, you do this by setting the SSL_VERIFYPEER option to false.
The request must specify query, in all lowercase, as the first element
following the hostname or IP address in the URL path.
HCP caches each query for a period of time on the server that receives
the request. If you use an IP address in the URL in each request, you access the cached query and avoid having to recreate the query with each request. This can significantly improve the performance of paged queries that get large amounts of data. Some HTTP libraries cache HTTP connections. Programs using these libraries may automatically reconnect to the same server for paged queries. In this case, using a DNS name to establish the connection provides the same performance benefit as using an IP address. For more information on the relative advantages of DNS names and IP addresses, see Namespace access by IP address on page 11-2.
Request format
You use the HTTP POST method to send a metadata query API request to HCP.
If the URL starts with an IP address and the request specifies any
namespaces, a Host header with the tenant name.
86
Request format
If the URL hostname or Host header starts with a tenant name, a data access account with read and search permission for the namespaces to be queried. If the URL hostname or Host header starts with admin, a systemlevel user account with the search role. In this case, the request can query all namespaces for all tenants to which system-level users have administrative access.
For information on the authorization cookie format, see Authenticating namespace access on page 3-8.
87
Request format
Request body
The body of the HTTP request consists of entries in XML or JSON format.
The XML request body has the format shown below. Elements at each hierarchical level can be in any order.
<queryRequest> <count>return-record-count</count> <lastResult> <urlName>object-url</urlName> <changeTimeMilliseconds>change-time-in-milliseconds.index </changeTimeMilliseconds> <version>version-id</version> </lastResult> <systemMetadata> <changeTime> <start>start-time-in-milliseconds</start> <end>end-time-in-milliseconds</end> </changeTime> <directories> <directory>directory-path</directory> ... </directories> <indexable>true|false</indexable> <namespaces> <namespace>namespace-name.tenant-name </namespace> ... </namespaces> <transactions> Any combination of the following <transaction>create</transaction> <transaction>delete</transaction> <transaction>purge</transaction> </transactions> </systemMetadata> <verbose>true|false</verbose> </queryRequest>
88
Request format
The JSON request body has the format shown below. Objects at each hierarchical level can be in any order.
{ "count":"return-record-count", "lastResult": { "urlName":"object-url", "changeTimeMilliseconds":"change-time-in-milliseconds.index", "version":version-id }, "systemMetadata": { "changeTime": { "start":start-time-in-milliseconds, "end":end-time-in-milliseconds }, "directories": { "directory":["directory-path",...] }, "indexable":"true|false", "namespaces": { "namespace":["namespace-name.tenant-name",...] }, "transactions": { "transaction":[Any combination of "create","delete", "purge"] } }, "verbose":"true|false" }
89
Request format
Valid values
Description
The maximum number of operation records to return. The default is 10,000.
lastResult N/A
verbose
One of: true Return detailed information about each object. false Return only the object URL, change time, version ID, and operation.
810
Request format
lastResult entry Use the lastResult entry only in the second through final requests of a paged query. This entry identifies the last record that was returned in the previous query so that HCP can retrieve the next set of records. The entry contains the child entries described in the table below.
Entry
urlName
Valid values
A fully qualified object URL
Description
The urlName value of the object in the last operation record returned in response to the previous query. The changeTimeMilliseconds value of the object in the last operation record returned in response to the previous query. For more information on this entry, see object entry on page 8-18.
changeTime Milliseconds
A timestamp in milliseconds since January 1, 1970, at 00:00:00 UTC, followed by a period and a twodigit suffix
version
A version ID
The version value of the last operation record returned in response to the previous query.
systemMetadata entry The systemMetadata entry specifies the criteria that the returned operation records must match. The entry contains the subentries listed in the table below. Some of the subentries, such as changeTime have children. In this table, child entries immediately follow the parent entries.
Entry
changeTime N/A
Valid values
Description
Container for start and end entries. Specifies the range of change times of the objects for which to return the operation records. This entry can contain neither, either, or both of the start and end subentries. If you omit this entry, HCP returns operation records for all objects that were ingested, changed, deleted, or purged until one minute before the time HCP receives the request.
811
Request format
(Continued)
Entry
start (child) One of:
Valid values
Description
Child of the changeTime entry. Requests operation records for objects with change times on or after the specified date and time. The default value is 0 (January 1, 1970, 00:00:00 UTC). In the ISO 8601 format, you cannot specify a millisecond value. The time corresponds to zero milliseconds into the specified second.
Milliseconds since January 1, 1970, 00:00:00 UTC An ISO 8601 datetime value in this format:
yyyy-MM-ddThh:mm:ssZ
For example, 2010-1116T14:27:20-0500 represents the start of the 20th second into 2:27 PM, November 16, 2010, EST. end (child) One of: Milliseconds since January 1, 1970, 00:00:00 UTC. An ISO 8601 datetime value in this format:
Child of the changeTime entry. Requests operation records for objects with change times before the specified date and time. The default value is one minute before the time HCP receives the request. In the ISO 8601 format, you cannot specify a millisecond value. The time corresponds to zero milliseconds into the specified second. If you use a value that is less than one minute before the current time, ensure that all writes completed at least one minute ago so that you get results for the most recent operations.
yyyy-MM-ddThh:mm:ssZ
directories
N/A
Container for zero or more directory entries. If you omit this entry, HCP returns operation records for objects in all directories in the specified namespaces.
directory (child)
The path to the directory containing the objects for which to retrieve operation records. Start the path with a forward slash (/) followed by the name of a directory immediately below rest. Do not include rest in the path.
Child of the directories entry. Requests operation records for objects in the specified directory and all its subdirectories, recursively. If you query multiple namespaces, HCP returns operation records for the directory contents in each namespace in which the directory occurs.
812
Request format
(Continued)
Entry
indexable One of:
Valid values
Description
A specification of whether to filter the returned operation records based on the object index setting. HCP returns deletion and purge records only for objects that had the specified setting at the time they were deleted or purged. If you omit this entry, HCP returns operation records for objects regardless of their index settings. Container for zero or more namespace entries. If you omit this entry and the URL starts with a tenant name, HCP returns operation records for all namespaces for which the user has search permission. If you omit this entry and the URL starts with admin, HCP returns operation records for the default namespace and the namespaces owned by each tenant that has granted system-level users administrative access to itself.
true Return operation records only for objects with an index setting of true. false Return operation records only for objects with index setting of false.
namespaces
N/A
namespace (child)
Namespace name along with the name of the owning tenant, in this format:
namespace-name.tenantname
transactions N/A Container for up to three transaction entries, each specifying a different operation type. If you omit this entry, HCP returns operation records for all operation types. transaction (child) One of: create delete purge For more information on operation types, see Operation records on page 8-3. Child of the transactions entry. Specifies a type of operation for which to return records.
813
Response format
Response format
The response to a query API request returns the results as XML or JSON, depending on the value of the Accept request header. The returned results are sorted in this order:
changeTimeMilliseconds value, with the oldest records first. Path within the namespace. Version ID of the operation record.
Meaning
Description
HCP successfully ran the query and returned the results. One of: The hcp-ns-auth cookie does not provide a valid username and password for a data access account with search permission or for a system-level user account with the search role. The tenant specified in the URL does not exist.
814
Response format
(Continued)
Code
400
Meaning
Bad Request.
Description
The request syntax is not valid. Possible reasons for this error include: Invalid URL query parameters A Content-Encoding header that specifies gzip used with data that is not in gzip-compressed format Invalid XML or JSON, including invalid element or object names Invalid element or object values, such as a malformed version ID or invalid directory path
If more information about the error is available, the response includes an HCP-specific X-HCP-ErrorMessage HTTP header. 403 Forbidden The request specifies a namespace to which the authenticated user does not have access. If the request includes multiple namespaces, HCP returns this error if the user does not have access to at least one of them. If more information about the error is available, the response includes an HCP-specific X-HCP-ErrorMessage HTTP header. 406 Not Acceptable One of: The request does not have an Accept header, or the Accept header does not specify application/xml or application/json. The request has an Accept-Encoding header that does not include gzip or specify *.
One of: The request does not have a Content-Type header, or the Content-Type header does not specify application/ xml or application/json. The request has a Content-Encoding header with a value other than gzip.
815
Response format
Content-Encoding header with a value of gzip X-HCP-ContentLength header with the length of the returned data
before it was compressed If HCP can provide information about an invalid request, the response has an X-HCP-ErrorMessage header describing the error.
Response body
The body of the HTTP response contains XML or JSON that lists the operation records that match the query. The order of entries in the response body may vary from one request to another. However, all entries in a response have the same order.
Additional attributes if the verbose entry specified true changeTimeMilliseconds="change-time-in-milliseconds.index" version=version-id urlName="object-url" operation="operation-type"
/>
816
Response format
Additional name/value pairs if the verbose entry specified true "urlName":"object-url", "operation":"operation-type", "changeTimeMilliseconds":"change-time-in-milliseconds.index", "version":version-id,
},
Description
The time period that this query covers. The results include only operation records for objects with change times during this period. For more information, see query entry below. A set of object entries representing the operation records that match the query. For more information, see resultSet entry on page 8-18. Information about the response, including the number of returned records and whether the response completes the query results. For more information, see status entry on page 8-21.
resultSet
status
817
Response format
query entry The query entry has the properties described in the table below.
Property
start
Description
The value of the request start entry in milliseconds since January 1, 1970, at 00:00:00 UTC. If you omitted the entry, the value is 0 (zero). The value of the request end entry, in milliseconds since January 1, 1970, at 00:00:00 UTC. If you omitted an end entry in the request, the value is one minute before the time HCP received the request.
end
resultSet entry The resultSet entry has one child object entry for each operation record that matches the query. Note: The metadata query API does not return records for open objects (that is, objects that are still being written or never finished being written). object entry In XML, the object entries are child object elements of the resultSet element. In JSON, the object entries are unnamed objects in the resultSet entry. Each object entry provides information about an individual create, delete, or purge operation and the object affected by the operation. The properties the entry contains depend on the value of the verbose request entry. The object entry has the properties listed in the table below.
Property
Returned in all responses
changeTimeMilliseconds For creation records, the time when the object or version was last changed. For deletion and purge records, the time when the object was deleted or purged. The value is the time in milliseconds since January 1, 1970, at 00:00:00 UTC, followed by a period and a twodigit suffix. The suffix ensures that the change time values for versions of an object are unique.
Description
818
Response format
(Continued)
Property
operation
Description
The type of operation the record represents: CREATED HCP ingested the object or version. DELETED HCP deleted the object. PURGED HCP purged all versions of the object.
urlName version
The fully qualified object URL. The version ID of the operation record. All operation records, including those for delete and purge operations and operations performed while versioning is disabled, have version IDs. For create operations, this value is the version ID of the version.
819
Response format
(Continued)
Property
ingestTimeString
Description
The time when HCP stored the object in ISO 8601 format:
yyyy-MM-ddThh:mm:ssZ
For more information on the format, see the changeTimeString entry, above. replicated A value of true or false indicating whether the object has been replicated. The value is true only if the object, including the current version and all metadata, has been replicated. The end of the retention period for the object, in seconds since January 1, 1970, at 00:00:00, 00:00:00 UTC. This value can also be 0, -1, or -2. The name of the retention class assigned to the object. This value is an empty string if the object is not assigned to a retention class. The end of the retention period for the object, in this format:
retention
retentionClass
retentionString
yyyy-MM-ddThh:mm:ssZ
This value can also be Deletion Allowed, Deletion Prohibited, or Initial Unspecified. For more information on the time format, see the changeTimeString entry, above. shred size type utf8Name A value of true or false indicating whether HCP should shred the object after it is deleted. The object size, in bytes. The object type. This value is always object. The UTF-8-encoded name of the object.
820
Examples
status entry The status entry has the properties listed in the table below.
Property
code
Description
An indication of whether all matching records have been returned: COMPLETE All matching records have been returned. This value is returned if the response includes all matching records or if the response includes the last record in a set of partial responses in a paged query. INCOMPLETE Not all matching records have been returned. This value is returned if the request count entry is smaller than the number of records that meet the query criteria or if the response is incomplete due to an error encountered in executing the query. You can get additional results by resubmitting the request with a lastResult entry identifying the last record in the returned response. For more information on this technique, see Paged queries on page 8-4.
message
Normally, an empty string. If HCP encounters an error, such as a server being unavailable while processing the request, this entry describes the error. The number of operation records returned.
results
Examples
The following examples show some of the ways you can use the query API to get information. They show how to:
Get metadata for objects that changed during a specific time span Use a paged query to get a large number of operation records
821
Examples
822
Examples
Request headers
POST /query HTTP/1.1 Host: europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Type: application/xml Accept: application/json Content-Length: 258
Response headers
HTTP/1.1 200 OK Transfer-Encoding: chunked
823
Examples
824
Examples
Are in any namespace accessible by the data access account Changed during 2010
The start entry specifies 12:00:00.00 a.m. on January 1, 2010, and the end entry specifies 12:00:00.00 a.m. on January 1, 2011. The response, which is in XML, includes records for all operations and object versions. The information returned for each operation record that meets the query criteria consists of the change time, version, operation, and URL.
825
Examples
Request headers
POST /query HTTP/1.1 Host: europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Type: application/json Accept: application/xml Content-Length: 81
Response headers
HTTP/1.1 200 OK Transfer-Encoding: chunked
Response body
To limit the example size, the XML below contains only two of the object elements of the response.
<?xml version='1.0' encoding='UTF-8'?> <queryResult xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/query-result.xsd"> <query start="1262304000000" end="1293840000000"/> <resultSet> <object changeTimeMilliseconds="1277923464679.00" version="81787101672577" urlName="https://finance.europe.hcp.example.com/rest/test2.txt" operation="CREATED"/> <object changeTimeMilliseconds="1277923478677.00" version="81787102472129" url="https://finance.europe.hcp.example.com/rest/test2.bak" operation="CREATED"/> . . . </resultSet> <status results="11" message="" code="COMPLETE"/> </queryResult>
826
Examples
This example requires two methods, fromDict and toDict, that are not part of the Python or PycURL distribution. These methods convert between XML text and Python dict structures. In this example, they are imported from the XML module of the arc.parser.xml_parser module. To use this example, you need to provide these or equivalent methods.
#!/usr/bin/env python # encoding: utf-8 import pycurl import StringIO # Import XML - dict conversion methods. from arc.parser.xml_parser import XML # The request class represents a query and executes the request. class Request(Exception): """The request class used in the queries.""" def __init__(self): Exception.__init__(self) self.header=None self.cookie=None self.urlName="" self.namespace=None self.directory=None self.count=None self.body=None self.verbose=None # The query dictionary, a container for the request # body entries. self.queryDict={'queryRequest':{}} def queryRequest(self, data, namespace, systemName): """Make a request to HCP.""" # Define the pycurl parameters. headers = {pycurl.HTTPHEADER :["Accept: application/xml", "Content-Type: application/xml", "Host: europe.%s" %(systemName)]} self.lastHeader = StringIO.StringIO() self.curl=pycurl.Curl() self.curl.setopt(pycurl.FAILONERROR, 1) self.curl.setopt(pycurl.COOKIE, self.cookie) self.curl.setopt(pycurl.HTTPHEADER, headers[pycurl.HTTPHEADER]) self.curl.setopt(pycurl.URL, self.urlName) cout=StringIO.StringIO() self.curl.setopt(pycurl.WRITEFUNCTION, cout.write) cin=StringIO.StringIO(data) fileSize=len(data) self.curl.setopt(pycurl.INFILESIZE, fileSize) self.curl.setopt(pycurl.READFUNCTION, cin.read)
827
Examples
self.curl.setopt(pycurl.PUT, 1) self.curl.setopt(pycurl.CUSTOMREQUEST, 'POST') self.curl.setopt(pycurl.SSL_VERIFYPEER, 0) self.curl.setopt(pycurl.SSL_VERIFYHOST, 0) self.curl.setopt(pycurl.VERBOSE, 0) for header, value in headers.iteritems(): self.curl.setopt(header, value) self.curl.perform() output=cout.getvalue() response=self.curl.getinfo(pycurl.RESPONSE_CODE) self.curl.close # Return the XML response and the HTTP response code. return output, response # Initialize the query dictionary - called by the class initializer. def makeQueryDict(self): """ Add the system metadata to the query dictionary.""" self.queryDict['queryRequest'].update({'systemMetadata':{}}) self.queryDict['queryRequest']['systemMetadata'].update \ ({'namespaces':{'namespace':[]}}) self.queryDict['queryRequest']['systemMetadata']['namespaces']\ ['namespace'].append(self.namespace) self.queryDict['queryRequest']['count']=self.count self.queryDict['queryRequest']['verbose']=self.verbose self.queryDict['queryRequest']['systemMetadata'].update \ ({'directories':{'directory':[]}}) self.queryDict['queryRequest']['systemMetadata']['directories']\ ['directory'].append("/%s" % self.directory) # Utility methods used by main. def getRecords(resultsDict): """ Returns a list of operation records from the query, plus the / result count.""" recordList = {} resultsCount=0 currResultCount = int(resultsDict['status']['results']) if currResultCount: resultsCount += currResultCount if resultsDict['resultSet']: recordList = resultsDict['resultSet']['object'] if isinstance(recordList, dict): recordList = [recordList] print "Got " + str(resultsCount) + " records." # Return the operation records as a list, number of results \ # received to date. return recordList, resultsCount
828
Examples
def getResults(currRecordList): """ Returns a results iterator. """ while currRecordList != None and len(currRecordList) > 0: obj = {} for obj in currRecordList: yield obj currRecordList = list() # Executed when you run python query_test.py. def main(): """Run a query using the Request class""" # Create a query request and set the request entries. testQuery=Request() testQuery.namespace="finance.europe" systemName="hds.example.com" testQuery.directory="" testQuery.count=3 testQuery.verbose=True testQuery.cookie="hcp-ns-auth=YWxscm9sZXM=:04EC9F614D89FF5C7126D..." testQuery.urlName="https://europe.%s/query" % systemName testQuery.makeQueryDict() # Make the first request to HCP. data, response=testQuery.queryRequest(XML.fromDict (testQuery.queryDict), testQuery.namespace, systemName) # If the response is invalid, raise an exception. if response!=200: raise Exception("Error. Response code is: %d" % response) # The response is valid. Convert XML to python dict. resultsDict=XML.toDict(data)['queryResult'][1] # Call getRecords to get the operation records and results count. recordList, resultCount = getRecords(resultsDict) # Show the operation records list on the command line. # Open a file in which to put the operation records. outRecords=open("recordList.txt", 'wb') try: # Write the first set of operation records to the file. outRecords.write("%s" % recordList) # Loop to make the requests and add the results to the file. # Executed until HCP returns a result with a COMPLETE status. while resultsDict['status']['code']!='COMPLETE': # Get the last result record in the response. for rowDict in getResults(recordList): continue # Add results to the queryDict for the next request. testQuery.queryDict['queryRequest']['lastResult']= {} testQuery.queryDict['queryRequest']['lastResult']\ ['urlName']=rowDict['urlName']
829
Examples
testQuery.queryDict['queryRequest']['lastResult']\ ['changeTimeMilliseconds']=rowDict\ ['changeTimeMilliseconds'] testQuery.queryDict['queryRequest']['lastResult']\ ['version']=rowDict['version'] # Make the new request and get the results. data, response=testQuery.queryRequest(XML.fromDict (testQuery.queryDict), testQuery.namespace, systemName) resultsDict=XML.toDict(data)['queryResult'][1] recordList, resultCount = getRecords(resultsDict) # If the HTTP response was not 200, raise an exception. if response!=200: raise Exception("Error. Response code is: %d" % response) # Add the results to the file. outRecords.write("%s" % recordList) # Close the file in all cases. finally: outRecords.close() if __name__== '__main__': main()
830
9
Retrieving namespace information
You can use HTTP GET requests to retrieve information about the namespaces you can access with your data access account. You can retrieve this information:
Namespaces you can access Retention classes supported by a namespace Permissions that apply to a namespace Statistics for a namespace
This chapter describes the procedures for getting this information and the format of the returned information.
91
The URL must include a namespace. You can use the name of any namespace owned by the tenant, even if your data access account doesnt include any permissions for that namespace. Request-specific return codes This operation does not have any request-specific return codes. For descriptions of all possible return codes, see HTTP return codes on page A-9. Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13.
92
Response body The HTTP response body consists of XML that lists the namespaces you can access, along with their settings. The XML has this format:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/proc-namespaces.xsl"?> <namespaces xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/proc-namespaces.xsd" tenantHostName="tenant-name.hcp-name.domain-name" httpScheme="http"> <namespace name="namespace-name" nameIDNA="namespace-url-name" versioningEnabled="true|false" searchEnabled="true|false" retentionMode="compliance|enterprise" defaultShredValue="true|false" defaultIndexValue="true|false" defaultRetentionValue="retention-setting" hashScheme="hash-algorithm" dpl="data-protection-level"> <description><![CDATA[
namespace-description
]]></description> </namespace> ... </namespaces>
Example: Listing accessible namespaces Heres a sample HTTP GET request that returns information about the namespaces owned by the europe tenant in the hcp.example.com system.
93
Request headers
GET /proc HTTP/1.1 Host: /finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 Content-Length: 1275
Response body
<namespaces xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/proc-namespaces.xsd" tenantHostName="europe.hcp.example.com" httpScheme="https"> <namespace name="finance" nameIDNA="finance" versioningEnabled="true" searchEnabled="true" retentionMode="enterprise" defaultShredValue="false" defaultIndexValue="true" defaultRetentionValue="0" hashScheme="SHA-256" dpl="2" > <description><![CDATA[ Finance department ]]></description> </namespace> <namespace name="support" nameIDNA="support" versioningEnabled="true" searchEnabled="true" retentionMode="enterprise" defaultShredValue="false" defaultIndexValue="true" defaultRetentionValue="0" hashScheme="SHA-256" dpl="2" > <description><![CDATA[ Technical Support department ]]></description> </namespace> </namespaces>
94
Request-specific return code If you request information about a namespace to which you do not have access, HCP returns 401 (Unauthorized). For descriptions of all possible return codes, see HTTP return codes on page A-9. Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Response body The body of the HTTP response consists of XML that lists the retention classes defined for the namespace. The XML has this format:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/proc-retention-classes.xsl"?> <retentionClasses xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/proc-retention-classes.xsd" namespaceName="namespace-name"> <retentionClass name="class-name" value="retention-value"> autoDelete="true|false" <description><![CDATA[
class-description
]]></description> </retentionClass> ... </retentionClasses>
95
Example: Listing retention classes for a namespace Heres a sample HTTP GET request that retrieves XML describing the retention classes defined for the finance namespace owned by the europe tenant in the hcp.example.com system. The example saves the results in a file named finance.europe.retentionclasses.xml.
Request headers
GET /proc/retentionClasses HTTP/1.1 Host: /finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 Content-Length: 1186
96
Response body
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/proc-retention-classes.xsl"?> <retentionClasses xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/proc-retention-classes.xsd" namespaceName="support"> <retentionClass name="deletable" value="-1" autoDelete="false"> <description><![CDATA[ Can be deleted at any time. ]]></description> </retentionClass> <retentionClass name="undeletable" value="-1" autoDelete="false"> <description><![CDATA[ Deletion is permanently disallowed. ]]></description> </retentionClass> <retentionClass name="HlthReg-107" value="A+21y" autoDelete="true"> <description><![CDATA[ Meets health information rule 107 for retention of 21 years. ]]></description> </retentionClass> <retentionClass name="sevenYears" value="A+7y" autoDelete="true"> <description><![CDATA[ Object can be deleted after seven years. ]]></description> </retentionClass> </retentionClasses>
97
Request-specific return code If you request information about a namespace to which you do not have access, HCP returns 401 (Unauthorized). For descriptions of all possible return codes, see HTTP return codes on page A-9. Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Response body The body of the HTTP response consists of XML that lists the permissions for the namespace and user. The table below describes the XML elements.
Element
namespacePermissions namespaceEffectivePermissions
Description
The permissions specified for the namespace. The permissions that are in effect for the namespace. This is the logical AND of the permissions specified at the system, tenant, and namespace levels. The permissions included in the users data access account. The permissions that are in effect for the current user in the namespace. This is the logical AND of the permissions specified for the system, tenant, namespace, and users data access account.
userPermissions userEffectivePermissions
98
Example: Listing permissions for a namespace and user Heres a sample HTTP GET request that retrieves namespace and user permissions for:
The myuser data access account. (The account is identified by the hcpns-auth cookie).
99
Request headers
GET /proc/permissions HTTP/1.1 Host: /finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 1286
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 Content-Length: 880
910
Response body
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/proc-permissions.xsl"?> <permissions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/proc-permissions.xsd" namespaceName="finance"> <namespacePermissions read="true" write="true" delete="true" privileged="true" purge="true" search="false"/> <namespaceEffectivePermissions read="true" write="true" delete="true" privileged="false" purge="false" search="false"/> <userPermissions read="true" write="true" delete="true" privileged="true" purge="true" search="false"/> <userEffectivePermissions read="true" write="true" delete="true" privileged="false" purge="false" search="false"/> </permissions>
911
Request-specific return code If you request information about a namespace to which you do not have access, HCP returns 401 (Unauthorized). For descriptions of all possible return codes, see HTTP return codes on page A-9. Request-specific response headers This operation does not return any request-specific response headers. For information on all HCP-specific response headers, see HCP-specific HTTP response headers on page A-13. Response body The body of the HTTP response consists of XML that lists the namespace statistics. The table below describes the XML elements.
Element
namespaceName totalCapacityBytes The namespace name. The number of bytes of storage allocated to the namespace. This is the total space available for all data stored in the namespace, including object data, metadata, and the redundant data required to satisfy the namespace DPL. The number of bytes currently occupied by all data stored in the namespace, including object data, metadata, and any redundant data required to satisfy the namespace DPL. The percent of the totalCapacityBytes value at which HCP notifies the tenant that free storage space for the namespace is running low.
Description
usedCapacityBytes
softQuotaPercent
912
Element
objectCount
Description
The number of objects and versions stored in the namespace. The object count includes individual versions of objects, including the versions of deleted objects, but not deleted versions. For example, if you create an object, add a version, delete the object, and then add a new version of the object, the object count increases by three.
The number of objects and versions that have been deleted and are waiting to be shredded. The number of bytes occupied by the objects and versions that are waiting to be shredded. The number of objects and versions that have custom metadata. The number of bytes occupied by custom metadata.
Example: Listing statistics for a namespace Heres a sample HTTP GET request that retrieves the statistics for the finance namespace owned by the europe tenant of the hcp.example.com system.
913
Request headers
GET /proc/statistics HTTP/1.1 Host: /finance.europe.hcp.example.com Cookie: hcp-ns-auth=bXl1c2Vy:3f3c6784e97531774380db177774ac8d Content-Length: 1286
Response headers
HTTP/1.1 200 OK X-HCP-Time: 1259584200 Content-Length: 565
Response body
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/static/xsl/proc-statistics.xsl"?> <statistics xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/static/xsd/proc-statistics.xsd" namespaceName="finance" totalCapacity="10737418240" usedCapacity="932454739" softQuota="85" objectCount="43230" shredObjectCount="0" shredObjectBytes="0" customMetadataObjectCount="6754" customMetadataObjectBytes="894893" />
914
10
Using the Namespace Browser
The HCP Namespace Browser lets you manage namespace content and view information about namespaces. This chapter describes the Browser interface and how to use it.
101
Logging in
To log into the Namespace Browser: 1. Open a web browser window. 2. In the address field, enter the URL for the namespace for which you want information. For example, to use HTTPS to browse the finance.europe.hcp.example.com namespace, enter:
https://finance.europe.hcp.example.com
The Namespace Browser login page appears. Note: You cannot use an IP address in the URL to access the Namespace Browser. 3. In the Username field, type your data access account username. 4. In the Password field, type your case-sensitive password.
102
5. Click on the Log In button. The Namespace Browser opens showing the rest directory for the namespace. You can log out at any time by clicking on the Log Out link in the Browser menu bar.
Common elements
All Namespace Browser pages have these options in the menu bar at the top of the page:
Content Displays the top-level directories in the current namespace. Namespace Displays the list of namespaces you can access and
provides access to information about these namespaces.
Search HCP Opens the HCP Search Console. For more information on
using this Console, see Searching Namespaces. This option appears only if search is enabled for the namespace.
Log Out Logs you out and displays the login page. Password Displays the Change Password page. For more information
on the Change Password page, see Changing your password below.
Are case sensitive. Can contain any valid UTF-8 characters, including white space. Must include at least one character from two of these three groups:
alphabetic, numeric, and other.
103
To change your password: 1. Click on Password in the Browser menu bar. 2. On the Change Password page:
In the Existing Password field, type your current password. In the New Password field, type your new password. In the Confirm New Password field, type your new password again.
3. Click on the Update Password button. After you change the password, the Change Password page continues to be displayed. To continue using the Namespace Browser, navigate to another page.
If versioning is enabled for the namespace, list versions of an object. View or retrieve the content of objects and, if versioning is enabled for
the namespace, the content of specific versions of objects.
104
Description
Click on this tab to add a new directory. For more information how to add a directory, see Creating directories on page 10-10.
Upload Object
Click on this tab to add an object to the namespace. For more information on how to add an object, see Adding objects and versions on page 10-10.
Directory listing
The directory listing displays information about the objects in a directory and provides access to subdirectories and object versions. These controls appear in a bar immediately above the directory listing:
If versioning is enabled for the namespace, the left side has a link that
alternately says Show deleted objects or Hide deleted objects. Click on the link to show or hide deleted objects and directories.
The right side has two buttons: Previous and Next. The Namespace
Browser displays up to 100 entries at a time. If a directory has more than 100 entries, these buttons are active and you can use them to page through the directory to see all its contents. Click on any column heading to sort the listing by the column values. Clicking on a heading alternately sorts in ascending and descending order.
105
The table below describes the information for each item in the directory listing.
Item
Type icon
Description
An icon indicating the item type: Directory Object Deleted object Click on the icon to display the directory contents or view or retrieve the object. Clicking on a deleted object icon has no effect. For more information, see Viewing and retrieving object and version content on page 10-9.
Action icon
A control for deleting an object. This icon appears only if the item can be deleted. For more information, see Deleting objects on page 10-10. A control for listing object versions. This icon appears if versioning is enabled for the namespace. For more information, see Listing object versions on page 10-7.
Name
The object or directory name. Names of deleted objects, if displayed, are in gray, crossed-out text and are not active. Click on the name to display the directory contents or view or retrieve the object. For more information, see Viewing and retrieving object and version content on page 10-9.
The object size, in bytes. The retention setting as a date and time or as a value such as Deletion Allowed. The retention class name, if any. The date and time the object or current version was added to the namespace. A representation of the state of the object metadata: The object is on hold. The object will be shredded when it is deleted. The object is marked for indexing. The object has custom metadata.
106
Item
DPL Hash (algorithmname)
Description
The data protection level. Hash information for the object. Move the mouse over the View Hash link to view the hash value. You can then highlight and copy it to the clipboard.
Tip: You can use the view-source option in the web browser to see the XML for the Namespace Browser page. For information on this XML, see Listing directory contents on page 5-5.
107
Click on any column heading to sort the list by the column values. Clicking on a heading alternately sorts in ascending and descending order. The table below describes the information displayed for each item in the version list.
Item
Type icon
Description
An icon indicating the version type: Normal version. Click on this entry to view or retrieve this version. Deleted version. This icon is inactive.
Version
The version ID. Click on this entry to view or retrieve this version. If the entry represents a deleted version, the ID is inactive.
The object size, in bytes. The retention setting as a date and time or as a value such as Deletion Allowed. The retention class name, if any. One of: For object versions, the date and time the version was added to the namespace. For deleted versions, the date and time the object was deleted.
Metadata
A representation of the state of the metadata for the version: The object is on hold. All versions of an object have the same hold setting. The version will be shredded when it is deleted. The version is marked for indexing. The version has custom metadata.
The data protection level. Hash information. Move the mouse over the View Hash link to view the hash value for the object. You can then highlight and copy the hash value to the clipboard.
108
Listing versions of deleted objects and directories with the same name A directory listing in the Namespace Browser can include only one item with a specific name, but a version listing for a deleted object can include entries for both the object and a deleted directory with the same name. The following example shows how this can happen: 1. Create a test1 directory. 2. In the test1 directory, create and delete a subdirectory named test2. 3. In the test1 directory, save and delete an object with the name test2. 4. List the contents of the test1 directory and select Show deleted objects. The test1 directory listing includes the deleted test2 object but not the deleted test2 directory. 5. List the versions of the test2 object and select Show deleted objects. The version list includes the deleted test2 directory. Note: If you create and delete the test2 object before you create the test2 directory (whether you delete the test2 directory or not), you cannot list versions of test2 in the Namespace Browser, and you cannot see the deleted object versions. You can however, use direct HTTP requests (for example, with curl) to get this information.
109
3. In the version list, click on the version ID or type icon for the version you want.
Deleting objects
To delete an object from the namespace: 1. Navigate to the directory that contains the object. 2. Click on the delete icon ( ) for the object you want to delete.
3. In response to the confirming message, click on the OK button. The object disappears from the list. For information on displaying deleted objects, see Listing directory contents on page 10-5.
Creating directories
To create a directory: 1. Navigate to the parent directory. 2. If the create directory interface is not active, click on the Create Directory tab. 3. Enter the name of the new directory in the entry field below the tab.
1010
Namespaces you can access Retention classes defined for a namespace Permissions for accessing a namespace Namespace statistics
Hash algorithm. Data protection level. Whether versioning is enabled. Whether search is enabled. Retention mode. This mode affects whether privileged operations are
allowed on objects. A namespace can be in either of two retention modes: compliance or enterprise. In compliance mode, privileged operations are allowed. In enterprise mode, they are not allowed.
1011
Default index setting for objects added to the namespace. Default retention setting for objects added to the namespace. Statistics A link that displays the namespace statistics. For more
information, see Viewing namespace statistics on page 10-13.
Namespace description.
Retention class name Retention setting Whether to automatically delete objects in the class after their
retention periods expire
1012
Viewing permissions
To view the data access permissions that apply to a namespace and your data access account: 1. In the Namespace Browser menu bar, click on Namespaces to open the list of accessible namespaces. 2. Click on the Permissions link for the namespace you want. The table below describes the information shown about permissions.
Item
Namespace permissions Namespace effective permissions Data access account username permissions Data access account username effective permissions
Description
The permissions specified for the namespace. The permissions that are in effect for the namespace. This is the logical AND of the permissions specified for the system, tenant, and namespace. The permissions included in your data access account. The permissions that are in effect for the combination of your user account and the namespace. This is the logical AND of the permissions specified for the system, tenant, namespace, and users data access account.
Description
The number of bytes of storage allocated to the namespace. This is the total space available for all data stored in the namespace, including object data, metadata, and the redundant data required to satisfy the namespace DPL.
1013
Statistic
Used Capacity (Bytes) Soft Quota
Description
The number of bytes currently occupied by all data stored in the namespace, including object data, metadata, and any redundant data required to satisfy the namespace DPL. The percentage of the Total Capacity value at which HCP notifies the tenant that free storage space for the namespace is running low. The number of objects and versions stored in the namespace. The object count includes individual versions of objects, including the versions of deleted objects, but not deleted versions. For example, if you create an object, add a version, delete the object, and then add a new version of the object, the object count increases by three.
Object Count
The number of objects and versions that have been deleted and are waiting to be shredded.
Shred Object (Bytes) The number of bytes occupied by the objects and versions that are waiting to be shredded. Custom Metadata Object Count Custom Metadata Object (Bytes) The number of objects and versions that have custom metadata. The number of bytes occupied by custom metadata.
1014
11
General usage considerations
This appendix contains usage considerations that affect namespace access in general.
111
If your client uses a hosts file to map HCP host names to IP addresses,
the client system has full responsibility for converting any hostnames to IP addresses. Therefore, HCP cannot spread the load or prevent attempts to connect to an unavailable server. For more information on using a hosts file, see Enabling URLs with hostnames on page 3-4.
When you access the HCP system by hostname, HCP ensures that
requests are distributed among servers, but it does not ensure that the resulting loads on the servers are evenly balanced.
Directory structures
Because of the way HCP stores objects, the directory structures you create and the way you store objects in them can have an impact on performance. Here are some guidelines for creating effective directory structures:
Plan your directory structures before storing objects. Make sure all
namespace users are aware of these plans.
112
Shredding considerations
Shredding considerations
Multiple objects with the same content should all have the same shred setting. If they don't and you delete the objects:
Each object is shredded or not according to its shred setting. If the last object deleted is marked for shredding, HCP shreds the
storage used for that object and for all the other objects that were not marked for shredding. As a result, none of the content remains on the system storage in any form.
If the last object deleted is not marked for shredding, HCP does not
shred the storage used for that object or any of the other objects that were not marked for shredding. As a result, the unshredded content remains on the system storage until the storage is reused.
113
The target server failed while the object was open for write. The TCP connection broke while the object was open for write (for
example, due to a network failure or the abnormal termination of the client application). Also, in some circumstances, a write operation is considered to have failed if system hardware failed while the object was open for write. HTTP causes a flush only at the end of a write operation, so an object left by a failed write:
Remains open Is empty Is not WORM Has no cryptographic hash value Is not subject to retention Cannot have custom metadata Is not indexed 114
Is not replicated
An object like this can be deleted or overwritten.
The open empty object from the failed write is removed, and no version
of the empty instance of the object is retained.
A deleted version of the object is created, even if the object did not
exist prior to the failed write.
While an object is open for write through one IP address, you cannot
open it for write through any other IP address.
While an object is open for write, you can read it from any IP address,
even though the object data may be incomplete. If the read is against the server hosting the write, it may return more data than reads against other servers.
115
Non-WORM objects
Non-WORM objects
The namespace can contain objects that are not WORM:
Objects that are open for write and have no data are not WORM. Objects left by certain failed write operations are not WORM.
Objects that are not WORM are not subject to retention. You can delete these objects or overwrite them without first deleting them.
Persistent connections
HCP supports HTTP persistent connections. Following a request for an operation, HCP keeps the connection open for 60 seconds so a subsequent request can use the same connection. Persistent connections enhance performance because they avoid the overhead of opening and closing multiple connections. In conjunction with persistent connections, using multiple threads so that operations can run concurrently provides still better performance. If the persistent connection timeout period is too short, tell your namespace administrator. Note: With persistent connections, if a single IP address has more than 254 concurrent open connections, those above the first 254 may have to wait as long as ten minutes to be serviced. This includes connections where the request explicitly targeted the IP address, as well as connections where the HCP hostname resolved to the target IP address. To avoid this issue, either dont use persistent connections or ensure that no more than 254 threads are working against a single server at any time.
116
If the original request used the hostname of the HCP system in the
URL, repeat the request in the same way.
Multithreading
HCP lets multiple threads access the namespace simultaneously. Using multiple threads can enhance performance, especially when accessing many small files across multiple directories. Here are some guidelines for the effective use of multithreading:
Only one client can write to a given object at one time. Similarly, a
multithreaded client can write to multiple objects at the same time but cannot have multiple threads writing to the same object simultaneously.
117
Multithreading
HCP has a limit of 255 concurrent HTTP connections per server, with
another 20 queued. Tip: For better performance, consider limiting the number of concurrent read threads per server to 200 and concurrent write threads per server to 50 for small objects. For large objects, consider using fewer threads.
118
A
HTTP reference
This appendix contains a reference of HTTP requests and responses for accessing a namespace and for using the metadata query API. For detailed information on using HTTP, see Chapters 4 through 7. For detailed information on using the metadata query API, see Chapter 8, Using the HCP metadata query API.
A1
HTTP methods
HTTP methods
The table below provides a quick reference to the HTTP methods you use to access and manage the namespace.
Summary / Permissions
Summary Deletes: Objects Versions (purge) Empty directories Custom metadata
Method
DELETE
Elements
For all operations: hcp-ns-auth cookie Object or directory URL
Permissions For all operations, delete For purge, purge For privileged operations, privileged
privileged=true reason=reason-text
To purge all versions, this as a URL query parameter or form-encoded data:
purge=true
To delete custom metadata, this as a URL query parameter:
type=custom-metadata
A2
HTTP methods
(Continued)
Method
GET of objects and objectrelated information
Summary / Permissions
Summary Retrieves: Objects Versions Version lists Directory listings Custom metadata
Elements
For all operations: hcp-ns-auth cookie Object, directory, or namespace information URL
Permission Read
To receive data in gzip format, an Accept-Encoding header Response headers that contains gzip or specifies * X-HCP-Time X-HCP-SoftwareVersion To retrieve a specific version, this X-HCP-ErrorMessage URL query parameter: (if an error occurred)
version=version
version=list
To list deleted objects or versions, this URL query parameter:
If response is in gzipcompressed format: Content-Encoding X-HCP-ContentLength Objects and versions: X-HCP-CustomMetadata X-HCP-DPL X-HCP-Hash X-HCP-Index X-HCP-IngestTime X-HCP-Replicated X-HCP-Retention X-HCP-RetentionClass X-HCP-RetentionHold X-HCP-Retention String X-HCP-Shred X-HCP-Size X-HCP-Type X-HCP-VersionID Objects and custom metadata: Preceding headers plus: X-HCP-Custom MetadataContent Type X-HCP-Custom MetadataFirst X-HCP-DataContent Type Directory listings: X-HCP-Type
deleted=true
For objects and versions, optionally, an HTTP Range header specifying any of these zeroindexed byte ranges: start-positionend-position start-position offset-from-end
To retrieve an object or version data and custom metadata as a single unit: This URL query parameter:
type=whole-object
To control the order of the returned information, an XHCP-CustomMetadataFirst header with a true or false (the default) value
type=custom-metadata
A3
HTTP methods
(Continued)
Method
GET of namespace information
Summary / Permissions
Summary Retrieves: Namespace list Namespace statistics Namespace and user permissions Namespace retention classes
Elements
For all operations: hcp-ns-auth cookie Object, directory, or namespace information URL
http://namespace.tenant. system.domain/proc
To get statistics:
Permission Search
A4
HTTP methods
(Continued)
Method
HEAD
Summary / Permissions
Summary Checks existence of: Objects Versions Directories Custom metadata
Elements
For all operations: hcp-ns-auth cookie To check a specific version, this URL query parameter:
version=version
To check custom metadata, this URL query parameter:
type=custom-metadata
Permission Read
A5
HTTP methods
(Continued)
Method
POST to /rest
Summary / Permissions
Summary Changes one or more of these metadata values: Hold status Index setting Retention setting Shred setting
Elements
For all operations: hcp-ns-auth cookie Object or directory URL
hold=true|false
To change the index setting, this URL query parameter:
index=true|false
To change the retention setting, this URL query parameter:
retention=retentionexpression
To specify that the object will be shredded, this URL query parameter:
shred=true
A6
HTTP methods
(Continued)
Method
POST to /query
Summary / Permissions
Summary Retrieves a set of operation records with metadata for objects that match query criteria Permission Search
Elements
hcp-ns-auth cookie To send a gzip-compressed query, a Content-Encoding header with a value of gzip and a chunked transfer encoding To receive a gzip-compressed response, an Accept-Encoding header that contains gzip or specifies * Content-Type header with one of these values: application/xml application/json
URL or Host header with a hostname starting with one of these: If the hcp-ns-auth cookie specifies a data access account, a tenant name If the hcp-ns-auth cookie specifies a system-level user account, admin
URL termination of /query XML or JSON request body specifying the query criteria
A7
HTTP methods
(Continued)
Method
PUT
Summary / Permissions
Summary Adds: Objects Versions Empty directories Custom metadata
Elements
For all operations: hcp-ns-auth cookie Object or directory URL To send a gzip-compressed query, a Content-Encoding header with a value of gzip and a chunked transfer encoding
For all operations except directory creation: A body containing the data To send a gzip-compressed query, a Content-Encoding header with a value of gzip and a chunked transfer encoding
type=directory
To store custom metadata, this URL query parameter:
type=custom-metadata
To store object or version data and custom metadata together: This URL query parameter:
type=whole-object
An X-HCP-Size header with the object size in bytes
To set hold, index, retention, or shred metadata when adding an object or version, see the query parameters described for the POST to /rest request above.
A8
Meaning
Methods
DELETE GET HEAD POST
Description
HCP successfully performed a request that does any of: Retrieves an object, a directory, metadata, or custom metadata Sets a metadata value Performs a metadata query
HCP successfully added an object, a version, a directory, or custom metadata to the namespace or replaced the custom metadata for an object. One of: For custom metadata, the specified object does not have any custom metadata. For a version, the requested version is a historic deleted version.
204
No Content
206 302 Partial Content Found GET with a Range header All
HCP successfully retrieved the requested byte range. The hcp-ns-auth cookie is missing or specifies invalid credentials. This type of error response also returns a Location header with a URL: For requests that specify a namespace, the returned Location header URL specifies the login page used by the Namespace Browser, in this form: http[s]:namespace.tenant.system.domain/login/ login.jsp For metadata query API requests, the Location header URL has the preceding format without the namespace name.
To fix the error, do not use the returned URL. Instead, use correct credentials in the hcp-ns-auth cookie.
A9
Code
400
Meaning
Bad Request All
Methods
Description
The request was not valid. These are some, but not all, of the possible reasons: The URL in the request is not well-formed. The namespace does not exist. The request contains an unsupported parameter or an invalid value for a parameter. A GET request has both a type=whole-object query parameter and a Range request header. A PUT or POST request has a Content-Encoding header that specifies gzip, but the content is not in gzip-compressed format. A PUT request has a type=whole-object query parameter but does not have an X-HCP-Size header or the X-HCP-Size header value is greater than the content length. HCP has custom metadata XML checking enabled, and a PUT request includes custom metadata that is not well formed XML. The request is trying to change the retention setting from a retention class to an explicit setting, such as a datetime value. The request is trying to change the retention setting and the hold setting at the same time. The request is trying to change the retention setting for an object on hold. The request is trying to change the shred setting from true to false. For a metadata query, the request body XML or JSON is not valid.
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header.
A10
Code
401
Meaning
Unauthorized All
Methods
Description
The user identified by the hcp-ns-auth cookie does not have the permission necessary to perform the requested operation. For more information on the required data access permissions, see Data access permissions on page 1-8. The requested operation was not allowed. These are some, but not all, of the possible reasons: The authenticated user doesnt have permission to perform the requested operation. The access method (HTTP or HTTPS) is disabled. For a DELETE request to delete an object, the object is under retention. For a DELETE request to delete a directory, the directory is not empty. For a PUT request to add a version of an existing object, the object is under retention. For a PUT request to add, replace, or delete custom metadata for an object that is under retention, the namespace does not allow the operation for objects under retention.
403
Forbidden
All
If more information about the error is available, the response headers include the HCP-specific X-HCPErrorMessage header. 404 Not Found DELETE GET HEAD POST PUT of custom metadata One of: For all methods, HCP could not find the specified object, version, or directory. For a GET request for a version, the specified version is the current version of a deleted object.
A11
Code
406
Meaning
Not Acceptable
Methods
GET POST One of:
Description
The request has an Accept-Encoding header that does not include gzip or specify *. A metadata query POST request does not have an Accept header, or the Accept header does not specify application/xml or application/json. A metadata query POST request has an Accept header that does not specify application/xml or application/json.
409
Conflict
DELETE PUT
One of: For a DELETE request, HCP could not delete or purge the specified object or delete the directory or custom metadata because it is currently being written to the namespace. For a PUT request, HCP could not add the object or directory to the namespace because it already exists and versioning is not enabled. For a PUT request to store a version of an existing object, HCP could not add a new version of the object because another version is currently being added.
413
PUT of an object
One of: Not enough space is available to store the object. Try the request again after objects or versions are deleted from the namespace or the namespace capacity is increased. The request is trying to save an object that is larger than two TB. HCP cannot store objects with sizes greater than two TB. The request is trying to store custom metadata that is larger than one GB. HCP cannot store custom metadata that is larger than one GB.
414
All
The portion of the URL following rest is longer than 4,095 bytes.
A12
Code
415
Meaning
Unsupported Media Type
Methods
POST PUT One of:
Description
The request has a Content-Encoding header with a value other than gzip. A metadata query POST request does not have a Content-Type header. A metadata query POST request Content-Type header does not specify application/xml or application/json.
416
One of: The specified start position is greater than the size of the requested data. The size of the specified range is zero.
An internal error occurred. If this happens repeatedly, contact your namespace administrator. One of: HCP is temporarily unable to handle the request, probably due to system overload or maintenance. HCP may also have tried to read the object from a replica that was not currently available. Try the request again in a little while. The request includes a Content-Length header with a value larger than the length of the message body.
Methods
GET with compressed transmission
Description
The length of the data before HCP compressed it.
X-HCP-Custom-Metadata HEAD and GET for objects, object versions, and custom metadata
A true or false value indicating whether the object has custom metadata.
A13
Header
X-HCP-CustomMetadata ContentType X-HCP-CustomMetadata First
Methods
GET for both object or version data and custom metadata GET for both object or version data and custom metadata
Description
The custom metadata type, always text/xml.
One of: true if the custom metadata precedes the object data false if the object data precedes the custom metadata
X-HCP-CustomMetadata Hash PUT that stores object data and custom metadata together.
The cryptographic hash algorithm HCP uses and the cryptographic hash value of the stored custom metadata, in this format: X-HCP-CustomMetadataHash: hash-algorithm hash-value You can use the returned hash value to verify that the stored custom metadata is the same as the metadata you sent. To do so, compare this value with a hash value that you generate from the original custom metadata.
X-HCP-DataContentType
GET for both object or version data and custom metadata HEAD and GET for objects, object versions, and custom metadata All
The Internet media type of the object, such as text/plain or image/jpg. The data protection level of the object or version.
X-HCP-DPL
X-HCP-ErrorMessage
Detailed information about the cause of an error. This header is returned only if HCP has specific information about the error cause. The cryptographic hash algorithm the namespace uses, along with a cryptographic hash value: X-HCP-Hash: hash-algorithm hash-value For PUT requests that store only custom metadata, hash-value is the hash value of the custom metadata. For all other requests, it is the hash value of the object. You can use the returned hash value to verify that the stored data is the same as the data you sent. To do so, compare this value with a hash value that you generate from the original data.
X-HCP-Hash
HEAD, GET, and PUT for objects, object versions, and custom metadata
A14
Header
X-HCP-Index
Methods
HEAD and GET for objects, object versions, and custom metadata HEAD and GET for objects, object versions, and custom metadata HEAD and GET for objects and object versions. HEAD and GET for objects, object versions, and custom metadata HEAD and GET for objects, object versions, and custom metadata HEAD and GET for objects, object versions, and custom metadata HEAD and GET for objects, object versions, and custom metadata
Description
A true or false value indicating whether the object is marked for indexing.
X-HCP-IngestTime
The time when HCP stored the object, in seconds since January 1, 1970, at 00:00:00 UTC.
X-HCP-Replicated
A true or false value indicating whether the object has been replicated. The value is true only if the current version of the object, its system metadata, and any custom metadata have been replicated. The end of the retention period for the object, in seconds since January 1, 1970, at 00:00:00 UTC. This value can also be 0, -1, or -2. The name of the retention class to which the object belongs. This value is an empty string if the object is not in a retention class. A true or false value indicating whether the object is on hold.
X-HCP-Retention
X-HCP-RetentionClass
X-HCP-RetentionHold
X-HCP-RetentionString
yyyy-MM-ddThh:mm:ssZ
For example, 2015-11-16T14:27:20-0500 represents the start of the 20th second into 2:27 PM, November 16, 2015, EST. The value can also be Deletion Allowed, Deletion Prohibited, or Initial Undefined. For more information on the datetime format, see Specifying a date and time on page 2-10.
X-HCP-Shred
HEAD and GET for objects, object versions, and custom metadata HEAD and GET for objects and object versions, and custom metadata
A true or false value indicating whether HCP will shred the object after it is deleted.
X-HCP-Size
The size of the object or version, in bytes, including when retrieving custom metadata.
A15
Header
X-HCP-SoftwareVersion
Methods
HEAD and GET for objects, object versions, and directories All except POST
Description
The version number of the HCP software.
X-HCP-Time
The time at which HCP sent the response to the request, in seconds since January 1, 1970, at 00:00:00 UTC. The entity type. The value is either directory or object.
X-HCP-Type
HEAD and GET for objects, object versions, and directories HEAD, GET, and PUT for objects and object versions
X-HCP-VersionId
Metadata query API requests return information as XML or JSON. For the structure of the returned information, see:
Listing object versions on page 4-16 Listing directory contents on page 5-5 Response format on page 8-14 for the XML and JSON returned in
response to a metadata query API request
Listing accessible namespaces on page 9-2 Listing retention classes on page 9-5 Listing namespace and user permissions on page 9-8 Listing namespace statistics on page 9-12
A16
B
Java classes for examples
This appendix contains the implementation of these Java classes that are used in examples in this book:
B1
GZIPCompressedInputStream class
GZIPCompressedInputStream class
package com.hds.hcp.examples; import import import import import java.io.IOException; java.io.InputStream; java.util.zip.CRC32; java.util.zip.Deflater; java.util.zip.DeflaterInputStream;
public class GZIPCompressedInputStream extends DeflaterInputStream { /** * This static class is used to hijack the InputStream * read(b, off, len) function to be able to compute the CRC32 * checksum of the content as it is read. */ static private class CRCWrappedInputStream extends InputStream { private InputStream inputStream; /** * CRC-32 of uncompressed data. */ protected CRC32 crc = new CRC32(); /** * Construct the object with the InputStream provided. * @param pInputStream - Any class derived from InputStream class. */ public CRCWrappedInputStream(InputStream pInputStream) { inputStream = pInputStream; crc.reset(); // Reset the CRC value. } /** * This group of methods are the InputStream equivalent methods * that just call the method on the InputStream provided during * construction. */ public int available() throws IOException { return inputStream.available(); }; public void close() throws IOException { inputStream.close(); }; public void mark(int readlimit) { inputStream.mark(readlimit); }; public boolean markSupported() { return inputStream.markSupported(); }; public int read() throws IOException { return inputStream.read(); }; public int read(byte[] b) throws IOException { return inputStream.read(b); }; public void reset() throws IOException { inputStream.reset(); }; public long skip(long n) throws IOException { return inputStream.skip(n); };
B2
GZIPCompressedInputStream class
/* * This function intercepts all read requests in order to * calculate the CRC value that is stored in this object. */ public int read(byte b[], int off, int len) throws IOException { // Do the actual read from the input stream. int retval = inputStream.read(b, off, len); // If we successfully read something, compute the CRC value // of it. if (0 <= retval) { crc.update(b, off, retval); } // All done with the intercept. return retval; }; Return the value.
/* * Function to retrieve the CRC value computed thus far while the * stream was processed. */ public long getCRCValue() { return crc.getValue(); }; } // End class CRCWrappedInputStream. /** * Creates a new input stream with the default buffer size of * 512 bytes. * @param pInputStream - Input Stream to read content for * compression. * @throws IOException if an I/O error has occurred. */ public GZIPCompressedInputStream(InputStream pInputStream) throws IOException { this(pInputStream, 512); } /** * Creates a new input stream with the specified buffer size. * @param pInputStream - Input Stream to read content for * compression. * @param size the output buffer size * @exception IOException If an I/O error has occurred. */ public GZIPCompressedInputStream(InputStream pInputStream, int size) throws IOException { super(new CRCWrappedInputStream(pInputStream), new Deflater(Deflater.DEFAULT_COMPRESSION, true), size); mCRCInputStream = (CRCWrappedInputStream) super.in; }
B3
GZIPCompressedInputStream class
// Indicator for if EOF has been reached for this stream. private boolean mReachedEOF = false; // Holder for the hi-jacked InputStream that computes the // CRC-32 value. private CRCWrappedInputStream mCRCInputStream; /* * GZIP Header structure and positional variable. */ private final static int GZIP_MAGIC = 0x8b1f; private final static byte[] mHeader = { (byte) GZIP_MAGIC, // Magic number (short) (byte)(GZIP_MAGIC >> 8), // Magic number (short) Deflater.DEFLATED, // Compression method (CM) 0, // Flags (FLG) 0, // Modification time MTIME (int) 0, // Modification time MTIME (int) 0, // Modification time MTIME (int) 0, // Modification time MTIME (int) 0, // Extra flags (XFLG) 0 // Operating system (OS) FYI. UNIX/Linux OS is 3 }; private int mHeaderPos = 0; // Keeps track of how much of the // header has already been read.
/* * GZIP trailer structure and positional indicator. * * Trailer consists of 2 integers: CRC-32 value and original file * size. */ private final static int TRAILER_SIZE = 8; private byte mTrailer[] = null; private int mTrailerPos = 0; /*** * Overridden functions against the DeflatorInputStream */ /* * Function to indicate whether there is any content available to * read. It is overridden because there are the GZIP header and * trailer to think about. */ public int available() throws IOException { return (mReachedEOF ? 0 : 1); }
B4
GZIPCompressedInputStream class
/* * This read function is the meat of the class. It handles passing * back the GZIP header, GZIP content, and GZIP trailer in that * order to the caller. */ public int read(byte[] outBuffer, int offset, int maxLength) throws IOException, IndexOutOfBoundsException { int retval = 0; // Contains the number of bytes read into // outBuffer and will be the return value of // the function. int bIndex = offset; // Used as current index into outBuffer. int dataBytesCount = 0; // Used to indicate how many data bytes // are in the outBuffer array. // Make sure we have a buffer. if (null == outBuffer) { throw new NullPointerException("Null buffer for read"); } // Make sure offset is valid. if (0 > offset || offset >= outBuffer.length) { throw new IndexOutOfBoundsException( "Invalid offset parameter value passed into function"); } // Make sure the maxLength is valid. if (0 > maxLength || outBuffer.length - offset < maxLength) throw new IndexOutOfBoundsException( "Invalid maxLength parameter value passed into function"); // Asked for nothing; you get nothing. if (0 == maxLength) return retval; /** * Put any GZIP header in the buffer if we haven't already returned * it from previous calls. */ if (mHeaderPos < mHeader.length) { // Get how much will fit. retval = Math.min(mHeader.length - mHeaderPos, maxLength); // Put it there. for (int i = retval; i > 0; i--) { outBuffer[bIndex++] = mHeader[mHeaderPos++]; }
B5
GZIPCompressedInputStream class
// Return the number of bytes copied if we exhausted the // maxLength specified. // NOTE: Should never be >, but... if (retval >= maxLength) { return retval; } } /** * At this point, the header has all been read or put into the * buffer. * * Time to add some GZIP compressed data, if there is some still * left. */ if (0 != super.available()) { // Get some data bytes from the DeflaterInputStream. dataBytesCount = super.read(outBuffer, offset+retval, maxLength-retval); // As long as we didn't get EOF (-1) update the buffer index and // retval. if (0 <= dataBytesCount) { bIndex += dataBytesCount; retval += dataBytesCount; } // Return the number of bytes copied during this call, if we // exhausted the maxLength requested. // NOTE: Should never be >, but... if (retval == maxLength) { return retval; } // If we got here, we should have read all that can be read from // the input stream, so make sure the input stream is at EOF just // in case someone tries to read it outside this class. byte[] junk = new byte[1]; if (-1 != super.read(junk, 0, junk.length)) { // Should never happen. But you know how that goes. throw new IOException( "Unexpected content read from input stream when EOF expected"); } } /** * Got this far; time to write out the GZIP trailer. */ // Have we already set up the GZIP trailer in a previous // invocation? if (null == mTrailer) {
B6
GZIPCompressedInputStream class
// Time to prepare the trailer. mTrailer = new byte[TRAILER_SIZE]; // Put the content in it. writeTrailer(mTrailer, 0); } // If there are still GZIP trailer bytes to be returned to the // caller, do as much as will fit in the outBuffer. if (mTrailerPos < mTrailer.length) { // Get the number of bytes that will fit in the outBuffer. int trailerSize = Math.min(mTrailer.length - mTrailerPos, maxLength - bIndex); // Move them in. for (int i = trailerSize; i > 0; i--) { outBuffer[bIndex++] = mTrailer[mTrailerPos++]; } // Return the total amount of bytes written during this call. return retval + trailerSize; } /** * If we got this far, we have already been asked to read * all content that is available. * * So we are at EOF. */ mReachedEOF = true; return -1; } /*** * Helper functions to construct the trailer. */ /* * Writes GZIP member trailer to a byte array, starting at a given * offset. */ private void writeTrailer(byte[] buf, int offset) throws IOException { writeInt((int)mCRCInputStream.getCRCValue(), buf, offset); // CRC-32 of uncompr. data writeInt(def.getTotalIn(), buf, offset + 4); // Number of uncompr. bytes }
B7
WholeIOInputStream class
/* * Writes integer in Intel byte order to a byte array, starting at * a given offset. */ private void writeInt(int i, byte[] buf, int offset) throws IOException { writeShort(i & 0xffff, buf, offset); writeShort((i >> 16) & 0xffff, buf, offset + 2); } /* * Writes short integer in Intel byte order to a byte array, * starting at a given offset */ private void writeShort(int s, byte[] buf, int offset) throws IOException { buf[offset] = (byte)(s & 0xff); buf[offset + 1] = (byte)((s >> 8) & 0xff); } }
WholeIOInputStream class
package com.hds.hcp.examples; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; /** * This class defines an InputStream that is composed of both a data * file and a custom metadata file. * * The class is used to provide a single stream of data and custom * metadata to be transmitted over HTTP for type=whole-object PUT * operations. */ public class WholeIOInputStream extends InputStream { /* * Constructor. Passed in an InputStream for the data file and the * custom metadata file. */ WholeIOInputStream( InputStream inDataFile, InputStream inCustomMetadataFile) { mDataFile = inDataFile; mCustomMetadataFile = inCustomMetadataFile; bFinishedDataFile = false; }
B8
WholeIOOutputStream class
// Indicates when all data file // content has been read. private InputStream mDataFile, mCustomMetadataFile;
/* * Base InputStream read function that reads from either the data * file or custom metadata, depending on how much has been read so * far. */ public int read() throws IOException { int retval = 0; // Assume nothing read. // Do we still need to read from the data file? if (! bFinishedDataFile ) { // Read from the data file. retval = mDataFile.read(); // If reached the end of the stream, indicate it is time to read // from the custom metadata file. if (-1 == retval) { bFinishedDataFile = true; } } // This should not be coded as an "else" because it may need to be // run after data file has reached EOF. if ( bFinishedDataFile ) { // Read from the custom metadata file. retval = mCustomMetadataFile.read(); } return retval; } }
WholeIOOutputStream class
package com.hds.hcp.examples; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; /** * This class defines an OutputStream that will create both the data * file and the custom metadata file for an object. The copy() method * is used to read an InputStream and create the two output files based * on the indicated size of the data file portion of the stream. * * The class is used to split and create content retrieved over HTTP as * a single stream for type=whole-object GET operations. */
B9
WholeIOOutputStream class
public class WholeIOOutputStream extends OutputStream { // Constructor. Passed output streams for the data file and the // custom metadata file. Allows specification of whether the custom // metadata comes before the data. WholeIOOutputStream(OutputStream inDataFile, OutputStream inCustomMetadataFile, Boolean inCustomMetadataFirst) { bCustomMetadataFirst = inCustomMetadataFirst; // Set up first and second file Output Streams based on whether // custom metadata is first in the stream. if (bCustomMetadataFirst) { mFirstFile = inCustomMetadataFile; mSecondFile = inDataFile; } else { mFirstFile = inDataFile; mSecondFile = inCustomMetadataFile; } bFinishedFirstPart = false; } // Member variables. private Boolean bFinishedFirstPart; private Boolean bCustomMetadataFirst; private OutputStream mFirstFile, mSecondFile; /** * This routine copies content in an InputStream and to this * output stream. The first inDataSize number of bytes are written * to the data file output stream. * * @param inStream - InputStream to copy content from. * @param inFirstPartSize - number of bytes of inStream that should * be written to the first output stream. * @throws IOException */ public void copy(InputStream inStream, Integer inFirstPartSize) throws IOException { int streamPos = 0; int readValue = 0; // Keep reading bytes until EOF has been reached. while (-1 != (readValue = inStream.read())) { // Have we read all the bytes for the data file? if (streamPos == inFirstPartSize) { // Yes. bFinishedFirstPart = true; }
B10
WholeIOOutputStream class
// Write the bytes read. write(readValue); streamPos++; } } /** * This is the core write function for the InputStream implementation. * It either writes to the data file stream or the custom metadata * stream. */ public void write(int b) throws IOException { // Write to first or second file depending on where we are in the // stream. if (! bFinishedFirstPart ) { mFirstFile.write(b); } else { mSecondFile.write(b); } } /** * flush() method to flush all files involved. */ public void flush() throws IOException { mFirstFile.flush(); mSecondFile.flush(); super.flush(); } /** * close() method to first close the data file and custom metadata * files. Then close itself. */ public void close() throws IOException { mFirstFile.close(); mSecondFile.close(); super.close(); } }
B11
WholeIOOutputStream class
B12
Glossary
A
access protocol
See namespace access protocol.
authentication
See namespace access authentication
C
compliance mode
The retention mode in which objects under retention cannot be deleted through any mechanism. This is the more restrictive retention mode.
custom metadata
One or more user-defined properties that provide descriptive information about an object. Custom metadata, which is normally specified as XML, enables future users and applications to understand and repurpose object content.
Glossary1
Using a Namespace
D
data access account
A set of credentials that give a user or application access to one or more HCP namespaces. For each namespace, the account specifies which operations the user or application can perform.
E
effective permissions
For a namespace, the permissions that are included in all three of the system-level, tenant-level, and namespace-level permission masks. For a user or application accessing a given namespace, the permissions that are included in both the data access account in use and the effective permission mask for the namespace.
enterprise mode
The retention mode in which these operations are allowed:
Privileged delete Changing the retention class of an object to one with a shorter duration Reducing retention class duration Deleting retention classes
expired object
An object that is no longer under retention.
Glossary2
Using a Namespace
F
fixed-content data
A digital asset ingested into HCP and preserved in its original form as the core part of an object. Once stored, fixed-content data cannot be modified.
H
hash value
See cryptographic hash value.
HCP
See Hitachi Content Platform (HCP).
HCP namespace
A namespace that requires user authentication for data access. An HCP system can have multiple HCP namespaces.
HDDS
See Hitachi Data Discovery Suite (HDDS).
Glossary3
Using a Namespace
hold
A condition that prevents an object from being deleted by any means and from having its metadata modified, regardless of its retention setting, until it is explicitly released.
HTTP
Hypertext Transfer Protocol. The protocol HCP uses to provide access to the contents of a namespace.
HTTPS
HTTP with SSL security. See HTTP and SSL.
I
index
See search index.
index setting
The property that specifies whether an object should be indexed.
M
metadata
System-generated and user-supplied information about an object. Metadata is stored as an integral part of the object it describes, thereby making the object self-describing.
Glossary4
Using a Namespace
permission
N
namespace
A logical partition of the objects stored in an HCP system. A namespace consists of a grouping of objects such that the objects in one namespace are not visible in any other namespace. Namespaces are configured independently of each other and, therefore, can have different properties.
O
object
An exact digital representation of data as it existed before it was ingested into HCP, together with the system and custom metadata that describes that data. An object is handled as a single unit by all transactions and internal processes, including shredding, indexing, versioning, and replication.
operation record
A record of a create, delete, or purge operation. The record identifies the object involved, the type of operation, and the time at which the operation occurred and also contains system metadata for the object HCP updates the applicable creation record when object metadata changes.
P
permission
One of these:
In a data access permission mask, the condition of allowing a specific type of operation to be performed in a namespace.
Glossary5
Using a Namespace
permission mask
In a data access account, the granted ability to perform a specific type of operation in a given namespace.
permission mask
See data access permission mask.
policy
One or more settings that influence how transactions and internal processes work on objects. Such a setting can be a property of an object, such as retention, or a property of a namespace, such as versioning.
POSIX
Portable Operating System Interface for UNIX. A set of standards that define an application programming interface (API) for software designed to run under heterogeneous operating systems.
privileged delete
A delete operation that works on objects regardless of their retention settings, except for objects on hold. This operation is available only to users and applications with explicit permission to perform it.
privileged purge
The operation that deletes all versions of an object regardless of whether the object is under retention, except if the object is on hold. This operation is available only to users and applications with explicit permission to perform it. Privileged purge operations work only in namespaces in enterprise mode.
protocol
See namespace access protocol.
purge
The operation that deletes all versions of an object.
Q
query
A request submitted to HCP to return operation records
Glossary6
Using a Namespace
retention period
query API
See metadata query API.
R
replication
The process of keeping selected tenants and namespaces in two HCP systems in sync with each other. This entails copying object creations, deletions, and metadata changes from one system to the other.
repository
The aggregate of the namespaces defined for an HCP system.
REST
Representational State Transfer. A software architectural style that defines a set of rules (called constraints) for client/server communication. In a REST architecture:
Resources (where a resource can be any coherent and meaningful concept) must be uniquely addressable. Representations of resources (for example, as XML) are transferred between clients and servers. Each representation communicates the current or intended state of a resource. Clients communicate with servers through a uniform interface (that is, a set of methods that resources respond to) such as HTTP.
retention class
A named retention setting. The value of a retention class can be a duration, Deletion Allowed, Deletion Prohibited, or Initial Unspecified.
retention hold
See hold.
retention mode
A namespace property that affects which operations are allowed on objects under retention. A namespace can be in either of two retention modes: compliance or enterprise.
retention period
The period of time during which an object cannot be deleted (except by means of a privileged delete).
Glossary7
Using a Namespace
retention setting
retention setting
The property that determines the retention period for an object.
S
Search Console
The web application that provides interactive access to the search functionality of the active search system.
search facility
An interface between the search functionality provided by a system such as HDDS or HCP and the HCP Search Console. Only one search facility can be enabled at any given time.
search index
An index of the metadata and key terms in namespace objects. The active search system builds, maintains, and stores this index.
shred setting
The property that determines whether a data object will be shredded or simply removed when its deleted from HCP.
shredding
The process of deleting a data object and overwriting the locations where its bytes were stored in such a way that none of its data or metadata can be reconstructed. Also called secure deletion.
SSL
Secure Sockets Layer. A key-based Internet protocol for transmitting documents through an encrypted link.
system metadata
System-managed properties that describe the content of an object. System metadata includes policies, such as retention and data protection level, that influence how transactions and internal processes affect the object.
Glossary8
Using a Namespace
WORM
T
tenant
An administrative entity created for the purpose of owning and managing namespaces and data access accounts. Tenants typically correspond to customers, business units, or individuals.
U
Unix
Any UNIX-like operating system (such as UNIX itself or Linux).
user account
A set of credentials that gives an HCP system administrator access to the metadata query API.
V
versioning
A feature that allows the creation and management of multiple versions of an object.
W
WORM
Write once, read many. A data storage property that protects the stored data from being modified or overwritten.
Glossary9
Using a Namespace
WORM
Glossary10
Using a Namespace
Index
Numbers
0 (retention setting) 2-5, 2-9 -1 (retention setting) 2-5, 2-9 -2 (retention setting) 2-5, 2-9 checking custom metadata existence 7-67-8 directory existence 5-35-5 object existence 4-124-15 chunking data 11-4 clear-text passwords 3-8 client tools 1-6 compressed data See gzip compressing data for transmission 1-7 connections handling failed 11-7 persistent 11-6 Content-Encoding HTTP header 1-7 Content-Type HTTP header 8-7 create operations, records for 8-3 creating See also storing directories 5-25-3 directories in the Namespace Browser 10-1010-11 cryptographic hash values 2-2 cURL 3-1 custom metadata about 1-2, 2-172-18 allowed operations 2-17 checking existence 7-67-8 deleting 7-117-13 with objects under retention 2-172-18 receiving in compressed format 7-9 retrieving 7-87-11 retrieving together with object data 4-22 4-23, 4-364-38 sending in compressed format 7-2 storing 7-27-5 storing in compressed format 1-7 storing together with object data 4-3, 4-9 4-11 transmitting in compressed format 1-7
A
Accept HTTP header 8-7 Accept-Encoding HTTP header 1-7 access, authenticating 3-83-9 active search system 1-5 adding See storing; creating assigning objects to retention classes 2-10 authenticating namespace access 3-83-9 authorization See hcp-ns-auth cookie
B
Base64 username encoding 3-83-9 browsing the namespace 10-110-14 byte range, retrieving about 4-224-24 example 4-304-32
C
change time about 2-3 in metadata query request 8-118-12 in metadata query response 8-18, 8-19 selector for metadata query API 8-2 changing index settings 2-17 passwords 10-310-4 retention settings 6-5 shred settings 2-13 system metadata 6-56-7
Index1
Using a Namespace
data access
D
data access 1-31-6 data access account 1-8 data access permission mask 1-8 data access permissions about 1-8 listing 9-89-11 privileged 2-52-6 viewing in the Namespace Browser 10-13 XML listing format 9-89-9 data chunking with HTTP 11-4 data compression See gzip data protection level 2-3 date and time, specifying for retention setting 2-9, 2-102-11 delete operations, records for 8-3 delete permission 1-8 deleted objects and directories listing 5-6 listing in the Namespace Browser 10-7 listing versions 4-16, 4-184-19 listing versions in the Namespace Browser 10-9 deleted versions 2-16, 4-38 deleting all versions of an object 4-434-46 custom metadata 7-117-13 directories 5-105-12 objects 2-16, 4-384-42 objects automatically 2-4 objects in the Namespace Browser 10-10 objects under repair 11-6 objects under retention 4-384-39 Deletion Allowed 2-9 Deletion Prohibited 2-9 directories checking existence of 5-35-5 creating 5-25-3 creating in the Namespace Browser 10-5, 10-1010-11 deleting 5-105-12 listing contents 5-55-10 listing contents in the Namespace Browser 10-510-7 listing deleted 5-6 listing deleted in the Namespace Browser 10-7, 10-9 listing deleted versions 4-184-19 selector for metadata query API 8-2 specifying in metadata API query requests 8-12 structuring 11-211-3
version IDs for deleted 2-14 XML listing format 5-65-8 DNS names, namespace access by 11-2
E
effective permissions, user and namespace 10-13 error codes See return codes examples changing object metadata 6-66-7 checking custom metadata existence 7-7 7-8 checking directory existence 5-45-5 checking object existence 4-144-15 creating directories 5-25-3 deleting custom metadata 7-13 deleting directories 5-115-12 deleting objects 4-404-42 listing accessible namespaces 9-39-4 listing directory contents 5-85-10 listing namespace statistics 9-139-14 listing object versions 4-194-21 listing permissions 9-99-11 listing retention classes 9-69-7 metadata query API 8-218-30 paged queries 8-268-30 privileged delete 4-414-42 privileged purge 4-46 purging objects 4-454-46 querying namespaces 8-218-30 retrieving custom metadata 7-107-11 retrieving historic versions 4-324-33 retrieving metadata for changed objects 8-258-26 retrieving metadata for indexable objects 8-228-24 retrieving object data and custom metadata together 4-364-38 retrieving objects and versions 4-294-38 retrieving objects in compressed format 4-334-36 retrieving partial objects 4-304-32 sending object data in compressed format 4-74-9 specifying metadata on object creation 6-4 storing custom metadata 7-5 storing objects 4-64-11
F
failed write operations 11-411-5 fixed-content data 1-2
Index2
Using a Namespace
HTTP GET
G
gzip compressing custom metadata data for retrieval 7-9 compressing custom metadata data for submission 7-2 compressing data for transmission 1-7 compressing metadata queries and responses 8-7 compressing object data and custom metadata for retrieval 4-23 compressing object data and custom metadata for submission 4-3 compressing object data for retrieval 4-22, 4-334-36 compressing object data for submission 4-2, 4-74-9 encoding data for operations 1-9 storing compressed data in HCP 1-7
H
hash values See cryptographic hash values HCP 1-1 HCP client tools 1-6 HCP Data Migrator 1-6 HCP search facility 1-5 hcp-ns-auth cookie format 3-8 generating 3-83-9 metadata query API 8-7 HDDS search facility 1-5 historic versions See versions Hitachi Content Platform 1-1 hold HTTP metadata parameter 6-2, 6-5 hold settings, for versions 2-14 holding objects 2-6 hostnames enabling use in URLs 3-43-5 namespace access by 11-2 hosts file 3-43-5 HTTP See also individual HTTP methods about 1-31-4 checking existence of directories 5-35-5 checking existence of objects and versions 4-124-15 compliance level 3-1 connection failure handling 11-7 creating directories 5-25-3 data chunking 11-4
deleting directories 5-105-12 deleting objects 4-384-42 failed write operations 11-411-5 HCP-specific response header summary A-13A-16 listing accessible namespaces 9-29-4 listing directory contents 5-55-10 listing namespace statistics 9-129-14 listing object versions 4-164-21 listing permissions 9-89-11 listing retention classes 9-59-7 method summary A-2A-8 namespace access by IP address 11-2 purging objects 4-434-46 retrieving objects and versions 4-224-38 return code summary A-9A-13 storing objects 4-24-11 transferring data in compressed format 1-7 URLs for metadata query API 8-48-6 URLs for namespace access 3-23-8 using self-signed server certificates 8-6 HTTP DELETE See also HTTP deleting custom metadata 7-117-13 deleting directories 5-105-12 deleting objects 4-384-42 examples 4-404-42, 4-454-46, 5-11 5-12, 7-13 method summary A-2 purging objects 4-434-46 HTTP GET See also HTTP byte-range requests 4-224-24 connection failure 11-7 examples 4-194-21, 4-294-38, 5-85-10, 7-107-11, 9-39-4, 9-69-7, 9-99-11, 9-139-14 listing accessible namespaces 9-29-4 listing directory contents 5-55-10 listing namespace statistics 9-129-14 listing object versions 4-164-21 listing permissions 9-89-11 listing retention classes 9-59-7 method summary A-3, A-4 Range request header 4-234-24 response headers 4-254-28 retrieving custom metadata 7-87-11 retrieving custom metadata in compressed format 7-9 retrieving data in compressed format 1-7 retrieving object data and custom metadata in compressed format 4-23
Index3
Using a Namespace
J
JSON metadata query API request body format 8-9 metadata query API response body format 8-17 specifying as metadata query request body format 8-7 specifying as metadata query response body format 8-7
L
libcurl 3-1 listing accessible namespaces 9-29-4 deleted objects and directories 5-6 directory contents 5-55-10 namespace statistics 9-129-14 object versions 4-164-21, 10-710-8 permissions 9-89-11 retention classes 9-59-7 Location header A-9 logging into Namespace Browser 10-210-3
M
Mac OS X hosts file 3-5 MD5 password hashing 3-83-9 metadata about 2-1 changing 6-56-7 custom 1-2 HTTP parameters A-6 retrieving 4-124-15 specifying on object creation 6-26-4 system 1-2 for versions 2-14 metadata query API See also queries about 1-4, 8-2 examples 8-218-30 request body contents 8-108-13 request body formats 8-88-9 request format 8-68-13 request specific return codes 8-148-15
I
index HTTP metadata parameter 6-2, 6-5 index settings changing 2-17, 6-5 overriding default 2-17, 6-2 selector for metadata query API 8-2 specifying in metadata query request 8-13
Index4
Using a Namespace
objects
response body contents 8-178-21 response body formats 8-168-17 response format 8-148-21 specifying request body format 8-7 specifying response body format 8-7 transmitting queries and responses in compressed format 8-7 methods, HTTP A-2A-8 multithreading 11-711-8
viewing retention classes in the Namespace Browser 10-12 viewing statistics in the Namespace Browser 10-1310-14 XML listing format 9-3 XML statistics listing format 9-129-13 naming objects 2-2, 3-7 non-ASCII, nonprintable characters 3-7 non-WORM objects 11-6
N
Namespace Browser about 1-41-5, 10-2 accessing object versions 10-910-10 accessing objects 10-9 adding objects and versions 10-10 common elements 10-3 creating directories 10-1010-11 deleting objects 10-10 listing deleted objects and directories 10-7 listing directory contents 10-510-7 listing object versions 10-710-8 listing versions of deleted objects and directories 10-9 logging in 10-210-3 viewing accessible namespaces 10-1110-12 viewing permissions 10-13 viewing retention classes 10-12 viewing statistics 10-1310-14 namespaces about 1-21-3 accessible 9-2 accessible, XML listing format 9-3 accessing 1-31-6, 3-13-9 accessing by DNS name 11-2 accessing by hostname 11-2 accessing by IP address 3-53-6, 11-2 authenticating access to 3-83-9 browsing 10-110-14 listing accessible 9-29-4 listing permissions 9-89-11 listing statistics 9-129-14 operations 1-91-11 querying 8-2 replicated 1-9 retrieving information 9-19-14 specifying in metadata query request 8-13 viewing data access permissions 10-13 viewing in the Namespace Browser 10-11 10-12 viewing permissions in the Namespace Browser 10-13
O
objects See also versions about 1-2 adding in the Namespace Browser 10-10 assigning retention classes to 2-10 change time 2-3 changing system metadata 6-56-7 checking existence 4-124-15 deleted versions 4-38 deleting 2-16, 4-384-42 deleting automatically 2-4 deleting in the Namespace Browser 10-10 deleting while under retention 4-384-39 holding 2-6 indexing 2-17 ingest time 2-3 listing deleted 5-6 listing deleted in the Namespace Browser 10-7, 10-9 listing deleted versions 4-16, 4-18 listing versions 4-164-21, 10-710-8 names with non-ASCII, nonprintable characters 3-7 naming 2-2 non-WORM 11-6 open 11-511-6 overriding default metadata 6-26-4 properties of 2-1 purging 2-16, 4-434-46 replicated 1-9 retention 2-32-13 retention setting formats 2-5, 2-82-13 retrieving 4-224-38 retrieving in compressed format 4-22, 4-33 4-36 retrieving in the Namespace Browser 10-9 retrieving object data and custom metadata together 4-224-23, 4-364-38 sending in compressed format 4-2, 4-74-9 shredding 2-13 storing 1-2, 4-24-11 storing in compressed format 1-7
Index5
Using a Namespace
storing object data and custom metadata together 4-3, 4-94-11 transmitting in compressed format 1-7 uploading in the Namespace Browser 10-5 viewing in the Namespace Browser 10-9 offset, specifying for retention setting 2-10, 2-112-12 open objects 11-511-6 operation records about 8-3 with versioning disabled 8-3 with versioning enabled 8-3 operations about 8-3 privileged 2-52-6 prohibited 1-11 query response values 8-19 restrictions 1-91-10 selector for metadata query API 8-2 specifying types in metadata query requests 8-13 supported 1-101-11 types of 8-3 using compressed data 1-9 overriding default index settings 2-17, 6-2 default metadata 6-26-4 default retention settings 2-4, 6-2 default shred settings 2-13, 6-2
privileged operations 2-52-6 privileged permission 1-8, 2-52-6 privileged purge about 2-6, 4-434-46 examples 4-46 /proc 3-33-4, 9-2 prohibited operations 1-11 pruning 2-14 purge operations, records for 8-3 purge permission 1-8 purging objects 2-16, 4-434-46
Q
queries See also metadata query API caching of 8-6 compressed format 1-7 examples of 8-218-30 hcp-ns-auth cookie 8-7 paged, using 8-4 response status 8-21 selection criteria 8-2 specifying operation types 8-13 URLs for 8-48-6 /query 8-6, A-7 query API See queries; metadata query API quotation marks with URLs 3-8
P
paged queries about 8-2 example 8-268-30 lastResult request parameter 8-10, 8-11 using 8-4 partial objects, retrieving about 4-224-24 example 4-304-32 passwords changing 10-310-4 clear-text 3-8 hashing 3-83-9 percent-encoding returned object names 3-7 in URLs 3-7 permissions See data access permissions persistent connections 11-6 primary system, replication 1-9 privileged delete about 2-5, 4-384-39 examples 4-414-42
R
read from replica 1-9 read permission 1-8 replica 1-9 replicated, in metadata query API response 8-20 replication about 1-9 metadata query API status entry 8-20 X-HCP-Replicated response header A-15 response headers See HTTP response headers /rest 3-3, A-6 REST interface 1-31-4 restrictions on operations 1-91-10 retention See also retention classes; retention settings about 2-3 changing hold setting 6-5 deleting objects under 4-384-39 hold 2-6 HTTP metadata parameter 6-2, 6-5 periods 2-4 purging objects under 4-434-46 specifying hold setting 6-2
Index6
Using a Namespace
versions
retention classes See also retention; retention settings about 2-4, 2-62-7 assigning to objects 2-10 listing 9-59-7 specifying 2-13 valid values 2-62-7 XML listing format 9-5 retention settings See also retention; retention classes changing 6-5 default 2-4 HTTP response headers for 2-8 overriding default 2-4, 6-2 representation formats 2-5 specifying 2-82-13 specifying a date and time 2-9, 2-102-11 specifying a retention class 2-13 specifying an offset 2-10, 2-112-12 for versions 2-14 retrieving custom metadata 7-87-11 historic versions 4-22, 4-324-33 object data and custom metadata together 4-224-23, 4-364-38 objects and versions 4-224-38 objects in the Namespace Browser 10-9 part of an object 4-224-24, 4-304-32 versions in the Namespace Browser 10-9 10-10 return codes, HTTP A-9A-13 status codes See return codes storing See also creating custom metadata 7-27-5 object data and custom metadata together 4-3, 4-94-11 objects 1-2, 4-24-11 objects and versions using the Namespace Browser 10-10 zero-sized files 11-3 structuring directories 11-211-3 supported operations 1-101-11 system metadata 1-2
T
tenants 1-3 transactions See operations
U
Unix hosts file 3-4 URLs enabling use of hostnames 3-43-5 formats for 3-23-4 HTTP access to the namespace 3-23-8 maximum length 3-7 percent-encoding 3-7 for metadata query API 8-48-6 usage considerations 11-111-8 user permissions See data access permissions username encoding 3-83-9 users, listing permissions 9-89-11 UTF-8 encoding 2-2, 3-7
S
Search Console about 1-5 holding objects 2-6 search facilities 1-5 search permission 1-8 search, object naming considerations 2-2 secure deletion 2-13 self-signed server certificates, using 8-6 shred HTTP metadata parameter 6-2, 6-5 shred settings about 2-13 changing 2-13, 6-5 overriding default 2-13, 6-2 for versions 2-14 shredding 2-13 SSL, using self signed server certificates 8-6 statistics listing for namespaces 9-129-14 viewing in the Namespace Browser 10-13 10-14
V
version See also versions in metadata query requests 8-11 in operation records 8-19 version IDs See also versions about 2-14 for deleted versions 2-16 for new versions 2-15 using to retrieve versions and version metadata 2-15 versioning, effect on operation records 8-3 versions See also objects about 2-1, 2-14
Index7
Using a Namespace
Z
zero-sized files, storing 11-3
W
whole-object URL query parameter 4-3 Windows hosts file 3-4 WORM 1-2 write operations, failed 11-411-5 write permission 1-8
X
X-HCP-Custom-Metadata response header A-13 X-HCP-CustomMetadataContentType response header A-14
Index8
Using a Namespace
Using a Namespace
Hitachi Data Systems Corporate Headquarters 750 Central Expressway Santa Clara, California 95050-2627 U.S.A. Phone: 1 408 970 1000 www.hds.com info@hds.com Asia Pacific and Americas 750 Central Expressway Santa Clara, California 95050-2627 U.S.A. Phone: 1 408 970 1000 info@hds.com Europe Headquarters Sefton Park Stoke Poges Buckinghamshire SL2 4HD United Kingdom Phone: + 44 (0)1753 618000 info.eu@hds.com
MK-99ARC023-04