Toolkit Data File Formats

Prepared by Geoff Podger and Robert Argent

Document History
Date Feb 2004 11/6/04 Jan 2005 19 Jan 2005 Author Podger Argent Argent Nick Murray Revision 0 1 1.1 1.1.1 Description of Change Content for RRL manual Initial development of content Small changes to match recent changes; addition of .flt Minor formatting changes; each example file format now fits completely on a page for convenience Added information of bsm Updated the CDT file format descriptions and examples to disambiguate month formatting. Added Appendix A - zero padding data in Excel. Added Appendix B - Times and Dates in Data Files Corrected information on cell size in .asc Combined file format examples with file format descriptions. Added link to SHP file format. Fixed several typos.

Oct 2005

Robert Argent, Nick Murray

1.1.2

February 2006 20 Feb 2006

Robert Argent Nick Murray

1.1.3 1.1.4

Copyright notice
CRC for Catchment Hydrology, Australia 2006

Legal Information
To the extend permitted by law, the CRC for Catchment Hydrology (including its employees and consultants) accepts no responsibility and excludes all liability whatsoever in respect of any person's use or reliance on this publication or any part of it.

Acknowledgements
The input and assistance of Catchment Modelling Toolkit Product Managers and TIME developers is gratefully acknowledged.

Contents

Toolkit Data File Formats

CONTENTS 1
1.1 1.2

Introduction...........................................................1
The user guide ...................................................................................................................1 File Formats.......................................................................................................................1 1.2.1 1.2.2 1.2.3 1.2.4 Overview ......................................................................................................1 Features........................................................................................................2 Audience ......................................................................................................2 Limitations ....................................................................................................2

2
2.1

File Format Overview ............................................3

Time Series Formats ...........................................................................................................3 2.1.1 .cdt Comma Delimited Column Time Series ....................................................3 TIMEExample.cdt ...........................................................................................4 Daily time series ............................................................................................4 Six minute time series .....................................................................................4 Annual time series..........................................................................................4 Daily time series in ISO format ........................................................................4 2.1.2 2.1.3 2.1.4 2.1.5 2.1.6 2.1.7 .sdt Space Delimited Column Time Series........................................................5 TIMEExample.sdt ...........................................................................................5 .tts Tarsier Daily Time Series...........................................................................6 TIMEExample.tts ............................................................................................7 .ar1 Annual Stochastic ..................................................................................8 TIMEExample.ar1 ..........................................................................................8 .awb AWBM Daily Time Series .......................................................................9 TIMEExample.awb .........................................................................................9 .bsb SWAT BSB File Time Series ...................................................................10 TIMEExample.bsb ........................................................................................10 .bsm Bureau of Meteorology 6 Minute (was .pluv ).........................................11 Data Records ..............................................................................................11 TIMEExample.bsm .......................................................................................12 2.1.8 2.1.9 2.1.10 2.1.11 .csv Comma Separated Variable ..................................................................13 TIMEExample.csv .........................................................................................13 .dat Fixed Format Time Series Files (F.Chiew) ................................................14 TIMEExample.dat.........................................................................................14 .iqqm IQQM Time Series ............................................................................15 TIMEExample.iqqm ......................................................................................16 .mrf MFM Monthly Rainfall Files ...................................................................17 TIMEExample.mrf.........................................................................................17

Contents

2.1.12 2.1.13 2.1.14 2.2

.pcp Swat Daily Time Series ........................................................................ 18 TIMEExample.pcp ....................................................................................... 18 .silo5 QDNR SILO Time Series (was .txt )...................................................... 19 TIMEExample.silo5...................................................................................... 19 .silo8 SILO Standard Time Series (8 Variable) ............................................... 20 TIMEExample.silo8...................................................................................... 20

Raster Formats ................................................................................................................ 21 2.2.1 2.2.2 2.2.3 2.2.4 .asc Arc ASCII Grids................................................................................... 21 TIMEExample.asc ........................................................................................ 21 .mwasc Map Window Ascii Grids................................................................. 22 TIMEExample.mwasc ................................................................................... 22 .tapesg TapesG Grid files ........................................................................... 23 TIMEExample.tapesg ................................................................................... 23 .flt ESRI Binary Raster Interchange format....................................................... 23

Appendix A Zero-padding data in Excel ...................... 24 Appendix B

B.1 B.2

Times and Dates in Data Files.................. 27

Formats for CDT and CSV files ......................................................................................... 27 The ISO8601 standard for representing dates and times ..................................................... 30

ii

Introduction

1 Introduction
Many Catchment Modelling Toolkit products and the TIME software development system have the capacity to load and save data in a wide range of formats. Adhering to these formats across Toolkit applications will increase the usability and sharing of data files, and improve the software user experience.

1.1

The user guide

This document provides an overview of file formats commonly used in TIME and relevant Catchment Modelling Toolkit products, with associated file extensions. Small examples of data files for relevant text formats are also provided.

1.2

File Formats
1.2.1 Overview

Clearly specifying file formats makes it easier to prepare files, transfer files between applications, and to interpret modelling results in output files. The file formats included in this manual are:

Time Series (text) .cdt .sdt .tts .ar1 .bsb .csv .dat Comma Delimited Column Time Series Space Delimited Column Time Series Tarsier Daily Time Series Annual Stochastic Swat BSB File Time Series Comma Separated Variable Fixed Format Time Series Files (F.Chiew)

.awb AWBM Daily Time Series .bsm Bureau of Meteorology 6 Minute (was .pluv )

Toolkit Data File Formats

.iqqm IQQM Daily Time Series .mrf .pcp MFM Monthly Rainfall Files Swat Daily Time Series

.silo5 QDNR SILO Time Series Files (was .txt ) .silo8 SILO Standard Time Series (8 Variable)

Raster (text) .asc Arc Ascii Grids Map Window Ascii Grids TapesG Grid files .mwasc .tapesg .tile .tra

Tiled Raster Files Tarsier Raster Files

Others There are several GIS, Graphics and other formats recognised by TIME but not included in this manual. These are: .flt .mif .shp .jpg .tif .tiff .tile .tra .tne .tsd ESRI Binary Raster Interchange format MapInfo Interchange ESRI Shape files (SHP) GEO JPG Image GEOTif Image GEOTiff Image tile raster file Tarsier raster file (binary) Tarsier Node Link Network Files Tarsier Sites Data Files

.jpeg GEO JPEG Image

1.2.2

Features

File types, formats and extensions are provided in this manual, covering time series and raster files.

1.2.3

Audience

This document is intended for use by software developers and users, as well as those preparing documentation for Toolkit products.

1.2.4

Limitations

This document does not provide specifications for common external file formats, such as .shp. For the shp file format, see http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf.

File Format Overview

2 File Format Overview

2.1 Time Series Formats
2.1.1 .cdt Comma Delimited Column Time Series

A comma delimited column time series format file is an ASCII text file that contains regular (periodic) time series data. The file commonly has no header line, but can support a single line header specifying Date, Time series 1 if required. There are two columns of data1, with the first column being a date string, followed by the time series value. The date string may be specified in two formats: the ISO date-time string (e.g. 2000-01-24, 1998-12), or the local culture settings of the machine (e.g. 24/01/2000 for Australia), i.e. typically what Microsoft Excel will produce by default. Both can be recognised unambiguously on a given machine, but the latter will not necessarily be transferable to another machine. You are thus encouraged to use the ISO format (see Appendix B for more details. See also Appendix B, Times and Dates in Data Files. All time series data readings assume that the reading is at time 00:00 at the beginning of the day/month/year "date" field. The columns are separated by a comma. Annual data can be entered using month number as 01 (e.g. 01/1995; 01/1996).

Note

We strongly recommend using zero padding, eg use "01/1996" for January 1996, not 1/1996. See Appendix A for instructions on how to zero-pad files in Microsoft Excel 2003 (the instructions will be similar for other versions.)

1 Many cultures use a comma as a decimal separator, and semi-column for the Excel separator for the equivalent of the CSV files (Arguably this is a better choice since comma may be the thousands separator in Australia)

In other words, the default output of Excel will likely be unreadable, or worse, values not read correctly, on many non English locale machines.

Toolkit Data File Formats TIMEExample.cdt

Daily time series

01/2000,2600 02/2000,2454 03/2000,2270 04/2000,2224 05/2000,2201 06/2000,2061 07/2000,1963 08/2000,1985 09/2000,71 10/2000,24 11/2000,23 12/2000,45 01/2001,43

Six minute time series

2000-01-01,00:00,0 2000-01-01,00:06,1 2000-01-01,00:12,2 2000-01-01,00:18,3 2000-01-01,00:24,4

Annual time series

2000,0 2001,1 2002,2

Daily time series in ISO format

2000-01-01,0.9,0.9 2000-02-01,0.7,0.9 2000-03-01,0.9,0.9 2000-04-01,0.4,0.9

File Format Overview

2.1.2

.sdt Space Delimited Column Time Series

A space delimited column time series format file is an ASCII text file that contains time series data. There is no header line in the file. There are four columns of data, being: Year (yyyy) Month (mm) Day (dd), and Value separated by a space or tab character. Monthly and annual data can be entered using month and/or day number as 01. These files can be created in a spreadsheet application by saving correctly formatted columns to a text (txt) format.

TIMEExample.sdt
2000 2000 2000 2000 2000 2000 2000 2000 2000 2000 1 1 1 1 1 1 1 1 1 1 1 14.000 2 2.000 3 16.000 4 9.000 5 13.000 6 15.000 7 1.000 8 19.000 9 45.000 10 0.000

Toolkit Data File Formats

2.1.3

.tts Tarsier Daily Time Series

The Tarsier daily time series format file is an ASCII text file that contains daily time series data. The file has a 21 line header followed by daily data values. The lines of information contained within the header are:
Line 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Purpose The Tarsier version number header Reference to author of Tarsier File path and name Name of software used to create the file Date and time file was created Tarsier timer series data class (e.g. TTimeSeriesData) File version number Number of header lines (set to 1) 1. Number of daily data entries in the file 'Xlabel' is always Date/Time for time series data 'Y1Label Y1' fixed field 'Y2Label Y2' fixed field Data units Format information (e.g. 1) Grid position east in metres Grid position north in metres 'Latitude' followed by the latitude of the site in degrees 'Longitude' followed by the longitude of the site in degrees 'Elevation' followed by the elevation of the site in metres Header character

The data is organized in four columns separated by spaces. The first column is the year, followed by the Julian day. The third column is the data value. The fourth column is a data quality code '.' is ok and '-' is missing.

File Format Overview TIMEExample.tts

Tarsier modelling framework, Version 2.0. : Created by Fred Watson. : File Name : C:\data\TIME\TIMEExample.tts : Generated from TIME Framework : Date : 24/12/2004 11:59:30 PM : File class: TTimeSeriesData. FileVersion unknown HeaderLines 1 1. NominalNumEntries 10 XLabel Date/Time Y1Label Y1 Y2Label Y2 Units mm.day^-1 Format 1 Easting 0.000000 Northing 0.000000 Latitude 0.000000 Longitude 0.000000 Elevation 0.000000 * 2001 363 0 . 2001 364 0 . 2001 365 10.2 . 2002 1 0 . 2002 2 0 . 2002 3 0 . 2002 4 16 . 2002 5 16 . 2002 6 56 . 2002 7 0 .

Toolkit Data File Formats

2.1.4

.ar1 Annual Stochastic

This format contains replicates of annual time series data generated using the AR(1) stochastic method. The file has two header lines, consisting of: the site name, and the number of years per replicate, and number of replicates, separated by space The replicate data sets then follow, each with a single line header with the replicate number, followed by a single line containing all the data for that replicate, separated by spaces.

Note

This format is not the same as the AR(1) format (.gen) generated and exported by the Stochastic Climate Library.

TIMEExample.ar1
Melbourne Stochastic Annual Rainfall 20 2 1 785.307 679.228 621.467 558.083 519.046 646.454 549.185 499.563 468.864 850.161 720.839 793.515 586.834 586.203 2 600.649 693.050 605.785 858.584 555.068 629.751 518.411 811.712 489.935 905.870 491.066 540.803 912.968 671.630

450.351 691.136

789.494 702.094

821.882 607.146

642.778 738.260

840.894 589.943

799.987 485.806

File Format Overview

2.1.5

.awb AWBM Daily Time Series

An AWBM daily time series format file is an ASCII text file that contains daily time series data. There is no header line in the file. The data are organized in rows of one month of data separated by spaces. The first entry in a row is the number of days in the month. This is followed by data values for each day in the month. The data values are followed by the year and month of the data.

Note

Dates were optional in the original AWBM file format, but are not optional in TIME.

TIMEExample.awb
31 28 31 30 31 30 31 31 30 31 0 0 0 0 0 0 0 0 0 0 0 0.8 0 0 0 0 0 0 1.2 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 2001 1 0 0 0 0 0 0 0 0.2 0 0 0 0 0 0 0 0 0 0.8 0 0 0 0 0 0 0 0 0 0 2001 2 0 0 1.2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2.2 0 0 0 0 0 0 0 0 0 0 2001 3 0 0 0 0 0 0 0.4 0 0 0 0 0 0 0 67.4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2001 4 45.3 0 0 0 0 0 0 0 0 0 1.2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2001 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3.6 0 0 0 0 0 0 0 0 0 0 2001 6 0 0 8.4 0 0 0 0 0 45.6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2001 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2001 8 0 0 0 0.6 6 12.3 45 0 4 0 0 0 0 0 0 0 0 0 0 3.5 0 0 0 0 0 0 0 0 0 2001 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7.2 0 0 0 0 0 0 0 0 0 0 0 0 0 2001 10

Toolkit Data File Formats

2.1.6

.bsb SWAT BSB File Time Series

This is a line-based fixed format file, typical of FORTRAN programmers. Each line consists of data for a particular basin and time step. The header line gives the fields for the file, with each data line providing basin, Identifier, MON, area, and values for each data value (e.g Precipitation). Data are given for each basin for each time step, followed by each basin for the next timestep. For more details refer to the SWAT manual.

TIMEExample.bsb
SUB GIS BoggyCk 1 SwampyCk 2 DampCk 3 SoggyCk 4 MuddyR 5 BoggyCk 1 SwampyCk 2 DampCk 3 SoggyCk 4 MuddyR 5 MON 0 0 0 0 0 0 0 0 0 0 AREAkm2 1.14170E+02 1.12000E+02 1.79000E+02 10.0200E+02 10.0200E+02 1.14170E+02 1.12000E+02 1.79000E+02 10.0200E+02 10.0200E+02 PRECIPmm 0.000 0.000 0.000 0.000 0.000 1.200 5.600 9.200 4.000 3.200

10

File Format Overview

2.1.7

.bsm Bureau of Meteorology 6 Minute (was .pluv )

A fixed format file type, typically supplied by the Australian Bureau of Meteorology for 6 minute pluviograph data. The file has two header lines, with the first containing the station identified and an integer (1), and the second the station number and name. Data lines start with the station number, followed by date (yyyymmdd), then 240 values of 6 minute data for that day. All values are on a fixed spacing when supplied, but spaced separated values can be read by TIME. There are a number of bsm file format details, as explained below, taken from a note from the Bureau of Meteorology.

Note 1

All data available in the computer archive are provided. However very few sites have a completely uninterrupted historical record, with no gaps. Such gaps or missing data may be due to many reasons from illness of the observer to a broken instrument. A site may have been closed, reopened, upgraded or downgraded during its existence, possibly causing breaks in the record of any particular element.

Note 2

Final quality control for any element usually occurs once the manuscript records have been received and processed, which may be 6-12 weeks after the end of the month. Thus quality controlled data will not normally be available immediately, in 'real time'.

Data Records

Record Type 1 Element No. 1 2 3 4 Start End Byte Byte 16 7 15 16 16 17 N Record Size 6 9 1 Description Station Number Blank Record type Blank Units/Values

Record Type 2 Element No. 5 6 7 8 9 10 Start End Record Byte Byte Size 16 7 15 16 16 17 20 21 54 55 N 6 9 1 4 34 Description Station Number Blank Record type Blank Station Name lank Units/Values

11

Toolkit Data File Formats

Record Type 3 Element No. 11 12 13 14 15 16 17 18 19 Start End Byte Byte 16 7 12 13 16 17 18 19 20 21 27 28 34 35 41 42 N Record Size 6 6 4 2 2 7 7 7 Description Station Number Blank Year Month Day (Midnight to Midnight) Rainfall value of first interval Rainfall value of second interval Rainfall value of third interval Rainfall value of other intervals YYYY MM DD See Notes. See Notes. See Notes. See Notes. Units/Values

Notes Fortran format F7.1 i.e. a field width of seven bytes with one decimal place. A rainfall of 2mm will show up as a figure of 20.0 tenths of mm. Rainfall values will be : 0 = Intervals of zero rain. ve = Value for intervals when total for the period is known. 9999.0 = Intervals when no data is available. 8888.0 = Intervals when rain fell but the total is known only for a period of several intervals. This total is entered as a negative value in the last interval of the accumulated period. The length of each record depends on the interval required. Record length (N) = 20 + 7 * (1440 / Interval) eg. Interval = 6 minutes, N = 20 + 7 * (1440 / 6) = 1700 Record types 1 and 2 occur at the beginning of each station's data and are followed by a number of records of type 3. If there is zero rain for the whole day, no record is written for that day. If the entire month is missing, no records are written unless padding indicator is set. Then records with days filled with missing values (-9999.0) are written. However if complete years are missing, no attempt is made to write these dummy records.

TIMEExample.bsm
61078 61078 61078 61078 61078 61078 61078 61078 61078 61078 61078 61078 1 2 WILLIAMTOWN 19521231 .0 1953 1 1 .0 1953 1 3 .0 1953 115 .0 1953 118 .0 1953 212 .0 1953 213 .0 1953 214 .0 19521231 .0 19521231 .0 RAAF .0 .0 .2 .0 .0 .0 .0 .0 .0 .0

.0 .0 .0 .2 .0 .0 .0 .0 .0 .0

[etc., [etc., [etc., [etc., [etc., [etc., [etc., [etc., [etc., [etc.,

240 240 240 240 240 240 240 240 240 240

values] values] values] values] values] values] values] values] values] values]

12

File Format Overview

2.1.8

.csv Comma Separated Variable

A comma separated variable file is an ASCII text file that contains regular (periodic) time series data. There are two columns of data, with the first column being a date string (e.g. 01/01/2003; 01/1995), followed by the time series value. The two values are separated by a comma. Annual data can be entered using month number as 01 (e.g. 01/1995; 01/1996). The file may have a header line with a title for each column. The first column header is usually "Date" followed by a relevant data column header (e.g. flow, velocity, natural, historic).

TIMEExample.csv
1/01/1974,336.9 2/01/1974,771.2 3/01/1974,837.7 4/01/1974,940.2 5/01/1974,1249.3 6/01/1974,1445.4 7/01/1974,974.7 8/01/1974,797.0 9/01/1974,849.8 10/01/1974,168.4

13

Toolkit Data File Formats

2.1.9

.dat Fixed Format Time Series Files (F.Chiew)

This two-column daily time series file has a fixed date format of yyyymmdd in the first column (starting at Col position 3 from the left margin), and value in the second column. Spaces are used alongside single digit days or month (e.g. 1997_227______3.00).

TIMEExample.dat
19691228 19691229 19691230 19691231 1970 1 1 1970 1 2 1970 1 3 1970 1 4 1970 1 5 1970 1 6 0.00 0.00 3.00 2.00 12.68 0.00 17.10 3.53 0.00 0.00

14

File Format Overview

2.1.10

.iqqm IQQM Time Series

An IQQM time series format file is an ASCII text file that contains daily, monthly or annual time series data. The file has a five line header followed by tables of data. The five lines of information contained within the header are: Title line a 40-character string detailing how the file was created. It is suffixed with a date and time of creation; Site name a 40 character; Type a 15 character string specifying the data type (eg. precipitation, evaporation or gauged flow); Units a 10 character string specifying the units of data (eg. mm, mm*0.1, ML/d); and Date specifies the time span of the time series data and the time interval of data stored within the file. In daily IQQM format files the daily data is grouped in tables of yearly data with 31 columns representing each day in a month and 12 rows for each month of the year. At the end of each row of daily data is a monthly total and at the end of each table is a yearly total. These tables are repeated for the number of years of time series data. The first line of each table specifies the year related to the data within the table. This year may be optionally followed by "factor=". Where the number following the factor is a factor that is applies to the whole of the table. If this factor does not exist then no factor is applied to the table. The time series data values may be suffixed by a special character, which modifies the time series data value. The special characters and their function is described below: '*' Indicates the time series value is to be multiplied by 1000; 'e' Indicates the time series value is estimated; 'E' Indicates the time series value is estimated and should be multiplied by 1000; NOTE TIME does not act on these quality codes at present. 'n' Indicates the time series value is negative. Note negative numbers not followed by a 'n' are assumed to be missing; 'N' Indicates the time series value is negative and is to be multiplied by 1000 Note negative numbers not followed by a 'N' are assumed to be missing; and '?' Indicates that the time series value is missing. Note typically missing values are flagged as "-1?". A negative value not suffixed with a "n" or "N" is also considered as missing e.g. "-1 " is considered as a missing value.

15

Toolkit Data File Formats TIMEExample.iqqm

Title: Site : Type : Units: Date : Winding River at Middle Gauge Date:30/07/2003 Time:17:47:24.66 Winding River Flow ML/d 01/01/1985 to 31/12/2000 Interval : Daily

Year:1985 Factor= 0.1E-01 ----------------------------------------- ---------------------01 02 03 04 05 06 30 31 Total ----------------------------------------- ---------------------Jan 0 0 0 0 165e 97 3070 2509 10956 Feb 2040 1666 1337 1081 -1? 870 -1? Mar 545 450 358 286 217 216 1987 7309 45937 Apr 6910 5528 4520 4818 4947 3982 9081 219146 Dec 33* 32558 31655 30729 29827 29154 19364 19051 767745 ----------------------------------------- ---------------------14627179

16

File Format Overview

2.1.11

.mrf MFM Monthly Rainfall Files

A text file format containing a header line followed by a line giving the number of months of data. Data are formatted in lines with year given first, followed by 12 monthly values, all space separated.

TIMEExample.mrf
Swiftflow River @ Wooden Bridge 10 1985 12 23 34 45 35 46 56 67 65 43 11 12 1986 13 23 34 45 5 46 6 67 65 3 11 72 1987 14 23 34 45 35 46 56 67 65 43 11 82 1988 15 3 45 35 46 56 6 65 43 11 12 2 1989 16 23 34 45 35 46 56 67 6 43 11 18 1990 22 2 34 45 35 46 5 67 65 43 11 15 1991 22 23 34 45 35 46 56 67 65 43 15 12 1992 25 3 34 45 3 46 5 67 65 4 11 17 1993 15 23 3 45 35 46 56 67 6 43 11 15 1994 16 2 34 45 35 46 5 67 65 43 11 14

17

Toolkit Data File Formats

2.1.12

.pcp Swat Daily Time Series

An SWAT daily rainfall time series format file is an ASCII text file that contains daily time series rainfall data. The file has a four line header followed by daily data values. The four lines of information contained within the header are: The swat file description header 'Lati' followed by the latitude of the site in degrees 'Long' followed by the longitude of the site in degrees 'Elev' followed by the elevation of the site in metres The data is organized in one column, in format yyyydddvvv.v, where yyyy is the year, ddd is the Julian day, and vvv.v is the data value to one decimal place (5 characters).

TIMEExample.pcp
Precipitation Input File TIMEExample.pcp 20041224 AVSWAT2000 - SWAT interface MDL Lati 14.77 Long 102.7 Elev 167 1985001000.0 1985002000.0 1985003000.0 1985004000.0 1985005000.0 1985006000.0 1985007000.0 1985008000.0 1985009000.0 1985010000.0

18

File Format Overview

2.1.13

.silo5 QDNR SILO Time Series (was .txt )

An QDNR SILO daily time series format file is an ASCII text file that contains daily time series data. There is no header line in the file. The data is organized in five columns separated by spaces. The first four columns are the date in year, month, day and Julian day format. The fifth column is the data value.

TIMEExample.silo5
2001 2001 2001 2002 2002 2002 2002 2002 2002 2002 12 29 12 30 12 31 1 1 1 1 2 2 1 3 3 1 4 4 1 5 5 1 6 6 1 7 7 363 0 364 0 365 10.2 0 0 0 0 12 11 8

19

Toolkit Data File Formats

2.1.14

.silo8 SILO Standard Time Series (8 Variable)

The .silo8 format contains the full 8 column daily data set from the SILO data base. The file can have multiple header lines, enclosed in inverted commas. Data, in order, are maximum temperature, minimum temperature, rainfall, evaporation, radiation, vapour pressure, maximum relative humidity, minimum relative humidity.

TIMEExample.silo8
Big River Station 12 23 34 67 65 43 13 23 34 45 65 3 14 23 34 45 35 46 15 3 45 35 46 56 16 23 34 45 35 46 22 2 34 46 5 67 22 23 34 45 65 43 45 3 46 5 67 65 15 45 35 46 56 67 34 45 35 46 5 67 11 11 56 6 11 65 15 4 6 65 12 72 67 65 18 43 12 11 43 34

20

File Format Overview

2.2

Raster Formats
2.2.1
ncols aa nrows bb xllcorner cc yllcorner dd cellsize ee NODATA_value ff Where: aa is the number of columns, bb the number of rows, cc and dd the coordinates of the lower left corner of the grid (may be set to 0), ee is cell side length, and ff in the null data string (e.g. -9999). Values are arranged in space delimited rows and columns, reflecting the structure of the grid. Units for cell size length depend upon the input data, and could be either geographic (eg degrees) or projected (eg. m, km). Units are generally determined by the using application, and m is common for most TIME-based applications. Arcinfo grid coverages can be converted to .asc files using Arc's GRIDASCII command. .asc files can be imported into Arc using the ASCIIGRID command.

.asc Arc ASCII Grids

A space delimited grid file, with a 6 line header. The header lines are:

TIMEExample.asc
ncols 10 nrows 10 xllcorner 0 yllcorner 0 cellsize 100 NODATA_value -9999 5 6 3 4 4 5 6 6 6 4 4 4 2 4 4 5 4 5 6 6 3 3 3 3 3 4 4 4 5 6 2 2 2 2 2 3 3 3 4 5 2 9 2 2 2 2 2 2 3 4 5 6 4 4 4 4 2 2 2 3 5 6 3 3 3 3 2 2 2 2 5 6 2 2 2 2 9 8 2 2 5 6 8 2 2 2 4 5 4 4 4 5 6 6 4 2 3 4 4 4

21

Toolkit Data File Formats

2.2.2

.mwasc Map Window Ascii Grids

The .mwasc ASCII grid is similar to .asc except the header rows xllcorner and yllcorner are replaced by xllcenter and yllcenter that is, the coordinates are offset by 1/2 cellsize. The header rows do not have titles. Thus there are six header rows with parameters only, followed by the gridded data.

TIMEExample.mwasc
10 10 0 0 1 -9999 5 6 3 4 4 2 3 3 3 2 2 2 2 9 2 5 6 4 5 6 3 5 6 2 5 6 8 4 5 6

4 4 3 2 2 4 3 2 2 6

4 4 3 2 2 4 3 2 2 4

5 5 4 3 2 4 3 2 2 2

6 4 4 3 2 2 2 9 4 3

6 5 4 3 2 2 2 8 5 4

6 6 5 4 3 2 2 2 4 4

4 6 6 5 4 3 2 2 4 4

22

File Format Overview

2.2.3

.tapesg TapesG Grid files

A three column raster data format, with space separated values. Each line consists of the X coordinate, Y coordinate, and value.

TIMEExample.tapesg
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 167 165 152 133 134 144 151 156 146 132 150 141 139 135 131 158 151 153 146 136 158 160 160 153 148

2.2.4

.flt ESRI Binary Raster Interchange format

Arcinfo grid coverages can be converted to .ftl files using Arc's GRIDFLOAT command. .flt files can be imported into Arc using the FLOATGRID command.

23

Toolkit Data File Formats

Appendix A Zero-padding data in Excel

Certain file formats require data to be zero-padded. In the table below, the first column represents months, and is not zero-padded. Some applications, including Toolkit products, will sort this data as is shown in the 2nd column - clearly this is not correct. The third column is zero-padded Non zero-padded 1 2 3 4 5 6 7 8 9 10 11 12 What happens after sorting (some applications) 1 10 11 12 2 3 4 5 6 7 8 9 Zero-padded data (will sort correctly) 01 02 03 04 05 06 07 08 09 10 11 12

How to zero-pad data in Excel 2003 In this example, we will pad numbers representing the months, for example in the CDT format (see page 3). 1 Select the column representing the data

24

Zero-padding data in Excel

Right-click the selection and select Format Cells...

Click the Number tab, then in the Categories list, click Custom, then select General in the Type list.

Type in as many zeroes as there are digits in the data field. For example, "months" has two digits, so type two zeroes:

25

Toolkit Data File Formats

In the Sample area at the top, you can see what the padded number looks like. If you wanted to pad Julian days (which have 3 digits), you would type three zeroes into the Type field. 5 Click OK, and Excel pads the data:

26

Times and Dates in Data Files

Appendix B Times and Dates in Data Files

B.1 Formats for CDT and CSV files
Line format The format of each line in a CDT, SDT or CSV file is

Date,value1,value2,value3,value4

Missing Entries Missing entries must be specified as -9999. Empty strings or whitespace between commas are also read as missing values.

Decimal separators Always use a dot as a decimal separator for numerical values, irrespective of the local culture/language/locale settings for Windows.

Headers There may be a header on the first line specifying the names of the time series in the columns. Each element in the header must also be separated by commas e.g.

Date,rainfall,evaporation,temperature

27

Toolkit Data File Formats

Dates Dates must comply with the ISO 8601 standard. The full date-time format used is: yyyy-MM-dd HH:mm:ss A subset of this format may be used for the sake of compactness if there is no ambiguity or loss of precision, for example: yyyy-MM-dd ie at most, seconds, minutes and hours can be omitted The TIME framework (used for most Toolkit products) will always write the following: yyyy-MM-dd though more compact formats will be read if unambiguous. The smallest time step that CSV files can currently handle is a second. See section B.2 for more details on the ISO date-time format.

Note

Microsoft Excel seems to recognise the ISO 8601 date format with at least the precision yyyy-MM-dd. This correct detection seems to be independent of the computer regional and language settings.

Dates read when loading a CSV file do not necessarily have to follow the ISO 8601 standard, but end-users should be encouraged to use it when generating CSV data files from another software tool. Local date-time culture settings are supported for the benefit of users for two main practical reasons: the default behaviour of Microsoft Excel and backward compatibility.

How the TIME framework determines date/time formats When reading a file, the TIME framework examines the first few lines to detect the date-time format and the time step of the time series: If the format is recognised as being ISO 8601 compliant, this format will be used to read all subsequent dates. Failing that, an attempt is made to detect the dates and time step with EnglishAustralia ("en-AU") settings, for backward-compatibility reasons. Last, an attempt is made to use the computer configuration for regional and language settings. An exception, LoadDataFileIOException, occurs if all methods fail. If you see this exception you will need to check your input data.

Note

The date-time format must be consistent throughout the file.

Possible problems with time steps The only known case where a time step may be incorrectly detected when reading a file is on a computer with U.S. settings, due to the mm/dd/yyyy date format. This may happen if the whole of a daily time series covers less than 13 days, or less than 12 months for a monthly time series. Users working on such computers should use the recommended ISO 8601 format in order to prevent any ambiguity.

28

Times and Dates in Data Files

Examples A CVS file with one daily time series, where the time of record is 00:00:00 every day will be written as:
2000-01-01,0.9 2000-01-02,0.7 2000-01-03,0.9 2000-01-04,0.4

An example of a daily time series with missing values at the second and fourth day:
2000-01-01,0.9 2000-01-02,-9999 2000-01-03,0.9 2000-01-04, 2000-01-05,1.1

Parsing the following file would throw an exception: missing entries are not allowed since their interpretation is likely ambiguous (missing data, or zero record?)
2000-01-01,0.9 2000-01-02,2.2 2000-01-05,1.1 2000-01-06,1.1

A time series with a six minutes time step:

2000-01-01 2000-01-01 2000-01-01 2000-01-01 2000-01-01 00:00,0 00:06,1 00:12,2 00:18,3 00:24,4

The following time series will be read as one monthly time series, with records on the first day of every month at time 00:00:00.
2000-01,0 2000-02,1 2000-03,2 2000-04,3 2000-05,4

The following time series will be read as one annual time series, with records on January the first, at time 00:00:00
2000,0 2001,1 2002,2

Any even time step is supported, e.g. a weekly time series would be written as:
2004-01-02,0 2004-01-09,1.2 2004-01-16,2.3 2004-01-23,3 2004-01-30,4 2004-02-06,4

A header may be specified:

Date,rainfall,evaporation 2000-01-01,0.9,0.9 2000-02-01,0.7,0.9 2000-03-01,0.9,0.9 2000-04-01,0.4,0.9

The following file, generated for instance with Excel, will be recognised as a monthly time series on a computer with U.S. culture (locale) settings, but loading it on a computer with different culture settings will likely fail:
01/01/2000,0.1 02/01/2000,0.8 03/01/2000,0.6 04/01/2000,0.9 05/01/2000,0.6

29

Toolkit Data File Formats

06/01/2000,0.4 07/01/2000,0.6 08/01/2000,0.2 09/01/2000,0.8 10/01/2000,0.6 11/01/2000,0.3 12/01/2000,0.1 01/01/2001,0.389 02/01/2001,0.389

The following file, irrespective of the computer's culture settings, will be read as a daily time series from the first of January to the fifth:
01/01/2000,0.1 02/01/2000,0.8 03/01/2000,0.6 04/01/2000,0.9 05/01/2000,0.6

B.2

The ISO8601 standard for representing dates and times

The TIME framework uses a subset of the ISO8601 standard. The central part of this subset is the use of the format string: yyyy-MM-ddTHH:mm:ss It appears that de facto 'standards' such as Microsoft Excel do not detect dates with the T symbol between the date and time parts of the date-time. However if the T symbol is replaced with a space, the date-time is recognised by Excel, irrespective of the regional settings of the computer. Houston (reference 2, below) reports that according to the ISO standard the T symbol "may be omitted by those interchanging data, if ambiguity can be avoided". The T symbol will thus be replaced by a space to ensure compatibility with de facto Microsoft 'standards'. The members of this class are qualified as 'compatible' if the T symbol is replaced by a whitespace, otherwise the term 'strict compliance' will be used.

References Kuhn M., "A Summary of the International Standard Date and Time Notation", http://www.cl.cam.ac.uk/~mgk25/iso-time.html Houston G., "ISO 8601:1988 Date/Time Representations", http://hydracen.com/dx/iso8601.htm

Note

The official ISO document is copyrighted and not freely available.

30