OPeNDAP and NetCDF

NetCDF was designed in way so that it can be used within a network envi- ronment (Nativi et al., 2005). Sgouros (2004) argues that the NetCDF API of OPeNDAP was initially designed to handle gridded data and that subse- quently limitations in accessing sequence data exist. The default mode for translating an OPeNDAP DAP 2.0 stream to NetCDF is according to He- imbigner (2009) and Rew et al. (2010) the NetCDF 3 classic model, but it is possible to change this setting by a modification of the client parameters. The translation from OPeNDAP to the NetCDF 4 enhanced data model is entirely new, as Rew et al. (2010) remark.

Since 2008, Unidata and OPeNDAP were working on a combination of the NetCDF API of Unidata that reads and writes NetCDF data to local disk files and of the NetCDF API of the OPeNDAP corporation that translates

NetCDF data to OPeNDAP’s data access protocol (DAP) and vice versa (Gallagher et al., 2008; Heimbigner, 2008). The primary goal, as mentioned by Heimbigner (2008), was the integration of the client side NetCDF API of DAP into the NetCDF 3 and later the NetCDF 4 library of Unidata. Since summer 2009, OPeNDAP’s DAP client library for NetCDF is integrated in Unidata’s standard NetCDF distribution and replaces the older library that was provided before by the OPeNDAP corporation (Heimbigner, 2009). From NetCDF version 4.1 it is thereof, as announced by OPeNDAP Inc. (2011), distributed by Unidata with optional built-in support for DAP as part of the NetCDF library (Rew et al., 2010).

Gallagher et al. (2008) clarify that the original data access protocol of OPeNDAP was partly based on NetCDF, before the newer version 2.0 of DAP was released. However, with the introduction of version 2.0, important differences between the data models of DAP 2.0 and NetCDF 3 came in existence, and some DAP 2.0 data became inaccessible within NetCDF 3. Unidata thereof introduced the Common Data Model (CDM) that is mostly implemented within the NetCDF 4 data model. One goal of the CMD is to unify again both data models of NetCDF and DAP so that as much as OPeNDAP data as possible can be accessed by a NetCDF-API. However, there are still differences in between the NetCDF classic as well as enhanced data models to the actual OPeNDAP data model (Heimbigner, 2009; Rew et al., 2010). The Common Data Model is implemented in the NetCDF-Java library, as documented by Caron (2004) and Gallagher et al. (2008).

Table 2.3 on page 56 gives an overview of the actual data type transfor- mation rules in case that an OPeNDAP dataset is read via the Common Data Model (NetCDF 4), as defined by Caron (2011). Rew et al. (2010) note that the translation from OPeNDAP to NetCDF 4 (CDM) is still experimental and might change.

Translation issues to OPeNDAP

Dimensions In comparison to the DAP of OPeNDAP, nested sequences

or arrays of sequences are not possible within the NetCDF 3 data model (Heimbigner, 2008). Caron (2004, 2011) remarks that OPeNDAP however does not have explicit shared dimensions, except for map arrays inside of the Grid data type. This problem can be avoided by approximating them by map vectors in the Grid data type Caron (2011).

Attributes Attributes within the CDM may only be scalar or as part of an

double or string, but not in the form of nested attributes, as it might be the case in OPeNDAP (Caron, 2011).

Variables Rew et al. (2010) declare that in case of the NetCDF 3 data

model, NetCDF variables are derived from OPeNDAP fields with primitive data types, as they can occur within the OPeNDAP Structure, Grid or Se- quence data types. The dimensions of a variable are obtained from the di- mensions of the fields that the variable represents, and may also be inherited from other dimensions of containing Structures. If a variable is contained di- rectly or indirectly by an OPeNDAP Sequence, special rules will be employed to determine its rank and its associated dimensions so that it gets translated correctly to NetCDF 3.

Data types Heimbigner (2009) explains that problems in data translation

between a DAP data stream and the NetCDF 3 format might occur, such as for the string data type that does not exist in the NetCDF 3 data model. It is therefore converted to an one-dimensional character array of a fix length (that is the string length) which represents this string. Moreover, unsigned data types do not exist in NetCDF 3, but in OPeNDAP. As documented by Caron (2011), OPeNDAP does not support – in contradiction to the Common Data Model of Unidata – the data types signed byte, char, long, opaque and enum. Contrary to the CDM, OPeNDAP’s DAP 2.0 does have no groups (Caron, 2011).

OPeNDAP Grids A Structure that contains the grid array and map vari-

ables forms a Grid object in OPeNDAP(Caron, 2004). Heimbigner (2008) explains that the OPeNDAP Grid data type is converted according to the coordinate variable convention of NetCDF 3. Hence the translation results in one-dimensional coordinate variables containing coordinate values for the corresponding dimensions and having the same name as these dimensions. These coordinate variables, as documented by Caron (2004), share their di- mensions with the array variables that contain the measurements.

OPeNDAP Sequences Heimbigner (2008) states that an OPeNDAP Se-

quence is converted to an one-dimensional array with an unlimited dimension in case that the NetCDF 3 data model is used. According to Caron (2011), the translation of an OPeNDAP Sequence within the Common Data Model is actually realized by either an one-dimensional variable-length Structure or a CDM Sequence. In an earlier publication Caron (2004) stated that access to Sequence data is not solved through the standard API of NetCDF.

OPeNDAP primitive data type CDM primitive data type Boolean boolean Byte byte (unsigned attr.) Float32 float Float64 double Int16 short Int32 int

UInt16 short (unsigned attr.)

UInt32 int (unsigned attr.)

OPeNDAP constructor type CDM data type Grid Variable List ignored

Sequence Structure (Sequence)

String String or Char

Structure Structure

URL ignored

Table 2.3: Transformation rules from OPeNDAP to Unidata’s Common Data Model (CDM), as defined by Caron (2011) and also mentioned by Caron (2004) (out of date)