TCD TCEH blog - skills share

Skill Share #1: NETCDF Files and our research

Author: Dr Cordula Scherer

1 Share

On Friday (8 June) we held our first Skill Share session – a new club we established in the TCEH for everyone to learn from and with colleagues and to share our knowledge with each other. The first session was about opening NetCDF files in R given by Dr Cordula Scherer. NetCDF (Network Common Data Form) is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

NetCDF files are commonly used in climatology, meteorology and oceanography applications (e.g., weather forecasting, climate change) and GIS applications. In our research in the CEH and especially in the NorFish project we analyse climate change and its effects on ecological dynamics on decadal and centennial scales across the North Atlantic in order to reconstruct the ocean conditions in the past. For this reason and for our attempt to visualize the reconstructed conditions alongside human interactions (e.g. fishing) through GIS deep mapping, our projects a very data hungry. Data stored in NetCFD formats are therefore very useful to us.

The format of NetCDFs was originally based on the conceptual model of the Common Data Format developed by NASA, but has since diverged and is not compatible with it. The Common Data Model has three layers, which build on top of each other to add successively richer semantics:

1. The data access layer, also known as the syntactic layer, handles data reading.

2. The coordinate system layer identifies the coordinates of the data arrays. Coordinates are a completely general concept for scientific data; specialized georeferencing coordinate systems, important to the Earth Science community, are specially annotated.

3. The scientific data type layer identifies specific types of data, such as grids, images, and point data, and adds specialized methods for each kind of data.

This array-oriented data format enables storage of close to endless data, taking up very little space. Therefore, it is used especially for spatial datasets. However, this is also the reason why it is often difficult to extract data from it with a traditional approach such as .txt or .xsl.

There are several help forums available on the internet assisting the opening of NetCDF files in various ways, however, R provides one of the best and most straight forward packages available on the R help forum website:
http://geog.uoregon.edu/bartlein/courses/geog490/week04-netCDF.html#reading-a-netcdf-data-set-using-the-ncdf4-package

After the session, a discussion arose on handling the time stamps on stored data – this is a common problem with NetCDF files. R seems to have a good solution for this problem while programmes such as ARCMap seem sadly deficient in its handling of time which will be (hopefully) improved soon.