International Journal of Scientific & Engineering Research, Volume 5, Issue 7, July-2014 983

ISSN 2229-5518

Reading and Writing NetCDF Datasets: Tools and Techniques

Gurbrinder Kaur

Abstract— GRIB (GRIdded Binary or General Regularly-distributed Information in Binary form) is a data format commonly used in meteorology to store historical and to forecast weather data. It is standard format made by the World Meteorological Organization's Commission for Basic Systems. GRIB is a W MO format for gridded data. GRIB data is used by the various meteorological centers for storage and the exchange of gridded fields. GRIB's major advantages are files are typically 1/2 to 1/3 of the size of normal binary files (floats), the fields are self describing, and GRIB is an open, international standard. A major advantage of GRIB is that is self describing. Each record has information such as: resolution of the grid, time, variable, level, who created the field. There are a number of GRIB decoders available.After decoding these files can be read in any language C, Fortran, Java or even Basic programs. This paper introduces the grib files, their decoders and tools to visualize the weather data.

Index Terms— Gridded Binary (GRIB),Network Common Format(NetCDF) ,Comma Seperated Values(CSV) ,Graphic User Interface(GUI), Netcdf Opetors(NCO),Common Data Language(CDL),Hierarchical Data Format(HDF),World Meteorological Organization(W MO).

1 INTRODUCTION

——————————  ——————————
RIB is a file format for the storage and transport of grid- ded meteorological data, such as Numerical Weathe Prediction model output. It is designed to be self- describing, compact and portable across computer architec- tures [4]. The GRIB standard was designed and is maintained by the World Meteorological OrganizationGRIB data is used by individuals, institutions and businesses who have the means and motivation to tap into numerical data from weath- er models in the rawest possible form [6]. GRIB data is often only one step removed from the original model output. Once Grib files are decoded, the data can be processed for visualiza- tion, or can be used as input for numerical weather prediction applications that require gridded data.Grib data can be decod- ed into various file formats depending upon the type of data
analysis required by the analysts.
The grib files can be converted into Netcdf files, Shape files,
polygon and simply flat database files. Grib Decoders availa- ble are wgrib, UGrib, GribUs and NDFD Degrib software. This paper provides details of decoders available and different file formats which Decoders can convert.Each format has its own
capability to interpret the weather data.The basic functionality of these decoders are:

1. make inventory and read grib-2 files

2. manipulate (reorder, extract) grib-2 records (messages)

3. convert to ieee, text, binary, comma separated values

(CSV)

4. convert to netcdf-3, netcdf-4 (optional package)

5. export to a mysql database (optional package)

6. geolocation of grid points

7. change grid-relative to earth relative winds (-

new_grid, optional package)

8. writing grib-2

9. parallel processing by using threads (OpenMP)

This paper provides details of different file formats and tools to read these formats.The focus will however will be on Netcdf file formats.

2 READING AND WRITING NETCDF FILES

2.1 Netcdf files:

NetCDF - short for Network Common Format is a set of inter- faces that was created in order to facilitate the sharing of elec- tronic scientific data, regardless of machine or operating sys- tem[10]. It was originally developed by the Unidata Program Center, part of the University Corporation for Atmospheric Research, in order for the program’s researchers to share their meteorological data and to create a reusable, cross-disciplined piece of software. [1] In the classic netCDF data model, data are stored in array data structures, called variables, which are shaped by dimensions and described by attributes.

————————————————

• Gurbrinder Kaur is pursuing Ph.d from Singhania Univesity,Rajasthan and is currently working as Asstistant Professor, MCA Department, Banarsidas Chandiwala Institute of Information Technology ,Delhi-India, PH-01149020147. E- mail:gurbrinderaulakh@yahoo.co.in

Fig. 1 Netcdf Classic Data Model[7] Explaining the structure of Netcdf file

International Journal of Scientific & Engineering Research, Volume 5, Issue 7, July-2014 984

ISSN 2229-5518

There are many resources for accessing netCDF files[12]. The following is a list of software tools and programs for working with netCDF files divided into two groups: command line interfaces and GUI interfaces.

2.2 Command Line Interfaces:

Ncdump: Ncdump is the netCDF file reader that is bundled with Unidata's netCDF product. The ncdump utility generates a text representation of a specified netCDF file on standard output, optionally excluding some or all of the variable data in the output[12]. The text representation is in the form of CDL (network Common Data Language) that can be viewed, edit- ed, or serve as input to ncgen, a companion program that can generate a binary netCDF file from a CDL file. Hence ncgen and ncdump can be used as inverses to trans- form the data representation between binary and text repre- sentations.

NetCDF Operators (NCO): The NCO are a suite of programs known as operators in which each operator is a standalone, command line program which is executed at the UNIX com- mand prompt. The netCDF Operators (NCO) comprise a doz- en standalone, command-line programs that take netCDF, HDF, and/or DAP files as input, then operate (e.g., derive new data, compute statistics, print, hyperslab, manipulate metadata) and output the results to screen or files in text, bina- ry, or netCDF formats. NCO aids analysis of gridded scientific data. The shell-command style of NCO allows users to manip- ulate and analyze files interactively, or with expressive scripts that avoid some overhead of higher-level programming envi- ronments.

2.3 GUI Interfaces:

Ncview

Ncview is a netCDF visual browser that allows the user to visually inspect NetCDF data files. Ncview is a visual browser for netCDF format files. Typically you would use ncview to get a quick and easy, push-button look at your netCDF files. You can view simple movies of the data, view along various dimensions, take a look at the actual data values, change color maps, invert the data, etc. It runs on UNIX platforms under X11, R4 or higher

IDL NetCDF Reader

For users who are familiar with IDL, David Fanning has creat-
ed a netCDF browser in IDL: ncdf_browser.pro. To download
this procedure, visit the NCDF Browser section of David Fan-
ning's Web site.

Panoply

Panoply is a JAVA application developed by NASA for viewing netCDF files. Panoply is a cross-platform application that plots geo-gridded and other arrays from netCDF, HDF, GRIB, and other datasets. Following are the features of Panoply:

1. Slice and plot geo-gridded latitude-longitude, lati-

tude-vertical, longitude-vertical, or time-latitude ar- rays from larger multidimensional variables. Slice and plot "generic" 2D arrays from larger multidimensional variables.

2. Slice 1D arrays from larger multidimensional varia- bles and create line plots.

3. Combine two geo-gridded arrays in one plot by dif- ferencing, summing or averaging.

4. Plot lon-lat data on a global or regional map using any of over 100 map projections or make a zonal av- erage line plot.

NcBrowse

NcBrowse is a Java application that offers flexible, interactive graphical displays of data and its attributes from a wide range of netCDF data file conventions.

IDV (Integrated Data Viewer)

Unidata's Integrated Data Viewer (IDV) is a Java application (for Java 1.4 or later) that can be used to display a variety of netCDF files, particularly well formatted, geolocated datasets.

2.4 Decoding and Displaying a Grib File:

This section will demonstrate how a grib file can be con- verted to netcdf file and how the netcdf file can be visualized to analyse the data.The tool used to decode grib file is Degrib Decoder and the tool used to read netcdf file is Java Netcdf User Interface.First the file is converted into .csv(comma sepa- rated value) file to view its contents then the file is converted to Netcdf file to visualize it . The grib file is converted into netfile file having extension .nc then contents of this file are displayed in Netcdf Java UI.

Table 1: Sample of the data converted to csv

Fig. 2 Data of netcdf file in Netcdf Grib Image viewer (Java

Netcdf UI Tool)

International Journal of Scientific & Engineering Research, Volume 5, Issue 7, July-2014 985

ISSN 2229-5518

Fig. 3 Displaying the netcdf file in Grib Viewer(Java
Netcdf tool)

3 CONCLUSION

The paper describes the various tools and techniques to read scientific weather data in grib file format. The information can be further used to visualize and predict various weather events.The data in the example is taken from the Indian Meterological department.

REFERENCES

[1] Brown, S. A, M. Folk, G. Goucher, and R. Rew, "Software for Portable Scientific Data Management," Computers in Physics, American Insti- tute of Physics, Vol. 7, No. 3, May/June 1993.

[2] Davies, H. L., "FAN - An array-oriented query language," Second

Workshop on Database Issues for Data Visualization (Visualization

1995), Atlanta, Georgia, IEEE, October 1995.

[3] Fahle, J., TeraScan Applications Programming Interface, SeaSpace, San

Diego, California, 1989.

[4] Fulker, D. W., "The netCDF: Self-Describing, Portable Files—a Basis

for 'Plug-Compatible' Software Modules Connectable by Networks," ICSU Workshop on Geophysical Informatics, Moscow, USSR, August

1988.

[5] Fulker, D. W., "Unidata Strawman for Storing Earth-Referencing Data," Seventh International Conference on Interactive Information and Pro- cessing Systems for Meteorology, Oceanography, and Hydrology, New Orleans, La., American Meteorology Society, January 1991.

[6] Gough, M. L., NSSDC CDF Implementer's Guide (DEC VAX/VMS) Version 1.1, National Space Science Data Center, 88-17, NASA/Goddard Space Flight Center, 1988.

[7] Jenter, H. L. and R. P. Signell, "NetCDF: A Freely-Available Software- Solution to Data-Access Problems for Numerical Modelers," Proceed- ings of the American Society of Civil Engineers Conference on Estua- rine and Coastal Modeling, Tampa, Florida, 1992.

[8] Raymond, D. J., "A C Language-Based Modular System for Analyzing and Displaying Gridded Numerical Data," Journal of Atmospheric and Oceanic Technology, 5, 501-511, 1988.

[9] Rew, R. K. and G. P. Davis, "The Unidata netCDF: Software for Scien- tific Data Access," Sixth International Conference on Interactive Infor- mation and Processing Systems for Meteorology, Oceanography, and

Hydrology, Anaheim, California, American Meteorology Society, Feb- ruary 1990.

[10] Rew, R. K. and G. P. Davis, "NetCDF: An Interface for Scientific Data

Access," Computer Graphics and Applications, IEEE, pp. 76-82, July

1990.

[11] Rew, R. K. and G. P. Davis, "Unidata's netCDF Interface for Data Ac- cess: Status and Plans," Thirteenth International Conference on Interac- tive Information and Processing Systems for Meteorology, Oceanogra- phy, and Hydrology, Anaheim, California, American Meteorology So- ciety, February 1997.

[12] Treinish, L. A. and M. L. Gough, "A Software Package for the Data In- dependent Management of Multi-Dimensional Data," EOS Transac-

tions, American Geophysical Union, 68, 633-635, 1987.

[13] [1] R. Rew, G. David, S. Emmerson, H. Davies, E. Hartnett and D.

Heimbigner, The NetCDF

[14] Users Guide, Boulder, CO: Unidata Program Center, June 2011.

[15] [2] L. A. Treinish and M. L. Gough, "A software package for the data- independent management

[16] of multidimensional data," EOS, Transactions, American Geophysical

Union, vol. 68, no. [17] 28, p. 633, 1987.

[18] [3] D. J. Raymon, "A C Language-Based Modular System for Analyz-

ing and Displaying

[19] Gridded Numerical Data," Physics Department and Geophysical Re- search Center, R&D

[20] Division, New Mexico Institute of Mining and Technology, Socorro,

1987.

[21] [4] "FAQ," Open Handset Alliance, 2007. [Online]. Available:

[22] http://www.openhandsetalliance.com/android_faq.html. [Accessed 20

March 2012].

[23] [5] R. Rew, "NetCDF FAQ," March 2012. [Online]. Available:

[24] http://www.unidata.ucar.edu/software/netcdf/docs/faq.html. [Accessed

15 March 2012].

[25] [6] J. Caron, "NetCDF Java Library," February 2012. [Online]. Avail- able:

[26] http://www.unidata.ucar.edu/software/netcdf-java/documentation.htm. [Accessed 15 March

[27] 2012].

[28] [7] "2008 Unidata NetCDF Workshop for Developers and Data Pro- viders," 2008.

[29] [8] G. Beavis, "A Complete History of Android," Tech Radar, 23 Sep- tember 2008. [Online].

[30] Available: http://www.techradar.com/news/phone-and- communications/mobile-phones/acomplete-

[31] history-of-android-470327. [Accessed 7 March 2012].

[32] [9] "Members," Open Handset Alliance, [Online]. Available:

[33] http://www.openhandsetalliance.com/oha_members.html. [Accessed

20 March 2012].

[34] [10] C. Ziegler, "Android: A Visual History," The Verge, 7 December

2011. [Online]. Available:

[35] http://www.theverge.com/2011/12/7/2585779/android-history. [Ac- cessed 18 March 2012].

[36] [11] "Dev Guide," Android Developers, 2012. [Online]. Available:

[37] http://developer.android.com/guide/index.html. [Accessed 10 March

2012].

[38] [12] "CDM File Types," Unidata, 2012. [Online]. Available: [39] http://www.unidata.ucar.edu/software/netcdf-

java/formats/FileTypes.html. [Accessed 30March 2012].