Basic use¶

efts-io is primarily about saving ensemble forecast time series to (resp. loading from) files on disk in netCDF STF 2.0 compliant format, from Python.

While most similar implementations in e.g. R, Matlab so var have been closely related to netCDF file handling, in Python xarray is a de facto standard for the high level manipulation of tensor-like, multidimensional data. There is a partial mismatch between the STF netCDF conventions devised ten years ago and limited by the capabilities of Fortran netCDF libraries at the time, and the best practices for xarray in-memory representations. efts-io is a package bridging the technical gap between these two representations, and reducing the risk of data handling bugs by users when trying to reconcile this technical gap.

Creating a new STF xarray dataset¶

There are several ways to create a dataset for EFTS with efts-io. One helper function to create a data set is xr_efts, particularly if you know upfront the geometry (dimensions) of your dataset:

In [2]:

Copied!

import pandas as pd
import numpy as np
import pandas as pd
import numpy as np

In [3]:

Copied!

from efts_io import wrapper as w
from efts_io import wrapper as w

In [4]:

Copied!





issue_times = pd.date_range("2010-01-01", periods=31, freq="D")
station_ids = ["410088","410776"]
lead_times = np.arange(start=1, stop=4, step=1)
lead_time_tstep = "hours"
ensemble_size = 10
station_names= ["GOODRADIGBEE B/BELLA", "Licking Hole Ck"]# None
nc_attributes = None
latitudes = None
longitudes = None
areas = None

d = w.xr_efts(
    issue_times,
    station_ids,
    lead_times,
    lead_time_tstep,
    ensemble_size,
    station_names,
    latitudes,
    longitudes,
    areas,
    nc_attributes,
)
issue_times = pd.date_range("2010-01-01", periods=31, freq="D")
station_ids = ["410088","410776"]
lead_times = np.arange(start=1, stop=4, step=1)
lead_time_tstep = "hours"
ensemble_size = 10
station_names= ["GOODRADIGBEE B/BELLA", "Licking Hole Ck"]# None
nc_attributes = None
latitudes = None
longitudes = None
areas = None

d = w.xr_efts(
    issue_times,
    station_ids,
    lead_times,
    lead_time_tstep,
    ensemble_size,
    station_names,
    latitudes,
    longitudes,
    areas,
    nc_attributes,
)

Let us have a look at the created Dataset:

In [5]:

Copied!

d
d

Out[5]:

<xarray.Dataset> Size: 608B
Dimensions:       (station_id: 2, time: 31, ens_member: 10, lead_time: 3)
Coordinates:
  * time          (time) datetime64[ns] 248B 2010-01-01 ... 2010-01-31
  * station_id    (station_id) <U6 48B '410088' '410776'
  * ens_member    (ens_member) int64 80B 1 2 3 4 5 6 7 8 9 10
  * lead_time     (lead_time) int64 24B 1 2 3
Data variables:
    station_name  (station_id) <U20 160B 'GOODRADIGBEE B/BELLA' 'Licking Hole...
    lat           (station_id) float64 16B nan nan
    lon           (station_id) float64 16B nan nan
    area          (station_id) float64 16B nan nan
Attributes:
    title:                   not provided
    institution:             not provided
    catchment:               not provided
    source:                  not provided
    comment:                 not provided
    STF_convention_version:  2.0
    STF_nc_spec:             https://github.com/csiro-hydroinformatics/efts/b...

xarray.Dataset

Dimensions:
- station_id: 2
- time: 31
- ens_member: 10
- lead_time: 3

Coordinates: (4)

time

(time)

datetime64[ns]

2010-01-01 ... 2010-01-31

standard_name :: time
long_name :: time
axis :: t

array(['2010-01-01T00:00:00.000000000', '2010-01-02T00:00:00.000000000',
       '2010-01-03T00:00:00.000000000', '2010-01-04T00:00:00.000000000',
       '2010-01-05T00:00:00.000000000', '2010-01-06T00:00:00.000000000',
       '2010-01-07T00:00:00.000000000', '2010-01-08T00:00:00.000000000',
       '2010-01-09T00:00:00.000000000', '2010-01-10T00:00:00.000000000',
       '2010-01-11T00:00:00.000000000', '2010-01-12T00:00:00.000000000',
       '2010-01-13T00:00:00.000000000', '2010-01-14T00:00:00.000000000',
       '2010-01-15T00:00:00.000000000', '2010-01-16T00:00:00.000000000',
       '2010-01-17T00:00:00.000000000', '2010-01-18T00:00:00.000000000',
       '2010-01-19T00:00:00.000000000', '2010-01-20T00:00:00.000000000',
       '2010-01-21T00:00:00.000000000', '2010-01-22T00:00:00.000000000',
       '2010-01-23T00:00:00.000000000', '2010-01-24T00:00:00.000000000',
       '2010-01-25T00:00:00.000000000', '2010-01-26T00:00:00.000000000',
       '2010-01-27T00:00:00.000000000', '2010-01-28T00:00:00.000000000',
       '2010-01-29T00:00:00.000000000', '2010-01-30T00:00:00.000000000',
       '2010-01-31T00:00:00.000000000'], dtype='datetime64[ns]')

station_id
(station_id)
<U6
'410088' '410776'
long_name :
station or node identification code
```
array(['410088', '410776'], dtype='<U6')
```
ens_member
(ens_member)
int64
1 2 3 4 5 6 7 8 9 10
standard_name :
ens_member
long_name :
ensemble member
units :
member id
axis :
u
```
array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
```
lead_time
(lead_time)
int64
1 2 3
standard_name :
lead time
long_name :
forecast lead time
axis :
v
units :
hours since time
```
array([1, 2, 3])
```

Data variables: (4)
- station_name
  (station_id)
  <U20
  'GOODRADIGBEE B/BELLA' 'Licking ...
  long_name :
  station or node name
```
array(['GOODRADIGBEE B/BELLA', 'Licking Hole Ck'], dtype='<U20')
```
- lat
  (station_id)
  float64
  nan nan
  long_name :
  latitude
  units :
  degrees_north
  axis :
  y
```
array([nan, nan])
```
- lon
  (station_id)
  float64
  nan nan
  long_name :
  longitude
  units :
  degrees_east
  axis :
  x
```
array([nan, nan])
```
- area
  (station_id)
  float64
  nan nan
  long_name :
  station area
  units :
  km^2
  standard_name :
  area
```
array([nan, nan])
```

Indexes: (4)

time

PandasIndex

PandasIndex(DatetimeIndex(['2010-01-01', '2010-01-02', '2010-01-03', '2010-01-04',
               '2010-01-05', '2010-01-06', '2010-01-07', '2010-01-08',
               '2010-01-09', '2010-01-10', '2010-01-11', '2010-01-12',
               '2010-01-13', '2010-01-14', '2010-01-15', '2010-01-16',
               '2010-01-17', '2010-01-18', '2010-01-19', '2010-01-20',
               '2010-01-21', '2010-01-22', '2010-01-23', '2010-01-24',
               '2010-01-25', '2010-01-26', '2010-01-27', '2010-01-28',
               '2010-01-29', '2010-01-30', '2010-01-31'],
              dtype='datetime64[ns]', name='time', freq='D'))

station_id

PandasIndex

PandasIndex(Index(['410088', '410776'], dtype='object', name='station_id'))

ens_member

PandasIndex

PandasIndex(Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype='int64', name='ens_member'))

lead_time

PandasIndex

PandasIndex(Index([1, 2, 3], dtype='int64', name='lead_time'))

Attributes: (7)
title :
not provided
institution :
not provided
catchment :
not provided
source :
not provided
comment :
not provided
STF_convention_version :
2.0
STF_nc_spec :
https://github.com/csiro-hydroinformatics/efts/blob/107c553045a37e6ef36b2eababf6a299e7883d50/docs/netcdf_for_water_forecasting.md

We did not provide custom global attributes to the functions. Defaults were created.

Note that while the intent is that the on-disk netCDF file created later will comply with the STF specs, but this is debatable whether the "in memory" data set just created should advertise the STF related attributes like STF_convention_version. If saved by the user with to_netcdf rather than via efts-io, it is a tad confusing.

In [6]:

Copied!

d.attrs
d.attrs

Out[6]:

{'title': 'not provided',
 'institution': 'not provided',
 'catchment': 'not provided',
 'source': 'not provided',
 'comment': 'not provided',
 'STF_convention_version': '2.0',
 'STF_nc_spec': 'https://github.com/csiro-hydroinformatics/efts/blob/107c553045a37e6ef36b2eababf6a299e7883d50/docs/netcdf_for_water_forecasting.md'}

In [7]:

Copied!

d.time.attrs
d.time.attrs

Out[7]:

{'standard_name': 'time', 'long_name': 'time', 'axis': 't'}

Climate and Forecast conventions¶

Placeholder section.

We may want to have the package cf-xarray as a dependency of efts-io, if it brings added value.

In [14]:

Copied!

import cf_xarray as cfx
import cf_xarray as cfx

In [15]:

Copied!

import xarray as xr
xr.set_options(display_expand_data=False)
xr.set_options(keep_attrs=True)
import xarray as xr
xr.set_options(display_expand_data=False)
xr.set_options(keep_attrs=True)

Out[15]:

<xarray.core.options.set_options at 0x7f7d981caa20>

In [16]:

Copied!

d.lon
d.lon

Out[16]:

<xarray.DataArray 'lon' (station_id: 2)> Size: 16B
nan nan
Coordinates:
  * station_id  (station_id) <U6 48B '410088' '410776'
Attributes:
    long_name:  longitude
    units:      degrees_east
    axis:       x

xarray.DataArray

'lon'

station_id: 2

nan nan
```
array([nan, nan])
```
Coordinates: (1)
- station_id
  (station_id)
  <U6
  '410088' '410776'
  long_name :
  station or node identification code
```
array(['410088', '410776'], dtype='<U6')
```

Indexes: (1)

station_id

PandasIndex

PandasIndex(Index(['410088', '410776'], dtype='object', name='station_id'))

Attributes: (3)
long_name :
longitude
units :
degrees_east
axis :
x

In [17]:

Copied!

d.cf
d.cf

Out[17]:

Coordinates:
             CF Axes: * T: ['time']
                        X, Y, Z: n/a

      CF Coordinates: * time: ['time']
                        longitude, latitude, vertical: n/a

       Cell Measures:   area, volume: n/a

      Standard Names: * ens_member: ['ens_member']
                      * lead time: ['lead_time']
                      * time: ['time']

              Bounds:   n/a

       Grid Mappings:   n/a

Data Variables:
       Cell Measures:   area, volume: n/a

      Standard Names:   area: ['area']

              Bounds:   n/a

       Grid Mappings:   n/a

In [12]:

Copied!

d.cf["lon"]
d.cf["lon"]

Out[12]:

<xarray.DataArray 'lon' (station_id: 2)> Size: 16B
nan nan
Coordinates:
  * station_id  (station_id) <U6 48B '410088' '410776'
Attributes:
    long_name:  longitude
    units:      degrees_east
    axis:       x

xarray.DataArray

'lon'

station_id: 2

nan nan
```
array([nan, nan])
```
Coordinates: (1)
- station_id
  (station_id)
  <U6
  '410088' '410776'
  long_name :
  station or node identification code
```
array(['410088', '410776'], dtype='<U6')
```

Indexes: (1)

station_id

PandasIndex

PandasIndex(Index(['410088', '410776'], dtype='object', name='station_id'))

Attributes: (3)
long_name :
longitude
units :
degrees_east
axis :
x

Saving as STF 2.0 compliant¶

Placeholder section

This key feature is under (re)construction. The statemements to do so will be as concise as possible while leaving future options open:

efts_ds = EftsDataset(d)
efts_ds.to_stf(path="/home/user/data/streamflow_forecasts.nc", version="2.0")

Validating compliance¶

The package includes functions to validate netCDF STF 2.0 files before loading

These will also be available from command line interfaces.

In [ ]:

Copied!

Was this page helpful?

Thanks for your feedback!

Let us know how we can improve this page.

Basic use¶

Creating a new STF xarray dataset¶

Climate and Forecast conventions¶

Saving as STF 2.0 compliant¶

Validating compliance¶

Feedback