Design rationale¶

The netCDF STF 2.0 compliant format is such that a file loaded from Python via xarray is not the most convenient data model for users.

This notebook illustrates interactively the behaviors, and informs the design choices made to reconcile the xarray view with the on-disk representation.

Loading an existing reference netCDF file¶

A file was created using (probably) a Matlab implementation of STF data handling and I/O. Let's load it via xarray as well as the netCDF4 package, as we are not sure which will be most adequate for efts-io for saving/loading operations.

In [1]:

Copied!

import xarray as xr
import xarray as xr

In [2]:

Copied!

import netCDF4 as nc
import netCDF4 as nc

xarray.open_dataset has arguments to turn on/off the decoding of climate and forecast (SF) and related conventions.

decode_times=False is a must, otherwise the statement fails. Decoding would work for the time dimension, but decoding lead_time fails.
decode_cf seems to influence at least how the station_name variable appears, notably whether it ends up of dimensions (station, strLen) if True, or (station,) if False.

In [3]:

Copied!

fn = "/home/per202/data/sf/sample/HT_swiftRain_daily_stfv2_2000111523+0000-2023111023+0000.nc"
rain_xr = xr.open_dataset(fn, decode_times=False, decode_cf=False)
fn = "/home/per202/data/sf/sample/HT_swiftRain_daily_stfv2_2000111523+0000-2023111023+0000.nc"
rain_xr = xr.open_dataset(fn, decode_times=False, decode_cf=False)

In [4]:

Copied!

rain_nc = nc.Dataset(fn)
rain_nc = nc.Dataset(fn)

xarray read¶

In [5]:

Copied!

rain_xr
rain_xr

Out[5]:

<xarray.Dataset> Size: 19MB
Dimensions:       (time: 8396, station: 563, lead_time: 1, strLen: 30,
                   ens_member: 1)
Coordinates:
  * time          (time) int32 34kB 1 2 3 4 5 6 ... 8392 8393 8394 8395 8396
  * station       (station) int32 2kB 1 2 3 4 5 6 7 ... 558 559 560 561 562 563
  * lead_time     (lead_time) int32 4B 0
  * ens_member    (ens_member) int32 4B 1
Dimensions without coordinates: strLen
Data variables:
    station_id    (station) int32 2kB ...
    station_name  (station, strLen) |S1 17kB ...
    lat           (station) float32 2kB ...
    lon           (station) float32 2kB ...
    area          (station) float32 2kB ...
    rain_obs      (time, ens_member, station, lead_time) float32 19MB ...
Attributes:
    title:                   Precip from Hydro Tasmania's observation network...
    institution:             CSIRO Land & Water
    source:                  
    catchment:               Hydro Tas
    STF_convention_version:  2.0
    STF_nc_spec:             https://wiki.csiro.au/display/wirada/NetCDF+for+...
    comment:                 
    history:                 2024-07-25 15:28:34 +10.0 - File created

xarray.Dataset

Dimensions:
- time: 8396
- station: 563
- lead_time: 1
- strLen: 30
- ens_member: 1
Coordinates: (4)
- time
  (time)
  int32
  1 2 3 4 5 ... 8393 8394 8395 8396
  standard_name :
  time
  long_name :
  time
  time_standard :
  UTC
  axis :
  t
  units :
  days since 2000-11-14 23:00:00.0 +0000
```
array([   1,    2,    3, ..., 8394, 8395, 8396], dtype=int32)
```
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```
- lead_time
  (lead_time)
  int32
  0
  standard_name :
  lead time
  long_name :
  forecast lead time
  axis :
  v
  units :
  days since time
```
array([0], dtype=int32)
```
- ens_member
  (ens_member)
  int32
  1
  standard_name :
  ens_member
  long_name :
  ensemble member
  units :
  member id
  axis :
  u
```
array([1], dtype=int32)
```
Data variables: (6)
- station_id
  (station)
  int32
  ...
  long_name :
  station or node identification code
```
[563 values with dtype=int32]
```
- station_name
  (station, strLen)
  |S1
  ...
  long_name :
  station or node name
```
[16890 values with dtype=|S1]
```
- lat
  (station)
  float32
  ...
  long_name :
  latitude
  units :
  degrees_north
  axis :
  y
```
[563 values with dtype=float32]
```
- lon
  (station)
  float32
  ...
  long_name :
  longitude
  units :
  degrees_east
  axis :
  x
```
[563 values with dtype=float32]
```
- area
  (station)
  float32
  ...
  units :
  sqkm
  _FillValue :
  -1.0
  standard_name :
  area
  long_name :
  station area
```
[563 values with dtype=float32]
```
- rain_obs
  (time, ens_member, station, lead_time)
  float32
  ...
  standard_name :
  rain_obs
  long_name :
  observed rainfall
  units :
  mm
  _FillValue :
  -9999.0
  type :
  2.0
  type_description :
  accumulated over the preceding interval
  dat_type :
  der
  dat_type_description :
  derived (from observations)
  location_type :
  area
```
[4726948 values with dtype=float32]
```

Indexes: (4)

time

PandasIndex

PandasIndex(Index([   1,    2,    3,    4,    5,    6,    7,    8,    9,   10,
       ...
       8387, 8388, 8389, 8390, 8391, 8392, 8393, 8394, 8395, 8396],
      dtype='int32', name='time', length=8396))

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

lead_time

PandasIndex

PandasIndex(Index([0], dtype='int32', name='lead_time'))

ens_member

PandasIndex

PandasIndex(Index([1], dtype='int32', name='ens_member'))

Attributes: (8)
title :
Precip from Hydro Tasmania's observation network areally averaged with inverse distance squared weighting
institution :
CSIRO Land & Water
source :
catchment :
Hydro Tas
STF_convention_version :
2.0
STF_nc_spec :
https://wiki.csiro.au/display/wirada/NetCDF+for+SWIFT
comment :
history :
2024-07-25 15:28:34 +10.0 - File created

If we use decode_cf=True, we seem to get a one dimensional array of array of bytes, rather than a matrix of bytes (type 'S1'):

In [6]:

Copied!

rain_cfdecode = xr.open_dataset(fn, decode_times=False, decode_cf=True)
rain_cfdecode = xr.open_dataset(fn, decode_times=False, decode_cf=True)

In [7]:

Copied!

rain_cfdecode.station_name
rain_cfdecode.station_name

Out[7]:

<xarray.DataArray 'station_name' (station: 563)> Size: 17kB
[563 values with dtype=|S30]
Coordinates:
  * station  (station) int32 2kB 1 2 3 4 5 6 7 8 ... 557 558 559 560 561 562 563
Attributes:
    long_name:  station or node name

xarray.DataArray

'station_name'

station: 563

...
```
[563 values with dtype=|S30]
```
Coordinates: (1)
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```

Indexes: (1)

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

Attributes: (1)
long_name :
station or node name

In [8]:

Copied!

rain_cfdecode.station_name.values[1]
rain_cfdecode.station_name.values[1]

Out[8]:

np.bytes_(b'19629011')

netCDF4 read¶

In [9]:

Copied!

rain_nc
rain_nc

Out[9]:

<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF3_CLASSIC data model, file format NETCDF3):
    title: Precip from Hydro Tasmania's observation network areally averaged with inverse distance squared weighting
    institution: CSIRO Land & Water
    source: 
    catchment: Hydro Tas
    STF_convention_version: 2.0
    STF_nc_spec: https://wiki.csiro.au/display/wirada/NetCDF+for+SWIFT
    comment: 
    history: 2024-07-25 15:28:34 +10.0 - File created
    dimensions(sizes): time(8396), station(563), lead_time(1), strLen(30), ens_member(1)
    variables(dimensions): int32 time(time), int32 station(station), int32 lead_time(lead_time), int32 station_id(station), |S1 station_name(station, strLen), int32 ens_member(ens_member), float32 lat(station), float32 lon(station), float32 area(station), float32 rain_obs(time, ens_member, station, lead_time)
    groups:

Modulo the value of decode_cf for xarray.open_dataset, the shape of the data in memory appears consistent between xarray and netCDF4

Requirements¶

Desired in-memory representation¶

See this discussion for background.

We assume that an "intuitive" data representation in an xarray dataset would have the following characteristics:

The time coordinate has values with python representations np.datetime64 or similar
A station_id coordinate has values as strings rather than bytes, so that slicing can be done with statements such as data.sel(station_id="407113A"). The STF representation is such that station is a dimension/coordinate, not station_id
In the example case loaded, the variable datatypes is 32-bits np.float32 rather than 64 bits np.float. The latter is probably more convenient in most use cases we can anticipate. However we may want to consider keeping a 32 bits representation: ensemble forecasting and modelling methods can be RAM-hungry even with 2024 typical machine setups.
coordinate data is in type int32. Memory footprint is not a consideration, we may want to change is to 64 bits, or not, based on other factors.
There should be a coordinate named "realisation" (or U.S. "realization"??) rather than "ens_member"

STF 2.0 compliance¶

It is imperative to be able to export the in-memory xarray representation to a netCDF file that complies with documented conventions and is readable by existing toolsets in Matlab, C++ or even Fortran. A key question is whether we can use xarray.to_netcdf or whether we need to use the lower level package netCDF4 to achieve that.

Implementation¶

This notebook will illustrate the various steps taken to bridge the gap between the on-disk and in-memory representations.

Reading from disk¶

In [10]:

Copied!

rain_cfdecode
rain_cfdecode

Out[10]:

<xarray.Dataset> Size: 19MB
Dimensions:       (time: 8396, station: 563, lead_time: 1, ens_member: 1)
Coordinates:
  * time          (time) int32 34kB 1 2 3 4 5 6 ... 8392 8393 8394 8395 8396
  * station       (station) int32 2kB 1 2 3 4 5 6 7 ... 558 559 560 561 562 563
  * lead_time     (lead_time) int32 4B 0
  * ens_member    (ens_member) int32 4B 1
Data variables:
    station_id    (station) int32 2kB ...
    station_name  (station) |S30 17kB b'18594010' b'19629011' ... b'28294677'
    lat           (station) float32 2kB ...
    lon           (station) float32 2kB ...
    area          (station) float32 2kB ...
    rain_obs      (time, ens_member, station, lead_time) float32 19MB ...
Attributes:
    title:                   Precip from Hydro Tasmania's observation network...
    institution:             CSIRO Land & Water
    source:                  
    catchment:               Hydro Tas
    STF_convention_version:  2.0
    STF_nc_spec:             https://wiki.csiro.au/display/wirada/NetCDF+for+...
    comment:                 
    history:                 2024-07-25 15:28:34 +10.0 - File created

xarray.Dataset

Dimensions:
- time: 8396
- station: 563
- lead_time: 1
- ens_member: 1
Coordinates: (4)
- time
  (time)
  int32
  1 2 3 4 5 ... 8393 8394 8395 8396
  standard_name :
  time
  long_name :
  time
  time_standard :
  UTC
  axis :
  t
  units :
  days since 2000-11-14 23:00:00.0 +0000
```
array([   1,    2,    3, ..., 8394, 8395, 8396], dtype=int32)
```
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```
- lead_time
  (lead_time)
  int32
  0
  standard_name :
  lead time
  long_name :
  forecast lead time
  axis :
  v
  units :
  days since time
```
array([0], dtype=int32)
```
- ens_member
  (ens_member)
  int32
  1
  standard_name :
  ens_member
  long_name :
  ensemble member
  units :
  member id
  axis :
  u
```
array([1], dtype=int32)
```
Data variables: (6)
- station_id
  (station)
  int32
  ...
  long_name :
  station or node identification code
```
[563 values with dtype=int32]
```
- station_name
  (station)
  |S30
  b'18594010' ... b'28294677'
  long_name :
  station or node name
```
array([b'18594010', b'19629011', b'18562015', ..., b'28286670', b'28294676',
       b'28294677'], dtype='|S30')
```
- lat
  (station)
  float32
  ...
  long_name :
  latitude
  units :
  degrees_north
  axis :
  y
```
[563 values with dtype=float32]
```
- lon
  (station)
  float32
  ...
  long_name :
  longitude
  units :
  degrees_east
  axis :
  x
```
[563 values with dtype=float32]
```
- area
  (station)
  float32
  ...
  units :
  sqkm
  standard_name :
  area
  long_name :
  station area
```
[563 values with dtype=float32]
```
- rain_obs
  (time, ens_member, station, lead_time)
  float32
  ...
  standard_name :
  rain_obs
  long_name :
  observed rainfall
  units :
  mm
  type :
  2.0
  type_description :
  accumulated over the preceding interval
  dat_type :
  der
  dat_type_description :
  derived (from observations)
  location_type :
  area
```
[4726948 values with dtype=float32]
```

Indexes: (4)

time

PandasIndex

PandasIndex(Index([   1,    2,    3,    4,    5,    6,    7,    8,    9,   10,
       ...
       8387, 8388, 8389, 8390, 8391, 8392, 8393, 8394, 8395, 8396],
      dtype='int32', name='time', length=8396))

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

lead_time

PandasIndex

PandasIndex(Index([0], dtype='int32', name='lead_time'))

ens_member

PandasIndex

PandasIndex(Index([1], dtype='int32', name='ens_member'))

Attributes: (8)
title :
Precip from Hydro Tasmania's observation network areally averaged with inverse distance squared weighting
institution :
CSIRO Land & Water
source :
catchment :
Hydro Tas
STF_convention_version :
2.0
STF_nc_spec :
https://wiki.csiro.au/display/wirada/NetCDF+for+SWIFT
comment :
history :
2024-07-25 15:28:34 +10.0 - File created

time¶

For background in issue https://jira.csiro.au/browse/WIRADA-635. We cannot have xarray automagically decoding this axis, so we need to do the work manually, but using as much as possible work already done. Not sure how I had figured out about CFDatetimeCoder, but:

In [11]:

Copied!

from xarray.coding import times
from xarray.coding import times

In [12]:

Copied!

decod = times.CFDatetimeCoder(use_cftime=True)
decod = times.CFDatetimeCoder(use_cftime=True)

In [13]:

Copied!

decod.decode?
decod.decode?

Signature: decod.decode(variable: 'Variable', name: 'T_Name' = None) -> 'Variable'
Docstring: Convert an decoded variable to a encoded variable
File:      ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/coding/times.py
Type:      method

We need to pass a "Variable", not a DataArray

In [14]:

Copied!

type(rain_cfdecode.coords['time'])
type(rain_cfdecode.coords['time'])

Out[14]:

xarray.core.dataarray.DataArray

In [15]:

Copied!

TIME_DIMNAME="time"
var = xr.as_variable(rain_cfdecode.coords[TIME_DIMNAME])
TIME_DIMNAME="time"
var = xr.as_variable(rain_cfdecode.coords[TIME_DIMNAME])

In [16]:

Copied!

time_zone = var.attrs["time_standard"]
time_coords = decod.decode(var, name=TIME_DIMNAME)
time_zone = var.attrs["time_standard"]
time_coords = decod.decode(var, name=TIME_DIMNAME)

In [17]:

Copied!

time_zone
time_zone

Out[17]:

'UTC'

In [18]:

Copied!

timestamp = time_coords.values[0]
timestamp
timestamp = time_coords.values[0]
timestamp

Out[18]:

cftime.DatetimeGregorian(2000, 11, 15, 23, 0, 0, 0, has_year_zero=False)

Date/time, calendar and time zone handling are a topic of underappreciated complexity, to put it mildly. Let's look at what we get here.

Unfamiliar with this type of time stamp. It seems not to have time zone from the decoding operation, but can have it:

In [19]:

Copied!

timestamp.tzinfo is None
timestamp.tzinfo is None

Out[19]:

True

Should our new time axis hold time zone info with each time stamp, or still rely on the coordinate attribute time_standard?

In [20]:

Copied!

from efts_io.wrapper import cftimes_to_pdtstamps
from efts_io.wrapper import cftimes_to_pdtstamps

In [21]:

Copied!





new_time_values = cftimes_to_pdtstamps(
    time_coords.values,
    time_zone,
)
new_time_values
new_time_values = cftimes_to_pdtstamps(
    time_coords.values,
    time_zone,
)
new_time_values

Out[21]:

array([Timestamp('2000-11-15 23:00:00+0000', tz='UTC'),
       Timestamp('2000-11-16 23:00:00+0000', tz='UTC'),
       Timestamp('2000-11-17 23:00:00+0000', tz='UTC'), ...,
       Timestamp('2023-11-08 23:00:00+0000', tz='UTC'),
       Timestamp('2023-11-09 23:00:00+0000', tz='UTC'),
       Timestamp('2023-11-10 23:00:00+0000', tz='UTC')], dtype=object)

This may be a suitable time axis. Depending on usage needs we may want to revisit though. In particular, users may create "naive" date time stamps from strings: how would ds.sel() then behave if time stamps have time zones??

In [22]:

Copied!

import pandas as pd
pd.Timestamp('2000-11-15 23:00:00+0000')
import pandas as pd
pd.Timestamp('2000-11-15 23:00:00+0000')

Out[22]:

Timestamp('2000-11-15 23:00:00+0000', tz='UTC')

In [23]:

Copied!

pd.Timestamp('2000-11-15 23:00:00+0000') == new_time_values[0]
pd.Timestamp('2000-11-15 23:00:00+0000') == new_time_values[0]

Out[23]:

True

In [24]:

Copied!

pd.Timestamp('2000-11-15 23:00:00')
pd.Timestamp('2000-11-15 23:00:00')

Out[24]:

Timestamp('2000-11-15 23:00:00')

In [25]:

Copied!

pd.Timestamp('2000-11-15 23:00:00') == new_time_values[0]
pd.Timestamp('2000-11-15 23:00:00') == new_time_values[0]

Out[25]:

False

As expected, the naive date time is not equal to the one with a time zone. Using time zone in the time stamps may be a fraught choice in practice. In particular there may be logical but unintuitive if we use a time slice to subset data

station_id¶

In [27]:

Copied!

station_ids = rain_cfdecode.station_id.values
station_ids.dtype
station_ids = rain_cfdecode.station_id.values
station_ids.dtype

Out[27]:

dtype('int32')

In [28]:

Copied!

station_ids[:3]
station_ids[:3]

Out[28]:

array([18594010, 19629011, 18562015], dtype=int32)

STF conventions are such that the station ID can only be an integer. We want a str in the in memory model. This is easy going one direction; when we consider going the other way (writing to STF 2.0) this will be trickier.

In [29]:

Copied!

station_ids_str = [str(x) for x in station_ids]
station_ids_str = [str(x) for x in station_ids]

In [30]:

Copied!

rain_cfdecode.station_id
rain_cfdecode.station_id

Out[30]:

<xarray.DataArray 'station_id' (station: 563)> Size: 2kB
array([18594010, 19629011, 18562015, ..., 28286670, 28294676, 28294677],
      dtype=int32)
Coordinates:
  * station  (station) int32 2kB 1 2 3 4 5 6 7 8 ... 557 558 559 560 561 562 563
Attributes:
    long_name:  station or node identification code

xarray.DataArray

'station_id'

station: 563

18594010 19629011 18562015 18562003 ... 28286670 28294676 28294677

array([18594010, 19629011, 18562015, ..., 28286670, 28294676, 28294677],
      dtype=int32)

Coordinates: (1)
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```

Indexes: (1)

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

Attributes: (1)
long_name :
station or node identification code

In [31]:

Copied!

type(rain_cfdecode.station_id)
type(rain_cfdecode.station_id)

Out[31]:

xarray.core.dataarray.DataArray

In [32]:

Copied!

rain_cfdecode.station
rain_cfdecode.station

Out[32]:

<xarray.DataArray 'station' (station: 563)> Size: 2kB
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
Coordinates:
  * station  (station) int32 2kB 1 2 3 4 5 6 7 8 ... 557 558 559 560 561 562 563

xarray.DataArray

'station'

station: 563

1 2 3 4 5 6 7 8 9 10 11 ... 554 555 556 557 558 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```
Coordinates: (1)
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```

Indexes: (1)

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

Attributes: (0)

In [33]:

Copied!

type(rain_cfdecode.station)
type(rain_cfdecode.station)

Out[33]:

xarray.core.dataarray.DataArray

A key thing here is that we will promote "station_id" which is a variable, to a coordinate, so we cannot just assign dimensions; we will need to reconstruct a new xarray.

station_name¶

In [34]:

Copied!

rain_cfdecode.station_name
rain_cfdecode.station_name

Out[34]:

<xarray.DataArray 'station_name' (station: 563)> Size: 17kB
array([b'18594010', b'19629011', b'18562015', ..., b'28286670', b'28294676',
       b'28294677'], dtype='|S30')
Coordinates:
  * station  (station) int32 2kB 1 2 3 4 5 6 7 8 ... 557 558 559 560 561 562 563
Attributes:
    long_name:  station or node name

xarray.DataArray

'station_name'

station: 563

b'18594010' b'19629011' b'18562015' ... b'28294676' b'28294677'

array([b'18594010', b'19629011', b'18562015', ..., b'28286670', b'28294676',
       b'28294677'], dtype='|S30')

Coordinates: (1)
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```

Indexes: (1)

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

Attributes: (1)
long_name :
station or node name

In [35]:

Copied!

x = b'18594010'
str(x, encoding="UTF-8")
x = b'18594010'
str(x, encoding="UTF-8")

Out[35]:

'18594010'

Using helper functions already included in the package at the time of writing:

In [36]:

Copied!

from efts_io.wrapper import byte_stations_to_str
from efts_io.wrapper import byte_stations_to_str

In [37]:

Copied!

station_name_str = byte_stations_to_str(rain_cfdecode.station_name.values)
station_name_str[:3]
station_name_str = byte_stations_to_str(rain_cfdecode.station_name.values)
station_name_str[:3]

Out[37]:

array(['18594010', '19629011', '18562015'], dtype='<U8')

creating a new dataset¶

The package already includes a function to create high level xarray

In [38]:

Copied!

from efts_io import wrapper as w
from efts_io import wrapper as w

In [39]:

Copied!

rain_cfdecode.ens_member.values
rain_cfdecode.ens_member.values

Out[39]:

array([1], dtype=int32)

In [40]:

Copied!





issue_times = new_time_values
station_ids = station_ids_str
lead_times = rain_cfdecode.lead_time.values
lead_time_tstep = "days"
ensemble_size = len(rain_cfdecode.ens_member.values)
station_names= station_name_str
nc_attributes = None
latitudes = rain_cfdecode.lat.values
longitudes = rain_cfdecode.lon.values
areas = rain_cfdecode.area.values

d = w.xr_efts(
    issue_times,
    station_ids,
    lead_times,
    lead_time_tstep,
    ensemble_size,
    station_names,
    latitudes,
    longitudes,
    areas,
    nc_attributes,
)
issue_times = new_time_values
station_ids = station_ids_str
lead_times = rain_cfdecode.lead_time.values
lead_time_tstep = "days"
ensemble_size = len(rain_cfdecode.ens_member.values)
station_names= station_name_str
nc_attributes = None
latitudes = rain_cfdecode.lat.values
longitudes = rain_cfdecode.lon.values
areas = rain_cfdecode.area.values

d = w.xr_efts(
    issue_times,
    station_ids,
    lead_times,
    lead_time_tstep,
    ensemble_size,
    station_names,
    latitudes,
    longitudes,
    areas,
    nc_attributes,
)

In [41]:

Copied!

d.sizes
d.sizes

Out[41]:

Frozen({'station': 563, 'time': 8396, 'ens_member': 1, 'lead_time': 1})

In [42]:

Copied!

d.sel(station_id="18594010", drop=True)
d.sel(station_id="18594010", drop=True)

Out[42]:

<xarray.Dataset> Size: 67kB
Dimensions:       (time: 8396, ens_member: 1, lead_time: 1)
Coordinates:
  * time          (time) datetime64[ns] 67kB 2000-11-15T23:00:00 ... 2023-11-...
  * ens_member    (ens_member) int64 8B 1
  * lead_time     (lead_time) int32 4B 0
Data variables:
    station_name  <U8 32B '18594010'
    lat           float32 4B -41.63
    lon           float32 4B 146.3
    area          float32 4B 8.743
Attributes:
    title:                   not provided
    institution:             not provided
    catchment:               not provided
    source:                  not provided
    comment:                 not provided
    history:                 not provided
    STF_convention_version:  2.0
    STF_nc_spec:             https://github.com/csiro-hydroinformatics/efts/b...

xarray.Dataset

Dimensions:
- time: 8396
- ens_member: 1
- lead_time: 1
Coordinates: (3)
- time
  (time)
  datetime64[ns]
  2000-11-15T23:00:00 ... 2023-11-...
  standard_name :
  time
  long_name :
  time
  axis :
  t
```
array(['2000-11-15T23:00:00.000000000', '2000-11-16T23:00:00.000000000',
       '2000-11-17T23:00:00.000000000', ..., '2023-11-08T23:00:00.000000000',
       '2023-11-09T23:00:00.000000000', '2023-11-10T23:00:00.000000000'],
      dtype='datetime64[ns]')
```
- ens_member
  (ens_member)
  int64
  1
  standard_name :
  ens_member
  long_name :
  ensemble member
  units :
  member id
  axis :
  u
```
array([1])
```
- lead_time
  (lead_time)
  int32
  0
  standard_name :
  lead time
  long_name :
  forecast lead time
  axis :
  v
  units :
  days since time
```
array([0], dtype=int32)
```
Data variables: (4)
- station_name
  ()
  <U8
  '18594010'
  long_name :
  station or node name
```
array('18594010', dtype='<U8')
```
- lat
  ()
  float32
  -41.63
  long_name :
  latitude
  units :
  degrees_north
  axis :
  y
```
array(-41.631687, dtype=float32)
```
- lon
  ()
  float32
  146.3
  long_name :
  longitude
  units :
  degrees_east
  axis :
  x
```
array(146.25577, dtype=float32)
```
- area
  ()
  float32
  8.743
  long_name :
  station area
  units :
  km^2
  standard_name :
  area
```
array(8.742642, dtype=float32)
```

Indexes: (3)

time

PandasIndex

PandasIndex(DatetimeIndex(['2000-11-15 23:00:00', '2000-11-16 23:00:00',
               '2000-11-17 23:00:00', '2000-11-18 23:00:00',
               '2000-11-19 23:00:00', '2000-11-20 23:00:00',
               '2000-11-21 23:00:00', '2000-11-22 23:00:00',
               '2000-11-23 23:00:00', '2000-11-24 23:00:00',
               ...
               '2023-11-01 23:00:00', '2023-11-02 23:00:00',
               '2023-11-03 23:00:00', '2023-11-04 23:00:00',
               '2023-11-05 23:00:00', '2023-11-06 23:00:00',
               '2023-11-07 23:00:00', '2023-11-08 23:00:00',
               '2023-11-09 23:00:00', '2023-11-10 23:00:00'],
              dtype='datetime64[ns]', name='time', length=8396, freq=None))

ens_member

PandasIndex

PandasIndex(Index([1], dtype='int64', name='ens_member'))

lead_time

PandasIndex

PandasIndex(Index([0], dtype='int32', name='lead_time'))

Attributes: (8)
title :
not provided
institution :
not provided
catchment :
not provided
source :
not provided
comment :
not provided
history :
not provided
STF_convention_version :
2.0
STF_nc_spec :
https://github.com/csiro-hydroinformatics/efts/blob/d7d43a995fb5e459bcb894e09b7bb89de03e285c/docs/netcdf_for_water_forecasting.md

In [43]:

Copied!

set(d.variables.keys())
set(d.variables.keys())

Out[43]:

{'area',
 'ens_member',
 'lat',
 'lead_time',
 'lon',
 'station',
 'station_id',
 'station_name',
 'time'}

In [44]:

Copied!

set(rain_cfdecode.variables.keys())
set(rain_cfdecode.variables.keys())

Out[44]:

{'area',
 'ens_member',
 'lat',
 'lead_time',
 'lon',
 'rain_obs',
 'station',
 'station_id',
 'station_name',
 'time'}

In [45]:

Copied!

rain_cfdecode.rain_obs.dims
rain_cfdecode.rain_obs.dims

Out[45]:

('time', 'ens_member', 'station', 'lead_time')

In [46]:

Copied!

da = rain_cfdecode.rain_obs
da = rain_cfdecode.rain_obs

Assigning the data variable straight is not possible due to the differing names for the coordinate(s) for the station ids: we'd end up with 5 dimensions:

In [47]:

Copied!

d
d

Out[47]:

<xarray.Dataset> Size: 114kB
Dimensions:       (station: 563, time: 8396, ens_member: 1, lead_time: 1)
Coordinates:
  * time          (time) datetime64[ns] 67kB 2000-11-15T23:00:00 ... 2023-11-...
  * station       (station) int64 5kB 1 2 3 4 5 6 7 ... 558 559 560 561 562 563
  * ens_member    (ens_member) int64 8B 1
  * lead_time     (lead_time) int32 4B 0
  * station_id    (station) <U8 18kB '18594010' '19629011' ... '28294677'
Data variables:
    station_name  (station) <U8 18kB '18594010' '19629011' ... '28294677'
    lat           (station) float32 2kB -41.63 -41.72 -41.78 ... -41.82 -41.85
    lon           (station) float32 2kB 146.3 146.4 146.2 ... 145.6 145.6 145.6
    area          (station) float32 2kB 8.743 8.05 24.13 ... 3.353 1.76 4.988
Attributes:
    title:                   not provided
    institution:             not provided
    catchment:               not provided
    source:                  not provided
    comment:                 not provided
    history:                 not provided
    STF_convention_version:  2.0
    STF_nc_spec:             https://github.com/csiro-hydroinformatics/efts/b...

xarray.Dataset

Dimensions:
- station: 563
- time: 8396
- ens_member: 1
- lead_time: 1
Coordinates: (5)
- time
  (time)
  datetime64[ns]
  2000-11-15T23:00:00 ... 2023-11-...
  standard_name :
  time
  long_name :
  time
  axis :
  t
```
array(['2000-11-15T23:00:00.000000000', '2000-11-16T23:00:00.000000000',
       '2000-11-17T23:00:00.000000000', ..., '2023-11-08T23:00:00.000000000',
       '2023-11-09T23:00:00.000000000', '2023-11-10T23:00:00.000000000'],
      dtype='datetime64[ns]')
```
- station
  (station)
  int64
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563])
```
- ens_member
  (ens_member)
  int64
  1
  standard_name :
  ens_member
  long_name :
  ensemble member
  units :
  member id
  axis :
  u
```
array([1])
```
- lead_time
  (lead_time)
  int32
  0
  standard_name :
  lead time
  long_name :
  forecast lead time
  axis :
  v
  units :
  days since time
```
array([0], dtype=int32)
```
- station_id
  (station)
  <U8
  '18594010' ... '28294677'
  long_name :
  station or node identification code
```
array(['18594010', '19629011', '18562015', ..., '28286670', '28294676',
       '28294677'], dtype='<U8')
```

Data variables: (4)

station_name

(station)

<U8

'18594010' ... '28294677'

long_name :: station or node name

array(['18594010', '19629011', '18562015', '18562003', '18562002',
       '18562020', '18562005', '18562007', '18562008', '18562012',
       '18562009', '18562010', '18562021', '18562013', '18562017',
       '18562014', '18562018', '18594027', '18594028', '18594029',
       '18594030', '18594031', '18594033', '18594034', '19631035',
       '18594036', '18594038', '18594040', '19629041', '19629042',
       '19629043', '19629044', '19629053', '19629046', '19629048',
       '19162011', '19629052', '18562019', '18562006', '18562001',
       '18562016', '27657058', '27657059', '27657060', '27657061',
       '27657062', '27657066', '27657067', '27657071', '27657072',
       '27656075', '27656077', '27656078', '27656079', '27656080',
       '27656081', '27656082', '26415015', '27742098', '27742099',
       '27742109', '27742101', '27742102', '27742103', '27742104',
       '27742105', '27742106', '27742108', '28286117', '28294118',
       '29370120', '11625124', '23759025', '22122130', '21211002',
       '10179003', '10170002', '10128137', '10190954', '10573143',
       '28286002', '29370001', '28294004', '10128001', '18594001',
       '11627001', '27657001', '11625001', '11613001', '11604001',
       '10170001', '10553001', '10580001', '10579001', '10573001',
       '10190001', '10592001', '27658001', '19629001', '19631001',
...
       '10179004', '10179005', '10179006', '10179007', '10179008',
       '10179010', '12001641', '24553002', '24553008', '24553004',
       '24553005', '24553006', '24553007', '10565009', '16799037',
       '18562004', '18562011', '19629045', '19661548', '19631549',
       '19006321', '10001231', '10190955', '26415018', '10565006',
       '10565053', '10565054', '10565055', '10565056', '10565057',
       '10565059', '10565058', '10565014', '32139002', '10167002',
       '10167003', '10176002', '10168002', '10169002', '10580142',
       '28294001', '32139001', '10169001', '10179001', '10168001',
       '10176001', '10167001', '11613604', '28294678', '15189002',
       '10128893', '10573992', '10573999', '10565003', '10565012',
       '10565023', '10565027', '10565034', '13155108', '13155109',
       '31418007', '31418010', '31418004', '10179009', '12158735',
       '30236015', '10565060', '10004151', '26415019', '26001801',
       '12158747', '16249123', '16249122', '10190945', '10190946',
       '10579916', '10007591', '10001711', '23171017', '23171015',
       '22122918', '22002111', '23004971', '23171012', '10579921',
       '12158740', '12004891', '12003011', '32005971', '32139003',
       '32139004', '10005071', '10000541', '10008261', '10565028',
       '28286670', '28294676', '28294677'], dtype='<U8')

lat

(station)

float32

-41.63 -41.72 ... -41.82 -41.85

long_name :: latitude
units :: degrees_north
axis :: y

array([-41.631687, -41.7165  , -41.776894, -41.93974 , -41.89406 ,
       -41.91518 , -41.913536, -41.860268, -41.802822, -41.85331 ,
       -41.881172, -41.86824 , -41.83606 , -41.843746, -41.799908,
       -41.80077 , -41.77146 , -41.781002, -41.750923, -41.72675 ,
       -41.712585, -41.697426, -41.74191 , -41.712852, -41.69828 ,
       -41.70118 , -41.674202, -41.648064, -41.749176, -41.74841 ,
       -41.72291 , -41.719963, -41.727497, -41.726566, -41.68101 ,
       -41.696434, -41.715874, -41.752   , -41.89081 , -41.775448,
       -41.80732 , -41.902855, -42.01951 , -41.968296, -41.9471  ,
       -41.851048, -41.770805, -41.8138  , -41.91512 , -41.848503,
       -41.77807 , -41.692287, -41.690037, -41.573517, -41.594967,
       -41.583652, -41.64903 , -41.92355 , -41.60278 , -41.528053,
       -41.537727, -41.545124, -41.547276, -41.587067, -41.625458,
       -41.675533, -41.668636, -41.711624, -41.866405, -41.932465,
       -42.002964, -41.564877, -42.082386, -42.086395, -42.174744,
       -42.22117 , -42.275208, -42.11975 , -42.397804, -42.605015,
       -41.87931 , -41.991184, -41.914623, -42.22994 , -41.650127,
       -41.30872 , -41.833553, -41.525406, -41.47772 , -41.38934 ,
       -42.2766  , -42.30613 , -42.49147 , -42.311035, -42.510754,
       -42.394913, -42.437775, -41.736443, -41.675716, -41.65877 ,
...
       -42.25616 , -42.27091 , -42.239876, -42.209545, -42.187366,
       -42.165   , -41.5289  , -42.252934, -42.28992 , -42.256786,
       -42.24237 , -42.22594 , -42.198124, -42.476032, -42.119785,
       -41.916996, -41.83074 , -41.737198, -41.633583, -41.656677,
       -41.657032, -42.39504 , -42.385357, -42.042328, -42.295185,
       -42.304424, -42.31551 , -42.36995 , -42.342033, -42.422092,
       -42.388145, -42.260147, -42.35363 , -41.81894 , -42.20476 ,
       -42.177784, -42.119568, -42.233315, -42.279427, -42.46545 ,
       -41.887608, -41.883717, -42.266857, -42.26225 , -42.23631 ,
       -42.15651 , -42.194283, -41.50659 , -41.878647, -41.726135,
       -42.19374 , -42.525448, -42.50113 , -42.033215, -42.172466,
       -42.53875 , -42.44366 , -42.149757, -41.83098 , -41.86145 ,
       -42.021538, -41.967735, -41.86054 , -42.137127, -41.470856,
       -41.97039 , -42.085102, -42.028996, -42.06697 , -42.016922,
       -41.554066, -42.768085, -42.732998, -42.347584, -42.366074,
       -42.1319  , -42.148197, -42.110012, -42.07383 , -42.018982,
       -42.165733, -42.181515, -42.036633, -42.033264, -42.24517 ,
       -41.688072, -41.701202, -41.7367  , -41.79801 , -41.885498,
       -41.790764, -42.27026 , -42.391933, -42.481453, -42.48671 ,
       -41.84537 , -41.818226, -41.85182 ], dtype=float32)

lon

(station)

float32

146.3 146.4 146.2 ... 145.6 145.6

long_name :: longitude
units :: degrees_east
axis :: x

array([146.25577, 146.4181 , 146.19212, 146.15002, 146.2372 , 146.199  ,
       146.1202 , 146.09334, 146.09958, 146.26053, 146.15012, 146.1904 ,
       146.14641, 146.23631, 146.31026, 146.25847, 146.26228, 146.13474,
       146.14459, 146.15536, 146.21718, 146.17564, 146.30244, 146.2579 ,
       146.35551, 146.30156, 146.34909, 146.28381, 146.3831 , 146.4212 ,
       146.34563, 146.35983, 146.38065, 146.45474, 146.41135, 146.38455,
       146.39626, 146.23596, 146.06766, 146.21133, 146.22327, 145.99011,
       145.96811, 145.97226, 145.88765, 145.88309, 145.8686 , 145.90535,
       145.79802, 145.63277, 145.73546, 145.88545, 145.80687, 145.87782,
       145.81328, 145.74689, 145.76532, 146.59389, 145.62363, 145.66489,
       145.54094, 145.48001, 145.4125 , 145.32181, 145.46033, 145.3697 ,
       145.2993 , 145.46347, 145.52838, 145.61986, 145.5864 , 146.10654,
       146.54851, 146.31183, 146.29903, 146.61267, 146.47269, 146.23853,
       146.51338, 146.56734, 145.55075, 145.58858, 145.56082, 146.23091,
       146.22769, 146.24933, 145.69202, 146.12682, 146.06894, 146.2166 ,
       146.46828, 146.41429, 146.62735, 146.46555, 146.67267, 146.5115 ,
       146.56155, 145.58855, 146.39082, 146.33511, 146.43759, 145.71982,
       145.6793 , 145.77908, 145.63249, 145.60204, 145.6054 , 145.56114,
       145.5198 , 145.54143, 145.46031, 145.34674, 145.28966, 145.28944,
       145.26875, 145.57275, 145.63557, 145.67822, 146.39017, 146.38783,
...
       147.69923, 147.90083, 147.96477, 148.14348, 148.02835, 147.90723,
       147.77087, 147.8108 , 147.97171, 147.72343, 146.30946, 146.21602,
       147.39684, 146.37997, 146.60793, 146.56194, 146.6224 , 146.5918 ,
       146.56328, 146.53462, 146.5808 , 147.00911, 146.28668, 146.38503,
       146.37201, 146.32549, 146.36899, 146.37764, 146.85728, 145.87483,
       146.1643 , 146.18391, 146.39412, 146.3155 , 146.31705, 146.33295,
       146.48418, 146.49727, 146.6232 , 147.00233, 147.02397, 146.99678,
       146.99535, 146.98616, 146.90967, 146.96208, 146.80684, 147.10713,
       146.78923, 146.47   , 146.50873, 146.62814, 146.51836, 146.5004 ,
       146.62518, 145.61536, 146.74213, 146.49141, 146.59415, 146.50864,
       146.64227, 146.48729, 146.05139, 145.5999 , 146.58469, 146.26138,
       146.66893, 146.65904, 146.8501 , 147.04012, 146.8955 , 146.64272,
       146.95453, 146.4743 , 146.53001, 146.94609, 146.98328, 146.91466,
       146.5447 , 147.07216, 146.55539, 146.69072, 146.68694, 146.65994,
       146.61472, 146.74748, 145.84705, 145.85425, 146.44633, 146.42802,
       146.48172, 146.49844, 146.49208, 146.48514, 146.47159, 146.37614,
       146.34831, 146.42378, 146.44463, 146.4135 , 147.09398, 147.09981,
       147.1228 , 146.67813, 146.67542, 146.65483, 146.87567, 147.00366,
       146.72874, 146.671  , 145.57736, 145.61763, 145.59996],
      dtype=float32)

area

(station)

float32

8.743 8.05 24.13 ... 1.76 4.988

long_name :: station area
units :: km^2
standard_name :: area

array([8.74264240e+00, 8.04953766e+00, 2.41292725e+01, 1.15444698e+01,
       1.48703051e+01, 1.13882227e+01, 2.03471794e+01, 2.85696564e+01,
       2.99010849e+01, 1.07939901e+01, 2.55833683e+01, 1.69574585e+01,
       1.20487185e+01, 2.63328037e+01, 2.36108913e+01, 6.97706556e+00,
       2.22069759e+01, 3.02010384e+01, 1.95261497e+01, 1.47493753e+01,
       1.03170156e+01, 2.31804771e+01, 2.01167450e+01, 2.09635601e+01,
       1.19352131e+01, 1.67128429e+01, 9.30091000e+00, 1.80477524e+01,
       6.29598665e+00, 7.82678986e+00, 3.79344845e+00, 4.03340101e+00,
       2.13854051e+00, 1.14826517e+01, 6.36217165e+00, 3.22781444e+00,
       8.10380077e+00, 1.10916786e+01, 1.11773901e+01, 8.82636738e+00,
       1.02675972e+01, 7.38301239e+01, 5.59130745e+01, 6.63101425e+01,
       5.00592957e+01, 5.95034599e+01, 9.46557770e+01, 4.62686958e+01,
       6.02089233e+01, 2.90027828e+01, 8.04257355e+01, 7.60305939e+01,
       7.20957108e+01, 3.77609901e+01, 6.03701935e+01, 4.43344460e+01,
       2.97676086e+01, 4.08927917e+01, 1.33005737e+02, 7.74885178e+01,
       9.13805389e+01, 5.68068733e+01, 6.57058792e+01, 4.17661247e+01,
       6.57868576e+01, 1.34466064e+02, 3.27444420e+01, 9.36494980e+01,
       7.36653709e+00, 7.15111685e+00, 4.17642927e+00, 1.22620497e+01,
       4.99790192e+01, 2.93087673e+01, 2.61270180e+01, 2.71608868e+01,
       2.65038657e+00, 6.68782349e+01, 8.03473949e+00, 3.01680851e+01,
...
       3.27399230e+00, 5.20085357e-02, 1.30760312e+00, 3.78825784e+00,
       1.91876564e+01, 9.14414883e+00, 4.55969667e+00, 2.69088554e+00,
       2.89496613e+01, 1.54010344e+01, 1.53724031e+01, 6.27398777e+00,
       1.91951656e+01, 3.99174919e+01, 1.82449081e+02, 2.85572529e+00,
       1.27389126e+01, 6.38944588e+01, 1.40638018e+01, 6.24922705e+00,
       1.58865681e+01, 3.07296109e+00, 1.72298004e+02, 3.98486948e+00,
       6.46440792e+00, 6.30638075e+00, 4.01565971e+01, 5.16905355e+00,
       1.60983715e+01, 7.91153669e+00, 1.53524078e+02, 2.27602577e+01,
       1.12519426e+01, 3.58527327e+00, 4.85922852e+01, 5.92527885e+01,
       4.52961235e+01, 1.80442352e+01, 6.32809563e+01, 3.66307220e+01,
       2.05543995e+00, 3.26414299e+00, 6.10807133e+00, 4.30361710e+01,
       1.78973427e+01, 4.45870514e+01, 5.17808266e+01, 1.05494299e+01,
       3.23782043e+01, 1.57228680e+01, 6.67126608e+00, 3.32986908e+01,
       1.70114079e+01, 1.83375323e+00, 2.42743516e+00, 4.89723778e+00,
       9.67607689e+00, 7.38988495e+00, 1.44319224e+00, 8.16402912e+00,
       2.63495426e+01, 5.74557781e+00, 5.29048777e+00, 3.75785398e+00,
       3.47913098e+00, 1.40595894e+01, 2.20178819e+00, 1.72933400e+00,
       9.82884026e+00, 9.44268715e-04, 5.00415192e+01, 1.87282810e+01,
       1.26958704e+01, 9.27362633e+00, 1.88401937e+00, 7.25890493e+00,
       3.35341191e+00, 1.76042402e+00, 4.98821402e+00], dtype=float32)

Indexes: (5)

time

PandasIndex

PandasIndex(DatetimeIndex(['2000-11-15 23:00:00', '2000-11-16 23:00:00',
               '2000-11-17 23:00:00', '2000-11-18 23:00:00',
               '2000-11-19 23:00:00', '2000-11-20 23:00:00',
               '2000-11-21 23:00:00', '2000-11-22 23:00:00',
               '2000-11-23 23:00:00', '2000-11-24 23:00:00',
               ...
               '2023-11-01 23:00:00', '2023-11-02 23:00:00',
               '2023-11-03 23:00:00', '2023-11-04 23:00:00',
               '2023-11-05 23:00:00', '2023-11-06 23:00:00',
               '2023-11-07 23:00:00', '2023-11-08 23:00:00',
               '2023-11-09 23:00:00', '2023-11-10 23:00:00'],
              dtype='datetime64[ns]', name='time', length=8396, freq=None))

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int64', name='station', length=563))

ens_member

PandasIndex

PandasIndex(Index([1], dtype='int64', name='ens_member'))

lead_time

PandasIndex

PandasIndex(Index([0], dtype='int32', name='lead_time'))

station_id

PandasIndex

PandasIndex(Index(['18594010', '19629011', '18562015', '18562003', '18562002', '18562020',
       '18562005', '18562007', '18562008', '18562012',
       ...
       '32005971', '32139003', '32139004', '10005071', '10000541', '10008261',
       '10565028', '28286670', '28294676', '28294677'],
      dtype='object', name='station_id', length=563))

Attributes: (8)
title :
not provided
institution :
not provided
catchment :
not provided
source :
not provided
comment :
not provided
history :
not provided
STF_convention_version :
2.0
STF_nc_spec :
https://github.com/csiro-hydroinformatics/efts/blob/d7d43a995fb5e459bcb894e09b7bb89de03e285c/docs/netcdf_for_water_forecasting.md

In [48]:

Copied!

d_tmp = d.copy()
d_tmp = d.copy()

In [49]:

Copied!

d_tmp.station
d_tmp.station

Out[49]:

<xarray.DataArray 'station' (station: 563)> Size: 5kB
array([  1,   2,   3, ..., 561, 562, 563])
Coordinates:
  * station     (station) int64 5kB 1 2 3 4 5 6 7 ... 558 559 560 561 562 563
  * station_id  (station) <U8 18kB '18594010' '19629011' ... '28294677'

xarray.DataArray

'station'

station: 563

1 2 3 4 5 6 7 8 9 10 11 ... 554 555 556 557 558 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563])
```
Coordinates: (2)
- station
  (station)
  int64
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563])
```
- station_id
  (station)
  <U8
  '18594010' ... '28294677'
  long_name :
  station or node identification code
```
array(['18594010', '19629011', '18562015', ..., '28286670', '28294676',
       '28294677'], dtype='<U8')
```

Indexes: (2)

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int64', name='station', length=563))

station_id

PandasIndex

PandasIndex(Index(['18594010', '19629011', '18562015', '18562003', '18562002', '18562020',
       '18562005', '18562007', '18562008', '18562012',
       ...
       '32005971', '32139003', '32139004', '10005071', '10000541', '10008261',
       '10565028', '28286670', '28294676', '28294677'],
      dtype='object', name='station_id', length=563))

Attributes: (0)

In [50]:

Copied!

da.station
da.station

Out[50]:

<xarray.DataArray 'station' (station: 563)> Size: 2kB
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
Coordinates:
  * station  (station) int32 2kB 1 2 3 4 5 6 7 8 ... 557 558 559 560 561 562 563

xarray.DataArray

'station'

station: 563

1 2 3 4 5 6 7 8 9 10 11 ... 554 555 556 557 558 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```
Coordinates: (1)
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```

Indexes: (1)

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

Attributes: (0)

In [51]:

Copied!

da = da.assign_coords(d_tmp.station.coords)
da = da.assign_coords(d_tmp.station.coords)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[51], line 1
----> 1 da = da.assign_coords(d_tmp.station.coords)

File ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/core/common.py:645, in DataWithCoords.assign_coords(self, coords, **coords_kwargs)
    642 else:
    643     results = self._calc_assign_results(coords_combined)
--> 645 data.coords.update(results)
    646 return data

File ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/core/coordinates.py:554, in Coordinates.update(self, other)
    546 # Discard original indexed coordinates prior to merge allows to:
    547 # - fail early if the new coordinates don't preserve the integrity of existing
    548 #   multi-coordinate indexes
    549 # - drop & replace coordinates without alignment (note: we must keep indexed
    550 #   coordinates extracted from the DataArray objects passed as values to
    551 #   `other` - if any - as those are still used for aligning the old/new coordinates)
    552 coords_to_align = drop_indexed_coords(set(other_coords) & set(other), self)
--> 554 coords, indexes = merge_coords(
    555     [coords_to_align, other_coords],
    556     priority_arg=1,
    557     indexes=coords_to_align.xindexes,
    558 )
    560 # special case for PandasMultiIndex: updating only its dimension coordinate
    561 # is still allowed but depreciated.
    562 # It is the only case where we need to actually drop coordinates here (multi-index levels)
    563 # TODO: remove when removing PandasMultiIndex's dimension coordinate.
    564 self._drop_coords(self._names - coords_to_align._names)

File ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/core/merge.py:556, in merge_coords(objects, compat, join, priority_arg, indexes, fill_value)
    554 _assert_compat_valid(compat)
    555 coerced = coerce_pandas_values(objects)
--> 556 aligned = deep_align(
    557     coerced, join=join, copy=False, indexes=indexes, fill_value=fill_value
    558 )
    559 collected = collect_variables_and_indexes(aligned, indexes=indexes)
    560 prioritized = _get_priority_vars_and_indexes(aligned, priority_arg, compat=compat)

File ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/core/alignment.py:946, in deep_align(objects, join, copy, indexes, exclude, raise_on_invalid, fill_value)
    943     else:
    944         out.append(variables)
--> 946 aligned = align(
    947     *targets,
    948     join=join,
    949     copy=copy,
    950     indexes=indexes,
    951     exclude=exclude,
    952     fill_value=fill_value,
    953 )
    955 for position, key, aligned_obj in zip(positions, keys, aligned):
    956     if key is no_key:

File ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/core/alignment.py:882, in align(join, copy, indexes, exclude, fill_value, *objects)
    686 """
    687 Given any number of Dataset and/or DataArray objects, returns new
    688 objects with aligned indexes and dimension sizes.
   (...)
    872 
    873 """
    874 aligner = Aligner(
    875     objects,
    876     join=join,
   (...)
    880     fill_value=fill_value,
    881 )
--> 882 aligner.align()
    883 return aligner.results

File ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/core/alignment.py:573, in Aligner.align(self)
    571 self.find_matching_indexes()
    572 self.find_matching_unindexed_dims()
--> 573 self.assert_no_index_conflict()
    574 self.align_indexes()
    575 self.assert_unindexed_dim_sizes_equal()

File ~/src/efts-io/.venv/lib/python3.12/site-packages/xarray/core/alignment.py:318, in Aligner.assert_no_index_conflict(self)
    314 if dup:
    315     items_msg = ", ".join(
    316         f"{k!r} ({v} conflicting indexes)" for k, v in dup.items()
    317     )
--> 318     raise ValueError(
    319         "cannot re-index or align objects with conflicting indexes found for "
    320         f"the following {msg}: {items_msg}\n"
    321         "Conflicting indexes may occur when\n"
    322         "- they relate to different sets of coordinate and/or dimension names\n"
    323         "- they don't have the same type\n"
    324         "- they may be used to reindex data along common dimensions"
    325     )

ValueError: cannot re-index or align objects with conflicting indexes found for the following dimensions: 'station' (2 conflicting indexes)
Conflicting indexes may occur when
- they relate to different sets of coordinate and/or dimension names
- they don't have the same type
- they may be used to reindex data along common dimensions

In [ ]:

Copied!

d_tmp['rain_obs'] = da
d_tmp['rain_obs'] = da

In [ ]:

Copied!

d_tmp
d_tmp

There is a DataArray.rename method to rename coordinates, but since we also have a change of values for the station and station_id coordinates, we need to do more work anyway.

In [52]:

Copied!





# make sure we manipulate the 4D dataset: do not assume a certain order in the dimensions:
coordinates_mapping = {
    "time": "time",
    "station": "station_id",
    "ens_member": "ens_member",
    "lead_time": "lead_time",
}
list(coordinates_mapping.keys())
# make sure we manipulate the 4D dataset: do not assume a certain order in the dimensions:
coordinates_mapping = {
    "time": "time",
    "station": "station_id",
    "ens_member": "ens_member",
    "lead_time": "lead_time",
}
list(coordinates_mapping.keys())

Out[52]:

['time', 'station', 'ens_member', 'lead_time']

In [ ]:

Copied!

In [53]:

Copied!

rain_obs = rain_cfdecode.rain_obs
rain_obs
rain_obs = rain_cfdecode.rain_obs
rain_obs

Out[53]:

<xarray.DataArray 'rain_obs' (time: 8396, ens_member: 1, station: 563,
                              lead_time: 1)> Size: 19MB
[4726948 values with dtype=float32]
Coordinates:
  * time        (time) int32 34kB 1 2 3 4 5 6 ... 8391 8392 8393 8394 8395 8396
  * station     (station) int32 2kB 1 2 3 4 5 6 7 ... 558 559 560 561 562 563
  * lead_time   (lead_time) int32 4B 0
  * ens_member  (ens_member) int32 4B 1
Attributes:
    standard_name:         rain_obs
    long_name:             observed rainfall
    units:                 mm
    type:                  2.0
    type_description:      accumulated over the preceding interval
    dat_type:              der
    dat_type_description:  derived (from observations)
    location_type:         area

xarray.DataArray

'rain_obs'

time: 8396
ens_member: 1
station: 563
lead_time: 1

...
```
[4726948 values with dtype=float32]
```
Coordinates: (4)
- time
  (time)
  int32
  1 2 3 4 5 ... 8393 8394 8395 8396
  standard_name :
  time
  long_name :
  time
  time_standard :
  UTC
  axis :
  t
  units :
  days since 2000-11-14 23:00:00.0 +0000
```
array([   1,    2,    3, ..., 8394, 8395, 8396], dtype=int32)
```
- station
  (station)
  int32
  1 2 3 4 5 6 ... 559 560 561 562 563
```
array([  1,   2,   3, ..., 561, 562, 563], dtype=int32)
```
- lead_time
  (lead_time)
  int32
  0
  standard_name :
  lead time
  long_name :
  forecast lead time
  axis :
  v
  units :
  days since time
```
array([0], dtype=int32)
```
- ens_member
  (ens_member)
  int32
  1
  standard_name :
  ens_member
  long_name :
  ensemble member
  units :
  member id
  axis :
  u
```
array([1], dtype=int32)
```

Indexes: (4)

time

PandasIndex

PandasIndex(Index([   1,    2,    3,    4,    5,    6,    7,    8,    9,   10,
       ...
       8387, 8388, 8389, 8390, 8391, 8392, 8393, 8394, 8395, 8396],
      dtype='int32', name='time', length=8396))

station

PandasIndex

PandasIndex(Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
       ...
       554, 555, 556, 557, 558, 559, 560, 561, 562, 563],
      dtype='int32', name='station', length=563))

lead_time

PandasIndex

PandasIndex(Index([0], dtype='int32', name='lead_time'))

ens_member

PandasIndex

PandasIndex(Index([1], dtype='int32', name='ens_member'))

Attributes: (8)
standard_name :
rain_obs
long_name :
observed rainfall
units :
mm
type :
2.0
type_description :
accumulated over the preceding interval
dat_type :
der
dat_type_description :
derived (from observations)
location_type :
area

In [54]:

Copied!

d.station_id.attrs
d.station_id.attrs

Out[54]:

{'long_name': 'station or node identification code'}

time axis¶

In [55]:

Copied!

axis = "hours since 2010-08-01 13:00:00 +0000"
axis = "hours since 2010-08-01 13:00:00 +0000"

In [56]:

Copied!

decod.decode(
decod.decode(

  Cell In[56], line 1
    decod.decode(
                 ^
SyntaxError: incomplete input

In [ ]:

Copied!

import cftime
import cftime

In [ ]:

Copied!

cftime.time2index
cftime.time2index

Was this page helpful?

Thanks for your feedback!

Let us know how we can improve this page.

Design rationale¶

Loading an existing reference netCDF file¶

xarray read¶

netCDF4 read¶

Requirements¶

Desired in-memory representation¶

STF 2.0 compliance¶

Implementation¶

Reading from disk¶

time¶

station_id¶

station_name¶

creating a new dataset¶

time axis¶

Feedback