xarray to and from Iris #1750

duncanwp · 2017-11-30T22:04:46Z

Closes An iris cube converter? #621 and Functions for converting DataArrays to and from iris.Cubes #37 - finishes xarray to and from iris #814
Tests added / passed
Passes git diff upstream/master **/*py | flake8 --diff
Fully documented, including whats-new.rst for all changes and api.rst for new API

…s.cube.Cube objects. Uses the same template as the cdms2 conversion.

…ributes

…rd and DimCoord correctly so 2d coords will work. Use dims variable to convert dimension numbers into names. Add 2d coord to test DataArray.

# Conflicts: # .travis.yml

# Conflicts: # ci/requirements-py27-cdat+iris+pynio.yml

# Conflicts: # xarray/test/test_dataarray.py

…convert

jhamman

We also should work this into the documentation somewhere. I'm not sure if the I/O page is the right place to mention this but it may be.

jhamman · 2017-11-30T23:26:21Z

xarray/convert.py

+    except ImportError:
+        # prior to v1.10
+        from iris.fileformats._pyke_rules.compiled_krb.fc_rules_cf_fc \
+            import _parse_cell_methods as parse_cell_methods


would it be wroth using importlib to clean this up?

I wonder if it's worth just removing this catch completeley actually and requiring iris > 1.10. It's at 1.13 now and nearly 2.0... @pelson?

jhamman · 2017-11-30T23:27:46Z

doc/whats-new.rst

@@ -23,7 +23,11 @@ Enhancements

 - :py:func:`~plot.contourf()` learned to contour 2D variables that have both a 1D co-ordinate (e.g. time) and a 2D co-ordinate (e.g. depth as a function of time).
  By `Deepak Cherian <https://github.com/dcherian>`_.
+- Added :py:meth:`DataArray.to_iris <xray.DataArray.to_iris>` for
+  converting a data array into an Iris_ Cube with the same data and coordinates (:issue:`621` and :issue:`37`).
+  By `Neil Parley <https://github.com/nparley>`_.


Also note the from_iris method.

jhamman · 2017-11-30T23:27:59Z

doc/api.rst

@@ -446,6 +446,7 @@ DataArray methods
   DataArray.to_index
   DataArray.to_masked_array
   DataArray.to_cdms2
+   DataArray.to_iris


also note the from_iris method

shoyer · 2017-12-01T02:55:41Z

@pelson any thoughts here?

I think this may be a reasonable place to start, though it is certainly not leveraging Iris's full metadata decoding capabilities.

It would also be nice to pass dask arrays back and forth, but that will probably need to wait until after Iris 2.0 and #1372 is solved.

duncanwp · 2017-12-01T08:33:50Z

I think this may be a reasonable place to start, though it is certainly not leveraging Iris's full metadata decoding capabilities.

It seams like a decent starting point, were there specific decodings you had in mind?

It would also be nice to pass dask arrays back and forth, but that will probably need to wait until after Iris 2.0 and #1372 is solved.

Agreed.

Do you have a feel for if/where else this should be documented? I'm not sure if really fits in I/O...

fmaussion · 2017-12-01T13:05:59Z

Do you have a feel for if/where else this should be documented? I'm not sure if really fits in I/O...

I think it's OK in I/O (for now). This is also the place where we document serialization in pickle, and the from_dict methods.

shoyer · 2017-12-01T17:38:42Z

I think it's OK in I/O (for now). This is also the place where we document serialization in pickle, and the from_dict methods.

Agreed. In the long term, the IO docs have gotten pretty long, it could make sense to split them up into several subpages.

…thods interface (this is already pretty old now)

shoyer · 2017-12-05T23:44:00Z

doc/io.rst

+----
+
+The Iris_ tool allows easy reading of common meteorological and climate model formats
+(including GRIB and UK MetOffice PP files) into ``Cube``s which are in many ways very


Write "Cube objects". Sphinx won't render Cubes properly.

shoyer · 2017-12-05T23:44:11Z

doc/io.rst

+The Iris_ tool allows easy reading of common meteorological and climate model formats
+(including GRIB and UK MetOffice PP files) into ``Cube``s which are in many ways very
+similar to to ``DataArray``s, while enforcing a CF-compliant data model. If iris is
+installaed xarray can convert a ``Cube`` into a ``DataArray`` using


installaed -> installed

shoyer · 2017-12-05T23:44:37Z

doc/io.rst

+    da_cube
+
+
+Conversly, We can create a new cube object from a ``DataArray`` using


shoyer · 2017-12-05T23:44:45Z

doc/io.rst

+
+
+Conversly, We can create a new cube object from a ``DataArray`` using
+:py:meth:`~xarray.Dataset.from_dict`:


shoyer · 2017-12-05T23:48:15Z

xarray/convert.py

+    if 'cell_methods' in dataarray.attrs:
+        args['cell_methods'] = parse_cell_methods(dataarray.attrs['cell_methods'])
+
+    cube = iris.cube.Cube(dataarray.to_masked_array(), **args)


Rather than loading every dataset into memory, can we raise NotImplementedError for now if a DataArray's data is a dask array? isinstance(dataarray.dask, dask_array_type) should do it, where dask_array_type is imported from xarray.core.pycompat.

Raising NotImplementedError seems a bit drastic, especially since Iris 2.0 is just around the corner. The latest commit lets Iris decide what to do with the array - it keeps it as a dask array if your version of Iris supports it but will read it into memory otherwise.

Otherwise we have to work out which version of Iris we're converting to/from which would be a pain... If you're happy with this approach I can add another set of tests for the various dask/numpy Cube/DataArray permutations.

OK, if it's easy enough to add support for dask, then that is great!

shoyer · 2017-12-05T23:49:29Z

xarray/convert.py

+        coord_attrs = _iris_obj_to_attrs(coord)
+        coord_dims = [dims[i] for i in cube.coord_dims(coord)]
+        if not coord.var_name:
+            raise ValueError('Coordinate has no var_name')


Can we print the offending coordinate in the error message? That makes things easier to debug.

shoyer · 2017-12-05T23:52:40Z

xarray/convert.py

+        try:
+            dim_coord = cube.coord(dim_coords=True, dimensions=(i,))
+            dims.append(dim_coord.var_name)
+        except iris.exceptions.CoordinateNotFoundError:


This line doesn't have an test coverage:
https://coveralls.io/builds/14513466/source?filename=xarray%2Fconvert.py

shoyer · 2017-12-05T23:52:51Z

xarray/convert.py

+        coord_attrs = _iris_obj_to_attrs(coord)
+        coord_dims = [dims[i] for i in cube.coord_dims(coord)]
+        if not coord.var_name:
+            raise ValueError('Coordinate has no var_name')


This line also need coverage in a test.

…sion of dask arrays which may or may not be masked.

…onvert # Conflicts: # ci/requirements-py27-cdat+iris+pynio.yml

duncanwp · 2017-12-18T11:22:24Z

OK, here's another go. It's not pretty, but it seems to work. Once the various issues referenced above are solved it should clean it up, but I'd rather not wait for those since this has been in the works a while now.

fmaussion · 2017-12-18T14:04:54Z

doc/io.rst

+
+The Iris_ tool allows easy reading of common meteorological and climate model formats
+(including GRIB and UK MetOffice PP files) into ``Cube`` objects which are in many ways very
+similar to to ``DataArray``s, while enforcing a CF-compliant data model. If iris is


double "to"

fmaussion · 2017-12-18T14:09:24Z

doc/io.rst

+
+    da_cube = xr.Dataset.from_iris(cube)
+    da_cube
+


Are these examples working? If you'd like this code to run at build time you'll have to add iris to the doc build environment and define the cube variable first.

Alternatively, you can use the :verbatim: environment (see OPeNDAP example below) which is static and therefore easier (and more error prone on the long term, though)

fmaussion

Thanks a lot! I just went over the doc section and have a couple of small comments.

duncanwp · 2017-12-18T15:04:53Z

Great - thanks!

shoyer · 2017-12-18T16:35:06Z

dask/dask#2977 needs to be resolved to make it possible to properly translate dask arrays back and forth between xarray/Iris.

I don't want to hold up this PR which is coming along very nicely (and I'm sure would already be useful), so we can we simply differ handling of dask arrays for now? I would suggest raising NotImplementedError and printing an error message with a link to a follow-up issue in xarray's issue tracker.

duncanwp · 2017-12-18T16:40:59Z

dask/dask#2977 would certianly make it cleaner, but 95b0197 should work in the meantime, unless there's a specific case I've missed?

shoyer

Actually, I think we're OK on the dask front, most likely :)

shoyer · 2017-12-18T16:36:31Z

doc/io.rst

+
+The Iris_ tool allows easy reading of common meteorological and climate model formats
+(including GRIB and UK MetOffice PP files) into ``Cube`` objects which are in many ways very
+similar to ``DataArray``s, while enforcing a CF-compliant data model. If iris is


Needs to be "DataArray object"

(RST can't handle anything adjacent to backticks which is why the color looks weird here in GitHub's syntax highlighting. Yes, it's not very intuitive.)

Good catch, thanks

shoyer · 2017-12-18T16:38:44Z

xarray/convert.py

+    from iris.fileformats.netcdf import parse_cell_methods
+    try:
+        from dask.array import ma as dask_ma
+        from dask.array import Array


Instead of testing for Array, use:

from xarray.core.pycompat import dask_array_type ... if isinstance(data, dask_array_type): # this is always defined, even if dask is not installed

shoyer · 2017-12-18T16:39:28Z

xarray/convert.py

+
+    # Create the right type of masked array (should be easier after #1769)
+    if isinstance(dataarray.data, Array):
+        masked_data = dask_ma.masked_invalid(dataarray)


just import from dask.array import ma as dask_ma right about this line to avoid the need for guarding the import statement in try/except.

Much nicer - thanks!

shoyer · 2017-12-18T16:41:02Z

xarray/convert.py

+    cube_data = cube.core_data() if hasattr(cube, 'core_data') else cube.data
+
+    # Deal with dask and numpy masked arrays
+    if dask_ma and isinstance(cube_data, Array):


Does Iris always use dask masked arrays? Or does it ever use dask numpy arrays? @pelson?

(I would guess not, since we can't tell these apart.)

I don't think it matters, if the given array isn't masked then ma.filled is just a no-op

OK, good to hear!

shoyer · 2017-12-18T16:42:58Z

xarray/convert.py

+    cube_data = cube.core_data() if hasattr(cube, 'core_data') else cube.data
+
+    # Deal with dask and numpy masked arrays
+    if dask_ma and isinstance(cube_data, Array):


Also, again it would be better to use dask_array_type andd import dask.array.ma here rather than above. That would ensure that an error is raised if somehow a cube contains a dask array but has the wrong version of dask installed, rather than letting the error pass silently.

shoyer · 2017-12-18T17:49:38Z

doc/io.rst

+:py:meth:`~xarray.Dataset.from_iris`:
+
+.. ipython:: python
+    :verbatim:


:verbatim: is basically designed for copying/pasting output from the IPython command line, so this isn't quite using it right yet.

I would definitely prefer adding Iris to our docs builds if that's feasible. See:
https://github.com/pydata/xarray/blob/master/doc/environment.yml

OK, I'll give it a shot. How does the environment work, are all the examples run in the same interpreter session? And are there pre-built datarrays I can just use as examples?

We have xr.tutorial.load_dataset('air_temperature') is a reasonable choice, though I'm not 100% sure it's sufficiently CF compliant for Iris :).

I'm OK without actually printing output here (maybe for now).... but in that case I would set things up a little differently so verbatim prints it properly, e.g.,

.. ipython:: python :verbatim: In [1]: da_cube = xr.Dataset.from_iris(cube)

Basically, don't include the line that would print the repr for the cube/array unless you actually would make the Python objects.

duncanwp · 2017-12-18T18:32:50Z

How's this? (I turned the example around so that I create the cube as a DataArray then convert it...)

shoyer · 2017-12-18T18:34:51Z

@duncanwp that example looks good to me. I assume it runs locally with this branch?

duncanwp · 2017-12-19T15:30:22Z

Yep, it's not a very exciting cube, but it runs fine.

shoyer

I think this is ready. @fmaussion @jhamman any further concerns?

fmaussion

Thanks a lot @duncanwp , this is a very important addition!

jhamman

This is great. Thanks for pushing this over the finish line. Can we fix a few style issues before merging:

git diff upstream/master **/*py | flake8 --diff
xarray/convert.py:12:1: F401 '.conventions.decode_cf_variable' imported but unused
xarray/convert.py:13:80: E501 line too long (81 > 79 characters)
xarray/convert.py:16:80: E501 line too long (118 > 79 characters)
xarray/convert.py:17:80: E501 line too long (118 > 79 characters)
xarray/convert.py:18:80: E501 line too long (107 > 79 characters)
xarray/convert.py:20:80: E501 line too long (100 > 79 characters)
xarray/convert.py:120:80: E501 line too long (82 > 79 characters)
xarray/convert.py:185:80: E501 line too long (94 > 79 characters)
xarray/convert.py:189:80: E501 line too long (81 > 79 characters)
xarray/tests/test_dataarray.py:2866:80: E501 line too long (83 > 79 characters)
xarray/tests/test_dataarray.py:2872:80: E501 line too long (80 > 79 characters)
xarray/tests/test_dataarray.py:2878:80: E501 line too long (80 > 79 characters)
xarray/tests/test_dataarray.py:2888:80: E501 line too long (80 > 79 characters)
xarray/tests/test_dataarray.py:2933:80: E501 line too long (105 > 79 characters)
xarray/tests/test_dataarray.py:2939:80: E501 line too long (80 > 79 characters)
xarray/tests/test_dataarray.py:2945:80: E501 line too long (80 > 79 characters)
xarray/tests/test_dataarray.py:2949:80: E501 line too long (89 > 79 characters)
xarray/tests/test_dataarray.py:2958:80: E501 line too long (80 > 79 characters)
xarray/tests/test_dataarray.py:2982:80: E501 line too long (84 > 79 characters)

duncanwp · 2017-12-19T20:59:54Z

Ah yes, sorry about that - fixed now.

duncanwp · 2017-12-20T09:41:31Z

This looks good to go now - thanks for all your help!

pelson · 2018-01-03T10:19:16Z

Apologies for the delay. This is a great first pass. Definitely more metadata sharing possible, but I'm glad this has been merged - let's grow the capability as required. If any issues crop up, please feel free to ping me & @bjlittle. 👍

nparley and others added 20 commits April 1, 2016 10:31

Added a first go at a converter from xarray.DataArrays objects to Iri…

f7842b3

…s.cube.Cube objects. Uses the same template as the cdms2 conversion.

Update tests to use original.coords and add extra tests for coord att…

e0498c5

…ributes

Remove set literals just in case. Replace has_key with in. Use AuxCoo…

c261845

…rd and DimCoord correctly so 2d coords will work. Use dims variable to convert dimension numbers into names. Add 2d coord to test DataArray.

Create dimensions if the Iris cube does not have any

edae053

Add code to convert cell_methods

cd92bca

Don't append blank cell method

44930af

Update cell method code to use internal Iris functions. Also add tests.

6bed306

Update the API for IRIS change

cd06a2e

Merge remote-tracking branch 'upstream/master'

15fbbba

# Conflicts: # .travis.yml

Move helper functions outside of main functions

e7f9cb1

Merge remote-tracking branch 'upstream/master'

03e9076

# Conflicts: # ci/requirements-py27-cdat+iris+pynio.yml

Update to build dims with mix of Dimension and Auxiliary coordinates

877d06f

Merge remote-tracking branch 'upstream/master'

f48de5a

# Conflicts: # xarray/test/test_dataarray.py

Fix import after merge

e42aeb2

Bug fix / refactoring

338ef6b

Change the dencode_cf method and raise error if coord has no var_name

46f68ff

Merge branch 'master' of https://github.com/nparley/xarray into iris_…

bfb2d16

…convert

Add missing test and two minor fixes

edcb3fd

Replacing function calls with set literals

fb724da

Adding what's new and updating api.rst

d510244

jhamman requested changes Nov 30, 2017

View reviewed changes

Adding from_cube method to docs

208d129

duncanwp added 2 commits December 5, 2017 17:23

Make Iris dependency >=1.10 so we can rely on the newer parse_cell_me…

5d933a8

…thods interface (this is already pretty old now)

Adding Iris section to the I/O docs

05760aa

shoyer reviewed Dec 5, 2017

View reviewed changes

Fixing typos

90ada4b

duncanwp added 2 commits December 18, 2017 11:16

Updates to remove the hard dependency on dask, and to test the conver…

95b0197

…sion of dask arrays which may or may not be masked.

Merge branch 'master' of https://github.com/pydata/xarray into iris_c…

239958b

…onvert # Conflicts: # ci/requirements-py27-cdat+iris+pynio.yml

fmaussion reviewed Dec 18, 2017

View reviewed changes

Minor doc fixes

1470bc5

shoyer reviewed Dec 18, 2017

View reviewed changes

duncanwp added 2 commits December 18, 2017 17:22

Minor typo

b457c74

Use dask_array_type for dask checks

441a84f

shoyer reviewed Dec 18, 2017

View reviewed changes

duncanwp added 2 commits December 18, 2017 18:24

Updating to run the iris docs

0dce7f3

Fix typo

301b2ef

shoyer approved these changes Dec 19, 2017

View reviewed changes

fmaussion approved these changes Dec 19, 2017

View reviewed changes

jhamman approved these changes Dec 19, 2017

View reviewed changes

Fixing long lines

b62367a

shoyer merged commit eeb109d into pydata:master Dec 20, 2017

pelson mentioned this pull request Jan 3, 2018

xarray to and from iris #814

Closed

Zac-HD mentioned this pull request Jan 7, 2018

Functions for converting DataArrays to and from iris.Cubes #37

Closed

martindurant mentioned this pull request Apr 5, 2018

we need a data catalog pangeo-data/pangeo#39

Closed

		da_cube


		Conversly, We can create a new cube object from a ``DataArray`` using



		Conversly, We can create a new cube object from a ``DataArray`` using
		:py:meth:`~xarray.Dataset.from_dict`:

xarray to and from Iris #1750

xarray to and from Iris #1750

Conversation

duncanwp commented Nov 30, 2017 • edited Loading

jhamman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duncanwp Dec 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shoyer commented Dec 1, 2017

duncanwp commented Dec 1, 2017 • edited Loading

fmaussion commented Dec 1, 2017

shoyer commented Dec 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duncanwp commented Dec 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmaussion left a comment

Choose a reason for hiding this comment

duncanwp commented Dec 18, 2017

shoyer commented Dec 18, 2017

duncanwp commented Dec 18, 2017

shoyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shoyer Dec 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duncanwp commented Dec 18, 2017

shoyer commented Dec 18, 2017

duncanwp commented Dec 19, 2017

shoyer left a comment

Choose a reason for hiding this comment

fmaussion left a comment

Choose a reason for hiding this comment

jhamman left a comment • edited Loading

Choose a reason for hiding this comment

duncanwp commented Dec 19, 2017

duncanwp commented Dec 20, 2017

pelson commented Jan 3, 2018

duncanwp commented Nov 30, 2017 •

edited

Loading

duncanwp Dec 1, 2017 •

edited

Loading

duncanwp commented Dec 1, 2017 •

edited

Loading

shoyer Dec 18, 2017 •

edited

Loading

jhamman left a comment •

edited

Loading