Publish to conda-forge

[0x1997]

I installed tensorflow from conda-forge and tensorflow_io with pip. However importing tensorflow_io failed with an error

NotImplementedError: unable to open file: libtensorflow_io.so, from paths: ['/data/miniconda/lib/python3.7/site-packages/tensorflow_io/co
re/python/ops/libtensorflow_io.so']
caused by: ['/data/miniconda/lib/python3.7/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so: undefined symbol: _ZN10tensor
flow10FileSystem10FilesExistERKSt6vectorISsSaISsEEPS1_INS_6StatusESaIS6_EE']

It seems the tensorflow_io wheel on pypi was not built with C++11 ABI. Could you publish a compatible version to conda-forge if you have to support older OSs? Thanks.

[yongtang]

We certainly want to support conda-forge. Though I don't have a lot of (or any) experience with conda-forge so it may take some time for me to get this thing going.

[yongtang]

If anyone can help getting this started or some pointers, will be much appreciated.

[yongtang]

With GitHub Actions supported (See #645), I think conda-forge support will be easier.

[BryanCutler]

This would be good to have. I don't have much experience with publishing to Conda, but I can give this a shot if no one is able to get to it first.

[yongtang]

@BryanCutler That would be great 👍 ! I will try to see if I can help as well.

[BryanCutler]

I'm looking into this now, will report back soon

[BryanCutler]

It looks like the latest version of TensorFlow in conda is v1.13.2 see https://anaconda.org/conda-forge/tensorflow/files?page=1. Would it be better to hold off on this until TF 2.0 is released in conda?

[seanpmorgan]

We've had some discussion about this for tf-addons that may be of use:
tensorflow/addons#201

Anaconda packages a tf2.0 but I'm not sure who is managing the conda-forge version.

[yongtang]

Thanks @seanpmorgan @BryanCutler . By looking at https://github.com/conda-forge/tensorflow-feedstock it looks like tensorflow-feedstock repo is not well maintained. The last update was 09/30/2019 which is 3 months old.

Wondering if there is a way to move forward the tf 2.0 in conda-forge?

[xhochy]

@yongtang There is already an open PR to update to 2.0 conda-forge/tensorflow-feedstock#97 generated by the bot. Either fix this PR or port the way Anaconda builds tensorflow to conda-forge: https://github.com/AnacondaRecipes/tensorflow_recipes

[yongtang]

Thanks @xhochy. Not very familiar with conda recipes/feedstock but I will try to take a look.

[yongtang]

Created a working in progress PR conda-forge/tensorflow-feedstock#101

Still too many issues to build. One dependency is the Bazel. When building Bazel gcc version is old. need more time to moving forward.

[yongtang]

After several fixes, the windows and OS X build in conda-forge/tensorflow-feedstock#101 are managed to run. However, it looks like after some time, there is no output anymore. (OS X timed out). Don't know what really happened. May need more time to figure it out.

[yongtang]

I gave another try with conda-forge recently conda-forge/staged-recipes#10913

unfortunately still face lots of confusion issues. Will have to see if they could be done later.

[strayedboat2013]

https://github.com/tensorflow/io have info of compatibility between tensorflow and tensorflow-io, for tf 2.0, you have to install tensorflow-io 0.10.0, please try "pip install tensorflow-io==0.10.0"

[emckenzi123]

I also installed tensorflow via conda install tensorflow-gpu (version 2.1.0)
and I installed tensorflow-io via pip install tensorflow-io (version 0.12.0)

I am running in linux (Ubuntu 18.04)

However if I try to import tensorflow_io in python I am also getting an error similar to the original post. Don't know if this additional info helps or not.

NotImplementedError: unable to open file: libtensorflow_io.so, from paths: ['.../.conda/envs/tensorflow2_gpu/lib/python3.7/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so']
caused by: ['.../.conda/envs/tensorflow2_gpu/lib/python3.7/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so: undefined symbol: _ZN10tensorflow2io19BufferedInputStream10ReadNBytesExPSs']

[chrissype]

I have exactly the same issue as above

[gabrielctn]

Same issue as above

[chrissype]

FWIW I was able to compile v0.12.0 from source against my conda installed tensorflow-gpu 2.1.0 and pip install the built package, and now everything seems to work correctly. This worked on my machine and inside a docker image (both linux distros). Obviously not ideal, but shows that it should be possible, and is a fix for anyone who needs one now.

[gabrielctn]

FWIW I was able to compile v0.12.0 from source against my conda installed tensorflow-gpu 2.1.0 and pip install the built package, and now everything seems to work correctly. This worked on my machine and inside a docker image (both linux distros). Obviously not ideal, but shows that it should be possible, and is a fix for anyone who needs one now.

Thanks for your suggestion @chrissype but I am stuck at the pip install step... I have this error maybe you know what the problem is since you juste went through the install ? (I am not used to install bazel compiled packages):

(tensorflow-gpu-2.1.0) root@cannelle:~/io-0.12.0# bazel build -s --verbose_failures //tensorflow_io/...
[...]
INFO: Elapsed time: 4387.942s, Critical Path: 225.79s
INFO: 4276 processes: 4276 linux-sandbox.
INFO: Build completed successfully, 4464 total actions
(tensorflow-gpu-2.1.0) root@cannelle:~/io-0.12.0# pip install .
Processing /root/io-0.12.0
    ERROR: Command errored out with exit status 1:
     command: /opt/apps/pkg/anaconda3/envs/tensorflow-gpu-2.1.0/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-dknthpdu/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-dknthpdu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-dknthpdu/pip-egg-info
         cwd: /tmp/pip-req-build-dknthpdu/
    Complete output (19 lines):
    setup.py - project = 'tensorflow-io'
    setup.py - package = 'tensorflow>=2.1.0,<2.2.0'
    setup.py - version = '0.12.0'
    setup.py - create /tmp/tmpvuph2qla and copy tensorflow_io
    setup.py - create /tmp/tmpvuph2qla/MANIFEST.in
    setup.py - create /tmp/tmpvuph2qla/setup.py with required = 'tensorflow>=2.1.0,<2.2.0', project_name = 'tensorflow-io' and __version__ = 0.12.0
    setup.py - run sandbox.run_setup /tmp/tmpvuph2qla/setup.py ['egg_info', '--egg-base', '/tmp/pip-req-build-dknthpdu/pip-egg-info']
    running egg_info
    creating /tmp/pip-req-build-dknthpdu/pip-egg-info/tensorflow_io.egg-info
    error: SandboxViolation: mkdir('/tmp/pip-req-build-dknthpdu/pip-egg-info/tensorflow_io.egg-info', 511) {}
    
    The package setup script has attempted to modify files on your system
    that are not within the EasyInstall build area, and has been aborted.
    
    This package cannot be safely installed by EasyInstall, and may not
    support alternate installation locations even if you run its setup
    script by hand.  Please inform the package's author and the EasyInstall
    maintainers to find out if a fix or workaround is available.
    
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Yet, when I do pip list I see tensorflow-io 0.12.0 but if I load the conda environment somewhere else I don't see it anymore...

[chrissype]

You have to run setup.py first to make the wheels binary I think, something like:

python setup.py --data bazel-bin bdist_wheel

And then:

pip install ./dist/tensorflow_io-0.12.0-cp37-cp37m-linux_x86_64.whl

And then you'll need to be in a different directory when you try and import tensorflow_io otherwise it'll still load the python version from the project source dir.

[yongtang]

@gabrielctn @chrissype Because tensorflow/io needs to carry the shared objects built by Bazel and located in a separate directory (bazel-bin), the setup.py has to create a temporarily directory, copy both python source code (in tensorflow_io) and shared object (bazel-bin) into the temporary directory, then invoke the setup.py from there.

As a result, the traditional pip install -e . does not work. It is fixable though, as long as the code path to get to pip install -e . could make the similar change. There is an issue tracking this one (#707). Though there has been quite a few thing recently so this issue has not been visited.

[duanbing]

I also meet the same case. I compile the TF 2.4.2 locally due to some special function added, and then install tensorflow-io with version 0.17.1.