python3-config --ldflags --embed does not report the correct directory on macOS

[mmomtchev]

Description:
python3-config --ldflags --embed does not report the library location

Action version:
actions/setup-python@v4

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:
3.8, 3.9, 3.10, 3.11

Repro steps:

python3-config --ldflags --embed
pkg-config --libs python3-embed

Expected behavior:

-L/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib -lpython3.8 -lintl -ldl -framework CoreFoundation
-L/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib -lpython3.8

The same command on my macbook produces:

-L/usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/config-3.10-darwin -lpython3.10 -ldl -framework CoreFoundation

Actual behavior:

-lpython3.8 -lintl -ldl -framework CoreFoundation
-L/Users/runner/hostedtoolcache/Python/3.8.14/x64/lib -lpython3.8

https://github.com/mmomtchev/pymport/actions/runs/3409283526/jobs/5670857537

[marko-zivic-93]

Hello @mmomtchev
Thank you for bringing this up. We will investigate and come back to you as soon as we get some information about this issue :)

[JonasGe]

We experienced the same issue with a self-hosted runner on Ubuntu 20.04 and Python 3.9.

What solved it was adding this to the workflow:

- run: |
          CPATH=$pythonLocation/include/python${{ matrix.python-version }}
          echo "CPATH=$CPATH" >> $GITHUB_ENV

[dsame]

hello @mmomtchev, it is expected behaviour

Since 3.11 python is installed from the official Python macos packages and hostedtoolcache files are nothing but a links to the default paths for sake of backward compatibility. See this output https://github.com/akv-demo/setup-python-test/actions/runs/3695552346/jobs/6258081360#step:3:26

But the versions below 3.11 are still installed from the custom prebuilt binares and reside within hostedtoolcache folders, see https://github.com/akv-demo/setup-python-test/actions/runs/3695552346/jobs/6258080844#step:3:26

Does this answer clarify the situation?

[mmomtchev]

I am afraid that this is not enough for embedded Python - it works by setting the environment variable so the Python executable finds the right library, but when linking, how would the linker pick up the the right Python library? Unless Frameworks has also been redirected? Anyway, the work-around is to use pkg-config and this works.

[dsame]

@mmomtchev , the reason why python3-config does not print -L option is the github provided pythons are built with --enable-shared and macos official binaries without them.

This is intended behaviour of the python3-config

 62         if opt == '--ldflags':
 63             if not getvar('Py_ENABLE_SHARED'):
 64                 libs.insert(0, '-L' + getvar('LIBPL'))

And i guess you should not provide the -L option for embedded python at all because the shared object is to be found in runtime whenever it resides. But some communities that are more python-specific can be more informative in this part.

Is the question answered now?

[mmomtchev]

I made a minimal reproduction of the problem:
https://github.com/mmomtchev/pymport/actions/runs/3715370531

The python interpreter can find its dynamic library because it knows its location.
But when linking to this library, its location must come from somewhere.

[dsame]

@mmomtchev, i have to agree with you the options provided by python3-config prevents from successful linking. But they do not depend on the setup-action and i really puzzled what is the sense of this branch in python3-config script

63             if not getvar('Py_ENABLE_SHARED'):
64                 libs.insert(0, '-L' + getvar('LIBPL'))

I can hardly help with the further investigation - this question should be addressed to the python community.

[mmomtchev]

Ok, the workaround is to use pkg-config