Tcl/Tk failure within venv on macOS/Linux

[tmontes]

Hi @indygreg,

Reporting this as a follow up of #43, after my progress with pup -- striving to automate Mu Editor's packaging into "native distributable artifacts/installers", pup does make use of Python Build Standalone -- ultimately it may become usable at packaging any Python GUI application, but that's speculative. :-)

As a beginner friendly Python editor, Mu itself is a particularly demanding Python project to package because itself runs and debugs user created Python code, including bringing in third party dependencies from PyPI installed into a "user-hidden" virtual environment, among other not-so-common duties.

The Surprise

Having reached the minimal-viable-product status, I took pup for a more thourough spin in packaging Mu only to find that one (human driven) test failed on me -- it failed running a minimal turtle-based program, the turtle graphics module commonly used with young beginners.

Digging in I noticed that it failed on macOS but did not fail on Windows.

Detail

I boiled it down to this minimal example, straight out of the available releases.

Extracted cpython-3.7.9-x86_64-apple-darwin-pgo-20200823T2228.tar.zst on my macOS system and ran:

tiago.montes (~/Temp/pbs-test): ./python/install/bin/python3
Python 3.7.9 (default, Aug 23 2020, 15:46:47) 
[Clang 10.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import turtle
>>> turtle.clear()
>>> ^D

...this worked nicely and the turtle window was displayed, as expected.

Repeating the test, under a freshly created venv:

tiago.montes (~/Temp/pbs-test): ./python/install/bin/python3 -m venv venv379
tiago.montes (~/Temp/pbs-test): ./venv379/bin/python
Python 3.7.9 (default, Aug 23 2020, 15:46:47) 
[Clang 10.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import turtle
>>> turtle.clear()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 6, in clear
  File "/Users/tiago.montes/Temp/pbs-test/python/install/lib/python3.7/turtle.py", line 3812, in __init__
    Turtle._screen = Screen()
  File "/Users/tiago.montes/Temp/pbs-test/python/install/lib/python3.7/turtle.py", line 3662, in Screen
    Turtle._screen = _Screen()
  File "/Users/tiago.montes/Temp/pbs-test/python/install/lib/python3.7/turtle.py", line 3678, in __init__
    _Screen._root = self._root = _Root()
  File "/Users/tiago.montes/Temp/pbs-test/python/install/lib/python3.7/turtle.py", line 434, in __init__
    TK.Tk.__init__(self)
  File "/Users/tiago.montes/Temp/pbs-test/python/install/lib/python3.7/tkinter/__init__.py", line 2023, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories: 
    /tools/deps/lib/tcl8.6 /Users/tiago.montes/Temp/pbs-test/venv379/lib/tcl8.6 /Users/tiago.montes/Temp/pbs-test/lib/tcl8.6 /Users/tiago.montes/Temp/pbs-test/venv379/library /Users/tiago.montes/Temp/pbs-test/library /Users/tiago.montes/Temp/pbs-test/tcl8.6.10/library /Users/tiago.montes/Temp/tcl8.6.10/library



This probably means that Tcl wasn't installed properly.

>>> ^D

It is clear that tkinter could not find init.tcl -- even though the file is included in the distribution, for some reason it is being looked for in non-existent directories.

Additional Info

• More recent releases behaved the same:
  • cpython-3.8.7-x86_64-apple-darwin-pgo-20210103T1125.tar.zst
  • cpython-3.9.1-x86_64-apple-darwin-pgo-20210103T1125.tar.zst
• Behaviour is similar on Linux -- did not test as thouroughly, but found same failure, under Ubuntu 18.04.
• Interestingly, the Windows builds do not fail and using turtle from within a venv works.

Work Around

After minor investigation, I learned that the TCL_LIBRARY environment variable can be used to drive Tcl into finding init.tcl.

With that in mind, and knowing that in my experiment the file was under /Users/tiago.montes/Temp/pbs-test/python/install/lib/tcl8.6/, I tried:

tiago.montes (~/Temp/pbs-test): ./venv379/bin/python
Python 3.7.9 (default, Aug 23 2020, 15:46:47) 
[Clang 10.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['TCL_LIBRARY'] = '/Users/tiago.montes/Temp/pbs-test/python/install/lib/tcl8.6/'                              
>>> import turtle
>>> turtle.clear()
>>> ^D

...which worked nicely.

The Request for Help

This is a somewhat unfortunate observation -- UNIX-like systems seem to be failing in this case, Windows works. In a way I suppose it may make sense, assuming that Tcl's build process -- or runtime decisions -- might differ across those platforms.

Then I thought about PSF's Python distribution where, as you imagine, turtle is perfectly usable from within a venv -- however, I'm not sure if that distribution is relocatable or not (so it might include some kind of build-time locked-down path to init.tcl, or some such).

Then again, not being familiar with the whole build process and Tcl's in particular, I wonder if there might be any kind of tweak that we could apply to make all of this "just work"...

I'll be glad to help out throught this, if you could please point me in the right direction.

Thanks again for this project and in advance for any assistance in this regard. :-)

[indygreg]

I just pushed a commit documenting the run-time quirks of the tcl/tk support files. You can view the rendered docs at https://python-build-standalone.readthedocs.io/en/latest/quirks.html#tcl-tk-support-files.

Does the new documentation answer all your questions? If so, feel free to close this issue. If not, I'd love to update the docs so your questions are answered!

[tmontes]

Thanks for the update. :-)

Note, however, that the issue I'm describing is more sublte -- at least from my reading of the docs, which are very good, I couldn't identify something more specific (and maybe they're ok that way!).

In particular, the failure I observe on UNIX-like systems in using tkinter/turtle from within a virtual environment, where something like what is documented does not work and, IME, doesn't seem to be needed otherwise:

os.environ["TCL_LIBRARY"] = os.path.join(os.path.dirname(sys.executable), "..", "lib", "tcl8.6")

...this fails because within the virtual environment sys.executable is a symbolic link to the actual binary, lying somewhere else on the filesystem. However, the underlying binary is the same.

I wonder if this relates to either the way Tcl itself is coded to locate its initialization files, being "sensitive" to argv[0] -- thus failing with the sym-linked binary -- or the way PBS is building either Tcl or Python, again, making it fail when argv[0] is not the actual binary -- the case when running Python from within a virtual environment.

I will probably move forward in having pup set the TCL_LIBRARY env var to a value produced from the distribution's PYTHON.json file, but would love to hear your thoughts on the above. :-)

Again, thanks a lot for PBS.

[thnee]

Ran into this issue as well.
I don't fully understand everything that is happening here, so apologies if this is adding noise.
But I thought I would just paste my debugging session here, for posterity.

λ mise install python@3.11.9
mise installing precompiled python from indygreg/python-build-standalone
mise if you experience issues with this python (e.g.: running poetry), switch to python-build
mise by running: mise settings set python_compile 1
mise python@3.11.9 ✓ installed                                                                                                                                                                                    

When using tkinter without a venv, the error is like this.

λ mise use python@3.11.9
λ python -m tkinter
[xcb] Unknown sequence number while appending request
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
python: ../../src/xcb_io.c:157: append_pending_request: Assertion `!xcb_xlib_unknown_seq_number' failed.
[1]    37814 IOT instruction  python -m tkinter

When using tkinter in a venv, the error is different.

λ mise use python@3.11.9
λ python -m venv venv
λ source venv/bin/activate
λ python -m tkinter
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/username/.local/share/mise/installs/python/3.11.9/lib/python3.11/tkinter/__main__.py", line 7, in <module>
    main()
  File "/home/username/.local/share/mise/installs/python/3.11.9/lib/python3.11/tkinter/__init__.py", line 4643, in _test
    root = Tk()
           ^^^^
  File "/home/username/.local/share/mise/installs/python/3.11.9/lib/python3.11/tkinter/__init__.py", line 2345, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_tkinter.TclError: Can't find a usable init.tcl in the following directories:
    /tools/deps/lib/tcl8.6 /home/username/venv/lib/tcl8.6 /home/username/lib/tcl8.6 /home/username/venv/library /home/username/library /home/username/tcl8.6.12/library /home/tcl8.6.12/library



This probably means that Tcl wasn't installed properly.

As a mise user, it was fairly easy to workaround the problem, once the details were understood.
Although, when asking mise to build from source, I guess it doesn't use this project, so this is more of a sidenote.

λ deactivate
λ rm -rf venv
λ mise uninstall python@3.11.9
mise python@3.11.9 ✓ uninstalled
λ sudo apt install $all_the_typical_python_build_deps
λ sudo apt install tk-dev
λ MISE_PYTHON_COMPILE="true" PYTHON_CONFIGURE_OPTS="--with-tcl-tk" mise install python@3.11.9
Downloading Python-3.11.9.tar.xz...
-> https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tar.xz
Installing Python-3.11.9...
Installed Python-3.11.9 to /home/username/.local/share/mise/installs/python/3.11.9
mise python@3.11.9 ✓ installed
λ mise use python@3.11.9
λ python -m tkinter
It works! There is no error, and a GUI window is displayed.
λ python -m venv venv
λ source venv/bin/activate
λ python -m tkinter
It works! There is no error, and a GUI window is displayed.

[charliermarsh]

This should be fixed in the next release (though we'll only re-build the latest patch versions): #421.