Add :meth:~.Scene.interactive_embed for OpenGL rendering

[eulertour]

Changelog / Overview

:meth:~.Scene.interactive_embed allows interaction with Scene via mouse and keyboard as well as dynamic commands via an iPython terminal.

Motivation

Scene.interactive_embed() allows screen interaction and dynamic commands simultaneously as opposed to Scene.embed(), which only allows dynamic commands and Scene.interact(), which only allows screen interaction.

Explanation for Changes

Scene.interactive_embed() runs an IPython terminal in a separate thread which communicates with the main thread via a shared queue. This allows screen interaction and dynamic control via the terminal simultaneously.

Testing Status

I tested with the following Scene:

class Test(Scene):
    def construct(self):
        s = OpenGLSquare()
        self.add(s)
        self.interactive_embed()

on the last line a live terminal will spawn and the screen will still be interactive.

Further Comments

Attempting to render from the terminal thread will result in a segfault. This is because OpenGL contexts are bound to a single thread. The add, remove, play, and wait methods are communicated to the renderer thread via a shared queue (so the action is performed on the renderer thread). This means that running self.play(...) (as opposed to play(...)) will crash.

Things like updating mobject attributes works as expected (even though data is updated on the terminal thread and read on the renderer thread) but I'm not sure whether this is actually safe or just luck.

Checklist

• I have read the Contributing Guidelines
• I have written a descriptive PR title (see top of PR template for examples)
• I have written a changelog entry for the PR or deem it unnecessary
• My new functions/classes either have a docstring or are private
• My new functions/classes have tests added and (optional) examples in the docs
• My new documentation builds, looks correctly formatted, and adds no additional build warnings

Reviewer Checklist

• The PR title is descriptive enough
• The PR is labeled correctly
• The changelog entry is completed if necessary
• Newly added functions/classes either have a docstring or are private
• Newly added functions/classes have tests added and (optional) examples in the docs
• Newly added documentation builds, looks correctly formatted, and adds no additional build warnings

[kilacoda-old]

Maybe a better name for this could be thought of? Such as interactive_embed etc. Also, why not just replace Scene.embed with this new method?

[eulertour]

I used Scene.interactive_embed() instead. I want to keep Scene.embed() for now just to prevent having to roll back if there's a bug, but it can be removed in the future.

[TRoboto]

The interactive part is working as it should, but the embed part does not function properly. Neither self.add nor add works. self.play and play also don't work for adding objects. But, they only work one time to animate an already added object but then don't return to the terminal. (check the image below). wait also works once like play, the good news is that remove works fine.

[eulertour]

What OS are you using?

[eulertour]

Also what command are you using to test with?

[TRoboto]

What OS are you using?

Ubuntu 20.04

Also what command are you using to test with?

manim render test.py -pql --renderer=opengl

Code:

class Test(Scene):
    def construct(self):
        square = OpenGLSquare()

        self.play(FadeIn(square))
        self.interactive_embed()

[eulertour]

Turns out that making it possible to prepend self will be a bit trickier. Try it now (without prepending self to use scene methods).

[TRoboto]

Turns out that making it possible to prepend self will be a bit trickier. Try it now (without prepending self to use scene methods).

add now works but play still doesn't work, it only plays one animation (same as the previous comment).

[eulertour]

That doesn't happen for me. It'd help if you could find out which line is hanging (adding in print statements should be enough to figure it out). The play will be executed here and you can try to narrow done the place where execution stops from there.

[TRoboto]

I don't know why but I solved the problem by only adding a print statement on line 983 as follows, does this make any sense?

while not (self.renderer.window.is_closing or self.quit_interaction):
            print("in loop")
            if not self.queue.empty():

[eulertour]

uh, not really :/
Did you try using the terminal after running an animation? It may have just not had the prompt visible.

[friedkeenan]

Prints fixing things is typically a sign of a race condition somewhere in my experience

[eulertour]

Mine too, but the multithreading here doesn't have anything to block on.

It'd probably be easiest for me to debug this over a discord call, but in the meantime there are a few questions that might be helpful:

1. Does the problem always occur without the print statement and/or never occur without it?
2. Are there any other places where you can add print statements with the same effect?

[jsonvillanueva]

Also what command are you using to test with?

manim render test.py -pql --renderer=opengl

Strange that this command works with the options supplied after the file/positional argument, I guess that's because I moved the render subcommand from a click.Group to a click.Command and the options are applied to the command if it's the last?... anyways, this seems to be working for me on Windows 10:

After further experimentation, I cannot use self.play(...) but can still use play(...). Also, if I use self.play(...) it will add the object to the screen but fail the animation.

[eulertour]

That's expected, though it may be worth adding some sort of guard against prepending self. to methods. Are you able to use play() multiple times?

[jsonvillanueva]

Yes, using the testing code's square, I can run play(Write/FadeIn/FadeOut/...) any number of times on square without issue. I can also supply other OpenGL Mobjects such as a circle to animate multiple things at the same time:

[TRoboto]

uh, not really :/
Did you try using the terminal after running an animation? It may have just not had the prompt visible.

It turns out to be an alignment problem :/. I just wrote another command while the terminal was hanging and it worked fine. I have no idea why the terminal switches to RTL only after the first play command. This is super strange and It might be related to my system.

Resizing the terminal window reveals In [5]:

[eulertour]

So can this be approved now?

[jsonvillanueva]

I approve but with a cautionary note that I've mainly just verified it functions on my machine and skimmed over scene.py's implementation. The other files seem fine.

I do find it odd that ipython_magic.py can make use of self.play, self.wait, self.add and self.remove but interactive_embed cannot. I've requested @behackl read over this due to writing ipython_magic.py and having an understanding of the package.

[eulertour]

It's probably because when manim is used in ipython the interpreter doesn't need to continually rerender the screen and listen for user input simultaneously.

[behackl]

It's probably because when manim is used in ipython the interpreter doesn't need to continually rerender the screen and listen for user input simultaneously.

I think that's essentially true. The %%manim IPython magic basically is a fancy wrapper around Scene.render that allows to circumvent calling manim via the CLI, but in a somewhat convenient way from the running interpreter.

Interactivity certainly makes things more complicated; this was also something we observed in the livestreaming approach over at #778 -- there it was also easier to make special manim.play etc. methods available instead of trying to get self to work.