Skip to content

Switch to compressed numpy arrays for test control data #623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 26, 2020

Conversation

behackl
Copy link
Member

@behackl behackl commented Oct 26, 2020

List of Changes

As pointed out by @kolibril13 in #599, we should store the control data for the graphical unit tests in a compressed format. This PR switches set_up_scene and _load_data to the compressed format, and changes all control files to the compressed format.

The size of the control_data directory actually reduces from 132MB to 620KB (!!!).

Testing Status

Tests pass locally -- and this should be easy to review: either the tests pass, or they don't. 🙃

Acknowledgement

@behackl behackl added testing Anything related to testing the library pr:easy review There is nothing particular (i.e, it's about a general/small thing) to know for review! labels Oct 26, 2020
@behackl behackl linked an issue Oct 26, 2020 that may be closed by this pull request
@leotrs
Copy link
Contributor

leotrs commented Oct 26, 2020

What a hero.

Copy link
Contributor

@yoshiask yoshiask left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@behackl
Copy link
Member Author

behackl commented Oct 26, 2020

To have this live somewhere, here is the code I used for converting the .npy files to their compressed counterparts:

import numpy as np
from pathlib import Path

p = Path(".")
control_files = [f for f in list(p.rglob("*")) if f.name[-4:] == ".npy"]

for file in control_files:
    data = np.load(file)
    np.savez_compressed(file.as_posix()[:-4], frame_data=data)

@behackl
Copy link
Member Author

behackl commented Oct 26, 2020

Docs don't only fail to build here, but apparently on master as well. This is not related to the changes here, seems we forgot to remove some PangoText somewhere. I'll have a look at that in a moment.

Copy link
Member

@naveen521kk naveen521kk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too many files to review 😂

@behackl behackl merged commit 84b3dee into ManimCommunity:master Oct 26, 2020
@behackl behackl deleted the compressed-test-data branch October 26, 2020 20:36
@huguesdevimeux
Copy link
Member

How is this miracle even possible

@kolibril13
Copy link
Member

One method is called Huffman coding:
https://www.youtube.com/watch?v=0kNXhFIEd_w
Actually, this topic would be very suitable to make a manim video of it :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr:easy review There is nothing particular (i.e, it's about a general/small thing) to know for review! testing Anything related to testing the library
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Considering smaller format for control_data
6 participants