Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trace why the built zip size keeps increasing #5977

Closed
bahmutov opened this issue Dec 16, 2019 · 15 comments
Closed

Trace why the built zip size keeps increasing #5977

bahmutov opened this issue Dec 16, 2019 · 15 comments
Labels
process: build Related to our internal build process process: dependencies Related to internal dependencies stage: needs investigating Someone from Cypress needs to look at this stale no activity on this issue for a long period type: performance 🏃‍♀️ Performance related

Comments

@bahmutov
Copy link
Contributor

The current zip of the Test Runner build in the Firefox branch is 191MB! Need to investigate what is driving the increases in size

Example for Mac

Find cached Cypress binary path and versions

$ npx cypress cache path
/Users/gleb/Library/Caches/Cypress
$ npx cypress cache list
3.0.1, 3.0.2, 3.0.3, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.2.0, 3.3.0, 3.3.1, 3.3.2, 3.4.0, 3.4.1, 3.5.0, 3.6.0, 3.6.1, 3.7.0, 3.8.0

You can see the size history by version

$ du -hs /Users/gleb/Library/Caches/Cypress/*
401M	/Users/gleb/Library/Caches/Cypress/3.0.1
430M	/Users/gleb/Library/Caches/Cypress/3.0.2
430M	/Users/gleb/Library/Caches/Cypress/3.0.3
494M	/Users/gleb/Library/Caches/Cypress/3.1.0
489M	/Users/gleb/Library/Caches/Cypress/3.1.1
488M	/Users/gleb/Library/Caches/Cypress/3.1.2
486M	/Users/gleb/Library/Caches/Cypress/3.1.3
490M	/Users/gleb/Library/Caches/Cypress/3.1.4
485M	/Users/gleb/Library/Caches/Cypress/3.1.5
453M	/Users/gleb/Library/Caches/Cypress/3.2.0
463M	/Users/gleb/Library/Caches/Cypress/3.3.0
463M	/Users/gleb/Library/Caches/Cypress/3.3.1
465M	/Users/gleb/Library/Caches/Cypress/3.3.2
465M	/Users/gleb/Library/Caches/Cypress/3.4.0
456M	/Users/gleb/Library/Caches/Cypress/3.4.1
476M	/Users/gleb/Library/Caches/Cypress/3.5.0
470M	/Users/gleb/Library/Caches/Cypress/3.6.0
475M	/Users/gleb/Library/Caches/Cypress/3.6.1
511M	/Users/gleb/Library/Caches/Cypress/3.7.0
559M	/Users/gleb/Library/Caches/Cypress/3.8.0

Take any version and see the resources sizes - for example 3.2.0 vs 3.8.0. Tip: sort -h sorts by human size in the first column

$ du -hs /Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/* | sort -h
8.0K	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/root
 16K	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/driver
136K	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/ts
144K	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/static
456K	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/coffee
1.1M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/example
2.9M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/socket
3.8M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/reporter
5.9M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/desktop-gui
6.2M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/extension
8.7M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/runner
8.9M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/launcher
9.6M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/https-proxy
 25M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/electron
253M	/Users/gleb/Library/Caches/Cypress/3.2.0/Cypress.app/Contents/Resources/app/packages/server

and for 3.8.0

$ du -hs /Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/* | sort -h
8.0K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/web-config
 12K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/driver
 12K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/root
 20K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/reporter
136K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/ts
144K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/static
456K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/coffee
1.1M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/example
3.9M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/socket
6.2M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/extension
7.7M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/proxy
7.8M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/https-proxy
8.9M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/launcher
 10M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/runner
 11M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/desktop-gui
 27M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/electron
 57M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network
251M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/server
@bahmutov bahmutov self-assigned this Dec 16, 2019
@bahmutov bahmutov added process: build Related to our internal build process process: dependencies Related to internal dependencies labels Dec 16, 2019
@bahmutov
Copy link
Contributor Author

So the size increase has a large component in packages/network which is all because of its dependencies

$ du -hs /Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/* | sort -h
...
716K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/parse-domain
728K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/pretty-format
752K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/neo-async
804K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/uglify-js
828K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/source-map
996K	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/snapdragon
1.1M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/acorn-globals
1.1M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/ajv
1.2M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/@jest
1.4M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/node-notifier
2.7M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/handlebars
2.9M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/jsdom
3.0M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/expect
3.3M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/fsevents
4.8M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/lodash
5.0M	/Users/gleb/Library/Caches/Cypress/3.8.0/Cypress.app/Contents/Resources/app/packages/network/node_modules/@babel

@bahmutov
Copy link
Contributor Author

ughh, looking at packages/network -

$ du -hs node_modules/\@babel/*
 20K	node_modules/@babel/code-frame
1.0M	node_modules/@babel/core
948K	node_modules/@babel/generator
 20K	node_modules/@babel/helper-function-name
 16K	node_modules/@babel/helper-get-function-arity
 20K	node_modules/@babel/helper-plugin-utils
 16K	node_modules/@babel/helper-split-export-declaration
 84K	node_modules/@babel/helpers
 16K	node_modules/@babel/highlight
1.7M	node_modules/@babel/parser
 16K	node_modules/@babel/plugin-syntax-object-rest-spread
 52K	node_modules/@babel/template
216K	node_modules/@babel/traverse
960K	node_modules/@babel/types

and then noticing that parse-domain includes jest in production dependencies!

$ cat node_modules/parse-domain/package.json
  "dependencies": {
    "got": "^8.3.2",
    "jest": "^24.9.0",
    "mkdirp": "^0.5.1",
    "npm-run-all": "^4.1.5"
  }

peerigon/parse-domain#89

A single Jest v24 install is 58MB

$ npm i -S jest
+ jest@24.9.0
added 533 packages from 376 contributors and audited 877518 packages in 20.963s
$ du -hs node_modules
 58M	node_modules

@bahmutov
Copy link
Contributor Author

Opened peerigon/parse-domain#91

@sainthkh
Copy link
Contributor

sainthkh commented Dec 16, 2019

TypeScript is 35~44MB. When we remove jest and TypeScript, we have about 100MB down.

Maybe we need bundlesize bot.

@bahmutov
Copy link
Contributor Author

bahmutov commented Dec 16, 2019 via email

@sainthkh
Copy link
Contributor

TypeScript is included in server with dependency-tree from #1463 (to watch plugin.js dependencies). dependency-tree uses filing-cabinet and precinct and they use typescript.

And network uses parse-domain and it requires jest and jest family is included in network node_modules.

And @babel family is included to server by @cypress/browserify-preprocessor.

@bahmutov
Copy link
Contributor Author

bahmutov commented Dec 16, 2019 via email

@sainthkh
Copy link
Contributor

sainthkh commented Dec 16, 2019

And when we're building Cypress, we install dependencies to each package. Because of that, we're including some packages several times like bluebird in 8 packages (server, proxy, network, launcher, https-proxy, extension, example, electron), lodash in 7 packages (electron, extension, https-proxy, launcher, network, proxy, server).

We can remove these redundant dependencies by merging the list of dependencies into one list under packages/package.json and install them from it. Then, we can install them only once.

@bahmutov
Copy link
Contributor Author

bahmutov commented Dec 17, 2019 via email

@jennifer-shehane
Copy link
Member

Isn't the yarn PR meant to address some of the redundant dependencies? cc @andrew-codes #5555

@cypress-bot cypress-bot bot added the stage: investigating Someone from Cypress is looking into this label Dec 17, 2019
@bahmutov
Copy link
Contributor Author

Ughh, server does rely on dependency-tree that brings TypeScript

$ npm ls typescript
@packages/server@0.0.0 /Users/gleb/git/cypress/packages/server
└─┬ dependency-tree@7.0.2
  ├─┬ filing-cabinet@2.5.1
  │ └── typescript@3.7.2 
  └─┬ precinct@6.2.0
    └─┬ detective-typescript@5.7.0
      └── typescript@3.7.2  deduped

@flotwig
Copy link
Contributor

flotwig commented Dec 17, 2019

dependency-tree is used to watch all the pluginsFile and all its dependencies for file changes, but it might be overkill

@sainthkh
Copy link
Contributor

I created TypeScript issue in #6051.

@cypress-app-bot
Copy link
Collaborator

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

@cypress-app-bot cypress-app-bot added the stale no activity on this issue for a long period label May 17, 2023
@cypress-app-bot
Copy link
Collaborator

This issue has been closed due to inactivity.

@cypress-app-bot cypress-app-bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
process: build Related to our internal build process process: dependencies Related to internal dependencies stage: needs investigating Someone from Cypress needs to look at this stale no activity on this issue for a long period type: performance 🏃‍♀️ Performance related
Projects
None yet
Development

No branches or pull requests

5 participants