Wheel binary package format

[dholth]

Experimental support for the wheel package format in pip. Can be used to realize dramatic reductions in install time.

[rubik]

I think adding a new command is reasonable here. Much like bundle does: since the core functionality are required that's doable by subclassing the install command and adding options to it.

@qwcode: I think idea #1 does not make much sense: why bothering with the finder (which, BTW, takes ages checking tons of useless links) when you have a cache out there. I'm +1 on idea 2: check the cache and then only if the package is not there look for links in the index.

[dholth]

@qwcode you would never get all the way to setup.py install and then check the wheel cache. Instead, the finder downloads and installs the wheel instead of the sdist.

We will have to make sure that the "populate a directory full of wheels" command will also download wheels directly from pypi and put them in that directory.

The new command is a more useful replacement for bundle.

[qwcode]

you would never get all the way to setup.py install and then check the wheel cache.

@dholth in the abstract world of idea1 you would : ). the point of idea1 was mostly to distinguish it from idea2, which is better, but also to be clear that neither idea1 or idea2 was what was in this pull. This was in response to Paul's post who seemed to be maybe thinking in idea1 terms.

why bothering with the finder (which, BTW, takes ages checking tons of useless links)
when you have a cache out there. I'm +1 on idea 2

@rubik , for the scope of this pull, I think we should/have-to be satisfied with the "--no-index --find-links" approximation I mentioned. Considering caching that's not just "download" caching is better left as a dedicated effort in another pull. here's a discussion that covers a lot of the territory: https://groups.google.com/forum/#!msg/python-virtualenv/J92cKpftSjA/i9zSae8zV3AJ

[carljm]

I think the idea of a local cache that precedes PackageFinder is interesting in its own right, but entirely orthogonal to wheels. Wheels are just another package distribution format. Any type of new caching feature should work equally well with wheels or sdists, and shouldn't be tied specifically to wheels in the UI or implementation.

[dholth]

PackageFinder is not preceded. It is still doing everything it did before, except it has a tiny patch belied by the large number of commits in the patch that lets it use .whl in addition to .tar.gz and .zip

You could and some people do populate the same local directory with sdists instead of wheels, and use --no-index --find-links there.

[carljm]

@dholth I wasn't talking about the current pull request, I was just replying to the speculation about hypothetical additional caching features in the last few posts (which should probably move to the mailing list).

I do use --no-install --find-links with sdists in all of my projects today; that's effectively similar to a "cache that precedes PackageFinder", though not quite - with something called a "cache" you'd expect it to do pass-through and auto-population.

[dholth]

Sure. We will stop calling it a cache post haste.

The two phase workflow is great for me. You only ping the index when you want to test new versions of dependencies.

On Sep 26, 2012, at 12:06 PM, Carl Meyer notifications@github.com wrote:

@dholth I wasn't talking about the current pull request, I was just replying to the speculation about hypothetical additional caching features in the last few posts (which should probably move to the mailing list).

I do use --no-install --find-links with sdists in all of my projects today; that's effectively similar to a "cache that precedes PackageFinder", though not quite - with something called a "cache" you'd expect it to do pass-through and auto-population.

—
Reply to this email directly or view it on GitHub.

[dholth]

If travis wasn't broken...

[qwcode]

@dholth , @carljm :
here's a thought. break this into 2 pulls. one for install and one for the build tool.
the install portion was ready I think, but the new subcommand could take awhile.
and ideally the install pull could have a lean set of commits (i.e < 62)

[qwcode]

@dholth , I'm willing to do the work to carve out a new pull just for the install portion.
I'd put your name on the commit(s) and get it merge ready. ok?

[dholth]

I'd appreciate it. Credit Rubik and pfmoore too of course.

On Sep 30, 2012, at 1:53 AM, Marcus Smith notifications@github.com wrote:

@dholth , I'm willing to do the work to carve out a new pull just for the install portion.
I'd put your name on the commit(s) and get it merge ready. ok?

—
Reply to this email directly or view it on GitHub.

[qwcode]

ok, will do. hopefully, in a couple of days. will shout out to the interested parties when it's ready.

[pnasrat]

@qwcode looking forward to it.

[pfmoore]

I've just tested the current patch and there's one issue that ought to be investigated. As it stands, it doesn't check the available wheels to determine which ones are compatible with the current Python. So it can install a 1.1 64-bit build into a 32-bit Python, because there's a 64-bit 1.1 available but only 1.0 in 32-bit.

Also, it doesn't rank wheels by compatibility, so it can install a "py33" wheel where there is also a "cp33" wheel. This could result in installing a version without C accelerators when the version with them is also available.

[dholth]

https://bitbucket.org/dholth/pep425tags has the relevant tags. See also https://github.com/dholth/pip/blob/develop/pip/index.py#L335

Note that I have a branch called 'wheel' that has the newest stuff; it is easy to pull develop by accident. The changes don't affect the compat tags issue at all.

[qwcode]

As it stands, it doesn't check the available wheels to determine which ones are compatible with the current Python

if you read way up in the conversation, we (or at least I was) were willing to accept the lack of tag awareness (other than python version) at first in the "experimental" support, but certainly would like to see it soon.

for awhile at atleast using wheels will largely be a local thing. the user will be the one generating their own wheels and using them locally, so they're going to know what they have.

[dholth]

Closing pull request. Rebased.