Cache is not purged after new release via Warehouse

[di]

There's been a couple reports that seem to indicate that the pypi-legacy cache is not getting correctly purged when a new release is made via Warehouse:

• pip install gets wrong version warehouse#2608
• I don't get my package with pip after "twine upload dist/*" pypa/twine#281
• pip fails to find package even though it is available on PyPI pypa/pip#4888

Manually purging the cache (e.g. curl -XPURGE) fixes the issue.

Warehouse attempts to purge the legacy cache after a new release here: https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L1046-L1062

Manually making this action request doesn't seem to work, while the status is 200, the content is just the pypi.python.org homepage, and the simple page remains the same. I had a typo, but it still fails to purge the cache.

requests.post(
    "https://pypi.python.org/pypi",
    data={"action": "purge", "project": "fontbakery"}
)

Has anything changed recently which would have broken this?

[berkerpeksag]

Another similar report is #736.

[ewdurbin]

will get this patched up, pypi-legacy needs to accept the post (it isn't currently)

[ewdurbin]

@di looks like the request is actually working, note you're missing a single :

>>> r = requests.post(
...     "https://pypi.python.org/pypi",
...     data={":action": "purge", "project": "fontbakery"}
... )
>>> r.content
'OK'

returns the expected OK string and indeed kicks off the purge job.

looks like we need to ensure that warehouse is actually issuing these, a bit more debugging to do :/

[ewdurbin]

Confirmed that the purge call from warehouse is working as expected using:

• 3aafbfa

Then followed up and determined that our internal redis cache for XMLRPC requests was not being purged due to a dumb typo stopping the enqueued task to fail (way too quietly):

• 4e8d3ee

[ewdurbin]

So this would absolutely cause some mismatches in state between our edge cache (Fastly CDN) and the XMLRPC/JSON caches (internal redis cache).

My assumption has always been that pip, uses /simple only, which should have a consistent status between warehouse and pypi-legacy, so this may not be the end of our investigation.

[dstufft]

Sometimes purging fails and legacy doesn't retry, it's probably just that.

[ewdurbin]

@dstufft thanks, i'll metricify that and see if we can fix with some simple retry logic in the task.

[dstufft]

There's a Web UI for RQ somewhere that you can look and see all the failed tasks I think... might be able to correlate them with that too.

[ewdurbin]

RQ history was horrendously messy thanks to the referenced typo.

I did some basic test uploads with one of my useless packages and everything seems to be working, OK.

I cleared it out and we'll keep an eye using rq-dashboard for errors.

[mauritsvanrees]

We had the same with several Plone packages that we released this weekend. See for example this Jenkins test output: http://jenkins.plone.org/job/plone-5.1-python-2.7/785/console

Due to this issue, I am now aware that a curl -X PURGE https://pypi.python.org/simple/my.package/ can help. I did that for two packages and that has fixed the errors for now. Someone else had the same problems earlier this weekend, and they went away after a couple of hours.

I have had problems very occasionally in earlier years, from which in hindsight I wonder if it was the same problem. I have created a new release a couple of times, and that fixed it. Legacy has been pretty stable for a long time now, which is good.

Is warehouse at https://pypi.org/simple/ already the recommended index? Or is that a bit premature?

[di]

Nice sleuthing @ewdurbin. Should we call this closed then?

[ewdurbin]

@di let's let it ride for a few more days and see if any Fastly purges fail on the pypi-legacy side!

[ewdurbin]

I haven't seen any failed purge jobs in the legacy infra since this rolled out. closing for now until we have another case of out of sync caches to investigate.

[schrockn]

Hello! @ewdurbin it looks like I have another case of this. I just pushed a new library of mine called dagster. https://pypi.org/project/dagster/ Pypi lists current rev at 0.1.4

However it cannot be found:

(clarify) [schrockn@mbp ~/code/superconductive/clarify-data-warehouse-201803/clarify_kickoff (mock-out-redshift *)]$ pip install dagster==0.1.4
Collecting dagster==0.1.4
  Could not find a version that satisfies the requirement dagster==0.1.4 (from versions: 0.1.0rc1, 0.1.1, 0.1.2, 0.1.3)
No matching distribution found for dagster==0.1.4

I also attempted to run

curl -X PURGE https://pypi.python.org/simple/dagster/

Not sure what is going on. Thanks!

[ewdurbin]

@schrockn the work around curl -X PURGE https://pypi.python.org/simple/<project name> no longer works, and this codebase has been replace entirely by https://github.com/pypa/warehouse.

However I just confirmed that I'm seeing releases for dagster@0.1.4 on both PyPI's main UI and simple index and don't see any failure in the cache purge task.

Can you provide the output of:

curl -v https://pypi.org/simple/dagster/

[di]

This is likely due to pypa/pip#5670

[di]

TL;DR: @schrockn You should be able to install it with pip install --no-cache dagster==0.1.4, or after waiting for at least 10 minutes.

[schrockn]

Totally right! It works now. Thanks so much for answering my question so quickly and being so responsive.

…

On Tue, Aug 7, 2018 at 1:59 PM, Dustin Ingram ***@***.***> wrote: TL;DR: @schrockn <https://github.com/schrockn> You should be able to install it with pip install --no-cache dagster==0.1.4, or after waiting for at least 10 minutes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#739 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbaFeXBRXh6uSj7upn5URwlJD5Juj2TLks5uOf-qgaJpZM4QqHiy> .