Fix remaining issues on the synchronization of the registry #1599

mssola · 2018-01-18T10:12:36Z

There are some pending issues from the synchronization of the registry. This issue tries to be an aggregate of old issues.

Crono deletes everything

Some issues pointed out that old crono might remove all the contents. See #663.

Crono is adding & removing images

See issues: #1184, #1261, #1160 and #1232.

Vesli · 2018-01-22T15:12:34Z

Hey mssola, how is this issue going on?

Here at my company I wanted to have a go with Portus and fortunately we are using it for our staging environment only at the moment.

I say fortunately because we are running it directly on an OpenSuse machine and over the weekend Portus deleted all the images, from its DB and from the registry!

I saw that for some people it pushes it again after some time, but on our case it did nothing of the sort.

I'm on Portus 2.2 and registry 2.6.
Should I move for the docker version instead or stop using Portus for now?

Thanks!

mssola · 2018-01-22T16:14:38Z

I say fortunately because we are running it directly on an OpenSuse machine and over the weekend Portus deleted all the images, from its DB and from the registry!

That is ... bad 😞 Could you possibly collect the logs, and possibly get into more detail on your deployment strategy. Even if it's a bit old, it might help us to fix this situation.

I'm on Portus 2.2 and registry 2.6. Should I move for the docker version instead or stop using Portus for now?

Well, I'd advise in favor of this solution. Bear in mind that we are in the process of releasing the 2.3 version, so you might want to take a look at this (since moreover it has some deployment changes). Check our examples directory for some production-ready examples (just use opensuse/portus:2.3 instead of opensuse/portus:head, since there are no stability guarantees for 2.3 😉).

marc0s · 2018-02-01T17:19:00Z

I'm using portus:head and registry:2.6 and I'm hitting this issue. Any workaround in the meantime?
What I see in the logs is

background_1  | Net::OpenTimeout: connection timed out.
background_1  | [catalog] Could not fetch manifest for 'redacted/image2' with tag 'latest': Net::OpenTimeout: connection timed out.
background_1  | Net::OpenTimeout: connection timed out.
background_1  | [catalog] Could not fetch manifest for 'redacted/image2' with tag 'latest': Net::OpenTimeout: connection timed out.
background_1  | [catalog] Removed the tag 'latest'.
background_1  | [catalog] Removed the image 'image2'.
background_1  | [catalog] Created the tag 'latest'.

Each of the images is around 650MB and there are just 5 of them (is a recently created registry).

Thanks!

mssola · 2018-02-09T10:38:18Z

@marc0s see this comment. This might even fix this issue (we'd need to identify some corner cases when enabled, but it should fix the situation for most people).

We have historically struggled with the synchronization of the Registry. This was developed at first because the following situation could happen: 1. User pushes image:tag. 2. Portus authorizes the transaction. 3. Registry sends the event to Portus with all the data, but Portus is unavailable. 4. Portus is up again but it missed the registry event. This is not the case anymore, because all the things that could make Portus unavailable have been either fixed or moved into the background process. Thus, this problem can be ignored in most cases. Moreover, this feature is quite dangerous because a bug on this code can make Portus wipe all the repositories, and historically "funny" issues have happened. This commit builds upon SUSE#1631 where background tasks can be disabled, and it disables the `sync` task by default. Moreover, it also adds a `sync-strategy` option which is available when users enable this feature. This option has four possible values: - `update-delete`: the same behavior we've had up until now. - `update`: similar to `update-delete` but this one will not delete repositories which no longer exist on the registry. This is useful for users which don't trust the risky `update-delete` strategy. - `on-start`: execute `update-delete` only once (on start). - `initial`: execute `update-delete` only once on start and only if the registry is empty. This is the default strategy since it might be convenient for new users that have already a running registry. This commit does not fix all the issues mentioned in SUSE#1631, but it hopefully does the trick for most users. Fixes SUSE#1650 Fixes SUSE#1664 See SUSE#1599 Depends on SUSE#1631 Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

We have historically struggled with the synchronization of the Registry. This was developed at first because the following situation could happen: 1. User pushes image:tag. 2. Portus authorizes the transaction. 3. Registry sends the event to Portus with all the data, but Portus is unavailable. 4. Portus is up again but it missed the registry event. This is not the case anymore, because all the things that could make Portus unavailable have been either fixed or moved into the background process. Thus, this problem can be ignored in most cases. Moreover, this feature is quite dangerous because a bug on this code can make Portus wipe all the repositories, and historically "funny" issues have happened. This commit builds upon SUSE#1631 where background tasks can be disabled. Moreover, it also adds a `sync-strategy` option which is available when users enable this feature. This option has four possible values: - `update-delete`: the same behavior we've had up until now. - `update`: similar to `update-delete` but this one will not delete repositories which no longer exist on the registry. This is useful for users which don't trust the risky `update-delete` strategy. - `on-start`: execute `update-delete` only once (on start). - `initial`: execute `update-delete` only once on start and only if the registry is empty. This is the default strategy since it might be convenient for new users that have already a running registry. This commit does not fix all the issues mentioned in SUSE#1631, but it hopefully does the trick for most users. Fixes SUSE#1650 Fixes SUSE#1664 See SUSE#1599 Depends on SUSE#1631 Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

We have historically struggled with the synchronization of the Registry. This was developed at first because the following situation could happen: 1. User pushes image:tag. 2. Portus authorizes the transaction. 3. Registry sends the event to Portus with all the data, but Portus is unavailable. 4. Portus is up again but it missed the registry event. This is not the case anymore, because all the things that could make Portus unavailable have been either fixed or moved into the background process. Thus, this problem can be ignored in most cases. Moreover, this feature is quite dangerous because a bug on this code can make Portus wipe all the repositories, and historically "funny" issues have happened. This commit builds upon SUSE#1631 where background tasks can be disabled. Moreover, it also adds a `sync-strategy` option which is available when users enable this feature. This option has four possible values: - `update-delete`: the same behavior we've had up until now. - `update`: similar to `update-delete` but this one will not delete repositories which no longer exist on the registry. This is useful for users which don't trust the risky `update-delete` strategy. - `on-start`: execute `update-delete` only once (on start). - `initial`: execute `update-delete` only once on start and only if the registry is empty. This is the default strategy since it might be convenient for new users that have already a running registry. This commit does not fix all the issues mentioned in SUSE#1599, but it hopefully does the trick for most users. Fixes SUSE#1650 Fixes SUSE#1664 See SUSE#1599 Depends on SUSE#1631 Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

We have historically struggled with the synchronization of the Registry. This was developed at first because the following situation could happen: 1. User pushes image:tag. 2. Portus authorizes the transaction. 3. Registry sends the event to Portus with all the data, but Portus is unavailable. 4. Portus is up again but it missed the registry event. This is not the case anymore, because all the things that could make Portus unavailable have been either fixed or moved into the background process. Thus, this problem can be ignored in most cases. Moreover, this feature is quite dangerous because a bug on this code can make Portus wipe all the repositories, and historically "funny" issues have happened. This commit builds upon #1631 where background tasks can be disabled. Moreover, it also adds a `sync-strategy` option which is available when users enable this feature. This option has four possible values: - `update-delete`: the same behavior we've had up until now. - `update`: similar to `update-delete` but this one will not delete repositories which no longer exist on the registry. This is useful for users which don't trust the risky `update-delete` strategy. - `on-start`: execute `update-delete` only once (on start). - `initial`: execute `update-delete` only once on start and only if the registry is empty. This is the default strategy since it might be convenient for new users that have already a running registry. This commit does not fix all the issues mentioned in #1599, but it hopefully does the trick for most users. Fixes #1650 Fixes #1664 See #1599 Depends on #1631 Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

mssola · 2018-02-15T13:19:29Z

We merged a PR yesterday (#1675) which fixes most issues (and works around existing ones). The image has not been built yet on the docker hub (it's in "Queue" for quite some time now ...), but once that is built you'll be able to pull that and have a better experience with this piece of Portus 😉

This commit is more explicit on what to do when a catalog error happens. In this case, if a plain catalog error happens (fetching the list of repositories and their manifests), then nothing will be done. If fetching the catalog worked, and fetching the list of tags then suddenly fails, before this commit that repo would've get nuked because it would've get a repository with no tags. This commit instructs to skip that repository when considering whether to remove or not some repositories that are found to be dangling. See SUSE#1293 See SUSE#1599 Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

mssola · 2018-04-17T14:40:12Z

See #1787 for some fixes on the update-delete policy. After some further tests, we will add these fixes into a new patch-level release (2.3.3) so everyone can pull the fixes right away.

This commit is more explicit on what to do when a catalog error happens. In this case, if a plain catalog error happens (fetching the list of repositories and their manifests), then nothing will be done. If fetching the catalog worked, and fetching the list of tags then suddenly fails, before this commit that repo would've get nuked because it would've get a repository with no tags. This commit instructs to skip that repository when considering whether to remove or not some repositories that are found to be dangling. See SUSE#1293 See SUSE#1599 Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

This commit is more explicit on what to do when a catalog error happens. In this case, if a plain catalog error happens (fetching the list of repositories and their manifests), then nothing will be done. If fetching the catalog worked, and fetching the list of tags then suddenly fails, before this commit that repo would've get nuked because it would've get a repository with no tags. This commit instructs to skip that repository when considering whether to remove or not some repositories that are found to be dangling. See #1293 See #1599 Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>

mssola · 2018-04-18T15:07:05Z

See #1787 for some fixes on the update-delete policy. After some further tests, we will add these fixes into a new patch-level release (2.3.3) so everyone can pull the fixes right away.

This has already been cherry picked into the v2.3 branch and now it's part of the 2.3 docker image.

mssola · 2018-04-19T14:35:01Z

I've deployed a Portus instance for a day, and after messing with it a couple of times, everything seems reliable. I'll do a couple of more tests and I'll take another look at the code, but if everything is fine I'll simply close this issue. Feedback is welcome 😄

mssola · 2018-04-25T08:45:32Z

I believe this can be closed. Let's create new issues for possible regressions or new bugs on this.

mssola added the bug label Jan 18, 2018

mssola added this to the Release 2.4 milestone Jan 18, 2018

This was referenced Feb 2, 2018

couldn't delete tag from registry by portus #1402

Closed

Background job is pushing and deleting images #1650

Closed

This was referenced Feb 20, 2018

Crono Job deletes all entries if it can't connect to the registry #1292

Closed

Sync existing docker repo with portus #1170

Closed

This was referenced Apr 11, 2018

portus keeps deleting and creating repos by itself #1771

Closed

Return nil for tags on unsuccessful catalogs #1293

Closed

mssola mentioned this issue Apr 17, 2018

sync: do not remove repositories on some errors #1787

Merged

mssola closed this as completed Apr 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix remaining issues on the synchronization of the registry #1599

Fix remaining issues on the synchronization of the registry #1599

mssola commented Jan 18, 2018 •

edited

Loading

Vesli commented Jan 22, 2018

mssola commented Jan 22, 2018

marc0s commented Feb 1, 2018

mssola commented Feb 9, 2018

mssola commented Feb 15, 2018

mssola commented Apr 17, 2018

mssola commented Apr 18, 2018

mssola commented Apr 19, 2018

mssola commented Apr 25, 2018

Fix remaining issues on the synchronization of the registry #1599

Fix remaining issues on the synchronization of the registry #1599

Comments

mssola commented Jan 18, 2018 • edited Loading

Crono deletes everything

Crono is adding & removing images

Vesli commented Jan 22, 2018

mssola commented Jan 22, 2018

marc0s commented Feb 1, 2018

mssola commented Feb 9, 2018

mssola commented Feb 15, 2018

mssola commented Apr 17, 2018

mssola commented Apr 18, 2018

mssola commented Apr 19, 2018

mssola commented Apr 25, 2018

mssola commented Jan 18, 2018 •

edited

Loading