Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support exclusive repositories #14208

Open
remal opened this issue Feb 14, 2022 · 9 comments · May be fixed by #34771
Open

Support exclusive repositories #14208

remal opened this issue Feb 14, 2022 · 9 comments · May be fixed by #34771
Labels
help wanted Help is needed or welcomed on this issue manager:gradle Gradle package manager priority-4-low Low priority, unlikely to be done unless it becomes important to more people type:feature Feature (new functionality)

Comments

@remal
Copy link

remal commented Feb 14, 2022

What would you like Renovate to be able to do?

Our projects uses io.confluent:kafka-avro-serializer dependency from https://packages.confluent.io/maven/ repository. The problem is that this repository has some dependencies that we would like to download from Maven Central only like org.apache.kafka:kafka-clients.

For this purpose we use a code like this in our Gradle build.gradle file:

repositories {
	mavenCentral() // Maven Central by default
	exclusiveContent {
		forRepository {
			maven {
				name = 'confluent'
				url 'https://packages.confluent.io/maven/'
			}
		}
		filter {
			includeGroupByRegex(/^io\.confluent(\..+)?$/) // io.confluent* dependencies exclusively from confluent repository
		}
	}
}

However, Renovate doesn't support Gradle's exclusiveContent or content and Confluent repository has org.apache.kafka:kafka-clients version 7.* (which we don't want to use) vs 3.1.0 in Maven Central. It leads to a situation where Renovate suggest upgrading kafka-clients to version 7.*, because this version can be find in Counfluent repository.

Currently, we deal with this issue by using this Renovate configuration:

{
  java: {
    packageRules: [
      {
        // Maven Central repository only:
        packagePatterns: [
          "^org\\.apache\\.kafka[.:]",
        ],
        registryUrls: ["https://repo.maven.apache.org/maven2/"],
      },
      {
        // Confluent repository only:
        packagePatterns: [
          "^io\\.confluent[.:]",
        ],
        registryUrls: ["https://packages.confluent.io/maven/"],
      },
    ],
  },
}

Personally, I wouldn't say it's very convenient.

Could you please add an ability to define exclusive repositories for some dependencies?

If you have any ideas on how this should be implemented, please tell us here.

A support for Gradle's exclusiveContent or content can be added.

Another option is to add exclusiveRegistryUrls config option.

Also, excludeRegistryUrls config option can be added that is supposed to be used like this:

{
  java: {
    packageRules: [
      {
        packagePatterns: [
          ".*",
        ],
        excludeRegistryUrls: ["https://packages.confluent.io/maven/"], // Don't use Confluent repo by default
      },
      {
        packagePatterns: [
          "^io\\.confluent[.:]",
        ],
        registryUrls: ["https://packages.confluent.io/maven/"], // Use Confluent repo for these dependencies
      },
    ],
  },
}

Is this a feature you are interested in implementing yourself?

No

@remal remal added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:feature Feature (new functionality) labels Feb 14, 2022
@viceice
Copy link
Member

viceice commented Feb 14, 2022

excludeRegistryUrls is an interesting idea. 🤔

@Churro Churro added the manager:gradle Gradle package manager label Oct 16, 2022
@rarkins rarkins added auto:reproduction A minimal reproduction is necessary to proceed priority-4-low Low priority, unlikely to be done unless it becomes important to more people and removed priority-5-triage labels Nov 25, 2022
@github-actions
Copy link
Contributor

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

@msfjarvis
Copy link

Here's the minimal reproduction: https://github.com/msfjarvis/renovate-repro-14208

It's not a valid Gradle project but it demonstrates the bug, Renovate sees the Sonatype snapshots repo declared in settings.gradle.kts and attempts to upgrade the dependency to v2.0.0-SNAPSHOT which will not work because Gradle will look for the dependency only in the Maven Central repository.

@viceice viceice added reproduction:provided and removed auto:reproduction A minimal reproduction is necessary to proceed labels Jan 21, 2023
@rarkins
Copy link
Collaborator

rarkins commented Jan 22, 2023

Reproduction forked to https://github.com/renovate-reproductions/14208

@jamesdh
Copy link

jamesdh commented Apr 1, 2023

Bumped into this while trying to make our renovate action run faster. It currently takes ~30 minutes to resolve against the various repositories despite making use of exclusiveContent.

Thanks for describing your workaround @remal, will give that a try right now!

@rarkins rarkins added help wanted Help is needed or welcomed on this issue status:ready and removed reproduction:provided status:requirements Full requirements are not yet known, so implementation should not be started labels Sep 26, 2023
@JuJup

This comment has been minimized.

@patrickneulichedldmde
Copy link

patrickneulichedldmde commented Mar 8, 2025

@Churro thanks so much for implementing the regular content filtering.
We just switched to exclusiveContent, because that makes it way easier to not leak any internals from our several in-house repositories.
But now renovate leaks internals to mavencentral all over the place.
I don't quite feel accustomed enough to the renovate code base to implement the exclusiveContent detection.
I would be happy to start a PR with a failing test in parser.spec.ts, but I don't quite get the whole setup to know how to populate and filter the registries then, would someone assist with the rest?

@Churro
Copy link
Collaborator

Churro commented Mar 8, 2025

@patrickneulichedldmde, compared to other additions, adding support for exclusiveContent is quite tricky and may require some preparatory refactoring. To prevent renovate from contacting mavenCentral you could define a packageRule and simply overwrite the list of parsed registry URLs, e.g.,

{
  "matchDatasources": ["maven"],
  "registryUrls": ["https://registryA", "https://registryB"]
}

@patrick-dedication
Copy link

@Churro thanks for the headsup, I'm trying to implement it since yesterady, let's see how it turns out.
We are using mavenCentral for some Dependencies so simply excluding won't do it.
What I could do is specify content { include... } in addition to exclusive content to get a 95% solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Help is needed or welcomed on this issue manager:gradle Gradle package manager priority-4-low Low priority, unlikely to be done unless it becomes important to more people type:feature Feature (new functionality)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants