Support exclusive repositories

[remal]

What would you like Renovate to be able to do?

Our projects uses io.confluent:kafka-avro-serializer dependency from https://packages.confluent.io/maven/ repository. The problem is that this repository has some dependencies that we would like to download from Maven Central only like org.apache.kafka:kafka-clients.

For this purpose we use a code like this in our Gradle build.gradle file:

repositories {
	mavenCentral() // Maven Central by default
	exclusiveContent {
		forRepository {
			maven {
				name = 'confluent'
				url 'https://packages.confluent.io/maven/'
			}
		}
		filter {
			includeGroupByRegex(/^io\.confluent(\..+)?$/) // io.confluent* dependencies exclusively from confluent repository
		}
	}
}

However, Renovate doesn't support Gradle's exclusiveContent or content and Confluent repository has org.apache.kafka:kafka-clients version 7.* (which we don't want to use) vs 3.1.0 in Maven Central. It leads to a situation where Renovate suggest upgrading kafka-clients to version 7.*, because this version can be find in Counfluent repository.

Currently, we deal with this issue by using this Renovate configuration:

{
  java: {
    packageRules: [
      {
        // Maven Central repository only:
        packagePatterns: [
          "^org\\.apache\\.kafka[.:]",
        ],
        registryUrls: ["https://repo.maven.apache.org/maven2/"],
      },
      {
        // Confluent repository only:
        packagePatterns: [
          "^io\\.confluent[.:]",
        ],
        registryUrls: ["https://packages.confluent.io/maven/"],
      },
    ],
  },
}

Personally, I wouldn't say it's very convenient.

Could you please add an ability to define exclusive repositories for some dependencies?

If you have any ideas on how this should be implemented, please tell us here.

A support for Gradle's exclusiveContent or content can be added.

Another option is to add exclusiveRegistryUrls config option.

Also, excludeRegistryUrls config option can be added that is supposed to be used like this:

{
  java: {
    packageRules: [
      {
        packagePatterns: [
          ".*",
        ],
        excludeRegistryUrls: ["https://packages.confluent.io/maven/"], // Don't use Confluent repo by default
      },
      {
        packagePatterns: [
          "^io\\.confluent[.:]",
        ],
        registryUrls: ["https://packages.confluent.io/maven/"], // Use Confluent repo for these dependencies
      },
    ],
  },
}

Is this a feature you are interested in implementing yourself?

No

[viceice]

excludeRegistryUrls is an interesting idea. 🤔

[github-actions]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[msfjarvis]

Here's the minimal reproduction: https://github.com/msfjarvis/renovate-repro-14208

It's not a valid Gradle project but it demonstrates the bug, Renovate sees the Sonatype snapshots repo declared in settings.gradle.kts and attempts to upgrade the dependency to v2.0.0-SNAPSHOT which will not work because Gradle will look for the dependency only in the Maven Central repository.

[rarkins]

Reproduction forked to https://github.com/renovate-reproductions/14208

[jamesdh]

Bumped into this while trying to make our renovate action run faster. It currently takes ~30 minutes to resolve against the various repositories despite making use of exclusiveContent.

Thanks for describing your workaround @remal, will give that a try right now!

[patrickneulichedldmde]

@Churro thanks so much for implementing the regular content filtering.
We just switched to exclusiveContent, because that makes it way easier to not leak any internals from our several in-house repositories.
But now renovate leaks internals to mavencentral all over the place.
I don't quite feel accustomed enough to the renovate code base to implement the exclusiveContent detection.
I would be happy to start a PR with a failing test in parser.spec.ts, but I don't quite get the whole setup to know how to populate and filter the registries then, would someone assist with the rest?

[Churro]

@patrickneulichedldmde, compared to other additions, adding support for exclusiveContent is quite tricky and may require some preparatory refactoring. To prevent renovate from contacting mavenCentral you could define a packageRule and simply overwrite the list of parsed registry URLs, e.g.,

{
  "matchDatasources": ["maven"],
  "registryUrls": ["https://registryA", "https://registryB"]
}

[patrick-dedication]

@Churro thanks for the headsup, I'm trying to implement it since yesterady, let's see how it turns out.
We are using mavenCentral for some Dependencies so simply excluding won't do it.
What I could do is specify content { include... } in addition to exclusive content to get a 95% solution.