Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support baseBranches when forkMode=true #7850

Open
rfkm opened this issue Dec 1, 2020 · 15 comments
Open

Support baseBranches when forkMode=true #7850

rfkm opened this issue Dec 1, 2020 · 15 comments
Labels
priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others type:feature Feature (new functionality)

Comments

@rfkm
Copy link

rfkm commented Dec 1, 2020

What Renovate type, platform and version are you using?

GHE, 23.89.3

Describe the bug

We are running a self-hosted renovate bot in fork mode. If we include a non-default branch in baseBranches, the same PR is created multiple times or conflicting PRs are created for that branch.
Non-default branches appear to be out of sync with fork and upstream. I wondered if this was creating bad PRs.

Relevant debug logs

 INFO: Repository started (repository=my-org/my-repo)
       "renovateVersion": "23.89.3"
DEBUG: Using localDir: renovate/workspace/repos/github/my-org/my-repo (repository=my-org/my-repo)
DEBUG: initRepo("my-org/my-repo") (repository=my-org/my-repo)
DEBUG: Overriding default GitHub endpoint (repository=my-org/my-repo)
       "endpoint": "https://my-ghe/api/v3/"
DEBUG: my-org/my-repo default branch = develop (repository=my-org/my-repo)
DEBUG: Bot is in forkMode (repository=my-org/my-repo)
DEBUG: Found existing fork (repository=my-org/my-repo)
       "repository_fork": "my-bot/my-repo"
DEBUG: Updating forked repository default sha to match upstream (repository=my-org/my-repo)

Updating forked repository default sha to match upstream

Synchronization is likely to be performed here, but only the default branch will be updated.

// This is a lovely "hack" by GitHub that lets us force update our fork's master
// with the base commit from the parent repository
try {
logger.debug(
'Updating forked repository default sha to match upstream'
);
await githubApi.patchJson(
`repos/${config.repository}/git/refs/heads/${config.defaultBranch}`,

To Reproduce

system conf:

module.exports = {
    platform: "github",
    endpoint: "https://my-ghe/api/v3/",
    token: process.env.GITHUB_TOKEN,
    forkToken: process.env.GITHUB_TOKEN,
    baseDir: "renovate/workspace",
    gitAuthor: "renovate <***>",
    includeForks: true,
    forkMode: true,
    logLevel: "debug"
};

repository conf:

{"baseBranches": ["master", "develop"]}

Additional context

@rarkins rarkins added type:bug Bug fix of existing functionality priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others labels Dec 1, 2020
@rarkins
Copy link
Collaborator

rarkins commented Dec 1, 2020

We currently only sync the default branch during repository initialisation so that would explain it. We'd need to do the syncing during base branch iteration but that's going to require quite a refactoring to achieve. We probably should note in the documentation that baseBranches is currently incompatible with forkMode

@rarkins
Copy link
Collaborator

rarkins commented Jan 13, 2022

@HonkingGoose could you also add this note regarding forking renovate? i.e. that baseBranches it not currently supported

@simonbasle
Copy link

Attempting to revive this issue. There is an API dedicated to syncing a fork with its upstream (branch by branch basis) since September 2021: https://docs.github.com/en/rest/reference/branches#sync-a-fork-branch-with-the-upstream-repository

Could this be used to synchronize baseBranches ? Or, as an MVP, to synchronize only if there is a single baseBranch in the array ?

With my limited understanding of how that part of Renovate works, I feel like it could be a viable solution with the following characteristics:

  • the config lives on the default branch of the upstream (main)
  • the config includes a single baseBranches (MVP) or more than one baseBranches (ideal case)
  • the initialization of the fork should still synchronize the default branch to get config updates (using the hack which AFAIK is here because we might not otherwise "know" the name of the default branch)
  • the initialization of the fork should then synchronize baseBranches using the above API

My javascript/typescript is rusty (that's an understatement) and I'm sure I'm missing something, but looking at lib/modules/platform/github/index.ts it feels like it could work.

@rarkins
Copy link
Collaborator

rarkins commented Mar 21, 2022

Another challenge to solve is knowing the config filename to avoid many try/catch/retry attempts to read it. Or limiting it to renovate.json if baseBranches support is necessary. Also I think the original "hack" might work for base branches too using your proposed approach

@simonbasle
Copy link

Another challenge to solve is knowing the config filename to avoid many try/catch/retry attempts to read it.

Isn't that already the case with the current arrangement? Or maybe the current expectation is that the config is read from each "base branch(es)"?

Having a constraint on the renovate config file location seems like a small price to pay 👍

@rarkins
Copy link
Collaborator

rarkins commented Mar 21, 2022

Another challenge to solve is knowing the config filename to avoid many try/catch/retry attempts to read it.

Isn't that already the case with the current arrangement? Or maybe the current expectation is that the config is read from each "base branch(es)"?

We already have that challenge in general, but we address it after clone. In other words much later than "initialization of the fork" as you wrote here:

  • the initialization of the fork should then synchronize baseBranches using the above API

So if you first clone the repo, discover the config file name, and THEN use the github API to sync each base branch, then the repo on github.com would be up to date but your local copy would not be (you'd need to pull each branch too, so it starts getting messy).

@rarkins rarkins changed the title Renovate doesn’t update non-default branches in a fork Forking Renovate doesn’t update non-default base branches Apr 21, 2023
@rarkins
Copy link
Collaborator

rarkins commented Apr 22, 2023

We need to revive and extend the changes I made in #13808

These were reverted out of caution, it turned out to be unnecessary.

@rarkins rarkins added type:feature Feature (new functionality) and removed type:bug Bug fix of existing functionality labels Oct 20, 2023
@rarkins rarkins changed the title Forking Renovate doesn’t update non-default base branches Support baseBranches when forkMode=true Oct 20, 2023
@janhoy
Copy link
Contributor

janhoy commented Jan 24, 2025

So I'm trying to figure out what's the current situation here. I made a comment #14253 (comment) to a discussion to see if there is a "hack" one can use to overcome the limitations.

@rarkins Can you sum up your understanding of where this stands now in 2025?

Background: For our project (Apache Solr) we tried dependabot but it could not handle our requirement to run postUpgradeTasks. Also, as only committers are allowed content-write access to the main code repository, we really need the forking mode to compy with policies, so we have our own @solrbot user running renovate, currenlty on default main branch only. I have attempted using baseBranches to add another branch, but that was catastropic, renovate went crazy, renamed all 50 open PRs with (main) suffix in titles, then autoclosed them all, and nothing happened on the other non-default branch. So I have reverted to old setting with only main-branch.

PS: I have already contributed to renovate (got the T-shirt 😉), so I could be willing to collaborate on this, or at least test out a feature-branch.

@rarkins
Copy link
Collaborator

rarkins commented Jan 24, 2025

@janhoy thanks for your interest

Copy-pasting my answer from #14253 (reply in thread) to here:

  • To work properly, each base branch in the fork must be up to date with the same branch in the origin, before Renovate starts detecting dependencies
  • The only way Renovate does this today is to use a REST API on github.com (not using git) and it's done for the default branch only, before the regular Renovate logic starts

Which brings us to #13586 (which is a prerequisite for this issue here).

I already wrote code for that long ago here: #13808

It was reverted in #13831 but that was out of misunderstanding/caution. It turned out that the errors came from a token which didn't have workflow permissions, so would have had problems using either approach of syncing branches.

So if you're feeling motivated to progress this, I'd recommend taking #13808 as a starting point and essentially just porting it to today's main branch, which might be a little different. Let me know if you'd like to be assigned!

@janhoy
Copy link
Contributor

janhoy commented Feb 25, 2025

So if you're feeling motivated to progress this, I'd recommend taking #13808 as a starting point and essentially just porting it to today's main branch, which might be a little different. Let me know if you'd like to be assigned!

Hi again. See my PR for an attempt of bringing your 3 year old commit to main branch. Some files had moved and some adaptions needed. We can continue discussion over in the PR.

@janhoy
Copy link
Contributor

janhoy commented Feb 26, 2025

I'd like to start a new PR on top of the git fetch one, to make baseBranches work in forking mode.

Q: Do we have a test harness capable of simulating multi multi branch github in forking mode? If so it would be great to first add a unit test to #34467 proving all branches get synced, and another (failing) test for full baseBranches support.

@rarkins
Copy link
Collaborator

rarkins commented Feb 26, 2025

We're right in the middle of switching from jest to vitest so it would be best to hold off temporarily from adding or modifying further tests until it's merged

@janhoy
Copy link
Contributor

janhoy commented Feb 27, 2025

My own understanding of the code involved is getting better.
I've setup a dummy repo with two branches, for testing, and I run the PR-branch code in my IDE, with a token that forks to another github user.

Syncing the main branch in the fork went well, but of course during extractDependencies we get error

WARN: Base branch does not exist - skipping (repository=cominvent/dummy, baseBranch=dev)

I first attempted adding sync to util/git#syncGit(), but soon found out as noted above, that we have not parsed repo config yet at this point. So I had a shot at repository/process#extractDependencies() where we do have access to a scm object. However, the scm tool, although similar to git, does not have capability of handling upstream/.

If at all possible to sync git repo in the handy for (const baseBranch of config.baseBranches) loop before extraction, that would be sweet. But we'd have to use the git tooling, and would probably need to validate first that the checked-out repo is in fact a git(hub) repo. Ideas?

@simonbasle Are you still interested in this feature?

@simonbasle
Copy link

simonbasle commented Feb 27, 2025 via email

@janhoy
Copy link
Contributor

janhoy commented Feb 27, 2025

I made progress in #34532 which succeeds in syncing each baseBranch in the exiting loop inside extractDependencies().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others type:feature Feature (new functionality)
Projects
None yet
Development

No branches or pull requests

4 participants