Support Maven 3.9.x to download dependency POMs in parallel

[Eskibear]

Maven 4.0.0-alpha-3 has been released.

One of the significant improvements is downloading poms in parallel, which has been shipped with maven-resolver-1.9.2, included in maven-4.0.0-alpha-3. It largely improves the performance when local maven repository is empty (common case for a new machine).

I tested with a simple spring-boot project, running mvn clean install with an empty local repository, ~85MB dependencies were downloaded and below are the time costs for different configs:

mvn 4.0.0-alpha-3

• bf: 2'55''
• df (default): 5'34''

mvn 3.8.6

• df (only): 6'54''

You can download and try from: https://dlcdn.apache.org/maven/maven-4/4.0.0-alpha-3/binaries/
Add -Daether.dependencyCollector.impl=bf to enable parallel downloading.

[laeubi]

@Eskibear you can use maven 4 already if you run a maven build, just add it as a maven installation and choose that as the default in the m2eclipse settings.

For m2eclipse embedded runtime, this is nothing we plan for the near future unless maven4 is final released, there is a major adoption of maven4 or there is sustainable contribution in form of development efforts and/or funding.

If this is crucial to your business and you likes to speed up the development of m2e in general a sponsoring would allow me to assign more time-slots to bug fixing or contact me for an individual contract for this specific issue.

[Eskibear]

Thank you for the information.

I am working on eclipse.jdt.ls project, which uses m2e to import Maven projects for users. We've received complaints about the startup performance, especially when they open Maven projects for the first time. I have been seeking ways to improve that for a while, and that's why I'm so eager to early investigate the possibility that m2e adopts upstream improvement in maven-resolver, at this stage.

Appreciate for your help! I'm also willing to contribute some effort if there's anything I can help to speed it up. I believe it would benefit a lot of users.

[laeubi]

I am working on eclipse.jdt.ls project, which uses m2e to import Maven projects for users. We've received complaints about the startup performance, especially when they open Maven projects for the first time.

Its always hard to guess what/if it could be improved, for that purpose we have opened:

• What is your favorite maven project that is NOT working with m2eclipse? #1064

but it has not gotten much feedback yet.

I have been seeking ways to improve that for a while, and that's why I'm so eager to early investigate the possibility that m2e adopts upstream improvement in maven-resolver, at this stage.

If the resolver part is that really improves things (I really doubt that first time performance is really something that will be a valuable thing to put effort in as usually one has to fill the maven cache only once), I think it would be better to integrate this into maven 3.9.x line as well m2e uses the maven resolver to fetch dependencies so it will then "just work" with minimal impact for m2e.

I'm also willing to contribute some effort if there's anything I can help to speed it up.

I think an important part would be:

• Setup a dev environment: https://github.com/eclipse-m2e/m2e-core/blob/master/CONTRIBUTING.md
• Integrate m4 and see what breaks and collect issues that might be addressed in the current codebase (e.g. some abstractions as proposed here: Get rid of maven API #1147) I have done this kind of work recently for the archetpye upgrade from 2.x > 3.x
• Prototype some ways to support maven 3.x and maven 4.x in parallel, as we most likely can not assume that all projects/plugins instantly upgrade to m4 and maintaining two distinct editions is currently out of scope of m2e and would also be hard for users that want m3/m4 projects in the same workspace.

I believe it would benefit a lot of users.

If only a few percent of these users would contribute to the tools they use I think this would speed up things dramatically...

[Eskibear]

I think it would be better to integrate this into maven 3.9.x line as well m2e uses the maven resolver to fetch dependencies so it will then "just work" with minimal impact for m2e.

Agree. It would be great if maven-resolver-1.9.2 can be integrated into 3.9.x line, but I see maven-3.8.7 (released on 2022-12-25) is still using maven-resolver-1.6.3. I'm concerning whether maven-resolver 1.9.2 is depending on some new architecture in maven-4.x.

[laeubi]

I think it would be better to integrate this into maven 3.9.x line as well m2e uses the maven resolver to fetch dependencies so it will then "just work" with minimal impact for m2e.

Agree. It would be great if maven-resolver-1.9.2 can be integrated into 3.9.x line, but I see maven-3.8.7 (released on 2022-12-25) is still using maven-resolver-1.6.3. I'm concerning whether maven-resolver 1.9.2 is depending on some new architecture in maven-4.x.

3.8.x is jsut maintain release, but 3.9.x is the next release line for new features.

[stbischof]

this is the PR
apache/maven#881

and this the List of todos fpr 3.9.0 release
https://github.com/apache/maven/milestone/2?closed=1

[Eskibear]

Maven 3.9.0 has been released, with maven-resolver 1.9.4. I tested the performance boost today, running mvn clean package -DskipTests against spring-petclinic project. Result as below:

Results:

• Maven 3.8.6
  • time: 15:59 min
• Maven 3.9.0 + -Daether.dependencyCollector.impl=bf
  • time: 04:35 min

I'll try creating a PR bumping maven-runtime to 3.9.0.

[laeubi]

As mentioned above we can't ugrade to 4.x or 3.9.x if not most of plugins have adopted to it 3.9 already contains some breaking changes, as I know there is some chance for a 3.8.8 release you probably want to help backporting the maven-resolver 1.9.4 to 3.8.x as well.

[laeubi]

@Eskibear by the way if you like to hack m2e a bit yourself and found that especially the resolving of dependencies is slow I recently came into my mind that m2e probably don't need to use "maven" but can use resolver directly, that way we can do whatever fits (and probably even show nice progress bars), here are some pointers:

m2e-core/org.eclipse.m2e.jdt/src/org/eclipse/m2e/jdt/internal/DefaultClasspathManagerDelegate.java

Lines 103 to 104 in 2ca5d4b

	MavenProject mavenProject = facade.getMavenProject(monitor);
	Set<Artifact> artifacts = mavenProject.getArtifacts();

This is where we compute the classpath, it would be good for a first step to not use MavenProject#getArtifacts() but have some new IMavenProjectFacade#getArtifacts() this can as a first step just delegate to the maven project but wrap them in an simple IMavenArtifact that only exposes what we really need here, this will decouple us from the maven internals (maven.model versus aether versus whatever in the future).

If this works, we then can head over to

m2e-core/org.eclipse.m2e.core/src/org/eclipse/m2e/core/internal/project/registry/ProjectRegistryManager.java

Lines 843 to 860 in 0b49dca

	private MavenProject readProjectWithDependencies(IMavenProjectFacade facade) {
	IFile pomFile = facade.getPom();
	IProjectConfiguration resolverConfiguration = facade.getConfiguration();
	Collection<MavenExecutionResult> results = readProjectsWithDependencies(pomFile, resolverConfiguration, null);
	if(results.size() != 1) {
	throw new IllegalStateException("Results should contain one entry.");
	}
	MavenExecutionResult result = results.iterator().next();
	MavenProject mavenProject = result.getProject();
	if(mavenProject != null && !hasError(result)) {
	return mavenProject;
	}
	MultiStatus status = new MultiStatus(IMavenConstants.PLUGIN_ID, 0, Messages.MavenProjectFacade_error);
	for(Throwable e : result.getExceptions()) {
	status.add(Status.error(e.getMessage(), e));
	}
	throw new IllegalStateException(new CoreException(status).fillInStackTrace());
	}

where currently we ask maven to resolve the dependencies of the project, but here we can just call resolver (or whatever other technique) directly and then set this in the model (for backward compat + direct maven usage).

[Eskibear]

Thank you for pointing above out, which helps me to get more familiar with the repo.

For consistency, hacking m2e with different impl from maven-resolver is the last thing I would resort to. Integrating the latest maven-resolver is my first choice for the moment. It would keep as same behavior with Maven CLI as possible, and reduce potential maintenance effort in the future (maybe). The tradeoff is that we will always have to wait upstream updates for new features/fixes, and limitation for further customization (like nice progress bars you mentioned)...

[caiwei-ebay]

@Eskibear
Thanks for the feedback about BF algorithm.
Echoed your comments to https://issues.apache.org/jira/browse/MRESOLVER-324

I think we should share the feedback to Maven and encourage them to make BF as the default option.

[HannesWell]

Since #1238 is merged, can this be closed?
Btw. you can try out the latest snapshot from:
https://download.eclipse.org/technology/m2e/snapshots/latest/

[Eskibear]

I tried snapshots from Jenkins CI for #1238 a few days ago, and it worked as expected. I've also created a PR in downstream jdt.ls project to consume it, let's close this issue.