Dokka Gradle Plugin: Implement Gradle Worker Daemon for Dokka generation #2740

aSemy · 2022-11-04T22:39:43Z

Refactor the way dokka-gradle executes a Dokka generation task to use a Gradle Worker Daemon. The Worker is configured to use the same classpath as the existing reflection-based process.

Using a Worker Daemon completely replaces the existing 'reflection magic'. I think this is a lot more simple and easier to understand than the previous method of using reflection to create an instance.

This PR is a further improvement upon #2709, which required a main method. Using the Gradle Worker API means the dokka-core code can be called idiomatically. However, I am not sure about the compatibility with Gradle versions. #2709 might be more compatible...

Benefits:

idiomatic Kotlin
idiomatic Gradle
no reflection magic
easier to configure/debug/test
no API changes for users - this should 'just work'

I haven't tested extensively so there might be some breaking changes. Of particular note:

is the logging output still present and as expected
is the Worker API compatible with the versions of Gradle that Dokka must support?

This PR includes

…ser matching usage in gradle.kts files)

…_plugin

…' into feat/gradle_dokka_executor_service

…ec {}

…sl_plugin

…o_task_package

…y_by_setter

…enerator in an isolated process

…orker_daemon

…sl_plugin

qwwdfsad · 2022-11-28T17:32:24Z

Thanks!
The contribution indeed looks very promising (reflection magic is something we strive to get away from), but unfortunately I can build neither coroutines nor Ktor with this change:

> A failure occurred while executing org.jetbrains.dokka.gradle.workers.DokkaGeneratorWorker
   > org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createForProduction(Lcom/intellij/openapi/Disposable;Lorg/jetbrains/kotlin/config/CompilerConfiguration;Lorg/jetbrains/kotlin/cli/jvm/compiler/EnvironmentConfigFiles;)Lorg/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment;

I've also verified that without your patch, coroutines can be built with Dokka from master branch. Could you please take a look?

qwwdfsad · 2022-11-28T17:34:48Z

It seems like Gradle Worker is missing some dependencies

aSemy · 2022-11-28T17:40:31Z

Thanks for trying it! Yes it's possible that dependencies might be missing now that the Dokka worker executes in a separate process. It should be easy to add them back in though.

(Missing dependencies isn't a big problem, even if it's annoying. Since Dokka creates a Gradle Configuration anyone would be able to fix this themselves, should a similar problem arise.)

It's been a while since I've looked at this.. I can try figuring out what's missing. Do you have the full stack trace? Are there any clues in there?

UPDATE: Ah I see, it's in the GitHub Actions

qwwdfsad · 2022-11-28T17:45:44Z

I've checked it manually

Stacktrace:

Caused by: java.lang.NoSuchMethodError: org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createForProduction(Lcom/intellij/openapi/Disposable;Lorg/jetbrains/kotlin/config/CompilerConfiguration;Lorg/jetbrains/kotlin/cli/jvm/compiler/EnvironmentConfigFiles;)Lorg/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment;
        at org.jetbrains.dokka.analysis.AnalysisEnvironment.createCoreEnvironment(AnalysisEnvironment.kt:107)
        at org.jetbrains.dokka.analysis.EnvironmentAndFacadeKt.createEnvironmentAndFacade(EnvironmentAndFacade.kt:30)
        at org.jetbrains.dokka.analysis.KotlinAnalysisKt.KotlinAnalysis(KotlinAnalysis.kt:13)
        at org.jetbrains.dokka.analysis.KotlinAnalysisKt.KotlinAnalysis$default(KotlinAnalysis.kt:11)
        at org.jetbrains.dokka.base.DokkaBase$defaultKotlinAnalysis$2$1.invoke(DokkaBase.kt:191)
        at org.jetbrains.dokka.base.DokkaBase$defaultKotlinAnalysis$2$1.invoke(DokkaBase.kt:43)
        at org.jetbrains.dokka.plugability.LazyEvaluated.get$core(LazyEvaluated.kt:8)
        at org.jetbrains.dokka.plugability.DokkaContextConfigurationImpl.single(DokkaContext.kt:159)
        at org.jetbrains.dokka.base.translators.psi.DefaultPsiToDocumentableTranslator.<init>(DefaultPsiToDocumentableTranslator.kt:818)
        at org.jetbrains.dokka.base.DokkaBase$psiToDocumentableTranslator$2$1.invoke(DokkaBase.kt:69)
        at org.jetbrains.dokka.base.DokkaBase$psiToDocumentableTranslator$2$1.invoke(DokkaBase.kt:43)
        at org.jetbrains.dokka.plugability.LazyEvaluated.get$core(LazyEvaluated.kt:8)
        at org.jetbrains.dokka.plugability.DokkaContextConfigurationImpl.actions(DokkaContext.kt:164)
        at org.jetbrains.dokka.plugability.DokkaContextConfigurationImpl.get(DokkaContext.kt:147)
        at org.jetbrains.dokka.base.generation.SingleModuleGeneration.translateSources(SingleModuleGeneration.kt:113)
        at org.jetbrains.dokka.base.generation.SingleModuleGeneration.access$translateSources(SingleModuleGeneration.kt:21)
        at org.jetbrains.dokka.base.generation.SingleModuleGeneration$createDocumentationModels$1$invokeSuspend$$inlined$parallelMap$1$1.invokeSuspend(parallelCollectionOperations.kt:17)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

Coroutines branch (note: Dokka is published locally via ./gradlew publishToMavenLocal with the corresponding version): https://github.com/Kotlin/kotlinx.coroutines/tree/dokka-update, command: ./gradlew cleanDokkaHtmlMultiModule dokkaHtmlMultiModule --no-daemon --stacktrace

Dokka is published from your PR, with two changes -- version for publishing and commented out Dispatchers.shutdown

aSemy · 2022-11-28T18:16:51Z

I get an error when looking at AnalysisEnvironment about missing dependencies...

I am very confused why this might be. I don't think this PR changes the dependencies.

UPDATE: This error is confusing, but unrelated. I think it's caused by the kotlin-analysis subproject only depending on the 'shadow' configuration, which does not expose the API dependencies. The error would probably go away if I did a full rebuild, then the shadowed jar would contain the missing dependencies.

dokka/kotlin-analysis/build.gradle.kts

Line 11 in f7691fe

api(project("compiler-dependency", configuration = "shadow"))

aSemy · 2022-11-28T19:01:16Z

@qwwdfsad I think it's fixed now, can you try again please?

The problem was that I didn't add the Dokka plugins to the worker's classpath.
c829043

qwwdfsad · 2022-11-28T19:01:16Z

The error would probably go away if I did a full rebuild, then the shadowed jar would contain the missing dependencies.

Yeah, you have to explicitly reimport your project into IDEA

…ckage' into feat/remove_property_by_setter # Conflicts: # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationSerializableTest.kt

…orker_daemon # Conflicts: # core/api/core.api # core/src/main/kotlin/DokkaBootstrapImpl.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaTask.kt

…rker_daemon

…and safeConvention() with convention()

…' into feat/expose_finalizeCoroutines_in_gradle # Conflicts: # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaTask.kt

…es_in_gradle' into feat/gradle_dokka_worker_daemon # Conflicts: # .github/workflows/qodana.yml

…y_by_setter-original # Conflicts: # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/GradleDokkaSourceSetBuilder.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/sourceSetKotlinGistConfiguration.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/utils.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationJsonTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationSerializableTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaMultiModuleFileLayoutTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/GradleDokkaSourceSetBuilderTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/KotlinDslDokkaTaskConfigurationTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/tasks/DokkaCollectorTaskTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/tasks/DokkaMultiModuleTaskTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/tasks/DokkaTaskTest.kt

…sl_plugin # Conflicts: # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/utils.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaCollectorTaskTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationJsonTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationSerializableTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaTaskTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/GradleDokkaSourceSetBuilderTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/KotlinDslDokkaTaskConfigurationTest.kt

…_task_package # Conflicts: # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationJsonTest.kt

…le_dokka_worker_daemon # Conflicts: # core/src/main/kotlin/DokkaBootstrapImpl.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/tasks/AbstractDokkaTask.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/tasks/DokkaCollectorTaskTest.kt

…te_gradle_utils_with_kotlin_dsl

…dle_utils_with_kotlin_dsl # Conflicts: # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationJsonTest.kt # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationSerializableTest.kt

…h_kotlin_dsl' into feat/gradle_dokka_worker_daemon # Conflicts: # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/tasks/AbstractDokkaTask.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/tasks/DokkaCollectorTask.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/tasks/DokkaTask.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/tasks/DokkaTaskPartial.kt

# Conflicts: # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/tasks/AbstractDokkaParentTaskTest.kt

adam-enko · 2024-08-22T12:12:08Z

Obsolete. Dokka Plugin v2 will use the worker API. #3695

aSemy added 18 commits October 11, 2022 18:40

rename DokkaPlugin file

da79426

apply kotlin-dsl plugin, update code to use idiomatic Gradle API (clo…

907fd7a

…ser matching usage in gradle.kts files)

Merge branch 'fix/rename_dokkaplugin_file' into feat/apply_kotlin_dsl…

fa2d7e5

…_plugin

move Gradle tasks into task package

bd82041

remove the 'by' helper function

b184405

Merge remote-tracking branch 'fork/aSemy/feat/apply_kotlin_dsl_plugin…

c83aa9b

…' into feat/gradle_dokka_executor_service

replace reflection access with a Gradle BuildService that uses javaex…

4de8616

…ec {}

Merge remote-tracking branch 'origin/master' into feat/apply_kotlin_d…

9f1db10

…sl_plugin

Merge remote-tracking branch 'origin/master' into feat/move_tasks_int…

bb31a7f

…o_task_package

add deprecated backwards compatibility for moved Dokka tasks

ac61520

Merge remote-tracking branch 'origin/master' into feat/remove_propert…

e47f1ec

…y_by_setter

try and fix 'Cannot fingerprint input property' issue

79d88bf

update API spec

899632f

mark DokkaExecutorService as @internal on task input

e02001d

use usesService to register DokkaExecutorService

76df6b4

Merge branch 'master' into feat/move_tasks_into_task_package

14719e7

replace DokkaExecutorService with Gradle Worker, for executing DokkaG…

42b4978

…enerator in an isolated process

Merge remote-tracking branch 'origin/master' into feat/gradle_dokka_w…

f8be7b4

…orker_daemon

aSemy mentioned this pull request Nov 4, 2022

Dokka Gradle Plugin: Implement Gradle BuildService for Dokka execution #2709

Closed

aSemy added 2 commits November 4, 2022 23:43

Merge remote-tracking branch 'origin/master' into feat/apply_kotlin_d…

ebdb4e7

…sl_plugin

Merge branch 'master' into feat/gradle_dokka_worker_daemon

e96e87b

include plugins in worker classpath

c829043

expose finalizeCoroutines as an option on Gradle tasks

0ff2edc

aSemy added 12 commits January 28, 2023 23:51

replaced subclasses with typealiases

49db2ad

Merge remote-tracking branch 'fork/aSemy/feat/move_tasks_into_task_pa…

c800ad0

…ckage' into feat/remove_property_by_setter # Conflicts: # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationSerializableTest.kt

Merge remote-tracking branch 'origin/master' into feat/gradle_dokka_w…

e49ece8

…orker_daemon # Conflicts: # core/api/core.api # core/src/main/kotlin/DokkaBootstrapImpl.kt # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaTask.kt

Merge branch 'feat/apply_kotlin_dsl_plugin' into feat/gradle_dokka_wo…

7b29c6c

…rker_daemon

make cast() an expression body fn

7362027

minor fixes regarding naming/spellings

a4172fc

make cast() an expression body fn

8273693

replace getSafe() with get()/orNull, safeProperty() with property(), …

01496ab

…and safeConvention() with convention()

update API dump

1683ef8

maintain API compatibility for moduleName

7cf2f39

Merge remote-tracking branch 'fork/aSemy/feat/apply_kotlin_dsl_plugin…

24b4b98

…' into feat/expose_finalizeCoroutines_in_gradle # Conflicts: # runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaTask.kt

Merge remote-tracking branch 'fork/aSemy/feat/expose_finalizeCoroutin…

c5730de

…es_in_gradle' into feat/gradle_dokka_worker_daemon # Conflicts: # .github/workflows/qodana.yml

aSemy mentioned this pull request Jan 30, 2023

Remove the dependency on internal Gradle API #2822

Closed

aSemy added 8 commits February 1, 2023 10:54

fix imports

0994afb

Merge branch 'feat/apply_kotlin_dsl_plugin' into feat/move_tasks_into…

8754121

…_task_package # Conflicts: # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/DokkaConfigurationJsonTest.kt

Merge branch 'feat/remove_property_by_setter-original' into feat/upda…

f8d0618

…te_gradle_utils_with_kotlin_dsl

IgnatBeresnev mentioned this pull request Feb 2, 2023

Improve compatibility with the Gradle API, and follow best practices #2700

Closed

aSemy added 2 commits February 16, 2023 09:56

Merge branch 'master' into feat/gradle_dokka_worker_daemon

de246ea

# Conflicts: # runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/tasks/AbstractDokkaParentTaskTest.kt

add some workaround for the SamWithReceiver bug

9d463fb

aSemy mentioned this pull request Feb 16, 2023

use a Gradle logger via LoggerAdapter #2875

Closed

vmishenev mentioned this pull request Mar 7, 2023

Use Gradle Worker API #2903

Closed

IgnatBeresnev mentioned this pull request Mar 16, 2023

Replace BiConsumer usages #2921

Closed

adam-enko closed this Aug 22, 2024

aSemy deleted the feat/gradle_dokka_worker_daemon branch August 22, 2024 16:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dokka Gradle Plugin: Implement Gradle Worker Daemon for Dokka generation #2740

Dokka Gradle Plugin: Implement Gradle Worker Daemon for Dokka generation #2740

aSemy commented Nov 4, 2022 •

edited

Loading

qwwdfsad commented Nov 28, 2022

qwwdfsad commented Nov 28, 2022

aSemy commented Nov 28, 2022 •

edited

Loading

qwwdfsad commented Nov 28, 2022

aSemy commented Nov 28, 2022 •

edited

Loading

aSemy commented Nov 28, 2022

qwwdfsad commented Nov 28, 2022

adam-enko commented Aug 22, 2024

Dokka Gradle Plugin: Implement Gradle Worker Daemon for Dokka generation #2740

Dokka Gradle Plugin: Implement Gradle Worker Daemon for Dokka generation #2740

Conversation

aSemy commented Nov 4, 2022 • edited Loading

qwwdfsad commented Nov 28, 2022

qwwdfsad commented Nov 28, 2022

aSemy commented Nov 28, 2022 • edited Loading

qwwdfsad commented Nov 28, 2022

aSemy commented Nov 28, 2022 • edited Loading

aSemy commented Nov 28, 2022

qwwdfsad commented Nov 28, 2022

adam-enko commented Aug 22, 2024

aSemy commented Nov 4, 2022 •

edited

Loading

aSemy commented Nov 28, 2022 •

edited

Loading

aSemy commented Nov 28, 2022 •

edited

Loading