Kotlin Scripting Support

[ligee]

https://github.com/Kotlin/KEEP/blob/master/proposals/scripting-support.md

replaces https://github.com/Kotlin/KEEP/blob/master/proposals/script-definition-template.md

[DaanDeMeyer]

This looks really promising!

Would this allow me to distribute a single self contained script template which could then be opened in Intellij without any other configuration being required? Ideally I could distribute a script template with a reference to the script definition and Intellij would automatically download this definition from Maven and provide rich code completion.

[ligee]

@DaanDeMeyer this is not yet in this KEEP, but it could be an interesting addition. I'll think about it. Thanks!

[artem-zinnatullin]

@ligee 2 questions:

• How is Spek related to scripting?
• What about unit testing for Kotlin Scripts? (tests for scripts themselves)?

Thanks!

[holgerbrandl]

Great that you guys are planning to push scripting to a new level. It's an amazing KEEP and I especially liked the bits about declarative bootstrapping of an IDE dependency and runtime environment. It's also great to read how you envision to cover the whole range from build-files to shell scripts with a unified framework addition.

Though, it reads like a very ambitious plan to me including lots of sub-projects. So I'd be curious about the indented time-frame.

1. You mentioned the difficulties of parameterized shebangs, and indeed they don't work across shells, so using a declarative approach is the way to go I think.

2. It would be helpful to spec out the declarations first, so that third party tools like my kscript (see https://github.com/holgerbrandl/kscript) could support these declarations rather soon in order to fill the gap until an official JB solutions is ready. Once the latter is done, kscript would be obsolete for good.

Looking forward to kontlinconf so that we can chat more about it!

[ligee]

@artem-zinnatullin:
About Spec - it is described here - https://github.com/Kotlin/KEEP/blob/scripting/proposals/scripting-support.md#script-based-dsl. Basically, it means using scripting infrastructure to produce regular project class files from script-like sources with reduced boilerplate, like necessity to place spec tests inside a class.
About unit testing scripts - it is, of course, possible to use script runner inside tests, but we do not plan any additional helpers/facilities to simplify scripts testing. Maybe we'll consider it later though.

[ligee]

@holgerbrandl, thank you for the feedback. We will try to keep the size of the work here under control. Significant parts of the infrastructure changes described here are, in fact, reshapings and unifyings of the existing scripting infrastructure. But there should be, of course, some particular implementations on top of that, and implementing them will take time, but with clearly defined interfaces we may be able to outsource it partly to the community.
But at the moment it is work in progress, so the plans could change.
As of the specs for declarations - we'll try to keep it mind. But first, we have to finish more general infrastructure specs.

[artem-zinnatullin]

@ligee

About Spec - it is described here - https://github.com/Kotlin/KEEP/blob/scripting/proposals/scripting-support.md#script-based-dsl

Yep, I saw Spek there, that's why I asked :)

like necessity to place spec tests inside a class

Spek is a pretty complex test framework that requires test engine/runner to work (currently Spek uses JUnit5 infrastructure). Also, usually you run tests through build system, not sure how scripting will handle that.

This can save one nesting level by removing top-level class (which doesn't look as a pain point atm), but on the other hand running tests seems to become much more complicated. Technically everything is possible, but I don't think this is direction we want to move Spek into at the moment.

cc @raniejade, @hhariri PTAL

[apatrida]

@ligee for script support, it looks like you have something roughed in for whitelist/blacklisting of functions/classes that can be accessed. This will be important for us to embed Kotlin scripting in Elasticsearch or other engines like Arango, Neo4j, etc. Ensuring what can be called at any given moment, or even what classes can be loaded. Lastly, time limiting of the script running, or loop iteration limiting protections as well (endless loops kill these type of systems). The more integrated this protection can be, the better.

[unlimitedsola]

[gtnarg]

Is there anywhere to follow the progress on this?

[ligee]

I'm working now on the updated proposal, along with the series of the prototypes. As soon as it will be ready, I'll publish the updated version. Hopefully, it will happen within a few weeks from now.

[sdeleuze]

I am interested to leverage these improvements in Spring and Riff.

[apatrida]

@ilya-g please add the requirement: Between compilation and class loading, there should be the option for a verifier that can approve or reject a script. This allows secure scripting to be tied into anything that executes those steps concurrently without needing to re-write every permutation of script runners that might be created. Allowing something like Cuerentena integration https://github.com/kohesive/cuarentena ... There should never be a hard-coded "compile and load class" without the ability to interject between those steps, before class loading.

[bhtek]

@apatrida Thanks. In cases where the script content came from outside of development, e.g. some simple formula for calculating P&L etc. It is critical to validate before loading for the feature to be usable in production.

[ligee]

The 1.2.50 is out with the new experimental scripting support. The KEEP is updated too, reflects now the approach taken with 1.2.50 and actual implementation status.
Not all feedback is incorporated (yet) into the KEEP, but I believe that what we have now is a good foundation that could be developed in the right direction relatively quick and easy.
Any feedback will be highly appreciated!

[ligee]

@apatrida, the current approach allows to write a scripting host that will get a compiled script before instantiating/evaluating it and can perform any checks at that point. Could it be an acceptable solution for your case, or you rather believe that we should introduce another entity - a Checker - with a predefined interface, that could be used with standard hosts?

[apatrida]

@ligee it should be after bytecode is generated but before any class is loaded to prevent attacks during static initialization. And would be nice if many implementations that might exist of scripting hosts don't all need rewritten to add verification. So some plugin point that can be exposed in the common hosts and hopefully whatever wrapping people do around those.

I'll look at what is in 1.2.50 and see how it is currently set up.

[ligee]

@apatrida it could be done now exactly at the moment you described, but having an extension point for it in the standard host interface looks like a good idea, I'll think about it.

[udalov]

@ligee As noted by @rgoldberg in #124, kotlin-scripting-common is listed twice in the "Implementation status" section, which is probably a typo

[yschimke]

It would be great if this optionally supported continuations via an implicit surrounding runBlocking call. Avoid nasty boilerplate and allow suspending functions to be called from top level (without nesting).

[LouisCAD]

@yschimke You can already have top level suspend fun in .kts files

[yschimke]

@LouisCAD to clarify, I really meant calling them at the toplevel e.g. without runBlocking here

https://github.com/yschimke/oksocial/blob/master/src/test/kotlin/commands/text-to-speech.kts#L71-L78

[cromefire]

That sounds great, would love to see that advance and be incorporated in some kind of API in order for it be much easier for users of scripting so that they don't have to worry about which stuff to block, what to allow and which compiler options to add in order to be safe (I'll probably try but as someone who's relatively familiar with the language, but not a developer it's probably quite hard to find all the "exploits")

[xGnoSiSx]

To be fair, I gave up expecting progress on this. I can have more success if I sit down and code some version of lisp in kotlin over the weekend, as a solution.

[Snipx]

Hi, could anyone give me some context on how can I connect ScriptEvaluationConfiguration and ScriptCompilationConfiguration? Namely, I want to pass the state/data from RefineScriptCompilationConfigurationHandler to ScriptEvaluationConfiguration somehow. The specific use case is setting providedProperties conditionally, based on presence of some annotation in a script. I am using RefineScriptCompilationConfigurationHandler to find the annotation of the given type in the ScriptCollectedData and based on whether I found that annotation, I add new property to providedProperties of the ScriptCompilationConfiguration. However, I also need to pass the property value into ScriptEvaluationConfiguration based on the very same condition and I cannot seem to find a way to do so (check whether the annotation is present in the source file). I have explored ScriptEvaluationConfigurationRefinementContext and I can't see a way to go back to the ScriptCollectedData or something like this. The closest I've got is context.compiledScript.getClass(context.evaluationConfiguration).valueOrNull()?.annotations but it returns empty list, probably because @file:Abc annotation does not end up in the compiled script.

I the documentation for Scripting host configuration I read that "These properties are defined by the scripting host and contain general parameters needed for all scripting services." but I don't see a way to store those options in the hosting config and it does not seem right at all.

Thanks. If the description is not clear I am happy to provide more context.

[ligee]

@Snipx, it seems that your scenario is not covered, and it makes sense to introduce a link from evaluation configuration to the compilation one. Could you please file the issue in YouTrack, basically with the description you provided.
As a workaround you can create a variable shared between configurations. See, e.g., how providedProperties property is defined, define your own similarly, make it pointing out to a separately allocated object from both configurations and modify this object in your handler.

[Snipx]

Leaving the link to the ticket for introducing the link from evaluation configuration to the compilation configuration: https://youtrack.jetbrains.com/issue/KT-48758

[aartiPl]

Hi, I am the current maintainer of 'kscript' and would like to align kscript with Kotlin Scripting implementation.
@ligee - what is the current status of Kotlin Scripting? Is there a list of annotations (with exact syntax) that, would be used in KS?

I also wonder if Kotlin Scripting can be used in kscript as backing implementation. Can you elaborate on that?

[ligee]

Hi, @aartiPl!
I'm glad to hear that you're thinking about using Kotlin Scripting API in kscript. A couple of years ago, I've tried to do some porting myself, but it ended up to be a bit more work than I anticipated. But it will be very nice if you'll do it, this will allow us to use common core implementation with all the relevant benefits of it.

The status of the relevant API is still "experimental", but we are planning to "deexperimentalize" it in the nearest future, maybe even in 1.8.x. This will mean that we'll move the API to the new packages without "experimental" part, and agree to support it in the foreseeable future. There are no significant changes planned in the API itself so far.

So I guess you can freely proceed with "porting" or aligning now, just take into account the future renaming.

There is very little documentation so far, unfortunately. The annotations and other part of the API could be extracted for the documented source code in the Kotlin repo. All relevant subprojects are located at https://github.com/JetBrains/kotlin/tree/master/libraries/scripting.

[aartiPl]

In regards to annotations - I can find only 3 in the source code:

But kscript is having much more:
MavenRepository, Repository -> (ok)
DependsOn, (DependsOnMaven) -> (ok)
Include -> can not find it; in some snippets, I could see Import annotation
EntryPoint ->(?)
KotlinOpts, KotlinOptions -> (?)
CompilerOpts, CompilerOptions -> (?)

In the new kscript (4.1) I have deprecated all comment-based annotations. Also, I have changed existing annotations to be aligned with Kotlin-scripting annotations:

1. added Repository (in addition to deprecated MavenRepository) -> (how to pass user/password in the form of Array)
2. added CompilerOptions (in addition to deprecated CompilerOpts) -> I have seen it in some snippets, but no exact syntax
3. added KotlinOptions - see p. 2

Will you support also additional annotations? What about Import/Include? Is there any artifact with annotations definition which can be imported into the project, so that annotations are recognized (like: https://github.com/holgerbrandl/kscript-annotations)?

[ligee]

@aartiPl, there are a couple of other annotations, defined in https://github.com/JetBrains/kotlin/blob/master/libraries/tools/kotlin-main-kts/src/org/jetbrains/kotlin/mainKts/annotations.kt, but they are not a part of the "official API" yet, although the underlying mechanisms are part of the API. We may make it official in the future, but it is not decided yet. For us as the API providers, it is important to have the underlying semantics to be common and reused, which annotations are used to trigger it is less important at this point.
But you're free, of course, to reuse the annotations from main-kts as well, we just cannot guarantee at the moment that these will stay the same for a long time.

So, to the list of annotations. We have:

• Repository probably corresponding to the MavenRepository one in kscript
• DependsOn - corresponding to DependsOnMaven
• CompilerOptions - from man-kts, not part of the API yet
• Import - from man-kts, not part of the API yet

We do not have anything analogous to KotlinOpts as far as I can see, at least in the current script processing model. These annotations processed during script compilation, and default scripting execution model is in-process.
But I can imagine an implementation that takes such an annotation and runs the script with all required JVM opts out-of-process, if needed.

About username/password - you can find more e.g. in this issue - https://youtrack.jetbrains.com/issue/KT-27701#focus=Comments-27-5659673.0-0 and the follow-up.

As of using these things, the best way to my mind to have a look at the main-kts implementation https://github.com/JetBrains/kotlin/blob/master/libraries/tools/kotlin-main-kts/src/org/jetbrains/kotlin/mainKts/scriptDef.kt#L45 or simplifide version here - https://github.com/Kotlin/kotlin-script-examples/blob/master/jvm/simple-main-kts/simple-main-kts/src/main/kotlin/org/jetbrains/kotlin/script/examples/simpleMainKts/scriptDef.kt#L49 and implement your version accordingly.
It is not recommended to use main-kts artefact directly, it is designed to be used standalone, and may mess up with your other dependencies.

[way-zer]

I have made a framework using kts as modules. But I encounter a problem, how to use other compiler plugin like kotlinx servialization for scripting with annotations.
I can't find any API to refine compiler configuration about plugins. And annotations can't be parsed before invoke compile.

[aartiPl]

@ligee - I have an initial implementation of kotlin scripting for kscript. Can you have a look here:
https://github.com/kscripting/kscript/tree/kscript_4.2

This implementation works for more than 50% of test cases, so not bad for the starter. However, some cases won't apply anymore, e.g., KotlinOptions, and kt files support (I am not sure here: it seems that Kotlin Scripting supports just a single extension for scripts).

But some cases fail, although they shouldn't. One notable example is passing the source file's location - currently, the compiler got the wrong path to the script file and can not resolve imports correctly. I tried manually providing the root script's location but did not succeed.

Here is the ticket in KScript for implementation of the Kotlin Scripting engine:
kscripting/kscript#379
I would suggest moving the discussion over there.

One more remark from my side, after initial implementation: the need to pass CompilationConfiguration and EvaluationConfiguration to @KotlinScript annotation IMHO makes the whole implementation much more complex than needed. From the KScript implementation perspective, it should be enough to create configurations once and then pass them to compile and evaluate methods.

[ligee]

@way-zer, plugins are supported in scripting, but you need to configure it with CLI arguments either via regular CLI compiler or via the compilerOptions configuration key.
There is a (quite outdated) test in the Kotlin repo with the CLI compiler usage - https://github.com/JetBrains/kotlin/blob/ca2f37f6ebe9dd7349cc4cab1e1ab1ddfdb5976e/libraries/tools/kotlin-main-kts-test/test/org/jetbrains/kotlin/mainKts/test/mainKtsIT.kt#L145 and https://github.com/JetBrains/kotlin/blob/a7b94b398abc1163ae525ca00c2040e3ad4cc88d/libraries/tools/kotlin-main-kts-test/testData/hello-kotlinx-serialization.main.kts#L1-L0
No IDE support yet unfortunately.

[way-zer]

@way-zer, plugins are supported in scripting, but you need to configure it with CLI arguments either via regular CLI compiler or via the compilerOptions configuration key. There is a (quite outdated) test in the Kotlin repo with the CLI compiler usage - https://github.com/JetBrains/kotlin/blob/ca2f37f6ebe9dd7349cc4cab1e1ab1ddfdb5976e/libraries/tools/kotlin-main-kts-test/test/org/jetbrains/kotlin/mainKts/test/mainKtsIT.kt#L145 and https://github.com/JetBrains/kotlin/blob/a7b94b398abc1163ae525ca00c2040e3ad4cc88d/libraries/tools/kotlin-main-kts-test/testData/hello-kotlinx-serialization.main.kts#L1-L0 No IDE support yet unfortunately.

How to use annotation to add plugin. I try CompilerOption:

using compilerOptions when beforeParse:

Is there any way to add compiler plugin using scripting host?

[ligee]

@way-zer, sorry for the delay.

You're right, the usage via compilerOptions is blocked by now, for no good reason. I created an issue about it - https://youtrack.jetbrains.com/issue/KT-54095/It-is-difficult-if-not-impossible-to-use-kotlin-compiler-plugins-with-scripting
will see what we can do about it.

The only way to do it now is to use CLI compiler as in my test.

[way-zer]

A new problem: World refineConfigurationBeforeCompiling be called when import scripting confuguration in IDE?
It seems not in the latest version(1.7.20), but it did in old version.
So, my library seems not work in new version.

Another problem, how to debug scripting confiiguration in IDE. This always costs much time due to no break point and even no log.

Update: beforeCompiling is called, and I see debug log in idea.log

[arkanovicz]

In terms of use cases, the proposed model doesn't feel very handy for templating.

By templating, I mean having a file in some alien language with several ${kotlin expression} inside it (or, why not, a construct like #if(kotlin boolean expression)).

From the compiler and renderer point of view, each kotlin block can be given an arbitrary filename, but I get the feeling that the IDE will be left aside. I don't see any easy way of defining an "uber format" that would contain kotlin code inside specific parsed constructs.

[gildor]

@arkanovicz

easy way of defining an "uber format" that would contain kotlin code inside specific parsed constructs

I think it's totally possible using your own tools which parses any format and extracts Kotlin snippets (for example see the existing kotlinx-knit).
I just don't think that there is a universal solution, for example for Markdown there is standard code block format (triple tick with language name), why one should use some arbitrary template instead?

So, how it can be universal for all possible formats and what is use case for it? One can create a utility that parses those blocks and executes them as programs

Maybe you could clarify your idea, just genuinely interested what do you mean

[arkanovicz]

@gildor I'm not speaking about a universal format. Most template engines are agnostic in regard to the generated format. It can be HTML or Cobol, but the template language will have the same syntax.

It's possible under Intellij Idea to edit Velocity HTML files, for instance, and have an appropriate syntax coloring in both code (HTML tags and Velocity references and directives). Of course there is a specific tool implied, the Velocity plugin, but it's not specific to the generated language.

I'm just dreaming aloud of a Kotlin-powered Velocity-like engine...

[rocketraman]

I'm just dreaming aloud of a Kotlin-powered Velocity-like engine...

https://github.com/casid/jte

[arkanovicz]

@rocketraman I took a look at jte. But it's not the subject here, jte can't incorporate code sections (and as a side note, a template engine which doesn't rely on a proper parser looks a bit frightful to me).

[way-zer]

The @file:Import seems support inport pure kt source. But it seems no code inspection for the kt.
Is there any plan for this feature. For now, I am using kts in src and using gradle for dependencies, but which is in deprecated.
I want to know whether is possible to config dependencies for imported source,

I am heavily using this feature in project, in which on kts is a module, and import some kt as lib.

[ligee]

@way-zer the importing of .kt from script is an unofficial and controversial feature, see https://youtrack.jetbrains.com/issue/KT-28916. We're not sure that it is consistent with the general module concept and worth the support cost.
If you have a compelling use case, please leave a comment in the issue above.

If you're using .kts in the sources, please consider using a dedicated definition for it with isStandalone(false) property in the configuration.

[way-zer]

I'm thinking how to using scripting config instead gradle to config project dependencies. But find it may be impossible.

To be specific, in my project, I'm using kts as mirco module, and a script can extend another script classpath. So configuration for one script require another script configuration, but now no refine support that.
May be provide a new refine with compileConfig of importScripts.

Another problem, I'd like importScripts don't affect implicitReceivers, as the usage above(just for compile configuration and classpath). I'm using extenal function to inject the importScripts like contextScipt<Parent>().xxx().

---

Or May provide another api to get metadata like annotations (just parse without compile)?
I see KT-54095 supporting plugin now but now staticly. Providing a api to get annotations may ease the problem.
And many things may need this api, as annotation seems more uniform than other formats(like shebang).

[ligee]

@way-zer, excuse me, but I cannot grasp the whole problem by your description. Do you have an example code of scripts compilation and refinements, that fails to achieve the results you like?

The refinements are designed with the gradle case in mind, in particular so you should be able to get a callback before compiling of a script, extract a subscript from it, that may affect the classpath and other properties of the main script (e.g. like gradle's buildScript section), compile and run it separately to get and updated compilation configuration, and then resume the compilation of the main script.

I do not think that we will provide a separate API for annotations extraction any time soon. This is quite controversial topic and require a lot of discussion and design.

[way-zer]

Thank you, I will try the buildScript way later.

Compare annotation with buildScript

1. annotation is simpler, easier, quicker to parse. While buildScript more complex (compile, gen class, and eval)
2. annotation only support simple type(I test enum before, not know whether works now). buildScript can use DSL
3. annotation share the classpath and defaultImport. buildScript in IDEA may need more lint (no access outer variable, no access to other library)

Some little Question

• Can I compile another script in refine callback? That seems in blocking context, while compiling a script needs coroutine context. Is this any problem when in IDE.
• Do I need bundle kotlin-compiler when in IDE, or using the one in kotlin plugin?

About the refine

My usage like this (This is actual project)

// ktor.kts
@Import("io.ktor:ktor-core:xxxx")
@ConfigDefaultImport("io.ktor.http.*")//more like this
//other thing, may init ktor

// b.kts
@Depends("ktor")
// No need to import, as parent config defaultImport
routing {
    get("/") {
        call.respondText("Hello, world!", ContentType.Text.Html)
    }
}

In runtime, I can compile ktor.kts first, get config for b.kts. But in ide, this is not easy to get support.

---

About importScripts

I'm using kts as module, which is the smallest unit to load and unload. Using contextScript<T>() for dependency inject. Before, I put all scripts in the same sources when develop, so they can access other's type info.
For example, two module A, B

// a.kts
fun hello(){
}
// b.kts
@Depends("a")
contextScript<A>().hello()

I want to use importScripts to compile they together, so B can use the class A. But I don't like A as the implicitReceivers because variable contamination, especially when B has more than one dependency module.

---

Another Thing

May provide compileOption to assign default package? As above, Multiple modules may have naming conflicts(runtime class conflicts), I want to use directory as package.
For example.

// a/module.kts
// no need to define package(more like script)

// b.kts
contextScript<a.Module>()

[way-zer]

Sorry for disturbing, I'm trying to implement an IDEA plugin to get better support for my framework ScriptAgent. But encountered some obstacles.

Two main feature in general:

Dependency between kts

When A.kts depends B.kts, A should see the B class (unlike @file:Import, A don't get B instance directly).
Now: I'm using isStandalone(false) as a workaround, but too many class in classpath.
In runtime: this is implemented by compiling one by one.(no cycling dependencies)

Import source(also TODO for LivePlugin)

When A.kts import C.kt(not kts), C will be compiled together with A, also A and B will see each other class(in same classpath). C.kt here not a inner class of A.kts, so it don't need A instance (cycling class dependency, no cycling instance dependency).
In this way, I could split some of A to C.kt or D.kt to keep A brief, and in C.kts I can expose some API that user don't need instance A to invoke.
Now: I'm using isStandalone(false) and @file:Import(".kt") as a workaround even they are unofficial and controversial features.
In runtime: use@file:Import(".kt") to compile together. (It's better to provide a specialized API to compile kt source together)

---

I am trying to implement IDEA plugin to support these features, getting rid of the dependence on unofficial and controversial features. But it seems hard to implement, I have tried ModuleInfoProviderExtension with custom IdeaModuleInfo, but failled cause to KotlinCacheServiceImpl and some internal implement in KT plugin.

So I'd like to know is there any way to resolve the problem, or would you like to give some support?

[ligee]

@way-zer, could you, please, ask your questions in the Kotlin Slack? There is a dedicated channel for scripting discussions - https://slack-chats.kotlinlang.org/c/scripting. Alternatively, please create an YouTrack issue, and we can discuss it there. Using this discussion for the particular questions like this is inconvenient.
Just in short - first, you probably need to implement your own variant (or two) of the @Import annotation with appropriate handling, so you don't need to hack around the existing Import from .main.kts and isStandalone flag.
Second - there are still missing pieces in the IDE support, namely imported scripts, but we can discuss, what to do with it, e.g. over slack.

[obj-p]

Hi all, I'm catching up to speed here. We are evaluating the use of kts on the server side and security is the biggest question mark. Wanted to follow up on the progress of the "restrictions" configuration. Looking at this previous comment...: #75 (comment)

Just wanted to see if there is any progress there? I'm navigating the YouTrack issues and will ask in the Kotlin Slack if I'm granted access. Thank you!

[ligee]

@jjgp, no, unfortunately there is no progress on that. We may return to this after completely switching to K2.