[Macros] In-process plugin server #73725

rintaro · 2024-05-17T22:55:48Z

Separate swift-syntax libs for the compiler and for the library plugins. Compiler communicates with library plugins using serialized messages just like executable plugins.

lib/swift/host/compiler/lib_Compiler*.dylib(source lib/CompilerSwiftSyntax): swift-syntax libraries for compiler. Library evolution is disabled.
Compiler (ASTGen and swiftIDEUtilsBridging) only depends on lib/swift/host/compiler libraries.
lib/swift/host/libSwiftInProcPluginServer.dylib (source tools/swift-plugin-server/Sources/SwiftInProcPluginServer): In-process plugin server shared library. This has one swift_inproc_plugins_handle_message entry point that receives a message and return the response.
In the compiler:
- Add -in-process-plugin-server-path front-end option, which specifies the SwiftInProcPluginServer shared library path.
- Remove LoadedLibraryPlugin, because all library plugins are managed by SwiftInProcPluginServer
- Introduce abstract CompilerPlugin class that has 2 subclasses:
  - LoadedExecutablePlugin existing class that represents an executable plugin
  - InProcessPlugins wraps dlopened SwiftInProcPluginServer and communicate with it via the swift_inproc_plugins_handle_message
- Unified the code path in TypeCheckMacros.cpp and ASTGen, the difference between executable plugins and library plugins are now abstracted by CompilerPlugin

rintaro · 2024-05-17T23:01:26Z

lib/CompilerSwiftSyntax/CMakeLists.txt

+# Make unified swift-syntax shared library.
+add_pure_swift_host_library(swiftSyntaxUnified SHARED)
+set_target_properties(swiftSyntaxUnified PROPERTIES LINKER_LANGUAGE Swift)
+
+macro(target_link_libraries_whole_archive target)
+  foreach(lib ${ARGN}) 
+    target_compile_options(${lib} PRIVATE "SHELL:-module-link-name ${target}")
+    target_sources(${target} PRIVATE $<TARGET_OBJECTS:${lib}>)
+    # Workaround for CMake doesn't recognize '.o' files as sources.
+    target_link_options(${target} PRIVATE $<TARGET_OBJECTS:${lib}>)
+    set_property(TARGET ${target} APPEND PROPERTY
+      INTERFACE_INCLUDE_DIRECTORIES "$<TARGET_PROPERTY:${lib},INTERFACE_INCLUDE_DIRECTORIES>"
+    )


Instead of making an aggregated swift-syntax libraries, can we link those static libraries with ASTGen and make libASTGen.dylib?

edit: no because ASTGen depends on C++ code.

rintaro · 2024-05-17T23:03:59Z

lib/ASTGen/Sources/ASTGen/PluginHost.swift

@@ -319,7 +319,7 @@ class PluginDiagnosticsEngine {
    messageSuffix: String? = nil
  ) {
    for diagnostic in diagnostics {
-      self.emit(diagnostic)
+      self.emit(diagnostic, messageSuffix: messageSuffix)


We had a bug in executable plugins where (from macro '\(macroName)') suffix wasn't added in the diagnostics from executable plugins 😞

include/swift/AST/PluginRegistry.h

lib/AST/PluginRegistry.cpp

lib/ASTGen/Sources/ASTGen/PluginHost.swift

tools/libSwiftScan/CMakeLists.txt

rintaro · 2024-05-24T04:31:12Z

cmake/modules/AddPureSwift.cmake

-        Swift_MODULE_NAME ${name}
-        Swift_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
-    set(module_file "${CMAKE_CURRENT_BINARY_DIR}/${name}.swiftmodule")
+    set(module_dir "${CMAKE_CURRENT_BINARY_DIR}/modules")


Fragile modules are now created in ${CMAKE_CURRENT_BINARY_DIR}/modules. This is to avoid PR testing failures in incremental bots. This PR changes _CompilerRegexParser to fragile, if we create ${CMAKE_CURRENT_BINARY_DIR}/_CompilerRegexParser.swiftmodule , subsequent PR testings will be confused because -I ${CMAKE_CURRENT_BINARY_DIR} is added by CMake. Pre-patch #73799 can avoid that, but just in case.

rintaro · 2024-05-24T04:37:02Z

test/Macros/macro_expand.swift

@@ -629,8 +629,8 @@ struct HasNestedType {
 #if TEST_DIAGNOSTICS
 @freestanding(expression)
 macro missingMacro() = #externalMacro(module: "MacroDefinition", type: "BluhBlah")
-// expected-warning@-1 {{external macro implementation type 'MacroDefinition.BluhBlah' could not be found for macro 'missingMacro()'; 'MacroDefinition.BluhBlah' could not be found in library plugin '}}
+// FIXME: xpected-warning@-1 {{external macro implementation type 'MacroDefinition.BluhBlah' could not be found for macro 'missingMacro()'; 'MacroDefinition.BluhBlah' could not be found in library plugin '}}


This is because the plugin message facilities (i.e. executable plugins) doesn't support "resolve macro type" at this point. Expansions would just fail and be diagnosed if the type is not found or not a macro type.

Maybe we can introduce case resolveMacroType(module: String, type: String) plugin message to recover this.

tools/libSwiftScan/CMakeLists.txt

kabiroberai · 2024-05-29T02:15:26Z

👋 does this PR remove support for the loadPluginLibrary approach entirely? Worth noting that we're relying on this mechanism in #73031 in order to enable support for WebAssembly-based plugins. We could move these in-process but I think it would be preferable to keep third-party Wasm plugins out-of-process for defence-in-depth/isolation reasons (especially if we eventually want to execute them with JIT), so IMO it's worth keeping the library plugin mechanism for that. Also, IIUC one of the ways we could improve macro performance is to reuse a single plugin library/executable across swiftc invocations. Would this PR preclude that change for library-based plugins?

I should mention that I don't fully understand the implications of this PR so please do correct me if I'm mistaken. In general, in-process execution for first-party plugins during one-off swiftc invocation does seem like a nice improvement!

rintaro · 2024-05-30T18:56:46Z

does this PR remove support for the loadPluginLibrary approach entirely?

I'm not 100% sure what "loadPluginLibrary approach" you mean, but either it's about -load-plugin-library compiler option or loadPluginLibrary plugin message, this PR doesn't remove them.

Since we need to keep supporting "in-process" plugins (i.e. -load-plugin-library and -plugin-path compiler options) for various reasons, we've been suffered by some limitations from that. The main intent of this PR is that separating swift-syntax libraries for the compiler and swift-syntax libraries for those "in-process" plugins. By that

We can unify the code path between in-process and out-of-process plugins
We can use different swift-syntax between the compiler and the in-process plugins. That let us compiling swift-syntax for the compiler without library-evolution, for the compiler performance

This PR doesn't have any intent of removing or deprecating out-of-process plugin mechanism.

IIUC one of the ways we could improve macro performance is to reuse a single plugin library/executable across swiftc invocations. Would this PR preclude that change for library-based plugins?

Yes, reusing plugin processes across multiple swift-frontend invocations is probably what we need to tackle on next. This PR does not preclude that approach.

I'm not sure I entirely understand your concerns. Let me know if you still have any questions!

kabiroberai · 2024-05-31T00:39:55Z

Ah I was under the impression that this removed support for external plugin servers in favour of in-process plugins, but looks like I was confused. Thanks for the clarification!

Separate swift-syntax libs for the compiler and for the library plugins. Compiler communicates with library plugins using serialized messages just like executable plugins. * `lib/swift/host/compiler/lib_Compiler*.dylib`(`lib/CompilerSwiftSyntax`): swift-syntax libraries for compiler. Library evolution is disabled. * Compiler (`ASTGen` and `swiftIDEUtilsBridging`) only depends on `lib/swift/host/compiler` libraries. * `SwiftInProcPluginServer`: In-process plugin server shared library. This has one `swift_inproc_plugins_handle_message` entry point that receives a message and return the response. * In the compiler * Add `-in-process-plugin-server-path` front-end option, which specifies the `SwiftInProcPluginServer` shared library path. * Remove `LoadedLibraryPlugin`, because all library plugins are managed by `SwiftInProcPluginServer` * Introduce abstract `CompilerPlugin` class that has 2 subclasses: * `LoadedExecutablePlugin` existing class that represents an executable plugin * `InProcessPlugins` wraps `dlopen`ed `SwiftInProcPluginServer` * Unified the code path in `TypeCheckMacros.cpp` and `ASTGen`, the difference between executable plugins and library plugins are now abstracted by `CompilerPlugin`

rintaro · 2024-06-17T20:18:03Z

swiftlang/llvm-project#8855
swiftlang/swift-syntax#2659
swiftlang/swift-driver#1616
swiftlang/swift-installer-scripts#304
@swift-ci Please smoke test

rintaro · 2024-06-20T17:42:45Z

swiftlang/llvm-project#8855
swiftlang/swift-syntax#2659
swiftlang/swift-driver#1616
swiftlang/swift-installer-scripts#304
@swift-ci Please smoke test

rintaro · 2024-06-20T17:45:30Z

swiftlang/llvm-project#8855
swiftlang/swift-syntax#2659
swiftlang/swift-driver#1616
swiftlang/swift-installer-scripts#304
@swift-ci Please Build Toolchain macOS

rintaro · 2024-06-20T18:27:18Z

swiftlang/llvm-project#8855
swiftlang/swift-syntax#2659
swiftlang/swift-driver#1616
swiftlang/swift-installer-scripts#304
@swift-ci Please smoke test

rintaro · 2024-06-20T18:27:28Z

swiftlang/llvm-project#8855
swiftlang/swift-syntax#2659
swiftlang/swift-driver#1616
swiftlang/swift-installer-scripts#304
@swift-ci Please Build Toolchain macOS

Not all driver can send '-in-process-plugin-server-path'. To keep existing '-plugin-path' working, infer default server path in the frontend.

rintaro · 2024-06-22T00:47:54Z

swiftlang/llvm-project#8855
swiftlang/swift-installer-scripts#304
@swift-ci Please test

rintaro · 2024-06-24T18:43:15Z

swiftlang/llvm-project#8855
swiftlang/swift-installer-scripts#304
@swift-ci Please test

rintaro · 2024-06-25T19:37:04Z

swiftlang/llvm-project#8855
swiftlang/swift-driver#1616
swiftlang/swift-installer-scripts#304
@swift-ci Please Build Toolchain Windows

…target PR swiftlang#73725 introduced the in-process plugin server library, but the selection of the library depends on the selected toolchain, which depends on the compiler target, not the host. When cross-compiling (for example from macOS to a embedded Unix target), the compiler will incorrectly chose the `.so` file, not find it, and fail to compile things like the `@debugDescription` macro. Move the in-process plugin server library code from the platform toolchains into the parent type, and code it so it uses the right name depending on the compiler host at compilation time. This discards the target and only relies on the compiler host for selecting the right library.

…target (#74785) PR #73725 introduced the in-process plugin server library, but the selection of the library depends on the selected toolchain, which depends on the compiler target, not the host. When cross-compiling (for example from macOS to a embedded Unix target), the compiler will incorrectly chose the `.so` file, not find it, and fail to compile things like the `@debugDescription` macro. Move the in-process plugin server library code from the platform toolchains into the parent type, and code it so it uses the right name depending on the compiler host at compilation time. This discards the target and only relies on the compiler host for selecting the right library.

…target PR swiftlang#73725 introduced the in-process plugin server library, but the selection of the library depends on the selected toolchain, which depends on the compiler target, not the host. When cross-compiling (for example from macOS to a embedded Unix target), the compiler will incorrectly chose the `.so` file, not find it, and fail to compile things like the `@debugDescription` macro. Move the in-process plugin server library code from the platform toolchains into the parent type, and code it so it uses the right name depending on the compiler host at compilation time. This discards the target and only relies on the compiler host for selecting the right library. (cherry picked from commit 55d9e74)

rintaro commented May 17, 2024

View reviewed changes

rintaro mentioned this pull request May 17, 2024

Add In-process plugin server path swiftlang/swift-driver#1616

Merged

rintaro force-pushed the macros-inproc-plugin branch 3 times, most recently from 9c69aa8 to abc1390 Compare May 21, 2024 19:59

hamishknight reviewed May 22, 2024

View reviewed changes

rintaro force-pushed the macros-inproc-plugin branch 2 times, most recently from 1d223b1 to 8e7b75e Compare May 24, 2024 04:16

rintaro commented May 24, 2024

View reviewed changes

tools/libSwiftScan/CMakeLists.txt Outdated Show resolved Hide resolved

rintaro force-pushed the macros-inproc-plugin branch 4 times, most recently from e64de00 to 9be379a Compare May 25, 2024 06:24

rintaro force-pushed the macros-inproc-plugin branch from 9be379a to 7dd4f52 Compare May 30, 2024 20:11

rintaro force-pushed the macros-inproc-plugin branch 3 times, most recently from d7a0d58 to db933a8 Compare June 5, 2024 21:51

rintaro mentioned this pull request Jun 5, 2024

[lldb] Update for _CompilerSwiftSyntax libraries swiftlang/llvm-project#8855

Merged

rintaro force-pushed the macros-inproc-plugin branch 2 times, most recently from bd5a47d to c0449ed Compare June 6, 2024 18:42

rintaro mentioned this pull request Jun 6, 2024

[Windows] Add _CompilerSwiftSyntax libs and plugin servers swiftlang/swift-installer-scripts#304

Merged

rintaro marked this pull request as ready for review June 6, 2024 22:53

rintaro requested review from ahoppen and bnbarham as code owners June 6, 2024 22:53

rintaro force-pushed the macros-inproc-plugin branch from 9769e8f to 2f7aa42 Compare June 17, 2024 18:37

hamishknight approved these changes Jun 17, 2024

View reviewed changes

cachemeifyoucan mentioned this pull request Jun 18, 2024

[ScanDependencies] Get accurate macro dependency #73421

Merged

rintaro force-pushed the macros-inproc-plugin branch from e9254dc to 4d42b59 Compare June 20, 2024 18:01

[Macros] Infer in-process plugin server in swift-frontend

f08d69c

Not all driver can send '-in-process-plugin-server-path'. To keep existing '-plugin-path' working, infer default server path in the frontend.

rintaro force-pushed the macros-inproc-plugin branch from 4d42b59 to f08d69c Compare June 21, 2024 16:00

bnbarham approved these changes Jun 24, 2024

View reviewed changes

rintaro merged commit 5ebd715 into swiftlang:main Jun 26, 2024
7 checks passed

rintaro mentioned this pull request Jun 26, 2024

[6.0][Macros] In-process plugin server #74740

Closed

drodriguez mentioned this pull request Jun 27, 2024

[Macros] In-process plugin server library tied to compiler host, not target #74785

Merged

kabiroberai mentioned this pull request Jun 29, 2024

Add support for WebAssembly Macros swiftlang/swift-syntax#2623

Merged

ahoppen mentioned this pull request Jul 4, 2024

[Macros] Fix memory leak #74941

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Macros] In-process plugin server #73725

[Macros] In-process plugin server #73725

rintaro commented May 17, 2024 •

edited

Loading

rintaro May 17, 2024 •

edited

Loading

rintaro May 17, 2024

rintaro May 24, 2024

rintaro May 24, 2024 •

edited

Loading

kabiroberai commented May 29, 2024 •

edited

Loading

rintaro commented May 30, 2024 •

edited

Loading

kabiroberai commented May 31, 2024 •

edited

Loading

rintaro commented Jun 17, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 22, 2024

rintaro commented Jun 24, 2024

rintaro commented Jun 25, 2024

[Macros] In-process plugin server #73725

[Macros] In-process plugin server #73725

Conversation

rintaro commented May 17, 2024 • edited Loading

rintaro May 17, 2024 • edited Loading

Choose a reason for hiding this comment

rintaro May 17, 2024

Choose a reason for hiding this comment

rintaro May 24, 2024

Choose a reason for hiding this comment

rintaro May 24, 2024 • edited Loading

Choose a reason for hiding this comment

kabiroberai commented May 29, 2024 • edited Loading

rintaro commented May 30, 2024 • edited Loading

kabiroberai commented May 31, 2024 • edited Loading

rintaro commented Jun 17, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 20, 2024

rintaro commented Jun 22, 2024

rintaro commented Jun 24, 2024

rintaro commented Jun 25, 2024

rintaro commented May 17, 2024 •

edited

Loading

rintaro May 17, 2024 •

edited

Loading

rintaro May 24, 2024 •

edited

Loading

kabiroberai commented May 29, 2024 •

edited

Loading

rintaro commented May 30, 2024 •

edited

Loading

kabiroberai commented May 31, 2024 •

edited

Loading