-
-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial implementation of lazy JLLs for LinearAlgebra #57719
base: master
Are you sure you want to change the base?
Conversation
This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. This must be paired with the appropriate base Julia changes [0]. [0] JuliaLang/julia#57719
85eb73d
to
d537c45
Compare
This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. This must be paired with the appropriate base Julia changes [0]. [0] JuliaLang/julia#57719
d537c45
to
b43712b
Compare
b43712b
to
3db34a3
Compare
d21736d
to
24bdde7
Compare
This is working locally now, I can confirm that Note that this is attempting to conform to the What this means is that for any realistic workload where there are JLLs that depend on LBT/OpenBLAS, everything is going to get eagerly loaded upon loading of that JLL anyway, until those JLLs are built to use |
The |
I've converted this to draft because I realized that while "real" JLLs that use |
This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. This must be paired with the appropriate base Julia changes [0]. [0] JuliaLang/julia#57719
b85e778
to
6109346
Compare
It's also worth noting that once we land this PR, |
94382bd
to
afc71c7
Compare
I think you might need a rebase for the diff --git i/base/libdl.jl w/base/libdl.jl
index 77f608e77e..d94a2fd0e2 100644
--- i/base/libdl.jl
+++ w/base/libdl.jl
@@ -333,7 +333,7 @@ struct LazyLibraryPath
pieces::Tuple{Vararg{Any}}
LazyLibraryPath(pieces...) = new(pieces)
end
-Base.string(llp::LazyLibraryPath) = joinpath(String[string(p) for p in llp.pieces])
+@inline Base.string(llp::LazyLibraryPath) = joinpath(String[string(p) for p in llp.pieces])
Base.cconvert(::Type{Cstring}, llp::LazyLibraryPath) = Base.cconvert(Cstring, string(llp))
# Define `print` so that we can wrap this in a `LazyString`
Base.print(io::IO, llp::LazyLibraryPath) = print(io, string(llp)) The robust solution to this will require a callable of some kind, but for now massaging is required to make sure that the components of the |
7dcb242
to
b4cc2f8
Compare
This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. This must be paired with the appropriate base Julia changes [0]. [0] JuliaLang/julia#57719
44bd276
to
41e8fb4
Compare
4919c69
to
a127439
Compare
This alters CompilerSupportLibraries_jll, OpenBLAS_jll and libblastrampoline_jll to use `LazyLibrary` objects and thereby be loaded only upon first `dlopen()` or `ccall()` to the individual library objects. Note that this is one of the more complicated cases, as `libblastrampoline` must have OpenBLAS_jll added as a dynamic dependency (as it does not actually have it listed in its shared object headers) and also has some on-load callbacks that must be invoked. Long-term, I would like to replace the bespoke JLLs here with actual JLL source code, and vendor a version of `LazyJLLWrappers` in-tree to do the actual code generation even for Base. That is left as future work. This must be paired with the appropriate `LinearAlgebra.jl` changes [0]. [0] JuliaLang/LinearAlgebra.jl#1235
This makes more sense, reduces work, and avoids that perennial thorn in my side, that in-tree builds store libraries in `${prefix}/lib`, whereas installed builds store libraries in `${prefix}/lib/julia`. Co-authored-by: Cody Tapscott <84105208+topolarity@users.noreply.github.com>
We can no longer depend on the fact that these libraries will be eagerly loaded (huzzah!) so we must teach these helper functions to either inspect the lazy CSL JLL for the necessary information, or just directly load the library.
a127439
to
3433d57
Compare
This alters CompilerSupportLibraries_jll, OpenBLAS_jll and
libblastrampoline_jll to use
LazyLibrary
objects and thereby be loadedonly upon first
dlopen()
orccall()
to the individual libraryobjects. Note that this is one of the more complicated cases, as
libblastrampoline
must have OpenBLAS_jll added as a dynamic dependency(as it does not actually have it listed in its shared object headers)
and also has some on-load callbacks that must be invoked.
Long-term, I would like to replace the bespoke JLLs here with actual JLL
source code, and vendor a version of
LazyJLLWrappers
in-tree to do theactual code generation even for Base. That is left as future work.
[0] JuliaLang/LinearAlgebra.jl#1235