Skip to content

ci: build and cache LLVM from source #552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 22, 2024
Merged

ci: build and cache LLVM from source #552

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 80 additions & 47 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,6 @@ jobs:
outputs:
matrix: ${{ steps.cpp-matrix.outputs.matrix }}
steps:
- name: Clone cpp-actions
uses: actions/checkout@v3

- name: Generate Test Matrix
uses: alandefreitas/cpp-actions/cpp-matrix@v1.8.2
id: cpp-matrix
Expand Down Expand Up @@ -59,15 +56,90 @@ jobs:
contents: write

steps:
- name: Install git
- name: Install Essential Packages
if: ${{ matrix.container }}
env:
DEBIAN_FRONTEND: 'noninteractive'
TZ: 'Etc/UTC'
uses: alandefreitas/cpp-actions/package-install@v1.8.2
with:
apt-get: git
apt-get: git build-essential python3

- name: Clone MrDocs
uses: actions/checkout@v3

- name: Setup CMake
uses: alandefreitas/cpp-actions/setup-cmake@v1.8.2
id: setup-cmake
with:
# Clang requires clang-scan-deps to work on the latest CMake versions
version: ${{ matrix.compiler == 'clang' && '3.26' || '>=3.26' }}
check-latest: 'true'
update-environment: 'true'

- name: Setup Ninja
uses: seanmiddleditch/gha-setup-ninja@v4
if: ${{ runner.os == 'Windows' }}

- name: LLVM Parameters
id: llvm-parameters
run: |
echo -E "llvm-hash=29b20829cc6ce3e6d9c3809164994c1659e0da56" >> $GITHUB_OUTPUT
echo -E "llvm-build-preset=${{ runner.os == 'Windows' && 'release-win' || 'release-unix' }}" >> $GITHUB_OUTPUT
cd ..
llvm_root=$(pwd)/third-party/llvm-project/install
if [[ ${{ runner.os }} == 'Windows' ]]; then
llvm_root=$(echo "$llvm_root" | sed 's/\\/\//g')
llvm_root=$(echo $llvm_root | sed 's|^/d/|D:/|')
echo "$llvm_root"
fi
echo -E "llvm-root=$llvm_root" >> $GITHUB_OUTPUT

- name: LLVM Binaries
id: llvm-cache
uses: actions/cache@v4
with:
path: ${{ steps.llvm-parameters.outputs.llvm-root }}
key: llvm-${{ runner.os }}-${{ steps.llvm-parameters.outputs.llvm-build-preset }}-${{ steps.llvm-parameters.outputs.llvm-hash }}

- name: Install LLVM
id: llvm-install
if: steps.llvm-cache.outputs.cache-hit != 'true'
shell: bash
run: |
# LLVM is be installed with the default compiler
set -x

# Shallow clone LLVM_HASH in ../third-party/llvm
cd ..
mkdir -p third-party/llvm-project
cd third-party/llvm-project
llvm_project_root=$(pwd)
git config --global init.defaultBranch master
git config --global advice.detachedHead false
git init
git remote add origin https://github.com/llvm/llvm-project.git
git fetch --depth 1 origin ${{ steps.llvm-parameters.outputs.llvm-hash }}
git checkout FETCH_HEAD

# Copy presets
cp ../../mrdocs/third-party/llvm/CMakePresets.json ./llvm
cp ../../mrdocs/third-party/llvm/CMakeUserPresets.json.example ./llvm/CMakeUserPresets.json

# Build
cd llvm
llvm_root=$(pwd)
cmake --version
cmake -S . -B ./build --preset=${{ steps.llvm-parameters.outputs.llvm-build-preset }}
if [[ ${{ runner.os }} == 'Linux' ]]; then
cmake --build ./build --target help
fi
N_CORES=$(nproc 2>/dev/null || echo 1)
cmake --build ./build --config Release --parallel $N_CORES
cmake --install ./build --prefix "$llvm_project_root"/install

# Setup C++ after installing LLVM to use the default compiler
# for LLVM and the specified compiler for MrDocs
- name: Setup C++
uses: alandefreitas/cpp-actions/setup-cpp@v1.8.2
id: setup-cpp
Expand All @@ -87,45 +159,6 @@ jobs:
cxx: ${{ steps.setup-cpp.outputs.cxx || matrix.cxx }}
cxxflags: ${{ matrix.cxxflags }}

- name: Install LLVM
id: llvm-install
shell: bash
run: |
set -xe
config_type="Release"
filename="${{ runner.os }}-$config_type-29b20829.${{ ( runner.os == 'Windows' && '7z' ) || 'tar.xz' }}"
url="https://mrdox.com/llvm+clang/$filename"

# Download
if command -v curl &> /dev/null
then
curl -L -o "$filename" "$url"
elif command -v wget &> /dev/null
then
wget -O "$filename" "$url"
else
echo "Neither curl nor wget are available"
exit 1
fi

# Extract
llvm_root="${{runner.tool_cache}}/llvm+clang"
llvm_root=$(echo "$llvm_root" | sed 's/\\/\//g')
mkdir -p "$llvm_root"
if [ "${{ runner.os }}" != "Windows" ]; then
tar -xvf "$filename" -C "$llvm_root" --strip-components=1
else
7z x "$filename"
cd "$config_type"
mv * "$llvm_root"
cd ..
rm -rf "$config_type"
fi

# Export
echo "llvm_root=$llvm_root"
echo -E "llvm-root=$llvm_root" >> $GITHUB_OUTPUT

- name: Install Node.js
uses: actions/setup-node@v3
with:
Expand All @@ -137,16 +170,16 @@ jobs:
cmake-version: '>=3.20'
cxxstd: ${{ matrix.cxxstd }}
cc: ${{ steps.setup-cpp.outputs.cc || matrix.cc }}
ccflags: ${{ matrix.ccflags }}
ccflags: ${{ matrix.ccflags }}${{ ( matrix.compiler == 'gcc' && ' -static-libstdc++') || '' }}${{ ( matrix.asan && ' -static-libasan') || '' }}${{ ( matrix.tsan && ' -static-libtsan') || '' }}
cxx: ${{ steps.setup-cpp.outputs.cxx || matrix.cxx }}
cxxflags: ${{ matrix.cxxflags }}${{ ( matrix.compiler == 'gcc' && ' -static-libstdc++') || '' }}${{ ( matrix.asan && ' -static-libasan') || '' }}${{ ( matrix.tsan && ' -static-libtsan') || '' }}
generator: Ninja
toolchain: ${{ steps.package-install.outputs.vcpkg_toolchain || steps.package-install.outputs.vcpkg-toolchain }}
build-type: ${{ matrix.build-type }}
install-prefix: .local
extra-args: |
-D LLVM_ROOT="${{ steps.llvm-install.outputs.llvm-root || '/usr/local' }}"
-D Clang_ROOT="${{ steps.llvm-install.outputs.llvm-root || '/usr/local' }}"
-D LLVM_ROOT="${{ steps.llvm-parameters.outputs.llvm-root || '../third-party/llvm-project/install' }}"
-D Clang_ROOT="${{ steps.llvm-parameters.outputs.llvm-root || '../third-party/llvm-project/install' }}"
export-compile-commands: true
run-tests: true
install: true
Expand Down
168 changes: 31 additions & 137 deletions docs/modules/ROOT/pages/install.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ Binary packages are available from our https://github.com/cppalliance/mrdocs/rel

== Source

The following instructions assume we are at a parent directory that's going to contain both the MrDocs and the third-party dependencies directories.

[source]
----
+ <parent-directory>
+ mrdocs
+ third-party
----

Clone the MrDocs repository with:

[source,bash]
Expand All @@ -33,128 +42,42 @@ Fell free to install them anywhere you want and adjust the main MrDocs configura
MrDocs uses LLVM to parse C++ code and extract documentation from it.
It depends on a recent version of LLVM: https://github.com/llvm/llvm-project/tree/29b20829cc6ce3e6d9c3809164994c1659e0da56[29b20829]

[#llvm-binaries]
**Binaries**:

Because building LLVM may take many hours to complete, we provide pre-built binaries for Windows and Linux:

|===
| | https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html[CMake Preset,window=_blank] | https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html[CMake Build,window=_blank] | Debug Info | Optimized | https://learn.microsoft.com/en-us/visualstudio/ide/understanding-build-configurations?view=vs-2022[MSVC Build,window=_blank]

| 🪟 https://mrdox.com/llvm+clang/Windows-Release-29b20829.7z[`Windows-Release-29b20829.7z`]
| 🚀 Release
| 🚀 Release
| ❌
| ✅
| 🚀 Release

| 🪟 https://mrdox.com/llvm+clang/Windows-Debug-29b20829.7z[`Windows-Debug-29b20829.7z`]
| 🐞 Debug
| 🐞 Debug
| ✅
| ❌
| 🐞 Debug

| 🪟 https://mrdox.com/llvm+clang/Windows-RelWithDebInfo-29b20829.7z[`Windows-RelWithDebInfo-29b20829.7z`]
| 🕵️‍♂️ RelWithDebInfo
| 🕵️‍♂️ RelWithDebInfo
| ✅
| ✅
| 🚀 Release

| 🪟 https://mrdox.com/llvm+clang/Windows-DebWithOpt-29b20829.7z[`Windows-DebWithOpt-29b20829.7z`]
| 🔬 DebWithOpt
| 🐞 Debug
| ✅
| ✅
| 🐞 Debug

| 🐧 https://mrdox.com/llvm+clang/Linux-Release-29b20829.tar.xz[`Linux-Release-29b20829.tar.xz`]
| 🚀 Release
| 🚀 Release
| ❌
| ✅
| N/A

| 🐧 https://mrdox.com/llvm+clang/Linux-Debug-29b20829.tar.xz[`Linux-Debug-29b20829.tar.xz`]
| 🐞 Debug
| 🐞 Debug
| ✅
| ❌
| N/A

| 🐧 https://mrdox.com/llvm+clang/Linux-RelWithDebInfo-29b20829.tar.xz[`Linux-RelWithDebInfo-29b20829.tar.xz`]
| 🕵️‍♂️ RelWithDebInfo
| 🕵️‍♂️ RelWithDebInfo
| ✅
| ✅
| N/A

| 🐧 https://mrdox.com/llvm+clang/Linux-DebWithOpt-29b20829.tar.xz[`Linux-DebWithOpt-29b20829.tar.xz`]
| 🔬 DebWithOpt
| 🐞 Debug
| ✅
| ✅
| N/A
|===

IMPORTANT: The Linux binaries are built on Ubuntu 22.04 and may not work on other distributions.

You can download the binaries and uncompress them in the `./third-party/llvm+clang` directory we created in the previous step.

LLVM binaries are provided in a number of preset configurations.
Here is a brief description of each preset:

- `Release`: this is the preset users will typically use.
It is optimized for speed and does not include debug information.
- `Debug`: this is a preset developers can use.
It includes debug information and no optimizations.
However, using a `Debug` build of LLVM to debug MrDocs might be too slow.
In this case, you can link MrDocs with `RelWithDebInfo` or `DebWithOpt`.
- `RelWithDebInfo`: this is a release build with debug information.
It is optimized for speed and includes debug information.
However, if you are working with Windows+MSVC, this preset has a `Release` build type at the MSVC level.
This means you can have conflicts with MrDocs in `Debug` mode because of LLVM setting flags such as the `_ITERATOR_DEBUG_LEVEL` and `/MDd`.
In this case, you can use `DebWithOpt` instead to avoid the conflict and subsequent workarounds.
- `DebWithOpt`: this is a debug build with optimizations.
It includes all the default Debug flags for LLVM, it's optimized for speed, includes debug information, and causes no conflicts with MrDocs in `Debug` mode.

If you chose to use the provided binaries instead of building LLVM from source, you can skip to the <<duktape>> section.

**Download**:

Alternatively, if building LLVM from source, you can clone the project from the official repository:
You can shallow-clone the project from the official repository:

[source,bash]
----
git clone https://github.com/llvm/llvm-project.git
mkdir -p llvm-project
cd llvm-project
git checkout 29b20829cc6ce3e6d9c3809164994c1659e0da56
cd llvm
git init
git remote add origin https://github.com/llvm/llvm-project.git
git fetch --depth 1 origin 29b20829cc6ce3e6d9c3809164994c1659e0da56
git checkout FETCH_HEAD
----

**Configure**:

There are two ways to configure LLVM: using <<llvm-configure-presets, CMake presets>> or using <<llvm-configure-cmd-line, CMake directly>>.

[#llvm-configure-presets]
_Configure with CMake Presets_
The `mrdocs/third-party/llvm` directory provides https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html[CMake presets,window=_blank] to build LLVM.
We recommend using preset files as they contain a replicable set of CMake configuration values that can be used for a project.
From `third-party/llvm-project`, you can copy the `CMakePresets.json` and `CMakeUserPresets.json` files to the `llvm-project/llvm` directory.

We recommend using https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html[CMake presets,window=_blank] to build LLVM.
Preset files contain a replicable set of CMake configuration values that can be used to configure a project.
[source,bash]
----
cp ../../mrdocs/third-party/llvm/CMakePresets.json ./llvm
cp ../../mrdocs/third-party/llvm/CMakeUserPresets.json.example ./llvm/CMakeUserPresets.json
----

Instead of passing all CMake configuration values on the command line, a template for the `CMakePresets.json` and `CMakeUserPresets.json` files is provided in the repository's `third-party/llvm` directory.
Copy these files to the `llvm-project/llvm` directory and run a command such as the following to configure LLVM:
Run a command such as the following to configure LLVM:

[source,bash]
----
cmake -preset=relwithdebinfo-win
cd llvm
cmake -S . -B ./build --preset=release-win
----

In the example above, we configure a `RelWithDebInfo` version of LLVM for MrDocs: a release build with debug information.

In the example above, we configure a `Release` version of LLVM for MrDocs.
Choose one of the presets from `CMakePresets.json` or edit the variants in `CMakeUserPresets.json` to customize the configurations.
The `CMakeUserPresets.json` file comes with presets for all the configurations described in the <<llvm-binaries,Binaries>> section.

[NOTE]
====
Expand All @@ -166,48 +89,19 @@ This should give you an optimized build with all debug features and flags, such
In other platforms, this should give you a release somewhat equivalent to `RelWithDebInfo` optimized for debugging experience. `-Og` offers a reasonable level of optimization while maintaining fast compilation and a good debugging experience.
====

[#llvm-configure-cmd-line]
_Configure with Command Line Arguments_:

You can also configure LLVM directly with the settings required by MrDocs:

Windows (from administrator `cmd.exe`, after running `vcvars64.bat`):

[source,commandline]
----
cmake -S llvm -B build/MSVC/RelWithDebInfo -G "Ninja" -A x64 -D LLVM_ENABLE_PROJECTS="clang" -D CMAKE_CONFIGURATION_TYPES="RelWithDebInfo" -D LLVM_ENABLE_RTTI=ON -D CMAKE_INSTALL_PREFIX=../llvm+clang/RelWithDebInfo -D LLVM_ENABLE_IDE=OFF -D LLVM_ENABLE_DIA_SDK=OFF
----

Unix variants:

[source,bash]
----
cmake -S llvm -B build/Linux/RelWithDebInfo -D LLVM_ENABLE_PROJECTS="clang" -D CMAKE_BUILD_TYPE=RelWithDebInfo -D LLVM_ENABLE_RTTI=ON -D CMAKE_INSTALL_PREFIX=../llvm+clang/RelWithDebInfo
----

Unlike the <<llvm-configure-presets,CMake presets>>, this command does not a number of parameters that removes features that are not required by MrDocs, thus increasing the build time and size of the installation.

**Build**:

Build and install the configured version of LLVM with:

[source,bash]
----
cd build
cmake --build . --config RelWithDebInfo
cmake --install . --prefix ../../llvm+clang/RelWithDebInfo" --config RelWithDebInfo
cmake --build ./build --config Release --parallel 4
cmake --install ./build --prefix ../install
----

If you prefer using the provided CMake presets, you can also use the `--preset` option for the `build` command:

[source,bash]
----
cd build
cmake --build --preset=relwithdebinfo-win
cmake --install MSVC/RelWithDebInfo --config RelWithDebInfo
----
Replace 4 with the number of cores you want to use for building LLVM.

Return from `./third-party/llvm-project/build` to the parent `third-party` directory to install other dependencies:
Return from `./third-party/llvm-project/llvm` to the parent `third-party` directory to install other dependencies:

[source,bash]
----
Expand Down
1 change: 1 addition & 0 deletions third-party/llvm/CMakePresets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
"LLVM_BUILD_TESTS": false,
"LLVM_BUILD_DOCS": false,
"LLVM_BUILD_EXAMPLES": false,
"LLVM_ENABLE_TERMINFO": false,
"CLANG_ENABLE_ARCMT": true,
"CLANG_ENABLE_HLSL": false,
"CLANG_ENABLE_OBJC_REWRITER": false,
Expand Down