Merge tag 'v1.8.0' of https://github.com/sql-js/sql.js

SQLite 3.38

add aggregate functions - maybe not wanted in this fork

pull more updates from 2021, 2022, etc.

keep FTS4 & JSON enabled with explicit flags

continue to keep extension functions out of this fork

Author: Christopher J. Brody

.devcontainer/Dockerfile

@@ -0,0 +1,137 @@
+# We build our DevContainer on MS' Typescript-Node Devcontainer
+# This gives us lots of standard stuff, and lets us layer a few custom things on top, like the Emscripten compiler, Puppeteer
+
+# --------------------------------------------------------------------
+# BEGIN Standard MS Devcontainer for Typescript-Node 
+
+# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.155.1/containers/typescript-node/.devcontainer/base.Dockerfile
+# [Choice] Node.js version: 14, 12, 10
+ARG VARIANT="14-buster"
+FROM mcr.microsoft.com/vscode/devcontainers/typescript-node:0-${VARIANT}
+
+# [Optional] Uncomment if you want to install an additional version of node using nvm
+# ARG EXTRA_NODE_VERSION=10
+# RUN su node -c "source /usr/local/share/nvm/nvm.sh && nvm install ${EXTRA_NODE_VERSION}"
+
+# [Optional] Uncomment if you want to install more global node packages
+# RUN su node -c "npm install -g <your-package-list -here>"
+
+# END Standard MS Devcontainer for Typescript-Node 
+# --------------------------------------------------------------------
+
+# --------------------------------------------------------------------
+# BEGIN EMSDK 
+# Install EMSDK to /emsdk just like the EMSDK Dockerfile: https://github.com/emscripten-core/emsdk/blob/master/docker/Dockerfile
+ENV EMSDK /emsdk
+# We pin the EMSDK version rather than 'latest' so that everyone is using the same compiler version
+ENV EMSCRIPTEN_VERSION 3.1.20
+
+RUN git clone https://github.com/emscripten-core/emsdk.git $EMSDK
+
+RUN echo "## Install Emscripten" \
+    && cd ${EMSDK} \
+    && ./emsdk install ${EMSCRIPTEN_VERSION} \
+    && echo "## Done"
+
+# Copied directly from https://github.com/emscripten-core/emsdk/blob/master/docker/Dockerfile
+RUN cd ${EMSDK} \
+    && echo "## Generate standard configuration" \
+    && ./emsdk activate ${EMSCRIPTEN_VERSION} \
+    && chmod 777 ${EMSDK}/upstream/emscripten \
+    && chmod -R 777 ${EMSDK}/upstream/emscripten/cache \
+    && echo "int main() { return 0; }" > hello.c \
+    && ${EMSDK}/upstream/emscripten/emcc -c hello.c \
+    && cat ${EMSDK}/upstream/emscripten/cache/sanity.txt \
+    && echo "## Done"
+
+ENV PATH $EMSDK:$EMSDK/upstream/emscripten/:$PATH
+
+# Cleanup Emscripten installation and strip some symbols
+# Copied directly from https://github.com/emscripten-core/emsdk/blob/master/docker/Dockerfile
+RUN echo "## Aggressive optimization: Remove debug symbols" \
+    && cd ${EMSDK} && . ./emsdk_env.sh \
+    # Remove debugging symbols from embedded node (extra 7MB)
+    && strip -s `which node` \
+    # Tests consume ~80MB disc space
+    && rm -fr ${EMSDK}/upstream/emscripten/tests \
+    # Fastcomp is not supported
+    && rm -fr ${EMSDK}/upstream/fastcomp \
+    # strip out symbols from clang (~extra 50MB disc space)
+    && find ${EMSDK}/upstream/bin -type f -exec strip -s {} + || true \
+    && echo "## Done"
+
+RUN echo ". /emsdk/emsdk_env.sh" >> /etc/bash.bashrc
+# We must set the EM_NODE_JS environment variable for a somewhat silly reason
+# We run our build scripts with `npm run`, which sets the NODE environment variable as it runs.
+# The EMSDK picks up on that environment variable and gives a deprecation warning: warning: honoring legacy environment variable `NODE`.  Please switch to using `EM_NODE_JS` instead`
+# So, we are going to put this environment variable here explicitly to avoid the deprecation warning.
+RUN echo 'export EM_NODE_JS="$EMSDK_NODE"' >> /etc/bash.bashrc
+
+# END EMSDK
+# --------------------------------------------------------------------
+
+# --------------------------------------------------------------------
+# BEGIN PUPPETEER dependencies
+# Here we install all of the packages depended upon by Chrome (that Puppeteer will use for headless tests).
+# We could also take a page from https://github.com/buildkite/docker-puppeteer/blob/master/Dockerfile instead,
+# and install the latest stable version of Chrome to get the right dependencies, but that version changes over time,
+# so the stable version of Chrome and the version installed by Puppeteer might diverge over time. 
+# It also means they end up having Chrome downloaded and installed twice.
+# We could install the particular version of Chrome that our version of Puppeteer would use and then tell Puppeteer not to download its own version of Chrome,
+# but then we'd have to rebuild our Docker container every time we revved Puppeteer, and that feels fiddly too.
+# For all of these reasons, it seems safer to simply install the explicit list packages depended upon by Chrome, assume that's unlikely to change
+# and move on.
+
+# List taken from:
+# https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix
+RUN apt-get update \
+     && apt-get install -y \
+        ca-certificates \
+        fonts-liberation \
+        libappindicator3-1 \
+        libasound2 \
+        libatk-bridge2.0-0 \
+        libatk1.0-0 \
+        libc6 \
+        libcairo2 \
+        libcups2 \
+        libdbus-1-3 \
+        libexpat1 \
+        libfontconfig1 \
+        libgbm1 \
+        libgcc1 \
+        libglib2.0-0 \
+        libgtk-3-0 \
+        libnspr4 \
+        libnss3 \
+        libpango-1.0-0 \
+        libpangocairo-1.0-0 \
+        libstdc++6 \
+        libx11-6 \
+        libx11-xcb1 \
+        libxcb1 \
+        libxcomposite1 \
+        libxcursor1 \
+        libxdamage1 \
+        libxext6 \
+        libxfixes3 \
+        libxi6 \
+        libxrandr2 \
+        libxrender1 \
+        libxss1 \
+        libxtst6 \
+        lsb-release \
+        wget \
+        xdg-utils
+
+# Installs the command "sha3sum", which is used check the download integrity of sqlite source.
+RUN apt-get install -y libdigest-sha3-perl
+
+# We set this env variable (RUN_WORKER_TEST_WITHOUT_PUPPETEER_SANDBOX=1) this to tell our sql.js test harness to run Puppeteer without the sandbox.
+# Otherwise, when we instantiate Puppeteer, we get this error: 
+# Puppeteer can't start due to a sandbox error. (Details follow.)
+#     [0321/173044.694524:FATAL:zygote_host_impl_linux.cc(117)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
+ENV RUN_WORKER_TEST_WITHOUT_PUPPETEER_SANDBOX=1
+
+# END PUPPETEER
+# --------------------------------------------------------------------

.devcontainer/devcontainer.json

@@ -0,0 +1,26 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
+// https://github.com/microsoft/vscode-dev-containers/tree/v0.155.1/containers/typescript-node
+{
+	"name": "Node.js & TypeScript",
+	"build": {
+		"dockerfile": "Dockerfile",
+		// Update 'VARIANT' to pick a Node version: 10, 12, 14
+		"args": {
+			"VARIANT": "14-buster"
+		},
+	},
+	// Set *default* container specific settings.json values on container create.
+	"settings": {},
+	// Add the IDs of extensions you want installed when the container is created.
+	"extensions": [
+		"dbaeumer.vscode-eslint"
+	],
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+	// Use 'postCreateCommand' to run commands after the container is created.
+	// We use `npm ci` instead of `npm install` because we want to respect the lockfile and ONLY the lockfile.
+	// That way, our devcontainer is more reproducible. --Taytay
+	"postCreateCommand": "npm ci",
+	// Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
+	"remoteUser": "node"
+}

.eslintrc.js

@@ -16,6 +16,7 @@ module.exports = {
     ignorePatterns: [
         "/dist/",
         "/examples/",
+        "/documentation/",
         "/node_modules/",
         "/out/",
         "/src/shell-post.js",

.github/actions/build-sqljs/action.yml

@@ -0,0 +1,7 @@
+# action.yml
+name: 'Build SQL.js'
+description: 'Builds sql.js using the .devcontainer/Dockerfile as its environment'
+runs:
+  using: 'docker'
+  image: '../../../.devcontainer/Dockerfile'
+  entrypoint: "/github/workspace/.github/actions/build-sqljs/entrypoint.sh"

.github/actions/build-sqljs/entrypoint.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+set -e
+
+cd /github/workspace/
+npm run rebuild

.github/workflows/CI.yml

@@ -7,18 +7,19 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
-    - uses: actions/cache@v1
-      id: cache
-      with:
-        path: '.emsdk-cache'
-        key: emscripten-1.39.20
-    - uses: mymindstorm/setup-emsdk@2a4a91b
-      with:
-        version: '1.39.20'
-        actions-cache-folder: '.emsdk-cache'
     - name: make
-      run: make
+      uses: ./.github/actions/build-sqljs
     - uses: actions/upload-artifact@v2
       with: {name: dist, path: dist}
     - name: test
       run: npm ci && npm test
+    - name: generate documentation
+      run: npm run doc
+    - name: Update github pages
+      if: github.event_name == 'push' && github.ref == 'refs/heads/master'
+      uses: JamesIves/github-pages-deploy-action@3.6.2
+      with:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        BRANCH: gh-pages # The branch the action should deploy to.
+        FOLDER: "." # The folder the action should deploy.
+        CLEAN: false # Automatically remove deleted files from the deploy branch

.github/workflows/release.yml

@@ -12,10 +12,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
-      - uses: mymindstorm/setup-emsdk@2a4a91b
-        with: {version: '1.39.20'}
       - name: make
-        run: make
+        uses: ./.github/actions/build-sqljs
       - name: Create Release
         id: create_release
         uses: actions/create-release@v1.0.0
@@ -28,7 +26,7 @@ jobs:
           prerelease: false
       - run: cd dist && zip sqljs-wasm.zip sql-wasm.{js,wasm}
       - name: Upload Release Asset (wasm)
-        uses: lovasoa/upload-release-asset@851d9cc
+        uses: lovasoa/upload-release-asset@851d9cc59fe8113912edffbd8fddaa09470a5ac0
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
@@ -38,7 +36,7 @@ jobs:
           asset_label: wasm version, best runtime performance, smaller assets, requires configuration
           asset_content_type: application/zip
       - name: Upload Release Asset (asm)
-        uses: lovasoa/upload-release-asset@851d9cc
+        uses: lovasoa/upload-release-asset@851d9cc59fe8113912edffbd8fddaa09470a5ac0
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
@@ -49,7 +47,7 @@ jobs:
           asset_content_type: text/javascript
       - run: cd dist && zip sqljs-worker-wasm.zip worker.sql-wasm.js sql-wasm.wasm
       - name: Upload Release Asset (worker wasm)
-        uses: lovasoa/upload-release-asset@851d9cc
+        uses: lovasoa/upload-release-asset@851d9cc59fe8113912edffbd8fddaa09470a5ac0
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
@@ -59,7 +57,7 @@ jobs:
           asset_label: webworker wasm version, to be loaded as a web worker
           asset_content_type: application/zip
       - name: Upload Release Asset (worker asm)
-        uses: lovasoa/upload-release-asset@851d9cc
+        uses: lovasoa/upload-release-asset@851d9cc59fe8113912edffbd8fddaa09470a5ac0
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:
@@ -70,7 +68,7 @@ jobs:
           asset_content_type: text/javascript
       - run: cd dist && zip sqljs-all.zip *.{js,wasm}
       - name: Upload Release Asset (all)
-        uses: lovasoa/upload-release-asset@851d9cc
+        uses: lovasoa/upload-release-asset@851d9cc59fe8113912edffbd8fddaa09470a5ac0
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         with:

.gitignore

@@ -4,6 +4,7 @@ node_modules/
 # Intermediary files:
 cache/
 out/
+.emsdk-cache/
 sqlite-src/
 tmp/
 c/

.jsdoc.config.json

@@ -6,5 +6,33 @@
         "include": [
             "src/api.js"
         ]
+    },
+    "opts": {
+        "encoding": "utf8",
+        "destination": "./documentation/",
+        "readme": "documentation_index.md",
+        "template": "./node_modules/clean-jsdoc-theme",
+        "theme_opts": {
+            "title": "sql.js",
+            "meta": [
+                "<title>sql.js API documentation</title>",
+                "<meta name=\"author\" content=\"Ophir Lojkine\">",
+                "<meta name=\"description\" content=\"Documentation for sql.js: an in-memory SQL database for the browser based on SQLite.\">"
+            ],
+            "menu": [
+                {
+                    "title": "Website",
+                    "link": "https://sql.js.org/"
+                },
+                {
+                    "title": "Github",
+                    "link": "https://github.com/sql-js/sql.js"
+                },
+                {
+                    "title": "Demo",
+                    "link": "https://sql.js.org/examples/GUI/"
+                }
+            ]
+        }
     }
-}
+}

.nojekyll

@@ -0,0 +1,66 @@
+
+# Compiling and Contributing
+
+General consumers of this library don't need to read any further. (The compiled files are available via the [release page](https://github.com/sql-js/sql.js/releases).)
+
+If you want to compile your own version of SQLite for WebAssembly, or want to contribute to this project, read on.
+
+## Setting up your Development Environment
+
+### Containerized Development Environment (Recommended) 
+
+This project defines a standardized development environment using Docker (and the .devcontainer spec in particular). This allows for anyone on any platform to get up and running quickly. (VSCode is not technically required to make use of this standardized environment, but it makes containerized development so seamless that the non-VSCode path is not currently documented here.)
+
+Standardizing our development environment has numerous benefits:
+- Allows anyone on ANY platform (Linux, Mac, and Windows) to contribute or compile their own build.
+- It's quicker and easier for any contributor to dive in and fix issues.
+- (Practically) eliminates configuration bugs that are difficult for maintainers to reproduce. Also known as "works on my machine" issues.
+- Allows us to write our scripts assuming that they're _always_ running in a single known environment of a single, known platform. 
+- Ensure that all contributors use a known, standardized installation of EMSDK.
+- Allows for a more clearly documented process for updating the EMSDK to a new version.
+- End-Users that simply want to compile and install their own version of SQLite don't have to bother with EMSDK installation in their particular environment.
+
+To get started:
+
+1. Follow the [Installation Steps for Containerized Development in VSCode](https://code.visualstudio.com/docs/remote/containers#_installation). This includes installing Docker, VSCode, and the [Remote Development extension pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) for VSCode)
+2. Clone this repository
+3. Open the repository folder in VSCode. It will detect the presence of a .devcontainer and prompt you: "Folder contains a Dev Container configuration file. Reopen folder to develop in a container." Click "Reopen in container"
+
+You're now ready to test the dev environment:
+
+4. Click on Terminal->New Terminal to be dropped into a terminal inside the dev environment.
+5. Run `$ npm install` to install the required modules
+6. Run `$ npm test` to ensure all tests pass
+7. Run `$ npm rebuild` to re-compile the project from scratch (using the version of EMSDK installed in the container).
+8. Run `$ npm test` to ensure all tests pass after said rebuild
+
+You're now ready for development!
+
+### Host-based configuration (Not recommended)
+
+If you're on a Mac or Linux-based host machine, you can install and use the EMSDK directly to perform a build.
+Note that if you run into bugs with this configuration, we highly encourage you to use the containerized development environment instead, as detailed above.
+
+Instructions:
+
+1. [Install the EMSDK](https://emscripten.org/docs/getting_started/downloads.html)
+2. Clone this repository
+3. Run `$ npm install` to install the required modules
+4. Run `$ npm test` to ensure all tests pass
+5. Run `$ npm rebuild` to re-compile the project from scratch (using the version of EMSDK installed in the container).
+6. Run `$ npm test` to ensure all tests pass after said rebuild
+
+## Compiling SQLite with different options
+
+In order to enable extensions like FTS5, change the CFLAGS in the [Makefile](Makefile) and run `npm run rebuild`:
+
+``` diff
+CFLAGS = \
+        -O2 \
+        -DSQLITE_OMIT_LOAD_EXTENSION \
+        -DSQLITE_DISABLE_LFS \
+        -DSQLITE_ENABLE_FTS3 \
+        -DSQLITE_ENABLE_FTS3_PARENTHESIS \
++       -DSQLITE_ENABLE_FTS5 \
+        -DSQLITE_THREADSAFE=0
+```