Run end-2-end tests in GitHub Actions CI (feast-dev#6)

Uses docker-compose to deploy alongside with Feast (Core, Kafka, PostgreSQL) containers, a CI container that runs end-to-end tests.

As docker-compose runs the container in the same network, it can access the other containers directly through the docker network.

With this PR we run a single basic integration test for Feast Core only, to validate the concept. Later on we will try to also deploy the serving container, and run more end-to-end tests.

Author: algattik

.github/workflows/integration-tests.yml

@@ -37,7 +37,6 @@ jobs:
         dockerfile: infra/docker/ci/Dockerfile
         tags: v${{ github.sha }}
 
-
     # Cache the Maven repository across runs.
     - name: Cache Maven repository
       id: cache-maven
@@ -49,27 +48,42 @@ jobs:
           ${{ runner.os }}-maven-
 
     # Generate the Maven cache if needed by running a throwaway build of Feast.
+    # --user: run as the host user, to avoid writing .m2 files to the host as root.
+    # -e MAVEN_CONFIG: See https://github.com/carlossg/docker-maven#running-as-non-root
+    # -v $PWD:/build: mount the current directory to write .m2 files onto the host.
     - name: Generate Maven cache
       if: steps.cache-maven.outputs.cache-hit != 'true'
-      run: docker run --rm --user $(id -u):$(id -g) -v $PWD:/build $IMAGE bash -c "cd /build && mvn -Dmaven.repo.local=/build/.m2/repository -Dgpg.skip=true -B verify -DskipTests"
+      run: docker run --rm --user $(id -u):$(id -g) -e MAVEN_CONFIG=/build/.m2 -v $PWD:/build $IMAGE bash -c "cd /build && mvn -Dmaven.repo.local=/build/.m2/repository -Dgpg.skip=true -B clean verify -DskipTests"
       env:
         IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-ci:v${{ github.sha }}
 
-    # "docker-compose up --build" builds and starts docker images.
+    # Build images in parallel.
+    - name: Build Feast runtime and end-to-end test images
+      run: |
+        set -x
+        docker-compose -f infra/docker-compose/docker-compose.yml build --parallel end-to-end-tests core online-serving
+      env:
+        FEAST_VERSION: v${{ github.sha }}
+        FEAST_CI_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-ci
+        FEAST_CORE_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-core
+        FEAST_SERVING_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-serving
+
+    # "docker-compose up" starts docker containers.
+    # "--abort-on-container-exit" ensures that the containers are shutdown when end-to-end
+    # tests complete, and that the task exit status indicates failure if there are test failures.  
     # Images fetched from a repository, such as Kafka, will not be built, so this
     # step will only build the Feast Core image.
-    # After this step, the specified containers have been started and Docker Core
-    # has successfully connected to its required services (such as PostgreSQL).
-    - name: Start local Feast environment
+    - name: Run Feast end-to-end test with Docker compose
       run: |
-        docker-compose -f infra/docker-compose/docker-compose.yml up -d --build core redis kafka db
-        for i in {1..12}; do nc -zv localhost 6565 && break; echo "Waiting for Feast Core to come online on port 6565..."; sleep 10; done
+        set -x
+        docker-compose -f infra/docker-compose/docker-compose.yml up --abort-on-container-exit end-to-end-tests
       env:
         COMPOSE_PROJECT_NAME: feast
         FEAST_VERSION: v${{ github.sha }}
+        FEAST_CI_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-ci
         FEAST_CORE_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-core
+        FEAST_SERVING_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-serving
         FEAST_CORE_CONFIG: direct-runner.yml
-        FEAST_SERVING_IMAGE: gcr.io/kf-feast/feast-serving
         FEAST_ONLINE_SERVING_CONFIG: online-serving.yml
         FEAST_ONLINE_STORE_CONFIG: redis-store.yml
 

docs/cicd/README.md

@@ -1,26 +1,36 @@
 # FarFetch CI setup
 
-The CI setup ensures that PRs on GitHub are automatically tested before being merging to *master*. The CI jobs defined under 
-
-[.github/workflows]: ../../.github/workflows
-
-run compilation, unit and end-to-end tests using GitHub Workflows.
+The CI setup ensures that PRs on GitHub are automatically tested before being merging to *master*. The CI jobs defined under [.github/workflows](../../.github/workflows) run compilation, unit and end-to-end tests using GitHub Workflows.
 
  ## Agent configuration
 
 1. Spin an Azure VM with Ubuntu 18.04. Choose e.g. a B4ms instance (burstable VM with 4 cores)
 2. Install prerequisites:
 
+Install **Docker**:
 
 ```
 sudo apt update
 sudo apt install docker.io
 sudo usermod -aG docker $USER
+```
+
+Install **docker-compose**:
+
+```
 
 sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
 sudo chmod +x docker-compose
 ```
 
+Upgrade **Git** to a more recent version to ensure repos are properly checked out including `.git` directory (used by Python SDK setup.py):
+
+```
+sudo add-apt-repository ppa:git-core/ppa -y
+sudo apt-get update
+sudo apt-get install git -y
+```
+
 3. Follow the actions to [add a self-hosted runner to your repository](https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners)
 4. Follow the actions to [configure the self-hosted runner as a service](https://help.github.com/en/actions/hosting-your-own-runners/configuring-the-self-hosted-runner-application-as-a-service)
 
@@ -38,7 +48,7 @@ Once the ACR instance is deployed, fetch the password from the Access keys secti
 
 Populate the GitHub secrets settings with the ACR configuration:
 
-![](create-secrets.png)
+![GitHub secrets](create-secrets.png)
 
 - CONTAINERREGISTRY_IMAGENAMEBASE=farfetchfeast
 - CONTAINERREGISTRY_PASSWORD=<ACR password>
@@ -50,5 +60,3 @@ Populate the GitHub secrets settings with the ACR configuration:
 The integration test job deploys the application components using Docker-compose. Later on, it will be extended to run end-to-end test scenarios.
 
 ![Integration test job](integration-test-job.png)
-
-

infra/docker-compose/docker-compose.yml

@@ -23,6 +23,11 @@ services:
       - -jar
       - /opt/feast/feast-core.jar
       - --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yaml
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8080/healthz"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
 
   online-serving:
     build:
@@ -119,4 +124,38 @@ services:
     environment:
       POSTGRES_PASSWORD: password
     ports:
-      - "5432:5342"
+      - "5432:5342"
+
+  end-to-end-tests:
+    build:
+      context: ../..
+      dockerfile: tests/e2e/Dockerfile
+      args:
+        FEAST_CI_IMAGE: ${FEAST_CI_IMAGE}:${FEAST_VERSION}
+    volumes:
+      - ../..:/feast
+    depends_on:
+      - core
+      - online-serving
+    command:
+      - bash
+      - -c
+      - |
+        set -ex
+        cd /feast
+
+        make compile-protos-python
+
+        # Ensure Feast Core is running and has initialized
+        for i in {1..60}; do curl -sf http://core:8080/healthz && break; echo "Waiting for Core..."; sleep 1; done
+        curl -f http://core:8080/healthz
+
+        # Run single end-to-end test
+        cd tests/e2e
+        pytest basic-ingest-redis-serving.py \
+          --core_url core:6565 \
+          --serving_url online-serving:6566 \
+          -k test_basic_register_feature_set_success  \
+          -sv \
+          --junitxml=build/junit.xml \
+          -o junit_family=xunit2

infra/docker/ci/Dockerfile

@@ -44,5 +44,8 @@ RUN PROTOC_ZIP=protoc-${PROTOC_VERSION}-linux-x86_64.zip && \
 	  go build && mkdir -p $HOME/bin && cp protoc-gen-docs $HOME/bin
 
 COPY Makefile /build/Makefile
-COPY sdk/python/requirements-ci.txt /build/sdk/python/requirements-ci.txt
-RUN cd /build && make install-python-ci-dependencies install-go-ci-dependencies && rm -rf /build
+COPY sdk/python/requirements-*.txt /build/sdk/python/
+RUN cd /build && \
+    make install-python-ci-dependencies install-go-ci-dependencies && \
+    pip install -r sdk/python/requirements-dev.txt && \
+    rm -rf /build

infra/docker/core/Dockerfile

@@ -4,8 +4,22 @@
 
 FROM maven:3.6-jdk-11-slim as builder
 ARG REVISION=dev
-COPY  . /build
+
 WORKDIR /build
+
+COPY datatypes datatypes
+COPY storage storage
+COPY sdk/java sdk/java
+COPY core core
+COPY ingestion ingestion
+COPY protos protos
+COPY serving serving
+COPY pom.xml pom.xml
+
+# Trick to copy .m2 directory only if it exists.
+# The LICENSE file is any file that actually exists, to make sure the command doesn't fail.
+COPY LICENSE .m[2] .m2/
+
 #
 # Setting Maven repository .m2 directory relative to /build folder gives the
 # user to optionally use cached repository when building the image by copying

infra/docker/serving/Dockerfile

@@ -4,8 +4,22 @@
 
 FROM maven:3.6-jdk-11 as builder
 ARG REVISION=dev
-COPY  . /build
+
 WORKDIR /build
+
+COPY datatypes datatypes
+COPY storage storage
+COPY sdk/java sdk/java
+COPY core core
+COPY ingestion ingestion
+COPY protos protos
+COPY serving serving
+COPY pom.xml pom.xml
+
+# Trick to copy .m2 directory only if it exists.
+# The LICENSE file is any file that actually exists, to make sure the command doesn't fail.
+COPY LICENSE .m[2] .m2/
+
 #
 # Setting Maven repository .m2 directory relative to /build folder gives the
 # user to optionally use cached repository when building the image by copying

tests/e2e/Dockerfile

@@ -0,0 +1,26 @@
+ARG FEAST_CI_IMAGE
+
+FROM ${FEAST_CI_IMAGE}
+
+WORKDIR /feast
+
+COPY Makefile .
+COPY protos protos
+COPY sdk/python sdk/python
+
+# Compile feast.core Python protobuf module
+# FIXME this is redundant with infra/docker/ci/Dockerfile
+RUN make compile-protos-python
+
+# Install Feast Python SDK and test requirements
+
+COPY tests/e2e/requirements.txt tests/e2e/requirements.txt
+COPY README.md README.md
+
+# setup.py in sdk/python grabs the version number from the .git
+# files, but we don't want to COPY .git so as to maximize
+# layer reuse. Fool it by initialize an empty .git repository.
+RUN git init .
+
+RUN pip install -qe sdk/python && \
+  pip install -qr tests/e2e/requirements.txt