Gt4py is used as a submodule #64
Conversation
|
Failed on final copy of caches with an error due to echoing gt4py version to the wrong file location, updated and resubmitting https://jenkins.ginko.ch/view/pace/job/pace-fv3core-cache-setup/143/ |
|
Cancelled cache build job for now, realized the caching infrastructure currently only supports one gt4py tag at a time and that this would disrupt other PRs. |
|
Updated gt4py caching plans to use multiple tar.gz's for each GT4PY_VERSION. This should avoid breaking other PRs when running the caching plan. Launched caching plan https://jenkins.ginko.ch/view/pace/job/pace-fv3core-cache-setup/144/ |
|
launch jenkins |
|
Caching plan likely passed since tests passed, but launched it again to see output in case an error happened later down the line. https://jenkins.ginko.ch/view/pace/job/pace-fv3core-cache-setup/158/ |
|
Resubmitting one caching configuration which had a "slurm error", possibly an intermittent issue. https://jenkins.ginko.ch/view/pace/job/pace-fv3core-cache-setup/160/ |
| @@ -111,13 +117,8 @@ echo "Benchmark directory: ${BENCHMARK_DIR}" | |||
| echo "Data directory: ${DATA_DIR}" | |||
| echo "Perf. artifact directory: ${TIMING_DIR}" | |||
| echo "Profile artifact directory: ${PROFILE_DIR}" | |||
| echo "Cache directory: ${CACHE_DIR}" | |||
There was a problem hiding this comment.
This was removed because CACHE_DIR is not used in this file, and because it gets printed out in the log already from the location where the variable is actually defined and used. Because this file doesn't use the variable, it could be incorrect without causing a failure.
| @@ -2,24 +2,23 @@ | |||
| BACKEND=$1 | |||
| EXPNAME=$2 | |||
| SANITIZED_BACKEND=`echo $BACKEND | sed 's/:/_/g'` #sanitize the backend from any ':' | |||
| CACHE_DIR="/scratch/snx3000/olifu/jenkins/scratch/store_gt_caches/" | |||
| CACHE_DIR="/scratch/snx3000/olifu/jenkins/scratch/store_gt_caches/${EXPNAME}/${SANITIZED_BACKEND}" | |||
There was a problem hiding this comment.
Here I updated the caching to use a tar.gz that has the gt4py version in its file path, so that we can have simultaneous caches for different gt4py versions. This was needed to be able to test this PR (which involves generating the caches) without breaking other PRs.
|
launch jenkins |
|
Launched performance test https://jenkins.ginko.ch/job/pace-fv3core-performance-test-PR/35/ |
|
Performance test passed. |
.jenkins/install_virtualenv.sh
Outdated
| @@ -11,20 +11,22 @@ exitError() | |||
|
|
|||
| # check a virtualenv path has been provided | |||
| test -n "$1" || exitError 1001 ${virtualenv_path} "must pass an argument" | |||
| pace_dir=`dirname $0`/../ | |||
There was a problem hiding this comment.
are we sure this always works (from wherever we call it?) Should we not aim for something like we do with the jenkins dir?
(I see that this is just moved - but just curious?)
There was a problem hiding this comment.
I am not sure that this always works, so I've been using the other pattern wherever I add new operations like this. But I've often kept the old pattern where it exists.
Should I update this to the new pattern?
There was a problem hiding this comment.
I'm ok with leaving it as is.
I was mostly wondering because we have several new places where you've added pieces along the lines of
SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
PACE_DIR=$SCRIPT_DIR/../..
And we could have used that here as well. But not really necessary
I've fixed it and now removed the symlink. We were not always using the top level file. This is also the case for fv3core, but I'm addressing it in the sphinx docs PR. |
|
sounds great, thanks |
|
Launched performance test https://jenkins.ginko.ch/job/pace-fv3core-performance-test-PR/37/. |
This PR updates pace to use gt4py pinned as a submodule, rather than using GT4PY_VERSION.txt. After this PR, the gt4py caches will use the gt4py sha rather than tag in their GT4PY_VERSION.txt files.
Changes:
git submodule update --initcommandsContinued from #56.