The `reproduce_ci_env.py` script reads in the yaml for the GitLab
runners and replicates the runs using Docker. Through a quirk of the
implemenation of MR !3191, some of the scripts, which are expected to be
a list of command strings, get defined as a list of lists. This is fixed
by flattening this list of commands.
MR !2598 introduced some fixes to `reproduce_ci_env.py`. One of
them handled newlines in command lists by separating the command
lists by newlines in the Dockerfile (which generated a script).
Although this worked for some of the docker containers, I found
others, like rhel8, where the newlines were not escaped correctly.
Correct this problem by moving back to separating commands with
`&&`. This requires replacing newlines in commands with `&&`.
That in itself is no problem except that if you have a blank
line, you get `&& &&`, which is a syntax error for the shell.
So, avoid this by stripping newlines from commands.
When `reproduce_ci_env.py` makes its docker container, it creates a pair
of scripts, `setup-gitlab-env.sh` and `run-gitlab-stage.sh`, inside the
container. These scripts came from the `before_script` and `script` CI
configuration parameters, respectively.
These two scripts were created by joining each item in the CI
configuration lists with `&&` onto a single line. However, this meant
that each list item had to be on its own line or it didn't work. A
recent configuration change meant that one of the configurations
contained multiple shell commands separated by newlines. This change
builds the script with multiple lines (which have to be carefully
escaped in the generated dockerfile).
Also modified these strings to escape quotes (`"`). This is important as
the dockerfile generates these scripts using an `echo` command that
needs to quote all of the arguments together.
When one CI build YAML spec extended another, the `reproduce_ci_env.py`
script was overwritting local configuration with the configuration being
extended. This change merges all configuration together (which appears
to be how the GitLab CI works).
Historically, the `/bin` directory on Unix has a reduced set of
commands used for small mounts while booting the system. As such,
it is common for `/bin` to be missing the `env` command.
For the same historical reasons, `/usr/bin` tends to have most if
not all commands provided to the user environment. Thus,
`/usr/bin/env` is more likely to exist than `/bin/env`. This is
in fact the case for Mac OSX, which is a very widely used version
of *nix.
Thus, it is better to use `#!/usr/bin/env` as the hash-bang in
scripts.
Previously stages like `doxygen` wouldn't be detected by
reproduce_ci_env as they didn't start with `build:` or `test:`.
Now we properly look for the `stage` key.
- It also adds Google's benchmarch compare.py script
- It is installed to the build directory.
- It add a wrapper script called compare-benchmarks.py which:
- Let you run each of the benchmarks with different devices
- It adds a README.md explaining how to run the benchmarks
- BenchmarkDeviceAdapter input size range parametrized at compile time
Signed-off-by: Vicente Adolfo Bolea Sanchez <vicente.bolea@kitware.com>
To simplify reproducing docker based CI workers locally, VTK-m has python program that handles all the
work automatically for you.
The program is located in `[Utilities/CI/reproduce_ci_env.py ]` and requires python3 and pyyaml.
To use the program is really easy! The following two commands will create the `build:rhel8` gitlab-ci
worker as a docker image and setup a container just as how gitlab-ci would be before the actual
compilation of VTK-m. Instead of doing the compilation, instead you will be given an interactive shell.
```
./reproduce_ci_env.py create rhel8
./reproduce_ci_env.py run rhel8
```
To compile VTK-m from the the interactive shell you would do the following:
```
> src]# cd build/
> build]# cmake --build .
```
Due to a bug in pthread, loguru would leak 12 bytes memory from the main
thread when `loguru::init` is in use. Since GCC does not support
blacklist file, the suppression logic is added to
ctest_memcheck_supp_file file.
See details in loguru github issue #59.
This includes updated to the latest master. To help with that,
I added an alias to SetupForDevelopment.sh to update the
master branch regardless of what branch you are on.
A recent merge request corrected several spelling errors in VTK-m.
One such correction was in the git-gitlab-sync script in the
Utilities/GitSetup directory. This commit reverts the change in
this specific file for two reasons.
1. The changed introduced a \' inside a single quote string (to
correct cant to can't). However, single quotes in shell scripts
do not allow you to escape characters like that, and thus this
causes an error when running the script.
2. This script actually comes from a separate repository
(https://gitlab.kitware.com/utils/gitsetup) that we occasionally
syncronize with. To prevent confusion, we should minimize the
divergence between this repository and that one. If someone wants
to make this change, it should really be made in the GitSetup
repository.
Specifically, several of the git setup scripts defined a function
named "egrep-q". This almost always works, but I happened to have
an install of git on windows/cygwin that gave the error:
`egrep-q': not a valid identifier
I always thought hyphens were allowed in script identifiers (and
usually they are), but apparently sometimes they are not. (See
for example
https://stackoverflow.com/questions/2821043/allowed-characters-in-linux-environment-variable-names).
This provides an easy fix by replacing egrep-q with egrep_q.
