The `reproduce_ci_env.py` script reads in the yaml for the GitLab
runners and replicates the runs using Docker. Through a quirk of the
implemenation of MR !3191, some of the scripts, which are expected to be
a list of command strings, get defined as a list of lists. This is fixed
by flattening this list of commands.
MR !2598 introduced some fixes to `reproduce_ci_env.py`. One of
them handled newlines in command lists by separating the command
lists by newlines in the Dockerfile (which generated a script).
Although this worked for some of the docker containers, I found
others, like rhel8, where the newlines were not escaped correctly.
Correct this problem by moving back to separating commands with
`&&`. This requires replacing newlines in commands with `&&`.
That in itself is no problem except that if you have a blank
line, you get `&& &&`, which is a syntax error for the shell.
So, avoid this by stripping newlines from commands.
When `reproduce_ci_env.py` makes its docker container, it creates a pair
of scripts, `setup-gitlab-env.sh` and `run-gitlab-stage.sh`, inside the
container. These scripts came from the `before_script` and `script` CI
configuration parameters, respectively.
These two scripts were created by joining each item in the CI
configuration lists with `&&` onto a single line. However, this meant
that each list item had to be on its own line or it didn't work. A
recent configuration change meant that one of the configurations
contained multiple shell commands separated by newlines. This change
builds the script with multiple lines (which have to be carefully
escaped in the generated dockerfile).
Also modified these strings to escape quotes (`"`). This is important as
the dockerfile generates these scripts using an `echo` command that
needs to quote all of the arguments together.
When one CI build YAML spec extended another, the `reproduce_ci_env.py`
script was overwritting local configuration with the configuration being
extended. This change merges all configuration together (which appears
to be how the GitLab CI works).
Historically, the `/bin` directory on Unix has a reduced set of
commands used for small mounts while booting the system. As such,
it is common for `/bin` to be missing the `env` command.
For the same historical reasons, `/usr/bin` tends to have most if
not all commands provided to the user environment. Thus,
`/usr/bin/env` is more likely to exist than `/bin/env`. This is
in fact the case for Mac OSX, which is a very widely used version
of *nix.
Thus, it is better to use `#!/usr/bin/env` as the hash-bang in
scripts.
Previously stages like `doxygen` wouldn't be detected by
reproduce_ci_env as they didn't start with `build:` or `test:`.
Now we properly look for the `stage` key.
To simplify reproducing docker based CI workers locally, VTK-m has python program that handles all the
work automatically for you.
The program is located in `[Utilities/CI/reproduce_ci_env.py ]` and requires python3 and pyyaml.
To use the program is really easy! The following two commands will create the `build:rhel8` gitlab-ci
worker as a docker image and setup a container just as how gitlab-ci would be before the actual
compilation of VTK-m. Instead of doing the compilation, instead you will be given an interactive shell.
```
./reproduce_ci_env.py create rhel8
./reproduce_ci_env.py run rhel8
```
To compile VTK-m from the the interactive shell you would do the following:
```
> src]# cd build/
> build]# cmake --build .
```
