Merge branch 'master' into pmd/7.0.x

This commit is contained in:
Andreas Dangel 2020-11-14 19:35:13 +01:00
commit bad14719c0
51 changed files with 1237 additions and 798 deletions

183
.ci/README.md Normal file
View File

@ -0,0 +1,183 @@
## PMD CI Scripts
This folder contains scripts used for CI.
## Secrets
One secret is required for decrypting the GPG Key with which the PMD Releases are signed and
for a ssh key, which is used to copy files to sourceforge.
## Environment variables
* PMD_CI_SECRET_PASSPHRASE
* CI_DEPLOY_USER
* CI_DEPLOY_PASSWORD
* CI_SIGN_KEY
* CI_SIGN_PASSPHRASE
* PMD_SF_USER
* PMD_SF_APIKEY
* GITHUB_OAUTH_TOKEN
* GITHUB_BASE_URL
* COVERALLS_REPO_TOKEN
* SONAR_TOKEN
* DANGER_GITHUB_API_TOKEN
* PMD_CI_CHUNK_TOKEN
## Encrypting
gpg --batch --symmetric --cipher-algo AES256 --passphrase="$PMD_CI_SECRET_PASSPHRASE" file.txt
## Known Issues
### Intermittent connection resets or timeouts while downloading dependencies from maven central
Root issue seems to be SNAT Configs in Azure, which closes long running [idle TCP connections
after 4 minutes](https://docs.microsoft.com/en-us/azure/load-balancer/troubleshoot-outbound-connection#idletimeout).
The workaround is described in [actions/virtual-environments#1499](https://github.com/actions/virtual-environments/issues/1499)
and [WAGON-545](https://issues.apache.org/jira/browse/WAGON-545)
and [WAGON-486](https://issues.apache.org/jira/browse/WAGON-486):
The setting `-Dmaven.wagon.httpconnectionManager.ttlSeconds=180 -Dmaven.wagon.http.retryHandler.count=3`
makes sure, that Maven doesn't try to use pooled connections that have been unused for more than 180 seconds.
These settings are placed as environment variable `MAVEN_OPTS` in all workflows, so that they are active for
all Maven executions (including builds done by regression tester).
Alternatively, pooling could be disabled completely via `-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false`.
This has the consequence, that for each dependency, that is being downloaded, a new https connection is
established.
More information about configuring this can be found at [wagon-http](https://maven.apache.org/wagon/wagon-providers/wagon-http/).
However, this doesn't work when [dokka-maven-plugin](https://github.com/Kotlin/dokka) is used: This plugin
downloads additional dokka plugins at runtime and reconfigures somehow Maven. After this plugin is loaded,
the above system properties have no effect anymore.
See [dokka/dokka-maven-plugin#1625](https://github.com/Kotlin/dokka/issues/1625) and
[dokka/dokka-maven-plugin#1626](https://github.com/Kotlin/dokka/issues/1626).
The workaround now in place is, to download all the dependencies first, see `inc/maven-dependencies.inc`.
## Hints
### Remote debugging
Debugging remotely is possible with <https://github.com/mxschmitt/action-tmate>.
Just add the following step into the job:
```
- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
```
The workflow `troubleshooting` can be started manually, which already contains the tmate action.
**Note**: This is dangerous for push/pull builds on pmd/pmd, because these have access to the secrets and the SSH session
is not protected. Builds triggered by pull requests from forked repositories don't have access to the secrets.
### Local tests with docker
Create a local docker container:
```
cd .ci/docker_ubuntu18.04
docker build -t pmd-ci .
```
This container is based on Ubuntu 18.04, which is used for `ubuntu-latest` github actions runner,
see [Virtual Environment](https://github.com/actions/virtual-environments).
You can run a local instance with docker:
```
docker run -it pmd-ci
```
You'll be dropped into a bash.
#### Testing a push build (snapshot)
Start docker without binding to local directory, so that we can do a fresh checkout: `docker run -it pmd-ci`.
You'll be dropped into a bash. Use the following script, to setup and start the build:
```
MAIN_BRANCH="master"
export MAVEN_OPTS="-Dmaven.wagon.httpconnectionManager.ttlSeconds=180 -Dmaven.wagon.http.retryHandler.count=3"
export PMD_CI_JOB_URL="manual job execution in docker"
export PMD_CI_PUSH_COMMIT_COMPARE=""
export PMD_CI_GIT_REF="refs/heads/${MAIN_BRANCH}"
export PMD_CI_SECRET_PASSPHRASE="xyz"
cd /workspaces/pmd
rmdir pmd && mkdir pmd
cd pmd
git init
git remote add origin https://github.com/pmd/pmd
git fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/${MAIN_BRANCH}:refs/remotes/origin/${MAIN_BRANCH}
git checkout --progress --force -B master refs/remotes/origin/${MAIN_BRANCH}
.ci/check-environment.sh
.ci/build.sh
```
#### Performing a release (push) build
Start docker without binding to local directory, so that we can do a fresh checkout: `docker run -it pmd-ci`.
You'll be dropped into a bash. Use the following script, to setup and start the build:
```
TAG_NAME="pmd_releases/0.0.0_release_test"
export MAVEN_OPTS="-Dmaven.wagon.httpconnectionManager.ttlSeconds=180 -Dmaven.wagon.http.retryHandler.count=3"
export PMD_CI_JOB_URL="manual job execution in docker"
export PMD_CI_PUSH_COMMIT_COMPARE=""
export PMD_CI_GIT_REF="refs/tags/${TAG_NAME}"
export PMD_CI_SECRET_PASSPHRASE="xyz"
cd /workspace/pmd
rmdir pmd && mkdir pmd
cd pmd
git init
git remote add origin https://github.com/pmd/pmd
git fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/tags/${TAG_NAME}:refs/tags/${TAG_NAME}
git checkout --progress --force refs/tags/${TAG_NAME}
.ci/check-environment.sh
.ci/build.sh
```
**Warning:** This will build and upload to maven central!
#### Testing a pull request
Start docker without binding to local directory, so that we can do a fresh checkout: `docker run -it pmd-ci`.
You'll be dropped into a bash. Use the following script, to setup and start the build:
```
export MAVEN_OPTS="-Dmaven.wagon.httpconnectionManager.ttlSeconds=180 -Dmaven.wagon.http.retryHandler.count=3"
export PMD_CI_BRANCH="master" # base branch
export PMD_CI_PULL_REQUEST_NUMBER=2913
# these are used by danger
export GITHUB_EVENT_PATH=/workspaces/event.json
export GITHUB_REPOSITORY=pmd/pmd
export GITHUB_ACTION=run1
export GITHUB_EVENT_NAME=pull_request
/home/pmd-ci/create-gh-pull-request-event.sh
cd /workspace/pmd
rmdir pmd && mkdir pmd
cd pmd
git init
git remote add origin https://github.com/pmd/pmd
git fetch --no-tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/pull/${PMD_CI_PULL_REQUEST_NUMBER}/merge:refs/remotes/pull/${PMD_CI_PULL_REQUEST_NUMBER}/merge
git checkout --progress --force refs/remotes/pull/${PMD_CI_PULL_REQUEST_NUMBER}/merge
.ci/check-environment.sh
.ci/build-pr.sh
```

37
.ci/build-coveralls.sh Executable file
View File

@ -0,0 +1,37 @@
#!/usr/bin/env bash
source $(dirname $0)/inc/logger.inc
source $(dirname $0)/inc/setup-secrets.inc
source $(dirname $0)/inc/install-openjdk.inc
source $(dirname $0)/inc/maven-dependencies.inc
set -e
log_group_start "Setup private env and OpenJDK"
pmd_ci_setup_private_env
install_openjdk_setdefault 11
export CI_NAME="github actions"
export CI_BUILD_URL="${PMD_CI_JOB_URL}"
export CI_BRANCH="${PMD_CI_GIT_REF##refs/heads/}"
log_group_end
log_group_start "Downloading maven dependencies"
maven_dependencies_resolve
log_group_end
log_group_start "Executing build with coveralls"
./mvnw \
-Dmaven.javadoc.skip=true \
-Dmaven.source.skip \
-Dcheckstyle.skip \
-DrepoToken=${COVERALLS_REPO_TOKEN} \
-B -V -e \
clean package jacoco:report \
coveralls:report -Pcoveralls
if [ $? -ne 0 ]; then
log_error "Error creating coveralls report"
else
log_success "New coveralls result: https://coveralls.io/github/pmd/pmd"
fi
log_group_end

34
.ci/build-pr-win-macos.sh Executable file
View File

@ -0,0 +1,34 @@
#!/usr/bin/env bash
source $(dirname $0)/inc/logger.inc
source $(dirname $0)/inc/install-openjdk.inc
source $(dirname $0)/inc/regression-tester.inc
source $(dirname $0)/inc/maven-dependencies.inc
set -e
log_group_start "Installing Java 8+11"
log_info "Install openjdk11 as default"
install_openjdk_setdefault 11
log_info "Install openjdk8 for integration tests and pmd-regression-tests"
install_openjdk 8
log_group_end
log_group_start "Downloading maven dependencies"
maven_dependencies_resolve
log_group_end
log_group_start "Building with maven"
./mvnw -e -V clean verify -Djava8.home=${HOME}/openjdk8
log_group_end
# Danger is executed only on the linux runner
case "$(uname)" in
Linux*)
log_group_start "Executing danger"
regression_tester_setup_ci
regression_tester_executeDanger
log_group_end
;;
esac

33
.ci/build-sonar.sh Executable file
View File

@ -0,0 +1,33 @@
#!/usr/bin/env bash
source $(dirname $0)/inc/logger.inc
source $(dirname $0)/inc/setup-secrets.inc
source $(dirname $0)/inc/install-openjdk.inc
source $(dirname $0)/inc/maven-dependencies.inc
set -e
log_group_start "Setup private env and OpenJDK"
pmd_ci_setup_private_env
install_openjdk_setdefault 11
log_group_end
log_group_start "Downloading maven dependencies"
maven_dependencies_resolve
log_group_end
log_group_start "Executing build with sonar"
./mvnw \
-Dmaven.javadoc.skip=true \
-Dmaven.source.skip \
-Dcheckstyle.skip \
-B -V -e \
clean package \
sonar:sonar -Dsonar.login=${SONAR_TOKEN} -Psonar
if [ $? -ne 0 ]; then
log_error "Error updating sonar..."
else
log_success "New sonar results: https://sonarcloud.io/dashboard?id=net.sourceforge.pmd%3Apmd"
fi
log_group_end

211
.ci/build.sh Executable file
View File

@ -0,0 +1,211 @@
#!/usr/bin/env bash
source $(dirname $0)/inc/logger.inc
source $(dirname $0)/inc/setup-secrets.inc
source $(dirname $0)/inc/sourceforge-api.inc
source $(dirname $0)/inc/pmd-doc.inc
source $(dirname $0)/inc/pmd-code-api.inc
source $(dirname $0)/inc/regression-tester.inc
source $(dirname $0)/inc/github-releases-api.inc
source $(dirname $0)/inc/maven-dependencies.inc
source $(dirname $0)/inc/install-openjdk.inc
set -e
function pmd_ci_build_main() {
log_group_start "Setting up private secrets"
pmd_ci_setup_private_env
pmd_ci_setup_gpg_key
pmd_ci_setup_ssh
log_group_end
log_group_start "Prepare Java 8+11, Maven, Bundler"
log_info "Install openjdk11 as default"
install_openjdk_setdefault 11
log_info "Install openjdk8 for integration tests and pmd-regression-tests"
install_openjdk 8
pmd_ci_build_setup_maven
pmd_ci_build_setup_bundler
pmd_ci_build_setup_env
log_group_end
log_group_start "Downloading maven dependencies"
maven_dependencies_resolve
log_group_end
log_group_start "Build and Deploy"
pmd_ci_build_run
pmd_ci_deploy_build_artifacts
log_group_end
log_group_start "Build and Upload documentation"
pmd_ci_build_and_upload_doc
log_group_end
if pmd_ci_build_isRelease; then
log_group_start "Publishing Release"
gh_release_publishRelease "$GH_RELEASE"
sourceforge_selectDefault "${VERSION}"
log_group_end
fi
log_group_start "Creating new baseline for regression tester"
regression_tester_setup_ci
regression_tester_uploadBaseline
log_group_end
exit 0
}
#
# Configures maven.
# Needed for deploy to central (both snapshots and releases)
# and for signing the artifacts.
#
function pmd_ci_build_setup_maven() {
mkdir -p ${HOME}/.m2
cp .ci/files/maven-settings.xml ${HOME}/.m2/settings.xml
}
#
# Installs bundler, which is needed for doc generation and regression tester
#
function pmd_ci_build_setup_bundler() {
log_info "Installing bundler..."
gem install bundler
}
#
# Setups common build parameters:
# * Determines the VERSION of PMD, that is being built
# * Determines the PMD_CI_BRANCH or PMD_CI_TAG, that is being built
#
function pmd_ci_build_setup_env() {
VERSION=$(pmd_ci_build_get_pom_version)
if [[ "${PMD_CI_GIT_REF}" == refs/heads/* ]]; then
PMD_CI_BRANCH=${PMD_CI_GIT_REF##refs/heads/}
unset PMD_CI_TAG
log_info "Building PMD ${VERSION} on branch ${PMD_CI_BRANCH}"
elif [[ "${PMD_CI_GIT_REF}" == refs/tags/* ]]; then
unset PMD_CI_BRANCH
PMD_CI_TAG=${PMD_CI_GIT_REF##refs/tags/}
log_info "Building PMD ${VERSION} on tag ${PMD_CI_TAG}"
else
log_error "Unknown branch/tag: PMD_CI_GIT_REF=${PMD_CI_GIT_REF}"
exit 1
fi
if [[ "${VERSION}" == *-SNAPSHOT && -z "$PMD_CI_BRANCH" ]]; then
log_error "Invalid combination: snapshot version ${VERSION} but no branch in PMD_CI_GIT_REF=${PMD_CI_GIT_REF}"
exit 1
fi
if [[ "${VERSION}" != *-SNAPSHOT && -z "$PMD_CI_TAG" ]]; then
log_error "Invalid combination: non-snapshot version ${VERSION} but no tag in PMD_CI_GIT_REF=${PMD_CI_GIT_REF}"
exit 1
fi
}
#
# Performs the actual build.
# Deploys the artifacts to maven central.
# Also generates rule documentation.
#
function pmd_ci_build_run() {
local mvn_profiles="ossrh,sign,generate-rule-docs"
if pmd_ci_build_isRelease; then
log_info "This is a release build"
mvn_profiles="${mvn_profiles},pmd-release"
else
log_info "This is a snapshot build"
fi
./mvnw clean deploy -P${mvn_profiles} -e -V -Djava8.home=${HOME}/openjdk8
}
#
# Deploys the binary distribution
#
function pmd_ci_deploy_build_artifacts() {
# Deploy to sourceforge files
sourceforge_uploadFile "${VERSION}" "pmd-dist/target/pmd-bin-${VERSION}.zip"
sourceforge_uploadFile "${VERSION}" "pmd-dist/target/pmd-src-${VERSION}.zip"
if pmd_ci_build_isRelease; then
# create a draft github release
gh_releases_createDraftRelease "${PMD_CI_TAG}" "$(git rev-list -n 1 ${PMD_CI_TAG})"
GH_RELEASE="$RESULT"
# Deploy to github releases
gh_release_uploadAsset "$GH_RELEASE" "pmd-dist/target/pmd-bin-${VERSION}.zip"
gh_release_uploadAsset "$GH_RELEASE" "pmd-dist/target/pmd-src-${VERSION}.zip"
fi
}
#
# Builds and uploads the documentation site
#
function pmd_ci_build_and_upload_doc() {
pmd_doc_generate_jekyll_site
pmd_doc_create_archive
sourceforge_uploadFile "${VERSION}" "docs/pmd-doc-${VERSION}.zip"
if pmd_ci_build_isRelease; then
gh_release_uploadAsset "$GH_RELEASE" "docs/pmd-doc-${VERSION}.zip"
fi
# Deploy doc to https://docs.pmd-code.org/pmd-doc-${VERSION}/
pmd_code_uploadDocumentation "${VERSION}" "docs/pmd-doc-${VERSION}.zip"
# Deploy javadoc to https://docs.pmd-code.org/apidocs/*/${VERSION}/
pmd_code_uploadJavadoc "${VERSION}" "$(pwd)"
if [[ "${VERSION}" == *-SNAPSHOT && "${PMD_CI_BRANCH}" == "master" ]]; then
# only for snapshot builds from branch master
pmd_code_createSymlink "${VERSION}" "snapshot"
# update github pages https://pmd.github.io/pmd/
pmd_doc_publish_to_github_pages
# rsync site to https://pmd.sourceforge.io/snapshot
sourceforge_rsyncSnapshotDocumentation "${VERSION}" "snapshot"
fi
if pmd_ci_build_isRelease; then
# documentation is already uploaded to https://docs.pmd-code.org/pmd-doc-${VERSION}
# we only need to setup symlinks for the released version
pmd_code_createSymlink "${VERSION}" "latest"
# remove old doc and point to the new version
pmd_code_removeDocumentation "${VERSION}-SNAPSHOT"
pmd_code_createSymlink "${VERSION}" "${VERSION}-SNAPSHOT"
# remove old javadoc
pmd_code_removeJavadoc "${VERSION}-SNAPSHOT"
# updating github release text
# renders, and skips the first 6 lines - the Jekyll front-matter
local rendered_release_notes=$(bundle exec .ci/render_release_notes.rb docs/pages/release_notes.md | tail -n +6)
local release_name="PMD ${VERSION} ($(date -u +%d-%B-%Y))"
gh_release_updateRelease "$GH_RELEASE" "$release_name" "$rendered_release_notes"
sourceforge_uploadReleaseNotes "${VERSION}" "${rendered_release_notes}"
publish_release_documentation_github
sourceforge_rsyncSnapshotDocumentation "${VERSION}" "pmd-${VERSION}"
fi
}
function pmd_ci_build_isRelease() {
if [[ "${VERSION}" != *-SNAPSHOT && -n "${PMD_CI_TAG}" && -z "${PMD_CI_BRANCH}" ]]; then
return 0
else
return 1
fi
}
function pmd_ci_build_get_pom_version() {
echo $(./mvnw -q -Dexec.executable="echo" -Dexec.args='${project.version}' --non-recursive org.codehaus.mojo:exec-maven-plugin:3.0.0:exec)
}
pmd_ci_build_main

60
.ci/check-environment.sh Executable file
View File

@ -0,0 +1,60 @@
#!/usr/bin/env bash
#
# This script should check, that all needed commands are available
# and are in the correct version.
#
source $(dirname $0)/inc/logger.inc
set -e
function check() {
local CMD=$1
local VERSION_CMD=$2
local VERSION_STRING=$3
echo -n "Checking ${CMD}..."
if hash "$CMD" 2>/dev/null; then
local VERSION_FULL=$(${VERSION_CMD} 2>&1)
local VERSION=$(echo "${VERSION_FULL}" | grep "${VERSION_STRING}" 2>&1)
if [ -n "${VERSION}" ]; then
echo -e "${COL_GREEN}OK${COL_RESET}"
echo " ${VERSION}"
else
echo -e "${COL_RED}wrong version${COL_RESET}. Expected: ${VERSION_STRING}"
echo " ${VERSION_FULL}"
fi
else
echo -e "${COL_RED}not found!${COL_RESET}"
fi
}
# every OS:
check "curl" "curl --version" "curl"
check "jq" "jq --version" "jq"
case "$(uname)" in
Linux*)
check "ruby" "ruby --version" "ruby 2.7"
check "gpg" "gpg --version" "gpg (GnuPG) 2."
check "printenv" "printenv --version" "printenv (GNU coreutils)"
check "rsync" "rsync --version" "version"
check "ssh" "ssh -V" "OpenSSH"
check "git" "git --version" "git version"
check "mvn" "mvn --version" "Apache Maven"
check "unzip" "unzip --version" "UnZip"
check "zip" "zip --version" "This is Zip"
#check "7z" "7z -version" "7-Zip"
;;
Darwin*)
;;
CYGWIN*|MINGW*)
check "7z" "7z -version" "7-Zip"
;;
*)
log_error "Unknown OS: $(uname)"
exit 1
;;
esac

View File

@ -0,0 +1,31 @@
# https://hub.docker.com/_/ubuntu/
FROM ubuntu:18.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get upgrade --yes && \
apt-get install --yes curl jq gpg rsync ssh git p7zip-full openjdk-8-jdk \
libgdbm-dev libncurses5-dev automake libtool bison libffi-dev \
sudo nano bash tzdata unzip zip && \
apt-get clean
RUN cd opt && \
curl https://mirror.checkdomain.de/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz | tar xz && \
ln -sf /opt/apache-maven-3.6.3/bin/mvn /usr/local/bin/mvn && \
cd ..
RUN groupadd --gid 1000 pmd-ci && useradd --gid 1000 --uid 1000 --groups sudo \
--shell /bin/bash --create-home --password "" \
pmd-ci
RUN mkdir -p /workspaces/pmd/pmd && chown -R pmd-ci:pmd-ci /workspaces && ln -sf /workspaces /home/pmd-ci/workspaces
USER pmd-ci:pmd-ci
WORKDIR /home/pmd-ci
COPY create-gh-pull-request-event.sh .
COPY install-ruby.sh .
RUN ./install-ruby.sh
CMD ["/bin/bash", "--login"]
#
# build with: docker build -t pmd-ci .
# run with: docker run -it pmd-ci
#

View File

@ -0,0 +1,10 @@
#!/usr/bin/env bash
cat > /workspaces/event.json <<EOF
{
"number": ${PMD_CI_PULL_REQUEST_NUMBER},
"repository": {
"clone_url": "https://github.com/pmd/pmd.git"
}
}
EOF

View File

@ -0,0 +1,8 @@
#!/usr/bin/env bash
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
curl -sSL https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
rvm install 2.7
rvm use 2.7 --default
ruby -v

BIN
.ci/files/id_rsa.gpg Normal file

Binary file not shown.

View File

@ -12,7 +12,7 @@
<servers>
<server>
<id>ossrh</id>
<username>${env.CI_DEPLOY_USERNAME}</username>
<username>${env.CI_DEPLOY_USER}</username>
<password>${env.CI_DEPLOY_PASSWORD}</password>
</server>
</servers>
@ -22,7 +22,7 @@
<profile>
<id>ossrh</id>
<properties>
<gpg.keyname>${env.CI_SIGN_KEYNAME}</gpg.keyname>
<gpg.keyname>${env.CI_SIGN_KEY}</gpg.keyname>
<gpg.passphrase>${env.CI_SIGN_PASSPHRASE}</gpg.passphrase>
</properties>
</profile>

BIN
.ci/files/private-env.gpg Normal file

Binary file not shown.

View File

@ -18,6 +18,8 @@ if test -e classpath.txt; then
exit
fi
set -e
mvn test-compile
mvn dependency:build-classpath -DincludeScope=test -Dmdep.outputFile=classpath.txt
]]></build-command>
@ -35,30 +37,56 @@ if test -e classpath.txt; then
exit
fi
set -e
# Note: openjdk8 will be installed by "before_install.sh"
JAVA_HOME=${HOME}/openjdk8
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME=${HOME}/openjdk8
export PATH=$JAVA_HOME/bin:$PATH
# keep the tabs!!
(cat <<EOF
--- build.gradle 2020-11-14 09:43:51.705417551 +0000
+++ build.gradle.patched 2020-11-14 09:43:27.265215303 +0000
@@ -1,5 +1,6 @@
buildscript {
repositories {
+ mavenCentral()
maven { url "https://repo.spring.io/plugins-release" }
}
dependencies {
@@ -138,6 +139,7 @@
}
repositories {
+ mavenCentral()
maven { url "https://repo.spring.io/libs-release" }
}
@@ -314,3 +316,20 @@
}
return version
}
+
+task createSquishClasspath {
+ doLast {
+ def dependencies = new HashSet()
+ dependencies.addAll(subprojects.configurations.compile.resolvedConfiguration.resolvedArtifacts.file.flatten())
+ dependencies.addAll(subprojects.configurations.optional.resolvedConfiguration.resolvedArtifacts.file.flatten())
+ dependencies.addAll(subprojects.configurations.testCompile.resolvedConfiguration.resolvedArtifacts.file.flatten())
+ dependencies.addAll(subprojects.configurations.testRuntime.resolvedConfiguration.resolvedArtifacts.file.flatten())
+
+ def paths = new ArrayList()
+ paths.addAll(subprojects.jar.outputs.files.asPath)
+ paths.addAll(subprojects.sourceSets.test.output.resourcesDir)
+ paths.addAll(subprojects.sourceSets.test.output.classesDirs.files.flatten())
+ paths.addAll(dependencies)
+ println paths.join(File.pathSeparator)
+ }
+}
EOF
) | patch
./gradlew build -x javadoc -x dokka -x asciidoctor -x test -x testNG -x api -x distZip
cat >> build.gradle <<EOF
task createSquishClasspath {
doLast {
def dependencies = new HashSet()
dependencies.addAll(subprojects.configurations.compile.resolvedConfiguration.resolvedArtifacts.file.flatten())
dependencies.addAll(subprojects.configurations.optional.resolvedConfiguration.resolvedArtifacts.file.flatten())
dependencies.addAll(subprojects.configurations.testCompile.resolvedConfiguration.resolvedArtifacts.file.flatten())
dependencies.addAll(subprojects.configurations.testRuntime.resolvedConfiguration.resolvedArtifacts.file.flatten())
def paths = new ArrayList()
paths.addAll(subprojects.jar.outputs.files.asPath)
paths.addAll(subprojects.sourceSets.test.output.resourcesDir)
paths.addAll(subprojects.sourceSets.test.output.classesDirs.files.flatten())
paths.addAll(dependencies)
println paths.join(File.pathSeparator)
}
}
EOF
./gradlew createSquishClasspath -q > classpath.txt
]]></build-command>
<auxclasspath-command>cat classpath.txt</auxclasspath-command>

BIN
.ci/files/public-env.gpg Normal file

Binary file not shown.

Binary file not shown.

View File

@ -1,6 +1,6 @@
#
# The functions here require the following scripts:
# logger.sh
# logger.inc
#
# The functions here require the following environment variables:
# GITHUB_OAUTH_TOKEN

View File

@ -0,0 +1,80 @@
# needs:
# inc/logger
#
# Downloads openjdk from AdoptOpenJDK by accessing the API.
# The API is documented at https://api.adoptopenjdk.net/swagger-ui/
#
function install_openjdk() {
OPENJDK_VERSION=$1
case "$(uname)" in
Linux*)
JDK_OS=linux
COMPONENTS_TO_STRIP=1 # e.g. openjdk-11.0.3+7/bin/java
;;
Darwin*)
JDK_OS=mac
COMPONENTS_TO_STRIP=3 # e.g. jdk-11.0.3+7/Contents/Home/bin/java
;;
CYGWIN*|MINGW*)
JDK_OS=windows
;;
*)
log_error "Unknown OS: $(uname)"
exit 1
;;
esac
DOWNLOAD_URL=$(curl --silent -X GET "https://api.adoptopenjdk.net/v3/assets/feature_releases/${OPENJDK_VERSION}/ga?architecture=x64&heap_size=normal&image_type=jdk&jvm_impl=hotspot&os=${JDK_OS}&page=0&page_size=1&project=jdk&sort_method=DEFAULT&sort_order=DESC&vendor=adoptopenjdk" \
-H "accept: application/json" \
| jq -r ".[0].binaries[0].package.link")
OPENJDK_ARCHIVE=$(basename ${DOWNLOAD_URL})
log_debug "Archive name: ${OPENJDK_ARCHIVE}"
CACHE_DIR=${HOME}/.cache/openjdk
TARGET_DIR=${HOME}/openjdk${OPENJDK_VERSION}
mkdir -p ${CACHE_DIR}
mkdir -p ${TARGET_DIR}
if [ ! -e ${CACHE_DIR}/${OPENJDK_ARCHIVE} ]; then
log_info "Downloading from ${DOWNLOAD_URL} to ${CACHE_DIR}"
curl --location --output ${CACHE_DIR}/${OPENJDK_ARCHIVE} "${DOWNLOAD_URL}"
else
log_info "Skipped download, file ${CACHE_DIR}/${OPENJDK_ARCHIVE} already exists"
fi
log_info "Extracting to ${TARGET_DIR}"
case "$OPENJDK_ARCHIVE" in
*.zip)
7z x ${CACHE_DIR}/${OPENJDK_ARCHIVE} -o${TARGET_DIR}
mv ${TARGET_DIR}/*/* ${TARGET_DIR}/
;;
*.tar.gz)
tar --extract --file ${CACHE_DIR}/${OPENJDK_ARCHIVE} -C ${TARGET_DIR} --strip-components=${COMPONENTS_TO_STRIP}
;;
*)
log_error "Unknown filetype: ${OPENJDK_ARCHIVE}"
exit 1
;;
esac
}
function install_openjdk_setdefault() {
OPENJDK_VERSION=$1
install_openjdk $OPENJDK_VERSION
log_info "Using OpenJDK ${OPENJDK_VERSION} as default"
TARGET_DIR=${HOME}/openjdk${OPENJDK_VERSION}
export JAVA_HOME="${TARGET_DIR}"
export PATH="${TARGET_DIR}/bin:${PATH}"
java -version
}

View File

@ -1,5 +1,3 @@
#!/bin/bash
COL_GREEN="\e[32m"
COL_RED="\e[31m"
COL_RESET="\e[0m"
@ -21,3 +19,12 @@ function log_debug() {
true
#echo -e "[DEBUG ] $*"
}
function log_group_start() {
echo "::group::$*"
log_info $*
}
function log_group_end() {
echo "::endgroup::"
}

View File

@ -0,0 +1,30 @@
#
# needs "inc/logger.inc"
#
#
# On azure, outgoing idle connection are dropped after 4 minutes.
# Usually, you can configure wagon with ttl. But these settings are
# ignored, as soon as dokka-maven-plugin is loaded.
# dokka-maven-plugin tries to load additional dependencies at runtime
# and injects a different http client, which is not configured correctly
# and thus maven fails if it tries to download later in the build process
# further dependencies.
#
# The workaround applied here is: first resolve all dependencies,
# then explicitly get dokka-maven-plugin and then resolve all plugins
# execpt for dokka-maven-plugin, as it does not play well with dependency-plugin.
#
function maven_dependencies_resolve() {
dokka_version=$(./mvnw -q -Dexec.executable="echo" -Dexec.args='${dokka.version}' \
--non-recursive org.codehaus.mojo:exec-maven-plugin:3.0.0:exec)
./mvnw dependency:resolve
./mvnw dependency:get -DgroupId=org.jetbrains.dokka \
-DartifactId=dokka-maven-plugin \
-Dversion=${dokka_version} \
-Dpackaging=jar \
-DremoteRepositories=jcenter::default::https://jcenter.bintray.com/
./mvnw dependency:resolve-plugins -DexcludeGroupIds=org.jetbrains.dokka -Psign
}

View File

@ -1,6 +1,6 @@
#
# The functions here require the following scripts:
# .travis/logger.sh
# inc/logger.inc
#
PMD_CODE_SSH_USER=pmd

111
.ci/inc/pmd-doc.inc Normal file
View File

@ -0,0 +1,111 @@
# Used env vars:
# PMD_CI_JOB_URL
# PMD_CI_PUSH_COMMIT_COMPARE
#
# Executes jekyll and generates the documentation
# The documentation will be generated in the directory "docs/_site".
#
function pmd_doc_generate_jekyll_site() {
pushd docs
echo -e "\n\n"
log_info "Building documentation using jekyll..."
bundle config set --local path vendor/bundle
bundle install
bundle exec jekyll build
popd
}
#
# Creates the pmd-doc.zip archive. It will be placed in "docs/".
#
function pmd_doc_create_archive() {
pushd docs
echo -e "\n\n"
log_info "Creating pmd-doc archive..."
mv _site pmd-doc-${VERSION}
zip -qr pmd-doc-${VERSION}.zip pmd-doc-${VERSION}/
log_success "Successfully created pmd-doc-${VERSION}.zip"
popd
}
#
# Publishes the site to https://pmd.github.io/pmd-${VERSION} and
# https://pmd.github.io/latest/
#
function publish_release_documentation_github() {
echo -e "\n\n"
log_info "Adding the new doc to pmd.github.io..."
# clone pmd.github.io. Note: This uses the ssh key setup earlier
# In order to speed things up, we use a sparse checkout - no need to checkout all directories here
mkdir pmd.github.io
(
cd pmd.github.io
git init
git config user.name "PMD CI (pmd-bot)"
git config user.email "andreas.dangel+pmd-bot@adangel.org"
git config core.sparsecheckout true
git remote add origin git@github.com:pmd/pmd.github.io.git
echo "/latest/" > .git/info/sparse-checkout
echo "/sitemap.xml" >> .git/info/sparse-checkout
git pull --depth=1 origin master
log_info "Copying documentation from ../docs/pmd-doc-${VERSION}/ to pmd-${VERSION}/ ..."
rsync -ah --stats ../docs/pmd-doc-${VERSION}/ pmd-${VERSION}/
git status
echo "Executing: git add pmd-${VERSION}"
git add pmd-${VERSION}
echo "Executing: git commit..."
git commit -q -m "Added pmd-${VERSION}"
log_info "Copying pmd-${VERSION} to latest ..."
git rm -qr latest
cp -a pmd-${VERSION} latest
echo "Executing: git add latest"
git add latest
echo "Executing: git commit..."
git commit -q -m "Copying pmd-${VERSION} to latest"
log_info "Generating sitemap.xml"
../docs/sitemap_generator.sh > sitemap.xml
echo "Executing: git add sitemap.xml"
git add sitemap.xml
echo "Executing: git commit..."
git commit -q -m "Generated sitemap.xml"
echo "Executing: git push origin master"
git push origin master
)
}
#
# Updates github pages of the main repository,
# so that https://pmd.github.io/pmd/ has the latest (snapshot) content
#
function pmd_doc_publish_to_github_pages() {
echo -e "\n\n"
log_info "Pushing the new site to github pages..."
git clone --branch gh-pages --depth 1 git@github.com:pmd/pmd.git pmd-gh-pages
# clear the files first
rm -rf pmd-gh-pages/*
# copy the new site
cp -a docs/pmd-doc-${VERSION}/* pmd-gh-pages/
(
cd pmd-gh-pages
git config user.name "PMD CI (pmd-bot)"
git config user.email "andreas.dangel+pmd-bot@adangel.org"
git add -A
MSG="Update documentation
${PMD_CI_JOB_URL}
${PMD_CI_PUSH_COMMIT_COMPARE}"
git commit -q -m "$MSG"
git push git@github.com:pmd/pmd.git HEAD:gh-pages
log_success "Successfully pushed site to https://pmd.github.io/pmd/"
)
}

View File

@ -1,10 +1,35 @@
#
# The functions here require the following scripts:
# .travis/logger.sh
# .travis/common-functions.sh
# inc/logger.inc
# inc/install-openjdk.inc
#
# The functions here require the following environment variables:
# PMD_SF_USER
# PMD_CI_BRANCH
#
# DANGER_GITHUB_API_TOKEN
# PMD_CI_CHUNK_TOKEN
function regression_tester_setup_ci() {
# note: building spring needs java8. This is setup already by build.sh/build-pr-win-macos.sh
gpg --batch --yes --decrypt --passphrase="GnxdjywUEPveyCD1RLiTd7t8CImnefYr" \
--output .ci/files/public-env .ci/files/public-env.gpg
source .ci/files/public-env >/dev/null 2>&1
rm .ci/files/public-env
if hash "bundler" 2>/dev/null; then
log_debug "Bundler is already installed"
else
log_info "Installing bundler..."
gem install bundler
fi
rm -f .bundle/config
bundle config set --local path vendor/bundle
bundle config set --local with release_notes_preprocessing
bundle install
}
#
# Generate a new baseline and upload it to sourceforge
@ -12,9 +37,8 @@
# Note: this function always succeeds, even if the upload fails.
# In that case, just a error logging is provided.
#
function regression-tester_uploadBaseline() {
change_ruby_version
log_debug "$FUNCNAME branch=${TRAVIS_BRANCH}"
function regression_tester_uploadBaseline() {
log_debug "$FUNCNAME branch=${PMD_CI_BRANCH}"
local targetUrl="https://sourceforge.net/projects/pmd/files/pmd-regression-tester/"
local errexitstate="$(shopt -po errexit)"
@ -33,11 +57,16 @@ function regression-tester_uploadBaseline() {
log_info "Generating and uploading baseline for pmdtester..."
cd ..
bundle config --local gemfile pmd/Gemfile
pmd/.travis/travis_wait "bundle exec pmdtester --mode single --local-git-repo ./pmd --patch-branch ${TRAVIS_BRANCH} --patch-config ./pmd/.travis/all-java.xml --list-of-project ./pmd/.travis/project-list.xml --html-flag"
pmd/.ci/travis_wait "bundle exec pmdtester
--mode single
--local-git-repo ./pmd
--patch-branch ${PMD_CI_BRANCH}
--patch-config ./pmd/.ci/files/all-java.xml
--list-of-project ./pmd/.ci/files/project-list.xml --html-flag"
cd target/reports
BRANCH_FILENAME="${TRAVIS_BRANCH/\//_}"
BRANCH_FILENAME="${PMD_CI_BRANCH/\//_}"
zip -q -r ${BRANCH_FILENAME}-baseline.zip ${BRANCH_FILENAME}/
../../pmd/.travis/travis_wait "rsync -avh ${BRANCH_FILENAME}-baseline.zip ${PMD_SF_USER}@web.sourceforge.net:/home/frs/project/pmd/pmd-regression-tester/"
../../pmd/.ci/travis_wait "rsync -avh ${BRANCH_FILENAME}-baseline.zip ${PMD_SF_USER}@web.sourceforge.net:/home/frs/project/pmd/pmd-regression-tester/"
log_success "Successfully uploaded ${BRANCH_FILENAME}-baseline.zip to ${targetUrl}"
)
# restore errexit state
@ -50,8 +79,7 @@ function regression-tester_uploadBaseline() {
# Note: this function always succeeds, even if the danger fails.
# In that case, just a error logging is provided.
#
function regression-tester_executeDanger() {
change_ruby_version
function regression_tester_executeDanger() {
log_debug "$FUNCNAME"
local errexitstate="$(shopt -po errexit)"
@ -67,13 +95,15 @@ function regression-tester_executeDanger() {
trap danger_failed ERR
# Create a corresponding remote branch locally
if ! git show-ref --verify --quiet refs/heads/${TRAVIS_BRANCH}; then
git fetch --no-tags origin +refs/heads/${TRAVIS_BRANCH}:refs/remotes/origin/${TRAVIS_BRANCH}
git branch ${TRAVIS_BRANCH} origin/${TRAVIS_BRANCH}
log_debug "Created local branch ${TRAVIS_BRANCH}"
if ! git show-ref --verify --quiet refs/heads/${PMD_CI_BRANCH}; then
git fetch --no-tags --depth=1 origin +refs/heads/${PMD_CI_BRANCH}:refs/remotes/origin/${PMD_CI_BRANCH}
git branch ${PMD_CI_BRANCH} origin/${PMD_CI_BRANCH}
log_debug "Created local branch ${PMD_CI_BRANCH}"
fi
# Fetch more commits of the PR for danger
git fetch --no-tags --depth=50 origin +$(git rev-parse HEAD^2):
log_info "Running danger on branch ${TRAVIS_BRANCH}"
log_info "Running danger on branch ${PMD_CI_BRANCH}"
bundle exec danger --verbose
log_success "Executing danger successfully"
)

62
.ci/inc/setup-secrets.inc Normal file
View File

@ -0,0 +1,62 @@
function pmd_ci_setup_private_env() {
log_info "Setting up secrets as environment variables..."
local -r ENV_FILE=.ci/files/private-env
printenv PMD_CI_SECRET_PASSPHRASE | gpg --batch --yes --decrypt \
--passphrase-fd 0 \
--output ${ENV_FILE} ${ENV_FILE}.gpg
source ${ENV_FILE} >/dev/null 2>&1
rm ${ENV_FILE}
}
function pmd_ci_setup_gpg_key() {
log_info "Setting up GPG release signing key..."
local -r GPG_FILE=.ci/files/release-signing-key-D0BF1D737C9A1C22.gpg
mkdir -p "${HOME}/.gpg"
printenv PMD_CI_SECRET_PASSPHRASE | gpg --batch --yes --decrypt \
--passphrase-fd 0 \
--output ${GPG_FILE} ${GPG_FILE}.gpg
gpg --batch --import ${GPG_FILE}
rm ${GPG_FILE}
}
function pmd_ci_setup_ssh() {
log_info "Setting up .ssh/id_rsa..."
local -r SSH_KEY_FILE=.ci/files/id_rsa
printenv PMD_CI_SECRET_PASSPHRASE | gpg --batch --yes --decrypt \
--passphrase-fd 0 \
--output ${SSH_KEY_FILE} ${SSH_KEY_FILE}.gpg
chmod 600 ${SSH_KEY_FILE}
mkdir -p ${HOME}/.ssh
chmod 700 "${HOME}/.ssh"
mv ${SSH_KEY_FILE} "${HOME}/.ssh/id_rsa"
log_info "Setting up .ssh/known_hosts..."
#
# https://sourceforge.net/p/forge/documentation/SSH%20Key%20Fingerprints/
#
# run locally:
# ssh-keyscan web.sourceforge.net | tee -a known_hosts
#
# verify fingerprints:
# ssh-keygen -F web.sourceforge.net -l -f known_hosts
# # Host web.sourceforge.net found: line 1
# web.sourceforge.net RSA SHA256:xB2rnn0NUjZ/E0IXQp4gyPqc7U7gjcw7G26RhkDyk90
# # Host web.sourceforge.net found: line 2
# web.sourceforge.net ECDSA SHA256:QAAxYkf0iI/tc9oGa0xSsVOAzJBZstcO8HqGKfjpxcY
# # Host web.sourceforge.net found: line 3
# web.sourceforge.net ED25519 SHA256:209BDmH3jsRyO9UeGPPgLWPSegKmYCBIya0nR/AWWCY
#
# then add output of `ssh-keygen -F web.sourceforge.net -f known_hosts`
#
echo 'web.sourceforge.net ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2uifHZbNexw6cXbyg1JnzDitL5VhYs0E65Hk/tLAPmcmm5GuiGeUoI/B0eUSNFsbqzwgwrttjnzKMKiGLN5CWVmlN1IXGGAfLYsQwK6wAu7kYFzkqP4jcwc5Jr9UPRpJdYIK733tSEmzab4qc5Oq8izKQKIaxXNe7FgmL15HjSpatFt9w/ot/CHS78FUAr3j3RwekHCm/jhPeqhlMAgC+jUgNJbFt3DlhDaRMa0NYamVzmX8D47rtmBbEDU3ld6AezWBPUR5Lh7ODOwlfVI58NAf/aYNlmvl2TZiauBCTa7OPYSyXJnIPbQXg6YQlDknNCr0K769EjeIlAfY87Z4tw==' >> "$HOME/.ssh/known_hosts"
echo 'web.sourceforge.net ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCwsY6sZT4MTTkHfpRzYjxG7mnXrGL74RCT2cO/NFvRrZVNB5XNwKNn7G5fHbYLdJ6UzpURDRae1eMg92JG0+yo=' >> "$HOME/.ssh/known_hosts"
echo 'web.sourceforge.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOQD35Ujalhh+JJkPvMckDlhu4dS7WH6NsOJ15iGCJLC' >> "$HOME/.ssh/known_hosts"
# add pmd-code.org (ssh-keyscan pmd-code.org)
echo 'pmd-code.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVsIeF6xU0oPb/bMbxG1nU1NDyBpR/cBEPZcm/PuJwdI9B0ydPHA6FysqAnt32fNFznC2SWisnWyY3iNsP3pa8RQJVwmnnv9OboGFlW2/61o3iRyydcpPbgl+ADdt8iU9fmMI7dC04UqgHGBoqOwVNna9VylTjp5709cK2qHnwU450F6YcOEiOKeZfJvV4PmpJCz/JcsUVqft6StviR31jKnqbnkZdP8qNoTbds6WmGKyXkhHdLSZE7X1CFQH28tk8XFqditX93ezeCiThFL7EleDexV/3+2+cs5878sDMUMzHS5KShTjkxzhHaodhtIEdNesinq/hOPbxAGkQ0FbD' >> $HOME/.ssh/known_hosts
}

View File

@ -1,6 +1,6 @@
#
# The functions here require the following scripts:
# .travis/logger.sh
# logger.inc
#
# The functions here require the following environment variables:
# PMD_SF_USER
@ -86,7 +86,7 @@ function sourceforge_uploadFile() {
trap upload_failed ERR
log_info "Uploading $filename to sourceforge..."
.travis/travis_wait "rsync -avh ${filename} ${PMD_SF_USER}@web.sourceforge.net:/home/frs/project/pmd/pmd/${pmdVersion}/"
.ci/travis_wait "rsync -avh ${filename} ${PMD_SF_USER}@web.sourceforge.net:/home/frs/project/pmd/pmd/${pmdVersion}/"
log_success "Successfully uploaded ${filename} to sourceforge: ${targetUrl}"
)
# restore errexit state
@ -158,7 +158,7 @@ function sourceforge_rsyncSnapshotDocumentation() {
trap upload_failed ERR
log_info "Uploading documentation to ${targetUrl}..."
.travis/travis_wait "rsync -ah --stats --delete docs/pmd-doc-${VERSION}/ ${PMD_SF_USER}@web.sourceforge.net:/home/project-web/pmd/htdocs/snapshot/"
.ci/travis_wait "rsync -ah --stats --delete docs/pmd-doc-${VERSION}/ ${PMD_SF_USER}@web.sourceforge.net:/home/project-web/pmd/htdocs/snapshot/"
log_success "Successfully uploaded documentation: ${targetUrl}"
)
# restore errexit state

View File

@ -1,30 +0,0 @@
name: Java CI
on: [push, pull_request]
jobs:
build:
runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental }}
if: "!contains(github.event.head_commit.message, '[skip ci]')"
strategy:
matrix:
os: [ ubuntu-latest , windows-latest , macos-latest ]
java: [ 11 ]
experimental: [ false ]
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
- name: Set up JDK ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}
- name: Build with mvnw
run: |
./mvnw -V clean install

40
.github/workflows/pull-requests.yml vendored Normal file
View File

@ -0,0 +1,40 @@
name: Pull Requests
on: pull_request
jobs:
build:
runs-on: ${{ matrix.os }}
continue-on-error: false
timeout-minutes: 30
strategy:
matrix:
os: [ ubuntu-latest, windows-latest, macos-latest ]
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 2
- uses: actions/cache@v2
with:
path: |
~/.m2/repository
~/.cache
vendor/bundle
key: ${{ runner.os }}-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-
- name: Set up Ruby 2.7
uses: actions/setup-ruby@v1
with:
ruby-version: 2.7
- name: Check Environment
run: .ci/check-environment.sh
shell: bash
- name: Build
run: .ci/build-pr-win-macos.sh
shell: bash
env:
MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=180 -Dmaven.wagon.http.retryHandler.count=3
PMD_CI_BRANCH: ${{ github.event.pull_request.base.ref }}
PMD_CI_PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }}

Some files were not shown because too many files have changed in this diff Show More