<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
<h1class="post-title-main">Testing your rules</h1>
</div>
<divclass="post-content">
<divclass="summary">Learn how to use PMD's simple test framework for unit testing rules.</div>
<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
<p>Good rules have tests. At least a positive test case - a code example, that triggers the rule and reports
a violation - and a negative test case - a code example, that doesn’t trigger the rule - should be created.
Of course, the more tests, the better the rule is verified. If the rule is more complex or defines properties,
with which the behavior can be modified, then these different cases can also be tested.</p>
<p>And if there is a bug fix for a rule, be it a false positive or a false negative case, it should be accompanied
with an additional test case, so that the bug is not accidentally reintroduced later on.</p>
<h2id="how-it-works">How it works</h2>
<p>PMD’s built-in rules are organized in rulesets, where all rules belonging to the same category are placed
in a single ruleset, such as “category/java/bestpractices.xml”.
Each category-ruleset has a single abstract base test class, from which the individual test classes inherit.
We have one test class per rule, which executes all test cases for a single rule. The actual test cases are
stored in separate XML files, for each rule a separate file is used.</p>
<p>All the test classes inherit from <codeclass="language-plaintext highlighter-rouge">net.sourceforge.pmd.testframework.PmdRuleTst</code>,
which provides the seamless integration with JUnit. This base class determines the language, the category name
and the rule name from the concrete test class. It then searches the test code on its own.
E.g. the individual rule test class
<codeclass="language-plaintext highlighter-rouge">net.sourceforge.pmd.lang.java.rule.bestpractices.AbstractClassWithoutAbstractMethodTest</code> tests the
rule with the name “AbstractClassWithoutAbstractMethod”, which is in the category “bestpractices” for the
language “java”.</p>
<p>The test code (see below <ahref="#test-xml-reference">Test XML Reference</a>) describes the test case completely with
the expected behavior like number of expected rule violations, where the violations are expected, and so on.</p>
<p>When you are running the test class in your IDE (e.g. Eclipse or IntelliJ IDEA) you can also select a single
test case and just execute this one.</p>
<h2id="where-to-place-the-test-code">Where to place the test code</h2>
<p>The <codeclass="language-plaintext highlighter-rouge">PmdRuleTst</code> class searches the XML file, that describes the test cases for a certain rule
using the following convention:
The XML file is a test resource, so it is searched in the tree under <codeclass="language-plaintext highlighter-rouge">src/test/resources</code>.</p>
<p>The sub package <codeclass="language-plaintext highlighter-rouge">xml</code> of the test class’s package should contain a file with the same name as the rule’s name
which is under test.</p>
<p>For example, to test the rule “AbstractClassWithoutAbstractMethod”, the fully qualified test class is:</p>
<divclass="alert alert-success"role="alert"><iclass="fa fa-check-square-o"></i><b>Tip:</b> This convention allows you to quickly find the test cases for a given rule:
Just search in the project for a file <codeclass="language-plaintext highlighter-rouge"><RuleName>.xml</code>. Search for a class <codeclass="language-plaintext highlighter-rouge"><Rule Name>Test</code> to find the
unit test class for the given rule.</div>
<divclass="alert alert-info"role="alert"><iclass="fa fa-info-circle"></i><b>Note:</b> If you want to use the test framework with a different package structure,
see <ahref="#using-the-test-framework-externally">Using the test framework externally</a>.</div>
<p>This class inherits from <codeclass="language-plaintext highlighter-rouge">PmdRuleTst</code> and is located in the package “bestpractices”, since the rule
<spanclass="c1">// no additional unit tests</span>
<spanclass="o">}</span>
</code></pre></div></div>
<divclass="alert alert-info"role="alert"><iclass="fa fa-info-circle"></i><b>Note:</b> You can also add additionally standard JUnit test methods annotated with <codeclass="language-plaintext highlighter-rouge">@Test</code> to
<p>Each test case is in an own <codeclass="language-plaintext highlighter-rouge"><test-code></code> element. The first defines 0 expected problems, means this code doesn’t
trigger the rule. The second test case expects 1 problem. Since the rule violations also report the exact AST node,
you can verify the line number, too.</p>
<h2id="test-xml-reference">Test XML Reference</h2>
<p>The root element is <codeclass="language-plaintext highlighter-rouge"><test-data></code>. It can contain one or more <codeclass="language-plaintext highlighter-rouge"><test-code></code> and <codeclass="language-plaintext highlighter-rouge"><code-fragment></code> elements.
Each <codeclass="language-plaintext highlighter-rouge"><test-code></code> element defines a single test case. <codeclass="language-plaintext highlighter-rouge"><code-fragment></code> elements are used to share code snippets
between different test cases.</p>
<divclass="alert alert-info"role="alert"><iclass="fa fa-info-circle"></i><b>Note:</b> The XML schema is available at <ahref="https://github.com/pmd/pmd/blob/master/pmd-test/src/main/resources/rule-tests_1_0_0.xsd">rule-tests.xsd</a>.</div>
<p>The <codeclass="language-plaintext highlighter-rouge"><test-code></code> elements understands three optional attributes:</p>
<ul>
<li>
<p><strong>reinitializeRule</strong>: By default, it’s <codeclass="language-plaintext highlighter-rouge">true</code>, so each test case starts with a fresh instantiated rule. Set it
to <codeclass="language-plaintext highlighter-rouge">false</code> to reproduce cases, where the previous run has influences.</p>
</li>
<li>
<p><strong>regressionTest</strong>: By default, it’s <codeclass="language-plaintext highlighter-rouge">true</code>. Set it to <codeclass="language-plaintext highlighter-rouge">false</code>, to ignore and skip a test case.</p>
</li>
<li>
<p><strong>useAuxClasspath</strong>: By default, it’s <codeclass="language-plaintext highlighter-rouge">true</code>. Set it to <codeclass="language-plaintext highlighter-rouge">false</code> to reproduce issues which only
<p><strong><codeclass="language-plaintext highlighter-rouge"><description></code></strong>: Short description of the test case. This will be the JUnit test name in the report.
If applicable, this description should contain a reference to the bug number, this test case reproduces.</p>
</li>
<li>
<p><strong><codeclass="language-plaintext highlighter-rouge"><rule-property></code></strong>: Optional rule properties, if the rule is configurable. Just add multiple elements, to
set multiple properties for one test case. For an example, see below.</p>
</li>
<li>
<p><strong><codeclass="language-plaintext highlighter-rouge"><expected-problems></code></strong>: The the raw number of expected rule violations, that this rule is expected to report.
For false-positive test cases, this is always “0”. For false-negative test cases, it can be any positive number.</p>
</li>
<li>
<p><strong><codeclass="language-plaintext highlighter-rouge"><expected-linenumbers></code></strong>: Optional element. It’s a comma separated list of line numbers.
If there are rule violations reported, then this allows you to
assert the line numbers. Useful if multiple violations should be detected and to be sure that
false positives and negatives don’t erase each other.</p>
</li>
<li>
<p><strong><codeclass="language-plaintext highlighter-rouge"><expected-messages></code></strong>: Optional element, with <codeclass="language-plaintext highlighter-rouge"><message></code> elements as children.
Can be used to validate the correct error message, e.g. if the error message references a variable name.</p>
</li>
<li>
<p><strong><codeclass="language-plaintext highlighter-rouge"><code></code></strong>: Either the <codeclass="language-plaintext highlighter-rouge"><code></code> element or the <codeclass="language-plaintext highlighter-rouge"><code-ref></code> element is required. It provides the actual code
snippet on which the rule is executed. The code itself is usually wrapped in a “CDATA” section, so that no
further XML escapes (entity references such as &lt;) are necessary.</p>
</li>
<li>
<p><strong><codeclass="language-plaintext highlighter-rouge"><code-ref id=...></code></strong>: Alternative to <codeclass="language-plaintext highlighter-rouge"><code></code>. References a <codeclass="language-plaintext highlighter-rouge"><code-fragment></code> defined earlier in the file.
This allows you to share the same code snippet with several test cases. The attribute <codeclass="language-plaintext highlighter-rouge">id</code> must match the
id of the references code fragment.</p>
</li>
<li>
<p><strong><codeclass="language-plaintext highlighter-rouge"><source-type></code></strong>: Optional element that specifies a specific language version. This can be used
to select a specific parser version for parsing the code snippet. If not given, the default version of
the rule’s language is used. This element can almost always be omitted.</p>
<p>The code fragment has just one required attribute: <strong>id</strong>. This is used to reference it via a <codeclass="language-plaintext highlighter-rouge"><code-ref></code> element
inside a <codeclass="language-plaintext highlighter-rouge"><test-code></code>. Similar like the <codeclass="language-plaintext highlighter-rouge"><code></code> element, the content of <codeclass="language-plaintext highlighter-rouge"><code-fragment></code> is usually wrapped
in a “CDATA” section, so that no further XML escapes (entity references such as &lt;) are necessary.</p>
<h3id="complete-xml-example">Complete XML example</h3>
<spanclass="nt"><description></span>Just a description, will be used as the test name for JUnit in the reports<spanclass="nt"></description></span>
<divclass="alert alert-info"role="alert"><iclass="fa fa-info-circle"></i><b>Note:</b> For better readability, the indentation should be 4 for spaces and no tabs.
Each test-code should be separated by a blank line. CDATA
sections are only required for the code snippets which should be formatted with indentation for
better readability. The description can be written directly without a CDATA section.</div>
<h2id="using-the-test-framework-externally">Using the test framework externally</h2>
<p>It is also possible to use the test framework for custom rules developed outside the PMD source base.
Therefore you just need to reference the dependency <codeclass="language-plaintext highlighter-rouge">net.sourceforge.pmd:pmd-test</code>.</p>
<p>This will then search for a rule named “CustomRule” in the ruleset, that is located in “src/main/resources” under
the path “com/example/pmd/ruleset.xml”.</p>
<p>The test data should be placed in an xml file located in “src/test/resources” under the path
“com/example/pmd/rules/xml/CustomRule.xml”.</p>
<h2id="how-the-test-framework-is-implemented">How the test framework is implemented</h2>
<p>The framework uses a custom JUnit test runner under the hood, among a couple of utility classes:</p>
<ul>
<li>
<p><codeclass="language-plaintext highlighter-rouge">PmdRuleTst</code>: This is the base class for tests in PMD’s code base. It is a subclass of <codeclass="language-plaintext highlighter-rouge">RuleTst</code> and just
contains the logic to determine the test resources based on the test class name.</p>
</li>
<li>
<p><codeclass="language-plaintext highlighter-rouge">SimpleAggregatorTst</code>: This is a more generic base class for the test classes and defines
the custom JUnit test runner. It doesn’t register any test cases on its own.
It itself is a subclass of <codeclass="language-plaintext highlighter-rouge">RuleTst</code>.</p>
</li>
<li>
<p><codeclass="language-plaintext highlighter-rouge">RuleTst</code>: contains the logic to parse the XML files and provide a list of <codeclass="language-plaintext highlighter-rouge">TestDescriptor</code>s. Each test descriptor
describes a single test case. It also contains the logic to execute such a test descriptor and assert the results.</p>
</li>
<li>
<p><codeclass="language-plaintext highlighter-rouge">PMDTestRunner</code>: A custom JUnit test runner, that combines two separate test runners: The custom <codeclass="language-plaintext highlighter-rouge">RuleTestRunner</code>
and the standard <codeclass="language-plaintext highlighter-rouge">JUnit4</code> test runner. This combination allows you to add additional standard unit test methods
annotated with <codeclass="language-plaintext highlighter-rouge">@Test</code> to your test class.</p>
<p><em>Note:</em> Since the test class is executed through two test runners, it is actually instantiated twice. Be aware
of this, if you do any initialization in the constructor. Also, the static hooks <codeclass="language-plaintext highlighter-rouge">@BeforeClass</code> and <codeclass="language-plaintext highlighter-rouge">@AfterClass</code>
will be executed twice.</p>
</li>
<li>
<p><codeclass="language-plaintext highlighter-rouge">RuleTestRunner</code>: This test runner executes the test descriptors with the help of <codeclass="language-plaintext highlighter-rouge">RuleTst</code>.</p>
