diff --git a/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md b/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md index 567db3d6c2..4ee9585405 100644 --- a/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md +++ b/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md @@ -3,7 +3,7 @@ title: Adding PMD support for a new ANTLR grammar based language short_title: Adding a new language with ANTLR tags: [devdocs, extending] summary: "How to add a new language to PMD using ANTLR grammar." -last_updated: October 2021 +last_updated: February 2023 (7.0.0) sidebar: pmd_sidebar permalink: pmd_devdocs_major_adding_new_language_antlr.html folder: pmd/devdocs @@ -21,7 +21,7 @@ folder: pmd/devdocs This is really a big contribution and can't be done with a drive by contribution. It requires dedicated passion and long commitment to implement support for a new language.

-This step by step guide is just a small intro to get the basics started and it's also not necessarily up-to-date +This step-by-step guide is just a small intro to get the basics started and it's also not necessarily up-to-date or complete and you have to be able to fill in the blanks.

Currently the Antlr integration has some basic limitations compared to JavaCC: The output of the @@ -86,7 +86,7 @@ definitely don't come for free. It is much effort and requires perseverance to i * You can add additional methods in your "InnerNode" (e.g. `SwiftInnerNode`) that are available on all nodes. But on most cases you won't need to do anything. -## 4. Generate your parser +## 4. Generate your parser (using ANTLR) * Make sure, you have the property `true` in your `pom.xml` file. * This is just a matter of building the language module. ANTLR is called via ant, and this step is added to the phase `generate-sources`. So you can just call e.g. `./mvnw generate-sources -pl pmd-swift` to @@ -116,23 +116,17 @@ definitely don't come for free. It is much effort and requires perseverance to i implementation that you need to extend to create your own adapter as we do with [`PmdSwiftParser`](https://github.com/pmd/pmd/blob/pmd/7.0.x/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/ast/PmdSwiftParser.java). -## 7. Create a rule violation factory -* This is an optional step. Most like, the default implementation will do what you need. - The default implementation is [`DefaultRuleViolationFactory`](https://github.com/pmd/pmd/blob/pmd/7.0.x/pmd-core/src/main/java/net/sourceforge/pmd/lang/rule/impl/DefaultRuleViolationFactory.java). -* The purpose of a rule violation factory is to create a rule violation instance for your handler (spoiler). - In case you want to provide additional data in your rule violation, you can create a custom one. However, - adding additional date here is discouraged, as you would need a custom renderer to actually use this - additional data. Such extensions are not language agnostic. - -## 8. Create a version handler +## 7. Create a language version handler * Now you need to create your version handler, as we did with [`SwiftHandler`](https://github.com/pmd/pmd/blob/pmd/7.0.x/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/SwiftHandler.java). -* This class is sort of a gateway between PMD and all parsing logic specific to your language. It has 2 purposes: - * `getRuleViolationFactory` method returns an instance of your rule violation factory *(see step #7)*. - By default, this returns the default rule violation factory. - * `getParser` returns an instance of your parser adapter *(see step #6)*. - That's the only method, that needs to be implemented here. +* This class is sort of a gateway between PMD and all parsing logic specific to your language. +* For a minimal implementation, it just needs to return a parser *(see step #6)*. +* It can be used to provide other features for your language like + * violation suppression logic + * violation decorators, to add additional language specific information to the created violations + * metrics + * custom XPath functions -## 9. Create a parser visitor adapter +## 8. Create a base visitor * A parser visitor adapter is not needed anymore with PMD 7. The visitor interface now provides a default implementation. * The visitor for ANTLR based AST is generated along the parser from the ANTLR grammar file. The @@ -142,21 +136,16 @@ definitely don't come for free. It is much effort and requires perseverance to i See [`SwiftVisitorBase`](https://github.com/pmd/pmd/blob/pmd/7.0.x/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/ast/SwiftVisitorBase.java) as an example. -## 10. Create a rule chain visitor -* This step is not needed anymore. For using rule chain, there is no additional adjustment necessary anymore - in the languages. -* This feature has been merged into AbstractRule via the overridable method - {% jdoc !!core::lang.rule.AbstractRule#buildTargetSelector() %}. Individual rules can make use of this optimization - by overriding this method and return an appropriate RuleTargetSelector. - -## 11. Make PMD recognize your language -* Create your own subclass of `net.sourceforge.pmd.lang.BaseLanguageModule`, see Swift as an example: +## 9. Make PMD recognize your language +* Create your own subclass of `net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase`, see Swift as an example: [`SwiftLanguageModule`](https://github.com/pmd/pmd/blob/pmd/7.0.x/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/SwiftLanguageModule.java). -* Add your default version with `addDefaultVersion` in your language module's constructor. -* Add for each additional version of your language a call to `addVersion` as well. -* Create the service registration via the text file `src/main/resources/META-INF/services/net.sourceforge.pmd.lang.Language`. Add your fully qualified class name as a single line into it. +* Add for each version of your language a call to `addVersion` in your language module’s constructor. + Use `addDefaultVersion` for defining the default version. +* You’ll need to refer the language version handler created in step #7. +* Create the service registration via the text file `src/main/resources/META-INF/services/net.sourceforge.pmd.lang.Language`. + Add your fully qualified class name as a single line into it. -## 12. Create an abstract rule class for the language +## 10. Create an abstract rule class for the language * You need to create your own `AbstractRule` in order to interface your language with PMD's generic rule execution. * See [`AbstractSwiftRule`](https://github.com/pmd/pmd/blob/pmd/7.0.x/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/AbstractSwiftRule.java) as an example. @@ -167,7 +156,7 @@ definitely don't come for free. It is much effort and requires perseverance to i via the method `buildVisitor()` for analyzing the AST. The provided visitor only implements the visit methods for specific AST nodes. The other node types use the default behavior and you don't need to care about them. -## 13. Create rules +## 11. Create rules * Creating rules is already pretty well documented in PMD - and it’s no different for a new language, except you may have different AST nodes. * PMD supports 2 types of rules, through visitors or XPath. @@ -179,15 +168,19 @@ definitely don't come for free. It is much effort and requires perseverance to i * To add an XPath rule you can follow our guide [Writing XPath Rules](pmd_userdocs_extending_writing_xpath_rules.html). ## 14. Test the rules -* See UnavailableFunctionRuleTest for example. Each rule has it's own test class. -* You have to create the category rule set for your language *(see pmd-swift/src/main/resources/bestpractices.xml for example)* -* When executing the test class - * this triggers the unit test to read the corresponding XML file with the rule test data - *(see `UnavailableFunctionRule.xml` for example)* - * This test XML file contains sample pieces of code which should trigger a specified number of - violations of this rule. The unit test will execute the rule on this piece of code, and verify - that the number of violations matches. -* To verify the validity of all the created rulesets, create a subclass of `AbstractRuleSetFactoryTest` (*see `RuleSetFactoryTest` in pmd-swift for example)*. +* Testing rules is described in depth in [Testing your rules](pmd_userdocs_extending_testing.html). + * Each rule has its own test class: Create a test class for your rule extending `PmdRuleTst` + *(see UnavailableFunctionTest for example)* + * Create a category rule set for your language *(see pmd-swift/src/main/resources/bestpractices.xml for example)* + * Place the test XML file with the test cases in the correct location + * When executing the test class + * this triggers the unit test to read the corresponding XML file with the rule test data + *(see `UnavailableFunction.xml` for example)* + * This test XML file contains sample pieces of code which should trigger a specified number of + violations of this rule. The unit test will execute the rule on this piece of code, and verify + that the number of violations matches. +* To verify the validity of all the created rulesets, create a subclass of `AbstractRuleSetFactoryTest` + (*see `RuleSetFactoryTest` in pmd-swift for example)*. This will load all rulesets and verify, that all required attributes are provided. *Note:* You'll need to add your ruleset to `categories.properties`, so that it can be found. diff --git a/docs/pages/pmd/devdocs/major_contributions/adding_a_new_javacc_based_language.md b/docs/pages/pmd/devdocs/major_contributions/adding_a_new_javacc_based_language.md index 7e14a56a24..1e4e523b1c 100644 --- a/docs/pages/pmd/devdocs/major_contributions/adding_a_new_javacc_based_language.md +++ b/docs/pages/pmd/devdocs/major_contributions/adding_a_new_javacc_based_language.md @@ -3,7 +3,7 @@ title: Adding PMD support for a new JavaCC grammar based language short_title: Adding a new language with JavaCC tags: [devdocs, extending] summary: "How to add a new language to PMD using JavaCC grammar." -last_updated: October 2021 +last_updated: February 2023 (7.0.0) sidebar: pmd_sidebar permalink: pmd_devdocs_major_adding_new_language_javacc.html folder: pmd/devdocs @@ -16,7 +16,7 @@ folder: pmd/devdocs This is really a big contribution and can't be done with a drive by contribution. It requires dedicated passion and long commitment to implement support for a new language.

-This step by step guide is just a small intro to get the basics started and it's also not necessarily up-to-date +This step-by-step guide is just a small intro to get the basics started and it's also not necessarily up-to-date or complete and you have to be able to fill in the blanks.

After the basic support for a language is there, there are lots of missing features left. Typical features @@ -39,68 +39,75 @@ definitely don't come for free. It is much effort and requires perseverance to i ## 2. Implement an AST parser for your language * Ideally an AST parser should be implemented as a JJT file *(see VmParser.jjt or Java.jjt for example)* -* There is nothing preventing any other parser implementation, as long as you have some way to convert an input stream into an AST tree. Doing it as a JJT simplifies maintenance down the road. +* There is nothing preventing any other parser implementation, as long as you have some way to convert an input + stream into an AST tree. Doing it as a JJT simplifies maintenance down the road. * See this link for reference: [https://javacc.java.net/doc/JJTree.html](https://javacc.java.net/doc/JJTree.html) ## 3. Create AST node classes * For each AST node that your parser can generate, there should be a class * The name of the AST class should be “AST” + “whatever is the name of the node in JJT file”. * For example, if JJT contains a node called “IfStatement”, there should be a class called “ASTIfStatement” -* Each AST class should have two constructors: one that takes an int id; and one that takes an instance of the parser, and an int id -* It’s a good idea to create a parent AST class for all AST classes of the language. This simplifies rule creation later. *(see SimpleNode for Velocity and AbstractJavaNode for Java for example)* +* Each AST class should have one package-private constructor, that takes an `int id`. +* It’s a good idea to create a parent AST class for all AST classes of the language. This simplifies rule + creation later. *(see SimpleNode for Velocity and AbstractJavaNode for Java for example)* * Note: These AST node classes are generated usually once by javacc/jjtree and can then be modified as needed. -## 4. Compile your parser (if using JJT) -* An ant script is being used to compile jjt files into classes. This is in `pmd-/src/main/ant/alljavacc.xml` file. -* Create `alljavacc.xml` file for your language, you can use one from `pmd-java` as an example. -* You would probably want to adjust contents of the `` tag: start with an empty `` and add there ``s for those AST nodes you had to manually rewrite (moving those node classes from autogenerated directory to the regular source tree). +## 4. Generate your parser (using JJT) +* An ant script is being used to compile jjt files into classes. This is in `javacc-wrapper.xml` file in the + top-level pmd sources. +* The ant script is executed via the `maven-antrun-plugin`. Add this plugin to your `pom.xml` file and configure + it the language name. You can use `pmd-java/pom.xml` as an example. +* The ant script is called in the phase `generate-sources` whenever the whole project is built. But you can + call `./mvnw generate-sources` directly for your module if you want your parser to be generated. -## 5. Create a TokenManager -* Create a new class that implements the `TokenManager` interface *(see VmTokenManager or JavaTokenManager for example)* - -## 6. Create a PMD parser “adapter” -* Create a new class that extends AbstractParser +## 5. Create a PMD parser “adapter” +* Create a new class that extends `JjtreeParserAdapter`. +* This is a generic class, and you need to declare the root AST node. * There are two important methods to implement - * `createTokenManager` method should return a new instance of a token manager for your language *(see step #5)* - * `parse` method should return the root node of the AST tree obtained by parsing the Reader source + * `tokenBehavior` method should return a new instance of `TokenDocumentBehavior` constructed with the list + of tokes in your language. The compile step #4 will generate a class `$langTokenKinds` which has + all the available tokens in the field `TOKEN_NAMES`. + * `parseImpl` method should return the root node of the AST tree obtained by parsing the CharStream source * See `VmParser` class as an example -## 7. Create a rule violation factory -* Extend `AbstractRuleViolationFactory` *(see VmRuleViolationFactory for example)* -* The purpose of this class is to create a rule violation instance specific to your language +## 6. Create a language version handler +* Extend `AbstractPmdLanguageVersionHandler` *(see VmHandler for example)* +* This class is sort of a gateway between PMD and all parsing logic specific to your language. +* For a minimal implementation, it just needs to return a parser *(see step #5)*. +* It can be used to provide other features for your language like + * violation suppression logic + * violation decorators, to add additional language specific information to the created violations + * metrics (see below "Optional features") + * custom XPath functions +* See `VmHandler` class as an example -## 8. Create a version handler -* Extend `AbstractLanguageVersionHandler` *(see VmHandler for example)* -* This class is sort of a gateway between PMD and all parsing logic specific to your language. It has 2 purposes: - * `getRuleViolationFactory` method returns an instance of your rule violation factory *(see step #7)* - * `getParser` returns an instance of your parser adapter *(see step #6)* +## 7. Create a base visitor +* A parser visitor adapter is not needed anymore with PMD 7. The visitor interface now provides a default + implementation. +* The visitor for JavaCC based AST is generated along the parser from the grammar file. The + base interface for a visitor is [`AstVisitor`](https://github.com/pmd/pmd/blob/pmd/7.0.x/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/AstVisitor.java). +* The generated visitor class for VM is called `VmVisitor`. +* In order to help use this visitor later on, a base visitor class should be created. + See `VmVisitorBase` as an example. -## 9. Create a parser visitor adapter -* If you use JJT to generate your parser, it should also generate an interface for a parser visitor *(see VmParserVisitor for example)* -* Create a class that implements this auto-generated interface *(see VmParserVisitorAdapter for example)* -* The purpose of this class is to serve as a pass-through `visitor` implementation, which, for all AST types in your language, just executes visit on the base AST type - -## 10. Create a rule chain visitor -* Extend `AbstractRuleChainVisitor` *(see VmRuleChainVisitor for example)* -* This class should `implement` two `important` methods: - * `indexNodes` generates a map of "node type" to "list of nodes of that type". This is used to visit all applicable nodes when a rule is applied. - * `visit` method should evaluate what kind of rule is being applied, and execute appropriate logic. Usually it will just check if the rule is a "parser visitor" kind of rule specific to your language, then execute the visitor. If it’s an XPath rule, then we just need to execute evaluate on that. - -## 11. Make PMD recognize your language -* Create your own subclass of `net.sourceforge.pmd.lang.BaseLanguageModule`. *(see VmLanguageModule or JavaLanguageModule as an example)* -* You’ll need to refer the rule chain visitor created in step #10. +## 8. Make PMD recognize your language +* Create your own subclass of `net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase`. *(see VmLanguageModule or + JavaLanguageModule as an example)* * Add for each version of your language a call to `addVersion` in your language module’s constructor. -* Create the service registration via the text file `src/main/resources/META-INF/services/net.sourceforge.pmd.lang.Language`. Add your fully qualified class name as a single line into it. + Use `addDefaultVersion` for defining the default version. +* You’ll need to refer the language version handler created in step #6. +* Create the service registration via the text file `src/main/resources/META-INF/services/net.sourceforge.pmd.lang.Language`. + Add your fully qualified class name as a single line into it. -## 12. Add AST regression tests +## 9. Add AST regression tests For languages, that use an external library for parsing, the AST can easily change when upgrading the library. -Also for languages, where we have the grammar under our control, it useful to have such tests. +Also for languages, where we have the grammar under our control, it is useful to have such tests. The tests parse one or more source files and generate a textual representation of the AST. This text is compared against a previously recorded version. If there are differences, the test fails. -This helps to detect anything in the AST structure, that changed, maybe unexpectedly. +This helps to detect anything in the AST structure that changed, maybe unexpectedly. * Create a test class in the package `net.sourceforge.pmd.lang.$lang.ast` with the name `$langTreeDumpTest`. * This test class must extend `net.sourceforge.pmd.lang.ast.test.BaseTreeDumpTest`. Note: This class @@ -116,7 +123,7 @@ This helps to detect anything in the AST structure, that changed, maybe unexpect Replace "$lang" and "$extension" accordingly. * Implement the method `getParser()`. It must return a subclass of `net.sourceforge.pmd.lang.ast.test.BaseParsingHelper`. See - `net.sourceforge.pmd.lang.ecmascript.ast.JsParsingHelper` for a example. + `net.sourceforge.pmd.lang.ecmascript.ast.JsParsingHelper` for an example. With this parser helper you can also specify, where the test files are searched, by using the method `withResourceContext(Class, String)`. * Add one or more test methods. Each test method parses one file and compares the result. The base @@ -139,24 +146,35 @@ The Scala module also has a test, written in Kotlin instead of Java: `net.sourceforge.pmd.lang.scala.ast.ScalaParserTests`. -## 13. Create an abstract rule class for the language +## 10. Create an abstract rule class for the language * Extend `AbstractRule` and implement the parser visitor interface for your language *(see AbstractVmRule for example)* -* All other rules for your language should extend this class. The purpose of this class is to implement visit methods for all AST types to simply delegate to default behavior. This is useful because most rules care only about specific AST nodes, but PMD needs to know what to do with each node - so this just lets you use default behavior for nodes you don’t care about. +* All other rules for your language should extend this class. The purpose of this class is to implement visit + methods for all AST types to simply delegate to default behavior. This is useful because most rules care only + about specific AST nodes, but PMD needs to know what to do with each node - so this just lets you use default + behavior for nodes you don’t care about. -## 14. Create rules -* Rules are created by extending the abstract rule class created in step 13 *(see `EmptyForeachStmtRule` for example)* -* Creating rules is already pretty well documented in PMD - and it’s no different for a new language, except you may have different AST nodes. +## 11. Create rules +* Rules are created by extending the abstract rule class created in step 9 *(see `EmptyForeachStmtRule` for example)* +* Creating rules is already pretty well documented in PMD - and it’s no different for a new language, + except you may have different AST nodes. -## 15. Test the rules -* See BasicRulesTest for example -* You have to create a rule set for your language *(see vm/basic.xml for example)* -* For each rule in this set you want to test, call `addRule` method in setUp of the unit test - * This triggers the unit test to read the corresponding XML file with rule test data *(see `EmptyForeachStmtRule.xml` for example)* - * This test XML file contains sample pieces of code which should trigger a specified number of violations of this rule. The unit test will execute the rule on this piece of code, and verify that the number of violations matches -* To verify the validity of the created ruleset, create a subclass of `AbstractRuleSetFactoryTest` (*see `RuleSetFactoryTest` in pmd-vm for example)*. +## 12. Test the rules +* Testing rules is described in depth in [Testing your rules](pmd_userdocs_extending_testing.html). + * Each rule has its own test class: Create a test class for your rule extending `PmdRuleTst` + *(see AvoidReassigningParametersTest in pmd-vm for example)* + * Create a category rule set for your language *(see category/vm/bestpractices.xml for example)* + * Place the test XML file with the test cases in the correct location + * When executing the test class + * this triggers the unit test to read the corresponding XML file with the rule test data + *(see `AvoidReassigningParameters.xml` for example)* + * This test XML file contains sample pieces of code which should trigger a specified number of + violations of this rule. The unit test will execute the rule on this piece of code, and verify + that the number of violations matches. +* To verify the validity of the created ruleset, create a subclass of `AbstractRuleSetFactoryTest` + (*see `RuleSetFactoryTest` in pmd-vm for example)*. This will load all rulesets and verify, that all required attributes are provided. - *Note:* You'll need to add your ruleset to `rulesets.properties`, so that it can be found. + *Note:* You'll need to add your category ruleset to `categories.properties`, so that it can be found. ## Debugging with Rule Designer