diff --git a/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md b/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md index 8f05cf0ac2..249da18808 100644 --- a/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md +++ b/docs/pages/pmd/devdocs/major_contributions/adding_a_new_antlr_based_language.md @@ -119,15 +119,15 @@ definitely don't come for free. It is much effort and requires perseverance to i ### 5. Create a TokenManager * This is needed to support CPD (copy paste detection) -* We provide a default implementation using [`AntlrTokenManager`](https://github.com/pmd/pmd/blob/master/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrTokenizer.java). -* You must create your own "AntlrTokenizer" such as we do with - [`SwiftTokenizer`](https://github.com/pmd/pmd/blob/master/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftTokenizer.java). +* We provide a default implementation using [`AntlrTokenManager`](https://github.com/pmd/pmd/blob/master/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrTokenManager.java). +* You must create your own "AntlrCpdLexer" such as we do with + [`SwiftCpdLexer`](https://github.com/pmd/pmd/blob/master/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftCpdLexer.java). * If you wish to filter specific tokens (e.g. comments to support CPD suppression via "CPD-OFF" and "CPD-ON") you can create your own implementation of [`AntlrTokenFilter`](https://github.com/pmd/pmd/blob/master/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrTokenFilter.java). You'll need to override then the protected method `getTokenFilter(AntlrTokenManager)` - and return your custom filter. See the tokenizer for C# as an exmaple: - [`CsTokenizer`](https://github.com/pmd/pmd/blob/master/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsTokenizer.java). + and return your custom filter. See the CpdLexer for C# as an exmaple: + [`CsCpdLexer`](https://github.com/pmd/pmd/blob/master/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsCpdLexer.java). If you don't need a custom token filter, you don't need to override the method. It returns the default `AntlrTokenFilter` which doesn't filter anything. diff --git a/docs/pages/pmd/devdocs/major_contributions/adding_new_cpd_language.md b/docs/pages/pmd/devdocs/major_contributions/adding_new_cpd_language.md index 665aa7ce18..9261c50e18 100644 --- a/docs/pages/pmd/devdocs/major_contributions/adding_new_cpd_language.md +++ b/docs/pages/pmd/devdocs/major_contributions/adding_new_cpd_language.md @@ -11,7 +11,7 @@ author: Matías Fraga, Clément Fournier ## Adding support for a CPD language CPD works generically on the tokens produced by a {% jdoc core::cpd.Tokenizer %}. -To add support for a new language, the crucial piece is writing a tokenizer that +To add support for a new language, the crucial piece is writing a CpdLexer that splits the source file into the tokens specific to your language. Thankfully you can use a stock [Antlr grammar](https://github.com/antlr/grammars-v4) or JavaCC grammar to generate a lexer for you. If you cannot use a lexer generator, for @@ -31,12 +31,12 @@ Use the following guide to set up a new language module that supports CPD. the lexer from the grammar. To do so, edit `pom.xml` (eg like [the Golang module](https://github.com/pmd/pmd/tree/master/pmd-go/pom.xml)). Once that is done, `mvn generate-sources` should generate the lexer sources for you. - You can now implement a tokenizer, for instance by extending {% jdoc core::cpd.impl.AntlrTokenizer %}. The following reproduces the Go implementation: + You can now implement a CpdLexer, for instance by extending {% jdoc core::cpd.impl.AntlrCpdLexer %}. The following reproduces the Go implementation: ```java // mind the package convention if you are going to make a PR package net.sourceforge.pmd.lang.go.cpd; - public class GoTokenizer extends AntlrTokenizer { + public class GoCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { @@ -64,9 +64,9 @@ If your language only supports CPD, then you can subclass {% jdoc core::lang.imp } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - // This method should return an instance of the tokenizer you created. - return new GoTokenizer(); + public Tokenizer createCpdLexer(LanguagePropertyBundle bundle) { + // This method should return an instance of the CpdLexer you created. + return new GoCpdLexer(); } } ``` @@ -77,7 +77,7 @@ If your language only supports CPD, then you can subclass {% jdoc core::lang.imp 4. Update the test that asserts the list of supported languages by updating the `SUPPORTED_LANGUAGES` constant in [BinaryDistributionIT](https://github.com/pmd/pmd/blob/master/pmd-dist/src/test/java/net/sourceforge/pmd/it/BinaryDistributionIT.java). -5. Add some tests for your tokenizer by following the [section below](#testing-your-implementation). +5. Add some tests for your CpdLexer by following the [section below](#testing-your-implementation). 6. Finishing up your new language module by adding a page in the documentation. Create a new markdown file `.md` in `docs/pages/pmd/languages/`. This file should have the following frontmatter: @@ -100,10 +100,10 @@ If your language only supports CPD, then you can subclass {% jdoc core::lang.imp {% endraw %} ``` -### Declaring tokenizer options +### Declaring CpdLexer options -To make the tokenizer configurable, first define some property descriptors using -{% jdoc core::properties.PropertyFactory %}. Look at {% jdoc core::cpd.Tokenizer %} +To make the CpdLexer configurable, first define some property descriptors using +{% jdoc core::properties.PropertyFactory %}. Look at {% jdoc core::cpd.CpdLexer %} for some predefined ones which you can reuse (prefer reusing property descriptors if you can). You need to override {% jdoc core::Language#newPropertyBundle() %} and call `definePropertyDescriptor` to register the descriptors. @@ -112,13 +112,13 @@ of {% jdoc core::cpd.CpdCapableLanguage#createCpdTokenizer(core::lang.LanguagePr To implement simple token filtering, you can use {% jdoc core::cpd.impl.BaseTokenFilter %} as a base class, or another base class in {% jdoc_package core::cpd.impl %}. -Take a look at the [Kotlin token filter implementation](https://github.com/pmd/pmd/blob/master/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinTokenizer.java), or the [Java one](https://github.com/pmd/pmd/blob/master/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaTokenizer.java). +Take a look at the [Kotlin token filter implementation](https://github.com/pmd/pmd/blob/master/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinCpdLexer.java), or the [Java one](https://github.com/pmd/pmd/blob/master/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaCpdLexer.java). ### Testing your implementation Add a Maven dependency on `pmd-lang-test` (scope `test`) in your `pom.xml`. -This contains utilities to test your tokenizer. +This contains utilities to test your CpdLexer. Create a test class extending from {% jdoc lang-test::cpd.test.CpdTextComparisonTest %}. To add tests, you need to write regular JUnit `@Test`-annotated methods, and diff --git a/docs/pages/pmd/languages/language_properties.md b/docs/pages/pmd/languages/language_properties.md index 91f396489f..9520ae1fd0 100644 --- a/docs/pages/pmd/languages/language_properties.md +++ b/docs/pages/pmd/languages/language_properties.md @@ -24,7 +24,7 @@ PropertyName is the name of the property converted to SCREAMING_SNAKE_CASE, that As a convention, properties whose name start with an *x* are internal and may be removed or changed without notice. -Properties whose name start with **CPD** are used to configure CPD tokenizer options. +Properties whose name start with **CPD** are used to configure CPD CpdLexer options. Programmatically, the language properties can be set on `PMDConfiguration` (or `CPDConfiguration`) before using the {%jdoc core::PmdAnalyzer %} (or {%jdoc core::cpd.CpdAnalyzer %}) instance diff --git a/docs/pages/release_notes.md b/docs/pages/release_notes.md index ce1e6af2f3..c1d89a3144 100644 --- a/docs/pages/release_notes.md +++ b/docs/pages/release_notes.md @@ -166,6 +166,7 @@ The rules have been moved into categories with PMD 6. * [#4723](https://github.com/pmd/pmd/issues/4723): \[cli] Launch fails for "bash pmd" * core * [#1027](https://github.com/pmd/pmd/issues/1027): \[core] Apply the new PropertyDescriptor<Pattern> type where applicable + * [#4065](https://github.com/pmd/pmd/issues/4065): \[core] Rename TokenMgrError to LexException, Tokenizer to CpdLexer * [#4313](https://github.com/pmd/pmd/issues/4313): \[core] Remove support for <lang>-<ruleset> hyphen notation for ruleset references * [#4314](https://github.com/pmd/pmd/issues/4314): \[core] Remove ruleset compatibility filter (RuleSetFactoryCompatibility) and CLI option `--no-ruleset-compatibility` * [#4378](https://github.com/pmd/pmd/issues/4378): \[core] Ruleset loading processes commented rules @@ -271,6 +272,15 @@ The following previously deprecated classes have been removed: * The node `ASTClassOrInterfaceBody` has been renamed to {% jdoc java::lang.ast.ASTClassBody %}. XPath rules need to be adjusted. +**Renamed classes and methods** + +* pmd-core + * {%jdoc_old core::lang.ast.TokenMgrError %} has been renamed to {% jdoc core::lang.ast.LexException %} + * {%jdoc_old core::cpd.Tokenizer %} has been renamed to {% jdoc core::cpd.CpdLexer %}. Along with this rename, + all the implementations have been renamed as well (`Tokenizer` -> `CpdLexer`), e.g. "CppCpdLexer", "JavaCpdLexer". + This affects all language modules. + * {%jdoc_old core::cpd.AnyTokenizer %} has been renamed to {% jdoc core::cpd.AnyCpdLexer %}. + **Removed functionality** * The CLI parameter `--no-ruleset-compatibility` has been removed. It was only used to allow loading @@ -684,6 +694,7 @@ See also [Detailed Release Notes for PMD 7]({{ baseurl }}pmd_release_notes_pmd7. * [#3919](https://github.com/pmd/pmd/issues/3919): \[core] Merge CPD and PMD language * [#3922](https://github.com/pmd/pmd/pull/3922): \[core] Better error reporting for the ruleset parser * [#4035](https://github.com/pmd/pmd/issues/4035): \[core] ConcurrentModificationException in DefaultRuleViolationFactory + * [#4065](https://github.com/pmd/pmd/issues/4065): \[core] Rename TokenMgrError to LexException, Tokenizer to CpdLexer * [#4120](https://github.com/pmd/pmd/issues/4120): \[core] Explicitly name all language versions * [#4204](https://github.com/pmd/pmd/issues/4204): \[core] Provide a CpdAnalysis class as a programmatic entry point into CPD * [#4301](https://github.com/pmd/pmd/issues/4301): \[core] Remove deprecated property concrete classes diff --git a/javacc-wrapper.xml b/javacc-wrapper.xml index 601b91bb66..ece4307099 100644 --- a/javacc-wrapper.xml +++ b/javacc-wrapper.xml @@ -280,6 +280,13 @@ + + + + + + + diff --git a/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/ApexLanguageModule.java b/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/ApexLanguageModule.java index acb0bb19c0..f3e8f4c64c 100644 --- a/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/ApexLanguageModule.java +++ b/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/ApexLanguageModule.java @@ -5,12 +5,12 @@ package net.sourceforge.pmd.lang.apex; import net.sourceforge.pmd.cpd.CpdCapableLanguage; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguageModuleBase; import net.sourceforge.pmd.lang.LanguageProcessor; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.PmdCapableLanguage; -import net.sourceforge.pmd.lang.apex.cpd.ApexTokenizer; +import net.sourceforge.pmd.lang.apex.cpd.ApexCpdLexer; public class ApexLanguageModule extends LanguageModuleBase implements PmdCapableLanguage, CpdCapableLanguage { private static final String ID = "apex"; @@ -47,7 +47,7 @@ public class ApexLanguageModule extends LanguageModuleBase implements PmdCapable } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new ApexTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new ApexCpdLexer(); } } diff --git a/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/cpd/ApexTokenizer.java b/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/cpd/ApexCpdLexer.java similarity index 93% rename from pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/cpd/ApexTokenizer.java rename to pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/cpd/ApexCpdLexer.java index 22319de686..688dce3d4e 100644 --- a/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/cpd/ApexTokenizer.java +++ b/pmd-apex/src/main/java/net/sourceforge/pmd/lang/apex/cpd/ApexCpdLexer.java @@ -12,16 +12,16 @@ import org.antlr.runtime.ANTLRStringStream; import org.antlr.runtime.Lexer; import org.antlr.runtime.Token; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.cpd.TokenFactory; -import net.sourceforge.pmd.cpd.Tokenizer; import net.sourceforge.pmd.lang.apex.ApexJorjeLogging; import net.sourceforge.pmd.lang.document.TextDocument; import apex.jorje.parser.impl.ApexLexer; -public class ApexTokenizer implements Tokenizer { +public class ApexCpdLexer implements CpdLexer { - public ApexTokenizer() { + public ApexCpdLexer() { ApexJorjeLogging.disableLogging(); } diff --git a/pmd-apex/src/test/java/net/sourceforge/pmd/lang/apex/cpd/ApexTokenizerTest.java b/pmd-apex/src/test/java/net/sourceforge/pmd/lang/apex/cpd/ApexCpdLexerTest.java similarity index 88% rename from pmd-apex/src/test/java/net/sourceforge/pmd/lang/apex/cpd/ApexTokenizerTest.java rename to pmd-apex/src/test/java/net/sourceforge/pmd/lang/apex/cpd/ApexCpdLexerTest.java index eee012f27f..d1f6a32143 100644 --- a/pmd-apex/src/test/java/net/sourceforge/pmd/lang/apex/cpd/ApexTokenizerTest.java +++ b/pmd-apex/src/test/java/net/sourceforge/pmd/lang/apex/cpd/ApexCpdLexerTest.java @@ -9,9 +9,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.apex.ApexLanguageModule; -class ApexTokenizerTest extends CpdTextComparisonTest { +class ApexCpdLexerTest extends CpdTextComparisonTest { - ApexTokenizerTest() { + ApexCpdLexerTest() { super(ApexLanguageModule.getInstance(), ".cls"); } diff --git a/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/CocoLanguageModule.java b/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/CocoLanguageModule.java index 6f293579f3..711a4a730b 100644 --- a/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/CocoLanguageModule.java +++ b/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/CocoLanguageModule.java @@ -4,10 +4,10 @@ package net.sourceforge.pmd.lang.coco; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.coco.cpd.CocoTokenizer; +import net.sourceforge.pmd.lang.coco.cpd.CocoCpdLexer; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; /** @@ -25,7 +25,7 @@ public class CocoLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new CocoTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new CocoCpdLexer(); } } diff --git a/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/cpd/CocoTokenizer.java b/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/cpd/CocoCpdLexer.java similarity index 80% rename from pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/cpd/CocoTokenizer.java rename to pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/cpd/CocoCpdLexer.java index 2467db44a0..cda9ab4b7f 100644 --- a/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/cpd/CocoTokenizer.java +++ b/pmd-coco/src/main/java/net/sourceforge/pmd/lang/coco/cpd/CocoCpdLexer.java @@ -7,13 +7,13 @@ package net.sourceforge.pmd.lang.coco.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.coco.ast.CocoLexer; /** * The Coco Tokenizer. */ -public class CocoTokenizer extends AntlrTokenizer { +public class CocoCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { diff --git a/pmd-coco/src/test/java/net/sourceforge/pmd/lang/coco/cpd/CocoTokenizerTest.java b/pmd-coco/src/test/java/net/sourceforge/pmd/lang/coco/cpd/CocoCpdLexerTest.java similarity index 85% rename from pmd-coco/src/test/java/net/sourceforge/pmd/lang/coco/cpd/CocoTokenizerTest.java rename to pmd-coco/src/test/java/net/sourceforge/pmd/lang/coco/cpd/CocoCpdLexerTest.java index bdb3735151..3ef139cc41 100644 --- a/pmd-coco/src/test/java/net/sourceforge/pmd/lang/coco/cpd/CocoTokenizerTest.java +++ b/pmd-coco/src/test/java/net/sourceforge/pmd/lang/coco/cpd/CocoCpdLexerTest.java @@ -9,8 +9,8 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.coco.CocoLanguageModule; -class CocoTokenizerTest extends CpdTextComparisonTest { - CocoTokenizerTest() { +class CocoCpdLexerTest extends CpdTextComparisonTest { + CocoCpdLexerTest() { super(CocoLanguageModule.getInstance(), ".coco"); } diff --git a/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/EcmascriptTokenizer.java b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/EcmascriptTokenizer.java index f75312c469..a63b348db2 100644 --- a/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/EcmascriptTokenizer.java +++ b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/EcmascriptTokenizer.java @@ -4,5 +4,5 @@ package net.sourceforge.pmd.cpd; -public class EcmascriptTokenizer extends net.sourceforge.pmd.lang.ecmascript.cpd.EcmascriptTokenizer { +public class EcmascriptTokenizer extends net.sourceforge.pmd.lang.ecmascript.cpd.EcmascriptCpdLexer implements Tokenizer { } diff --git a/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JSPTokenizer.java b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JSPTokenizer.java index 76b768afd8..2e6f8d112b 100644 --- a/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JSPTokenizer.java +++ b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JSPTokenizer.java @@ -4,5 +4,5 @@ package net.sourceforge.pmd.cpd; -public class JSPTokenizer extends net.sourceforge.pmd.lang.jsp.cpd.JSPTokenizer { +public class JSPTokenizer extends net.sourceforge.pmd.lang.jsp.cpd.JspCpdLexer implements Tokenizer { } diff --git a/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JavaTokenizer.java b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JavaTokenizer.java index 113663cffa..1f71aa8b6c 100644 --- a/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JavaTokenizer.java +++ b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/JavaTokenizer.java @@ -9,7 +9,7 @@ import java.util.Properties; import net.sourceforge.pmd.lang.java.JavaLanguageModule; import net.sourceforge.pmd.lang.java.internal.JavaLanguageProperties; -public class JavaTokenizer extends net.sourceforge.pmd.lang.java.cpd.JavaTokenizer { +public class JavaTokenizer extends net.sourceforge.pmd.lang.java.cpd.JavaCpdLexer implements Tokenizer { public JavaTokenizer(Properties properties) { super(convertLanguageProperties(properties)); } diff --git a/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/Tokenizer.java b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/Tokenizer.java new file mode 100644 index 0000000000..ece7fc3618 --- /dev/null +++ b/pmd-compat6/src/main/java/net/sourceforge/pmd/cpd/Tokenizer.java @@ -0,0 +1,8 @@ +/** + * BSD-style license; for more info see http://pmd.sourceforge.net/license.html + */ + +package net.sourceforge.pmd.cpd; + +public interface Tokenizer extends CpdLexer { +} diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/AnyTokenizer.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/AnyCpdLexer.java similarity index 91% rename from pmd-core/src/main/java/net/sourceforge/pmd/cpd/AnyTokenizer.java rename to pmd-core/src/main/java/net/sourceforge/pmd/cpd/AnyCpdLexer.java index 0d3de7a45d..db0988c0ff 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/AnyTokenizer.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/AnyCpdLexer.java @@ -20,9 +20,10 @@ import net.sourceforge.pmd.util.StringUtil; * Higher-quality lexers should be implemented with a lexer generator. * *

In PMD 7, this replaces AbstractTokenizer, which provided nearly - * no more functionality. + * no more functionality.

+ *

Note: This class has been called AnyTokenizer in PMD 6.

*/ -public class AnyTokenizer implements Tokenizer { +public class AnyCpdLexer implements CpdLexer { private static final Pattern DEFAULT_PATTERN = makePattern(""); @@ -40,15 +41,15 @@ public class AnyTokenizer implements Tokenizer { private final Pattern pattern; private final String commentStart; - public AnyTokenizer() { + public AnyCpdLexer() { this(DEFAULT_PATTERN, ""); } - public AnyTokenizer(String eolCommentStart) { + public AnyCpdLexer(String eolCommentStart) { this(makePattern(eolCommentStart), eolCommentStart); } - private AnyTokenizer(Pattern pattern, String commentStart) { + private AnyCpdLexer(Pattern pattern, String commentStart) { this.pattern = pattern; this.commentStart = commentStart; } diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdAnalysis.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdAnalysis.java index fc1e131825..7668a29a61 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdAnalysis.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdAnalysis.java @@ -23,7 +23,7 @@ import net.sourceforge.pmd.internal.util.IOUtil; import net.sourceforge.pmd.lang.Language; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.ast.FileAnalysisException; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; import net.sourceforge.pmd.lang.document.FileCollector; import net.sourceforge.pmd.lang.document.FileId; import net.sourceforge.pmd.lang.document.TextDocument; @@ -137,10 +137,10 @@ public final class CpdAnalysis implements AutoCloseable { this.listener = cpdListener; } - private int doTokenize(TextDocument document, Tokenizer tokenizer, Tokens tokens) throws IOException, TokenMgrError { + private int doTokenize(TextDocument document, CpdLexer cpdLexer, Tokens tokens) throws IOException, LexException { LOGGER.trace("Tokenizing {}", document.getFileId().getAbsolutePath()); int lastTokenSize = tokens.size(); - Tokenizer.tokenize(tokenizer, document, tokens); + CpdLexer.tokenize(cpdLexer, document, tokens); return tokens.size() - lastTokenSize - 1; /* EOF */ } @@ -152,12 +152,12 @@ public final class CpdAnalysis implements AutoCloseable { public void performAnalysis(Consumer consumer) { try (SourceManager sourceManager = new SourceManager(files.getCollectedFiles())) { - Map tokenizers = + Map tokenizers = sourceManager.getTextFiles().stream() .map(it -> it.getLanguageVersion().getLanguage()) .distinct() .filter(it -> it instanceof CpdCapableLanguage) - .collect(Collectors.toMap(lang -> lang, lang -> ((CpdCapableLanguage) lang).createCpdTokenizer(configuration.getLanguageProperties(lang)))); + .collect(Collectors.toMap(lang -> lang, lang -> ((CpdCapableLanguage) lang).createCpdLexer(configuration.getLanguageProperties(lang)))); Map numberOfTokensPerFile = new HashMap<>(); @@ -170,7 +170,7 @@ public final class CpdAnalysis implements AutoCloseable { int newTokens = doTokenize(textDocument, tokenizers.get(textFile.getLanguageVersion().getLanguage()), tokens); numberOfTokensPerFile.put(textDocument.getFileId(), newTokens); listener.addedFile(1); - } catch (TokenMgrError | IOException e) { + } catch (LexException | IOException e) { if (e instanceof FileAnalysisException) { // NOPMD ((FileAnalysisException) e).setFileId(textFile.getFileId()); } diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdCapableLanguage.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdCapableLanguage.java index 34f854268d..6484713f14 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdCapableLanguage.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdCapableLanguage.java @@ -16,7 +16,7 @@ public interface CpdCapableLanguage extends Language { /** - * Create a new {@link Tokenizer} for this language, given + * Create a new {@link CpdLexer} for this language, given * a property bundle with configuration. The bundle was created by * this instance using {@link #newPropertyBundle()}. It can be assumed * that the bundle will never be mutated anymore, and this method @@ -26,7 +26,7 @@ public interface CpdCapableLanguage extends Language { * * @return A new language processor */ - default Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new AnyTokenizer(); + default CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new AnyCpdLexer(); } } diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/Tokenizer.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdLexer.java similarity index 73% rename from pmd-core/src/main/java/net/sourceforge/pmd/cpd/Tokenizer.java rename to pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdLexer.java index e1bd1e84e2..554c12b4f4 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/Tokenizer.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/CpdLexer.java @@ -10,8 +10,10 @@ import net.sourceforge.pmd.lang.document.TextDocument; /** * Tokenizes a source file into tokens consumable by CPD. + * + *

Note: This interface has been called Tokenizer in PMD 6.

*/ -public interface Tokenizer { +public interface CpdLexer { /** * Tokenize the source code and record tokens using the provided token factory. @@ -22,9 +24,9 @@ public interface Tokenizer { * Wraps a call to {@link #tokenize(TextDocument, TokenFactory)} to properly * create and close the token factory. */ - static void tokenize(Tokenizer tokenizer, TextDocument textDocument, Tokens tokens) throws IOException { + static void tokenize(CpdLexer cpdLexer, TextDocument textDocument, Tokens tokens) throws IOException { try (TokenFactory tf = Tokens.factoryForFile(textDocument, tokens)) { - tokenizer.tokenize(textDocument, tf); + cpdLexer.tokenize(textDocument, tf); } } } diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/GUI.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/GUI.java index 033f40822c..54e98722b2 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/GUI.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/GUI.java @@ -142,8 +142,8 @@ public class GUI implements CPDListener { .extensions(extension) .name("By extension...")) { @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new AnyTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new AnyCpdLexer(); } }; } diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/TokenFactory.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/TokenFactory.java index 4a7e5a3e49..cfc2eba638 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/TokenFactory.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/TokenFactory.java @@ -7,12 +7,12 @@ package net.sourceforge.pmd.cpd; import org.checkerframework.checker.nullness.qual.NonNull; import org.checkerframework.checker.nullness.qual.Nullable; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; import net.sourceforge.pmd.lang.document.FileLocation; import net.sourceforge.pmd.lang.document.TextDocument; /** - * Proxy to record tokens from within {@link Tokenizer#tokenize(TextDocument, TokenFactory)}. + * Proxy to record tokens from within {@link CpdLexer#tokenize(TextDocument, TokenFactory)}. */ public interface TokenFactory extends AutoCloseable { @@ -43,7 +43,7 @@ public interface TokenFactory extends AutoCloseable { recordToken(image, location.getStartLine(), location.getStartColumn(), location.getEndLine(), location.getEndColumn()); } - TokenMgrError makeLexException(int line, int column, String message, @Nullable Throwable cause); + LexException makeLexException(int line, int column, String message, @Nullable Throwable cause); /** * Sets the image of an existing token entry. @@ -57,7 +57,7 @@ public interface TokenFactory extends AutoCloseable { /** * This adds the EOF token, it must be called when - * {@link Tokenizer#tokenize(TextDocument, TokenFactory)} is done. + * {@link CpdLexer#tokenize(TextDocument, TokenFactory)} is done. */ @Override void close(); diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/Tokens.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/Tokens.java index 3f314121e5..7bb8aab527 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/Tokens.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/Tokens.java @@ -14,7 +14,7 @@ import org.checkerframework.checker.nullness.qual.NonNull; import org.checkerframework.checker.nullness.qual.Nullable; import net.sourceforge.pmd.annotation.InternalApi; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; import net.sourceforge.pmd.lang.document.FileId; import net.sourceforge.pmd.lang.document.TextDocument; @@ -93,7 +93,7 @@ public class Tokens { /** * Creates a token factory to process the given file with - * {@link Tokenizer#tokenize(TextDocument, TokenFactory)}. + * {@link CpdLexer#tokenize(TextDocument, TokenFactory)}. * Tokens are accumulated in the {@link Tokens} parameter. * * @param file Document for the file to process @@ -117,8 +117,8 @@ public class Tokens { } @Override - public TokenMgrError makeLexException(int line, int column, String message, @Nullable Throwable cause) { - return new TokenMgrError(line, column, fileId, message, cause); + public LexException makeLexException(int line, int column, String message, @Nullable Throwable cause) { + return new LexException(line, column, fileId, message, cause); } @Override diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrTokenizer.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrCpdLexer.java similarity index 82% rename from pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrTokenizer.java rename to pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrCpdLexer.java index 07e2610d06..3c06996210 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrTokenizer.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/AntlrCpdLexer.java @@ -10,16 +10,16 @@ import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.CharStreams; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.antlr4.AntlrToken; import net.sourceforge.pmd.lang.ast.impl.antlr4.AntlrTokenManager; import net.sourceforge.pmd.lang.document.TextDocument; /** - * Generic implementation of a {@link Tokenizer} useful to any Antlr grammar. + * Generic implementation of a {@link CpdLexer} useful to any Antlr grammar. */ -public abstract class AntlrTokenizer extends TokenizerBase { +public abstract class AntlrCpdLexer extends CpdLexerBase { @Override protected final TokenManager makeLexerImpl(TextDocument doc) throws IOException { CharStream charStream = CharStreams.fromReader(doc.newReader(), doc.getFileId().getAbsolutePath()); diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/TokenizerBase.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/CpdLexerBase.java similarity index 87% rename from pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/TokenizerBase.java rename to pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/CpdLexerBase.java index e43627bba7..22478fa344 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/TokenizerBase.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/CpdLexerBase.java @@ -6,16 +6,16 @@ package net.sourceforge.pmd.cpd.impl; import java.io.IOException; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.cpd.TokenFactory; -import net.sourceforge.pmd.cpd.Tokenizer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.GenericToken; import net.sourceforge.pmd.lang.document.TextDocument; /** - * Generic base class for a {@link Tokenizer}. + * Generic base class for a {@link CpdLexer}. */ -public abstract class TokenizerBase> implements Tokenizer { +public abstract class CpdLexerBase> implements CpdLexer { protected abstract TokenManager makeLexerImpl(TextDocument doc) throws IOException; diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/JavaCCTokenizer.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/JavaCCTokenizer.java deleted file mode 100644 index 319d9dd0f3..0000000000 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/JavaCCTokenizer.java +++ /dev/null @@ -1,15 +0,0 @@ -/** - * BSD-style license; for more info see http://pmd.sourceforge.net/license.html - */ - -package net.sourceforge.pmd.cpd.impl; - -import net.sourceforge.pmd.cpd.Tokenizer; -import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; - -/** - * Base class for a {@link Tokenizer} for a language implemented by a JavaCC tokenizer. - */ -public abstract class JavaCCTokenizer extends TokenizerBase { - -} diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/JavaccCpdLexer.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/JavaccCpdLexer.java new file mode 100644 index 0000000000..7f5641aedb --- /dev/null +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/JavaccCpdLexer.java @@ -0,0 +1,15 @@ +/** + * BSD-style license; for more info see http://pmd.sourceforge.net/license.html + */ + +package net.sourceforge.pmd.cpd.impl; + +import net.sourceforge.pmd.cpd.CpdLexer; +import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; + +/** + * Base class for a {@link CpdLexer} for a language implemented by a JavaCC tokenizer. + */ +public abstract class JavaccCpdLexer extends CpdLexerBase { + +} diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/package-info.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/package-info.java index 7e2f66591f..aa699a7319 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/package-info.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/impl/package-info.java @@ -3,6 +3,6 @@ */ /** - * Utilities to implement a CPD {@link net.sourceforge.pmd.cpd.Tokenizer}. + * Utilities to implement a CPD {@link net.sourceforge.pmd.cpd.CpdLexer}. */ package net.sourceforge.pmd.cpd.impl; diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/package-info.java b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/package-info.java index ecb7637667..224bcc8f6a 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/cpd/package-info.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/cpd/package-info.java @@ -6,6 +6,6 @@ * Token-based copy-paste detection. * * @see net.sourceforge.pmd.cpd.CpdAnalysis - * @see net.sourceforge.pmd.cpd.Tokenizer + * @see net.sourceforge.pmd.cpd.CpdLexer */ package net.sourceforge.pmd.cpd; diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/lang/PlainTextLanguage.java b/pmd-core/src/main/java/net/sourceforge/pmd/lang/PlainTextLanguage.java index 5c4c95cc08..c510759c10 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/lang/PlainTextLanguage.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/lang/PlainTextLanguage.java @@ -5,9 +5,9 @@ package net.sourceforge.pmd.lang; import net.sourceforge.pmd.annotation.Experimental; -import net.sourceforge.pmd.cpd.AnyTokenizer; +import net.sourceforge.pmd.cpd.AnyCpdLexer; import net.sourceforge.pmd.cpd.CpdCapableLanguage; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.ast.AstInfo; import net.sourceforge.pmd.lang.ast.Parser; import net.sourceforge.pmd.lang.ast.Parser.ParserTask; @@ -47,8 +47,8 @@ public final class PlainTextLanguage extends SimpleLanguageModuleBase implements } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new AnyTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new AnyCpdLexer(); } private static final class TextLvh implements LanguageVersionHandler { diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/FileAnalysisException.java b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/FileAnalysisException.java index 94214a0bf4..d843d80f68 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/FileAnalysisException.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/FileAnalysisException.java @@ -16,7 +16,7 @@ import net.sourceforge.pmd.lang.document.FileLocation; /** * An exception that occurs while processing a file. Subtypes include *
    - *
  • {@link TokenMgrError}: lexical syntax errors + *
  • {@link LexException}: lexical syntax errors *
  • {@link ParseException}: syntax errors *
  • {@link SemanticException}: exceptions occurring after the parsing * phase, because the source code is semantically invalid diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/TokenMgrError.java b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/LexException.java similarity index 84% rename from pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/TokenMgrError.java rename to pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/LexException.java index 28bee8a79e..3aea6b656d 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/TokenMgrError.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/LexException.java @@ -16,8 +16,10 @@ import net.sourceforge.pmd.util.StringUtil; /** * An error thrown during lexical analysis of a file. + * + *

    Note: This exception has been called TokenMgrError in PMD 6.

    */ -public final class TokenMgrError extends FileAnalysisException { +public final class LexException extends FileAnalysisException { private final int line; private final int column; @@ -31,7 +33,7 @@ public final class TokenMgrError extends FileAnalysisException { * @param message Message of the error * @param cause Cause of the error, if any */ - public TokenMgrError(int line, int column, @Nullable FileId filename, String message, @Nullable Throwable cause) { + public LexException(int line, int column, @Nullable FileId filename, String message, @Nullable Throwable cause) { super(message, cause); this.line = max(line, 1); this.column = max(column, 1); @@ -44,7 +46,7 @@ public final class TokenMgrError extends FileAnalysisException { * Constructor called by JavaCC. */ @InternalApi - public TokenMgrError(boolean eofSeen, String lexStateName, int errorLine, int errorColumn, String errorAfter, char curChar) { + public LexException(boolean eofSeen, String lexStateName, int errorLine, int errorColumn, String errorAfter, char curChar) { super(makeReason(eofSeen, lexStateName, errorAfter, curChar)); line = max(errorLine, 1); column = max(errorColumn, 1); @@ -76,7 +78,7 @@ public final class TokenMgrError extends FileAnalysisException { * @return A new exception */ @Override - public TokenMgrError setFileId(FileId fileId) { + public LexException setFileId(FileId fileId) { super.setFileId(fileId); return this; } diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrTokenManager.java b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrTokenManager.java index 6233d29a11..5828195850 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrTokenManager.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/antlr4/AntlrTokenManager.java @@ -10,7 +10,7 @@ import org.antlr.v4.runtime.RecognitionException; import org.antlr.v4.runtime.Recognizer; import net.sourceforge.pmd.lang.TokenManager; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; import net.sourceforge.pmd.lang.document.TextDocument; /** @@ -63,7 +63,7 @@ public class AntlrTokenManager implements TokenManager { final int charPositionInLine, final String msg, final RecognitionException ex) { - throw new TokenMgrError(line, charPositionInLine, textDoc.getFileId(), msg, ex); + throw new LexException(line, charPositionInLine, textDoc.getFileId(), msg, ex); } } diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/javacc/JavaccTokenDocument.java b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/javacc/JavaccTokenDocument.java index 5887805ec4..5b727d6dd3 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/javacc/JavaccTokenDocument.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/lang/ast/impl/javacc/JavaccTokenDocument.java @@ -10,7 +10,7 @@ import java.util.List; import org.checkerframework.checker.nullness.qual.NonNull; import org.checkerframework.checker.nullness.qual.Nullable; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.ast.impl.TokenDocument; import net.sourceforge.pmd.lang.document.TextDocument; @@ -18,7 +18,7 @@ import net.sourceforge.pmd.lang.document.TextDocument; * Token document for Javacc implementations. This is a helper object * for generated token managers. Note: the extension point is a custom * implementation of {@link TokenDocumentBehavior}, see {@link JjtreeParserAdapter#tokenBehavior()}, - * {@link JavaCCTokenizer#tokenBehavior()} + * {@link JavaccCpdLexer#tokenBehavior()} */ public final class JavaccTokenDocument extends TokenDocument { diff --git a/pmd-core/src/main/java/net/sourceforge/pmd/lang/impl/CpdOnlyLanguageModuleBase.java b/pmd-core/src/main/java/net/sourceforge/pmd/lang/impl/CpdOnlyLanguageModuleBase.java index b66848e2ed..5da636b741 100644 --- a/pmd-core/src/main/java/net/sourceforge/pmd/lang/impl/CpdOnlyLanguageModuleBase.java +++ b/pmd-core/src/main/java/net/sourceforge/pmd/lang/impl/CpdOnlyLanguageModuleBase.java @@ -5,7 +5,7 @@ package net.sourceforge.pmd.lang.impl; import net.sourceforge.pmd.cpd.CpdCapableLanguage; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguageModuleBase; import net.sourceforge.pmd.lang.LanguagePropertyBundle; @@ -27,5 +27,5 @@ public abstract class CpdOnlyLanguageModuleBase extends LanguageModuleBase imple } @Override - public abstract Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle); + public abstract CpdLexer createCpdLexer(LanguagePropertyBundle bundle); } diff --git a/pmd-core/src/test/java/net/sourceforge/pmd/cpd/AnyTokenizerTest.java b/pmd-core/src/test/java/net/sourceforge/pmd/cpd/AnyCpdLexerTest.java similarity index 87% rename from pmd-core/src/test/java/net/sourceforge/pmd/cpd/AnyTokenizerTest.java rename to pmd-core/src/test/java/net/sourceforge/pmd/cpd/AnyCpdLexerTest.java index 467cbc2a5c..c91e3ebd33 100644 --- a/pmd-core/src/test/java/net/sourceforge/pmd/cpd/AnyTokenizerTest.java +++ b/pmd-core/src/test/java/net/sourceforge/pmd/cpd/AnyCpdLexerTest.java @@ -17,23 +17,23 @@ import net.sourceforge.pmd.lang.DummyLanguageModule; import net.sourceforge.pmd.lang.document.FileId; import net.sourceforge.pmd.lang.document.TextDocument; -class AnyTokenizerTest { +class AnyCpdLexerTest { @Test void testMultiLineMacros() throws IOException { - AnyTokenizer tokenizer = new AnyTokenizer("//"); + AnyCpdLexer tokenizer = new AnyCpdLexer("//"); compareResult(tokenizer, TEST1, EXPECTED); } @Test void testStringEscape() throws IOException { - AnyTokenizer tokenizer = new AnyTokenizer("//"); + AnyCpdLexer tokenizer = new AnyCpdLexer("//"); compareResult(tokenizer, "a = \"oo\\n\"", listOf("a", "=", "\"oo\\n\"", "EOF")); } @Test void testMultilineString() throws IOException { - AnyTokenizer tokenizer = new AnyTokenizer("//"); + AnyCpdLexer tokenizer = new AnyCpdLexer("//"); Tokens tokens = compareResult(tokenizer, "a = \"oo\n\";", listOf("a", "=", "\"oo\n\"", ";", "EOF")); TokenEntry string = tokens.getTokens().get(2); assertEquals("\"oo\n\"", string.getImage(tokens)); @@ -53,10 +53,10 @@ class AnyTokenizerTest { */ @Test void testTokenPosition() throws IOException { - AnyTokenizer tokenizer = new AnyTokenizer(); + AnyCpdLexer tokenizer = new AnyCpdLexer(); TextDocument code = TextDocument.readOnlyString("a;\nbbbb\n;", FileId.UNKNOWN, DummyLanguageModule.getInstance().getDefaultVersion()); Tokens tokens = new Tokens(); - Tokenizer.tokenize(tokenizer, code, tokens); + CpdLexer.tokenize(tokenizer, code, tokens); TokenEntry bbbbToken = tokens.getTokens().get(2); assertEquals(2, bbbbToken.getBeginLine()); assertEquals(1, bbbbToken.getBeginColumn()); @@ -64,10 +64,10 @@ class AnyTokenizerTest { } - private Tokens compareResult(AnyTokenizer tokenizer, String source, List expectedImages) throws IOException { + private Tokens compareResult(AnyCpdLexer tokenizer, String source, List expectedImages) throws IOException { TextDocument code = TextDocument.readOnlyString(source, FileId.UNKNOWN, DummyLanguageModule.getInstance().getDefaultVersion()); Tokens tokens = new Tokens(); - Tokenizer.tokenize(tokenizer, code, tokens); + CpdLexer.tokenize(tokenizer, code, tokens); List tokenStrings = new ArrayList<>(); for (TokenEntry token : tokens.getTokens()) { diff --git a/pmd-core/src/test/java/net/sourceforge/pmd/cpd/MatchAlgorithmTest.java b/pmd-core/src/test/java/net/sourceforge/pmd/cpd/MatchAlgorithmTest.java index 4eac716b9a..2ddee6712a 100644 --- a/pmd-core/src/test/java/net/sourceforge/pmd/cpd/MatchAlgorithmTest.java +++ b/pmd-core/src/test/java/net/sourceforge/pmd/cpd/MatchAlgorithmTest.java @@ -38,13 +38,13 @@ class MatchAlgorithmTest { @Test void testSimple() throws IOException { DummyLanguageModule dummy = DummyLanguageModule.getInstance(); - Tokenizer tokenizer = dummy.createCpdTokenizer(dummy.newPropertyBundle()); + CpdLexer cpdLexer = dummy.createCpdLexer(dummy.newPropertyBundle()); FileId fileName = FileId.fromPathLikeString("Foo.dummy"); TextFile textFile = TextFile.forCharSeq(getSampleCode(), fileName, dummy.getDefaultVersion()); SourceManager sourceManager = new SourceManager(listOf(textFile)); Tokens tokens = new Tokens(); TextDocument sourceCode = sourceManager.get(textFile); - Tokenizer.tokenize(tokenizer, sourceCode, tokens); + CpdLexer.tokenize(cpdLexer, sourceCode, tokens); assertEquals(44, tokens.size()); MatchAlgorithm matchAlgorithm = new MatchAlgorithm(tokens, 5); diff --git a/pmd-core/src/test/java/net/sourceforge/pmd/lang/ast/TokenMgrErrorTest.java b/pmd-core/src/test/java/net/sourceforge/pmd/lang/ast/LexExceptionTest.java similarity index 71% rename from pmd-core/src/test/java/net/sourceforge/pmd/lang/ast/TokenMgrErrorTest.java rename to pmd-core/src/test/java/net/sourceforge/pmd/lang/ast/LexExceptionTest.java index 806d5e743e..33bbcc972f 100644 --- a/pmd-core/src/test/java/net/sourceforge/pmd/lang/ast/TokenMgrErrorTest.java +++ b/pmd-core/src/test/java/net/sourceforge/pmd/lang/ast/LexExceptionTest.java @@ -8,30 +8,30 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import org.junit.jupiter.api.Test; -class TokenMgrErrorTest { +class LexExceptionTest { @Test void invalidLocation() { - TokenMgrError error = new TokenMgrError(2, 0, null, "test", null); + LexException error = new LexException(2, 0, null, "test", null); // this shouldn't throw a IllegalArgumentException assertEquals("line 2, column 1", error.location().startPosToString()); } @Test void invalidLocationJavaCC() { - TokenMgrError error = new TokenMgrError(false, "DEFAULT", 2, 0, "}", '\n'); + LexException error = new LexException(false, "DEFAULT", 2, 0, "}", '\n'); // this shouldn't throw a IllegalArgumentException assertEquals("line 2, column 1", error.location().startPosToString()); } @Test void validLocation() { - TokenMgrError error = new TokenMgrError(1, 1, null, "test", null); + LexException error = new LexException(1, 1, null, "test", null); assertEquals("line 1, column 1", error.location().startPosToString()); } @Test void validLocationJavaCC() { - TokenMgrError error = new TokenMgrError(false, "DEFAULT", 1, 1, "}", '\n'); + LexException error = new LexException(false, "DEFAULT", 1, 1, "}", '\n'); assertEquals("line 1, column 1", error.location().startPosToString()); } } diff --git a/pmd-cpp/etc/grammar/Cpp.jj b/pmd-cpp/etc/grammar/Cpp.jj index 85aca73e42..cf89e9188f 100644 --- a/pmd-cpp/etc/grammar/Cpp.jj +++ b/pmd-cpp/etc/grammar/Cpp.jj @@ -32,9 +32,6 @@ options { PARSER_BEGIN(CppParserImpl) package net.sourceforge.pmd.lang.cpp.ast; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; - public final class CppParserImpl { } diff --git a/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/CppLanguageModule.java b/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/CppLanguageModule.java index d155ae5c0f..e3f1643aee 100644 --- a/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/CppLanguageModule.java +++ b/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/CppLanguageModule.java @@ -5,10 +5,10 @@ package net.sourceforge.pmd.lang.cpp; import net.sourceforge.pmd.cpd.CpdLanguageProperties; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.cpp.cpd.CPPTokenizer; +import net.sourceforge.pmd.lang.cpp.cpd.CppCpdLexer; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; import net.sourceforge.pmd.properties.PropertyDescriptor; import net.sourceforge.pmd.properties.PropertyFactory; @@ -51,7 +51,7 @@ public class CppLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new CPPTokenizer(bundle); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new CppCpdLexer(bundle); } } diff --git a/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/cpd/CPPTokenizer.java b/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/cpd/CppCpdLexer.java similarity index 96% rename from pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/cpd/CPPTokenizer.java rename to pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/cpd/CppCpdLexer.java index 37c2bb166d..5d0e423a51 100644 --- a/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/cpd/CPPTokenizer.java +++ b/pmd-cpp/src/main/java/net/sourceforge/pmd/lang/cpp/cpd/CppCpdLexer.java @@ -9,8 +9,8 @@ import java.util.regex.Pattern; import org.apache.commons.lang3.StringUtils; import net.sourceforge.pmd.cpd.CpdLanguageProperties; +import net.sourceforge.pmd.cpd.impl.CpdLexerBase; import net.sourceforge.pmd.cpd.impl.JavaCCTokenFilter; -import net.sourceforge.pmd.cpd.impl.TokenizerBase; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; @@ -23,8 +23,10 @@ import net.sourceforge.pmd.lang.document.TextDocument; /** * The C++ tokenizer. + * + *

    Note: This class has been called CPPTokenizer in PMD 6

    . */ -public class CPPTokenizer extends TokenizerBase { +public class CppCpdLexer extends CpdLexerBase { private boolean skipBlocks; private Pattern skipBlocksStart; @@ -32,7 +34,7 @@ public class CPPTokenizer extends TokenizerBase { private final boolean ignoreIdentifierAndLiteralSeqences; private final boolean ignoreLiteralSequences; - public CPPTokenizer(LanguagePropertyBundle cppProperties) { + public CppCpdLexer(LanguagePropertyBundle cppProperties) { ignoreLiteralSequences = cppProperties.getProperty(CpdLanguageProperties.CPD_IGNORE_LITERAL_SEQUENCES); ignoreIdentifierAndLiteralSeqences = cppProperties.getProperty(CpdLanguageProperties.CPD_IGNORE_LITERAL_AND_IDENTIFIER_SEQUENCES); String skipBlocksPattern = cppProperties.getProperty(CppLanguageModule.CPD_SKIP_BLOCKS); diff --git a/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CppCharStreamTest.java b/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CppCharStreamTest.java index e3e760d081..9d16e62bae 100644 --- a/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CppCharStreamTest.java +++ b/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CppCharStreamTest.java @@ -20,7 +20,7 @@ class CppCharStreamTest { public CharStream charStreamFor(String source) { CppLanguageModule cpp = CppLanguageModule.getInstance(); TextDocument textDoc = TextDocument.readOnlyString(source, FileId.UNKNOWN, cpp.getDefaultVersion()); - CPPTokenizer tokenizer = new CPPTokenizer(cpp.newPropertyBundle()); + CppCpdLexer tokenizer = new CppCpdLexer(cpp.newPropertyBundle()); return tokenizer.newCharStream(textDoc); } diff --git a/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CPPTokenizerTest.java b/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CppCpdLexerTest.java similarity index 91% rename from pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CPPTokenizerTest.java rename to pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CppCpdLexerTest.java index dc7eb09153..8e7b1b22d5 100644 --- a/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CPPTokenizerTest.java +++ b/pmd-cpp/src/test/java/net/sourceforge/pmd/lang/cpp/cpd/CppCpdLexerTest.java @@ -10,15 +10,15 @@ import org.checkerframework.checker.nullness.qual.NonNull; import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.CpdLanguageProperties; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.cpd.Tokens; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.cpd.test.LanguagePropertyConfig; import net.sourceforge.pmd.lang.cpp.CppLanguageModule; -class CPPTokenizerTest extends CpdTextComparisonTest { +class CppCpdLexerTest extends CpdTextComparisonTest { - CPPTokenizerTest() { + CppCpdLexerTest() { super(CppLanguageModule.getInstance(), ".cpp"); } @@ -29,8 +29,8 @@ class CPPTokenizerTest extends CpdTextComparisonTest { @Test void testUTFwithBOM() { - Tokenizer tokenizer = newTokenizer(dontSkipBlocks()); - Tokens tokens = tokenize(tokenizer, sourceCodeOf("\ufeffint start()\n{ int ret = 1;\nreturn ret;\n}\n")); + CpdLexer cpdLexer = newCpdLexer(dontSkipBlocks()); + Tokens tokens = tokenize(cpdLexer, sourceCodeOf("\ufeffint start()\n{ int ret = 1;\nreturn ret;\n}\n")); assertEquals(15, tokens.size()); } @@ -71,7 +71,7 @@ class CPPTokenizerTest extends CpdTextComparisonTest { @Test void testWrongUnicodeInIdentifier() { - expectTokenMgrError(" void main() { int ⚜ = __; }"); + expectLexException(" void main() { int ⚜ = __; }"); } @Test @@ -107,7 +107,7 @@ class CPPTokenizerTest extends CpdTextComparisonTest { @Test void testLexicalErrorFilename() { - expectTokenMgrError(sourceText("issue-1559"), dontSkipBlocks()); + expectLexException(sourceText("issue-1559"), dontSkipBlocks()); } diff --git a/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/CsLanguageModule.java b/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/CsLanguageModule.java index fd53a85d4c..0e458d8a65 100644 --- a/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/CsLanguageModule.java +++ b/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/CsLanguageModule.java @@ -5,10 +5,10 @@ package net.sourceforge.pmd.lang.cs; import net.sourceforge.pmd.cpd.CpdLanguageProperties; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.cs.cpd.CsTokenizer; +import net.sourceforge.pmd.lang.cs.cpd.CsCpdLexer; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; /** @@ -38,7 +38,7 @@ public class CsLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new CsTokenizer(bundle); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new CsCpdLexer(bundle); } } diff --git a/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsTokenizer.java b/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsCpdLexer.java similarity index 97% rename from pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsTokenizer.java rename to pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsCpdLexer.java index 8d28908150..2628adfe00 100644 --- a/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsTokenizer.java +++ b/pmd-cs/src/main/java/net/sourceforge/pmd/lang/cs/cpd/CsCpdLexer.java @@ -8,8 +8,8 @@ import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; import net.sourceforge.pmd.cpd.CpdLanguageProperties; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.cpd.impl.AntlrTokenFilter; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; import net.sourceforge.pmd.cpd.impl.BaseTokenFilter; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.TokenManager; @@ -18,14 +18,16 @@ import net.sourceforge.pmd.lang.cs.ast.CSharpLexer; /** * The C# tokenizer. + * + *

    Note: This class has been called CsTokenizer in PMD 6

    . */ -public class CsTokenizer extends AntlrTokenizer { +public class CsCpdLexer extends AntlrCpdLexer { private final boolean ignoreUsings; private final boolean ignoreLiteralSequences; private final boolean ignoreAttributes; - public CsTokenizer(LanguagePropertyBundle properties) { + public CsCpdLexer(LanguagePropertyBundle properties) { ignoreUsings = properties.getProperty(CpdLanguageProperties.CPD_IGNORE_IMPORTS); ignoreLiteralSequences = properties.getProperty(CpdLanguageProperties.CPD_IGNORE_LITERAL_SEQUENCES); ignoreAttributes = properties.getProperty(CpdLanguageProperties.CPD_IGNORE_METADATA); diff --git a/pmd-cs/src/test/java/net/sourceforge/pmd/lang/cs/cpd/CsTokenizerTest.java b/pmd-cs/src/test/java/net/sourceforge/pmd/lang/cs/cpd/CsCpdLexerTest.java similarity index 93% rename from pmd-cs/src/test/java/net/sourceforge/pmd/lang/cs/cpd/CsTokenizerTest.java rename to pmd-cs/src/test/java/net/sourceforge/pmd/lang/cs/cpd/CsCpdLexerTest.java index 60cfd37e1e..49e9e81cc1 100644 --- a/pmd-cs/src/test/java/net/sourceforge/pmd/lang/cs/cpd/CsTokenizerTest.java +++ b/pmd-cs/src/test/java/net/sourceforge/pmd/lang/cs/cpd/CsCpdLexerTest.java @@ -12,11 +12,11 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.CpdLanguageProperties; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.cpd.test.LanguagePropertyConfig; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; -class CsTokenizerTest extends CpdTextComparisonTest { +class CsCpdLexerTest extends CpdTextComparisonTest { - CsTokenizerTest() { + CsCpdLexerTest() { super("cs", ".cs"); } @@ -37,7 +37,7 @@ class CsTokenizerTest extends CpdTextComparisonTest { @Test void testOpenString() { - assertThrows(TokenMgrError.class, () -> doTest("unlexable_string")); + assertThrows(LexException.class, () -> doTest("unlexable_string")); } @Test diff --git a/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/DartLanguageModule.java b/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/DartLanguageModule.java index 11fa573f31..96b4a4d19c 100644 --- a/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/DartLanguageModule.java +++ b/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/DartLanguageModule.java @@ -4,10 +4,10 @@ package net.sourceforge.pmd.lang.dart; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.dart.cpd.DartTokenizer; +import net.sourceforge.pmd.lang.dart.cpd.DartCpdLexer; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; /** @@ -26,7 +26,7 @@ public class DartLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new DartTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new DartCpdLexer(); } } diff --git a/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/cpd/DartTokenizer.java b/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/cpd/DartCpdLexer.java similarity index 93% rename from pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/cpd/DartTokenizer.java rename to pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/cpd/DartCpdLexer.java index 9ac4a74b71..73c33e448b 100644 --- a/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/cpd/DartTokenizer.java +++ b/pmd-dart/src/main/java/net/sourceforge/pmd/lang/dart/cpd/DartCpdLexer.java @@ -7,8 +7,8 @@ package net.sourceforge.pmd.lang.dart.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.cpd.impl.AntlrTokenFilter; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; import net.sourceforge.pmd.cpd.impl.BaseTokenFilter; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.antlr4.AntlrToken; @@ -16,8 +16,10 @@ import net.sourceforge.pmd.lang.dart.ast.DartLexer; /** * The Dart Tokenizer + * + *

    Note: This class has been called DartTokenizer in PMD 6

    . */ -public class DartTokenizer extends AntlrTokenizer { +public class DartCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { diff --git a/pmd-dart/src/test/java/net/sourceforge/pmd/lang/dart/cpd/DartTokenizerTest.java b/pmd-dart/src/test/java/net/sourceforge/pmd/lang/dart/cpd/DartCpdLexerTest.java similarity index 94% rename from pmd-dart/src/test/java/net/sourceforge/pmd/lang/dart/cpd/DartTokenizerTest.java rename to pmd-dart/src/test/java/net/sourceforge/pmd/lang/dart/cpd/DartCpdLexerTest.java index 02106add09..7d85f57be4 100644 --- a/pmd-dart/src/test/java/net/sourceforge/pmd/lang/dart/cpd/DartTokenizerTest.java +++ b/pmd-dart/src/test/java/net/sourceforge/pmd/lang/dart/cpd/DartCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class DartTokenizerTest extends CpdTextComparisonTest { +class DartCpdLexerTest extends CpdTextComparisonTest { - DartTokenizerTest() { + DartCpdLexerTest() { super("dart", ".dart"); } diff --git a/pmd-fortran/src/main/java/net/sourceforge/pmd/lang/fortran/FortranLanguageModule.java b/pmd-fortran/src/main/java/net/sourceforge/pmd/lang/fortran/FortranLanguageModule.java index 72960be9fd..9ad4a81e0a 100644 --- a/pmd-fortran/src/main/java/net/sourceforge/pmd/lang/fortran/FortranLanguageModule.java +++ b/pmd-fortran/src/main/java/net/sourceforge/pmd/lang/fortran/FortranLanguageModule.java @@ -4,8 +4,8 @@ package net.sourceforge.pmd.lang.fortran; -import net.sourceforge.pmd.cpd.AnyTokenizer; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.AnyCpdLexer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; @@ -30,7 +30,7 @@ public class FortranLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new AnyTokenizer("!"); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new AnyCpdLexer("!"); } } diff --git a/pmd-fortran/src/test/java/net/sourceforge/pmd/lang/fortran/cpd/FortranTokenizerTest.java b/pmd-fortran/src/test/java/net/sourceforge/pmd/lang/fortran/cpd/FortranCpdLexerTest.java similarity index 80% rename from pmd-fortran/src/test/java/net/sourceforge/pmd/lang/fortran/cpd/FortranTokenizerTest.java rename to pmd-fortran/src/test/java/net/sourceforge/pmd/lang/fortran/cpd/FortranCpdLexerTest.java index ed49ed7030..4174204f44 100644 --- a/pmd-fortran/src/test/java/net/sourceforge/pmd/lang/fortran/cpd/FortranTokenizerTest.java +++ b/pmd-fortran/src/test/java/net/sourceforge/pmd/lang/fortran/cpd/FortranCpdLexerTest.java @@ -12,9 +12,9 @@ import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; * @author rpelisse * */ -class FortranTokenizerTest extends CpdTextComparisonTest { +class FortranCpdLexerTest extends CpdTextComparisonTest { - FortranTokenizerTest() { + FortranCpdLexerTest() { super("fortran", ".for"); } diff --git a/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/GherkinLanguageModule.java b/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/GherkinLanguageModule.java index 4bc116d406..1710282749 100644 --- a/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/GherkinLanguageModule.java +++ b/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/GherkinLanguageModule.java @@ -4,10 +4,10 @@ package net.sourceforge.pmd.lang.gherkin; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.gherkin.cpd.GherkinTokenizer; +import net.sourceforge.pmd.lang.gherkin.cpd.GherkinCpdLexer; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; /** @@ -28,7 +28,7 @@ public class GherkinLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new GherkinTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new GherkinCpdLexer(); } } diff --git a/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinTokenizer.java b/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinCpdLexer.java similarity index 71% rename from pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinTokenizer.java rename to pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinCpdLexer.java index 1bbe22cf51..73120670e2 100644 --- a/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinTokenizer.java +++ b/pmd-gherkin/src/main/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinCpdLexer.java @@ -7,13 +7,15 @@ package net.sourceforge.pmd.lang.gherkin.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.gherkin.ast.GherkinLexer; /** * The Gherkin Tokenizer. + * + *

    Note: This class has been called GherkinTokenizer in PMD 6

    . */ -public class GherkinTokenizer extends AntlrTokenizer { +public class GherkinCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { diff --git a/pmd-gherkin/src/test/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinTokenizerTest.java b/pmd-gherkin/src/test/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinCpdLexerTest.java similarity index 83% rename from pmd-gherkin/src/test/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinTokenizerTest.java rename to pmd-gherkin/src/test/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinCpdLexerTest.java index 38c4d88595..6fde983b27 100644 --- a/pmd-gherkin/src/test/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinTokenizerTest.java +++ b/pmd-gherkin/src/test/java/net/sourceforge/pmd/lang/gherkin/cpd/GherkinCpdLexerTest.java @@ -8,8 +8,8 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class GherkinTokenizerTest extends CpdTextComparisonTest { - GherkinTokenizerTest() { +class GherkinCpdLexerTest extends CpdTextComparisonTest { + GherkinCpdLexerTest() { super("gherkin", ".feature"); } diff --git a/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/GoLanguageModule.java b/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/GoLanguageModule.java index 966d6300ee..d408688c1f 100644 --- a/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/GoLanguageModule.java +++ b/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/GoLanguageModule.java @@ -4,10 +4,10 @@ package net.sourceforge.pmd.lang.go; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.go.cpd.GoTokenizer; +import net.sourceforge.pmd.lang.go.cpd.GoCpdLexer; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; public class GoLanguageModule extends CpdOnlyLanguageModuleBase { @@ -22,7 +22,7 @@ public class GoLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new GoTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new GoCpdLexer(); } } diff --git a/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/cpd/GoTokenizer.java b/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/cpd/GoCpdLexer.java similarity index 69% rename from pmd-go/src/main/java/net/sourceforge/pmd/lang/go/cpd/GoTokenizer.java rename to pmd-go/src/main/java/net/sourceforge/pmd/lang/go/cpd/GoCpdLexer.java index a4d9526a52..454c8e4daf 100644 --- a/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/cpd/GoTokenizer.java +++ b/pmd-go/src/main/java/net/sourceforge/pmd/lang/go/cpd/GoCpdLexer.java @@ -7,10 +7,13 @@ package net.sourceforge.pmd.lang.go.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.go.ast.GolangLexer; -public class GoTokenizer extends AntlrTokenizer { +/** + *

    Note: This class has been called GoTokenizer in PMD 6

    . + */ +public class GoCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { diff --git a/pmd-go/src/test/java/net/sourceforge/pmd/lang/go/cpd/GoTokenizerTest.java b/pmd-go/src/test/java/net/sourceforge/pmd/lang/go/cpd/GoCpdLexerTest.java similarity index 88% rename from pmd-go/src/test/java/net/sourceforge/pmd/lang/go/cpd/GoTokenizerTest.java rename to pmd-go/src/test/java/net/sourceforge/pmd/lang/go/cpd/GoCpdLexerTest.java index 6ae9366f0b..e40d2c2b23 100644 --- a/pmd-go/src/test/java/net/sourceforge/pmd/lang/go/cpd/GoTokenizerTest.java +++ b/pmd-go/src/test/java/net/sourceforge/pmd/lang/go/cpd/GoCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class GoTokenizerTest extends CpdTextComparisonTest { +class GoCpdLexerTest extends CpdTextComparisonTest { - GoTokenizerTest() { + GoCpdLexerTest() { super("go", ".go"); } diff --git a/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/GroovyLanguageModule.java b/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/GroovyLanguageModule.java index a740daea45..de46895863 100644 --- a/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/GroovyLanguageModule.java +++ b/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/GroovyLanguageModule.java @@ -4,10 +4,10 @@ package net.sourceforge.pmd.lang.groovy; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.groovy.cpd.GroovyTokenizer; +import net.sourceforge.pmd.lang.groovy.cpd.GroovyCpdLexer; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; /** @@ -28,7 +28,7 @@ public class GroovyLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new GroovyTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new GroovyCpdLexer(); } } diff --git a/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/ast/impl/antlr4/GroovyTokenManager.java b/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/ast/impl/antlr4/GroovyTokenManager.java index 313f386a80..79c3b5d2da 100644 --- a/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/ast/impl/antlr4/GroovyTokenManager.java +++ b/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/ast/impl/antlr4/GroovyTokenManager.java @@ -7,7 +7,7 @@ package net.sourceforge.pmd.lang.groovy.ast.impl.antlr4; import org.apache.groovy.parser.antlr4.GroovyLexer; import net.sourceforge.pmd.lang.TokenManager; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; import net.sourceforge.pmd.lang.ast.impl.antlr4.AntlrTokenManager; import net.sourceforge.pmd.lang.document.TextDocument; @@ -81,7 +81,7 @@ public class GroovyTokenManager implements TokenManager { final int charPositionInLine, final String msg, final RecognitionException ex) { - throw new TokenMgrError(line, charPositionInLine, textDoc.getFileId(), msg, ex); + throw new LexException(line, charPositionInLine, textDoc.getFileId(), msg, ex); } } diff --git a/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyTokenizer.java b/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyCpdLexer.java similarity index 82% rename from pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyTokenizer.java rename to pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyCpdLexer.java index b180108fed..b27a2d1c44 100644 --- a/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyTokenizer.java +++ b/pmd-groovy/src/main/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyCpdLexer.java @@ -8,7 +8,7 @@ import java.io.IOException; import org.apache.groovy.parser.antlr4.GroovyLexer; -import net.sourceforge.pmd.cpd.impl.TokenizerBase; +import net.sourceforge.pmd.cpd.impl.CpdLexerBase; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.groovy.ast.impl.antlr4.GroovyToken; @@ -19,8 +19,10 @@ import groovyjarjarantlr4.v4.runtime.CharStreams; /** * The Groovy Tokenizer + * + *

    Note: This class has been called GroovyTokenizer in PMD 6

    . */ -public class GroovyTokenizer extends TokenizerBase { +public class GroovyCpdLexer extends CpdLexerBase { @Override protected final TokenManager makeLexerImpl(TextDocument doc) throws IOException { diff --git a/pmd-groovy/src/test/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyTokenizerTest.java b/pmd-groovy/src/test/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyCpdLexerTest.java similarity index 82% rename from pmd-groovy/src/test/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyTokenizerTest.java rename to pmd-groovy/src/test/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyCpdLexerTest.java index 10570f4652..647020c363 100644 --- a/pmd-groovy/src/test/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyTokenizerTest.java +++ b/pmd-groovy/src/test/java/net/sourceforge/pmd/lang/groovy/cpd/GroovyCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class GroovyTokenizerTest extends CpdTextComparisonTest { +class GroovyCpdLexerTest extends CpdTextComparisonTest { - GroovyTokenizerTest() { + GroovyCpdLexerTest() { super("groovy", ".groovy"); } diff --git a/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/HtmlLanguageModule.java b/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/HtmlLanguageModule.java index 16b3fbee78..26d02710a9 100644 --- a/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/HtmlLanguageModule.java +++ b/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/HtmlLanguageModule.java @@ -4,10 +4,10 @@ package net.sourceforge.pmd.lang.html; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; -import net.sourceforge.pmd.lang.html.ast.HtmlTokenizer; +import net.sourceforge.pmd.lang.html.ast.HtmlCpdLexer; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; public final class HtmlLanguageModule extends SimpleLanguageModuleBase { @@ -27,7 +27,7 @@ public final class HtmlLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new HtmlTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new HtmlCpdLexer(); } } diff --git a/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/ast/HtmlTokenizer.java b/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/ast/HtmlCpdLexer.java similarity index 91% rename from pmd-html/src/main/java/net/sourceforge/pmd/lang/html/ast/HtmlTokenizer.java rename to pmd-html/src/main/java/net/sourceforge/pmd/lang/html/ast/HtmlCpdLexer.java index a14661a83d..f7f0668de6 100644 --- a/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/ast/HtmlTokenizer.java +++ b/pmd-html/src/main/java/net/sourceforge/pmd/lang/html/ast/HtmlCpdLexer.java @@ -7,8 +7,8 @@ package net.sourceforge.pmd.lang.html.ast; import java.io.IOException; import java.io.UncheckedIOException; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.cpd.TokenFactory; -import net.sourceforge.pmd.cpd.Tokenizer; import net.sourceforge.pmd.lang.LanguageProcessor; import net.sourceforge.pmd.lang.LanguageProcessorRegistry; import net.sourceforge.pmd.lang.ast.Parser.ParserTask; @@ -16,7 +16,10 @@ import net.sourceforge.pmd.lang.ast.SemanticErrorReporter; import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.html.HtmlLanguageModule; -public class HtmlTokenizer implements Tokenizer { +/** + *

    Note: This class has been called HtmlTokenizer in PMD 6

    . + */ +public class HtmlCpdLexer implements CpdLexer { @Override public void tokenize(TextDocument document, TokenFactory tokens) { diff --git a/pmd-html/src/test/java/net/sourceforge/pmd/lang/html/cpd/HtmlTokenizerTest.java b/pmd-html/src/test/java/net/sourceforge/pmd/lang/html/cpd/HtmlCpdLexerTest.java similarity index 84% rename from pmd-html/src/test/java/net/sourceforge/pmd/lang/html/cpd/HtmlTokenizerTest.java rename to pmd-html/src/test/java/net/sourceforge/pmd/lang/html/cpd/HtmlCpdLexerTest.java index 02dd1be448..2039a2f72b 100644 --- a/pmd-html/src/test/java/net/sourceforge/pmd/lang/html/cpd/HtmlTokenizerTest.java +++ b/pmd-html/src/test/java/net/sourceforge/pmd/lang/html/cpd/HtmlCpdLexerTest.java @@ -10,9 +10,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.html.HtmlLanguageModule; -class HtmlTokenizerTest extends CpdTextComparisonTest { +class HtmlCpdLexerTest extends CpdTextComparisonTest { - HtmlTokenizerTest() { + HtmlCpdLexerTest() { super(HtmlLanguageModule.getInstance(), ".html"); } diff --git a/pmd-java/etc/grammar/Java.jjt b/pmd-java/etc/grammar/Java.jjt index 675db4e96c..df8f248e1f 100644 --- a/pmd-java/etc/grammar/Java.jjt +++ b/pmd-java/etc/grammar/Java.jjt @@ -310,12 +310,8 @@ import java.util.EnumSet; import java.util.List; import java.util.Set; import java.util.Map; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.GenericToken; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; import net.sourceforge.pmd.lang.ast.Node; -import net.sourceforge.pmd.lang.ast.TokenMgrError; -import net.sourceforge.pmd.lang.ast.Node; import net.sourceforge.pmd.lang.java.ast.JavaParserImplTokenManager.TokenContext; import net.sourceforge.pmd.lang.java.types.JPrimitiveType.PrimitiveTypeKind; diff --git a/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/JavaLanguageModule.java b/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/JavaLanguageModule.java index 928773dc75..562b6365e6 100644 --- a/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/JavaLanguageModule.java +++ b/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/JavaLanguageModule.java @@ -5,13 +5,13 @@ package net.sourceforge.pmd.lang.java; import net.sourceforge.pmd.cpd.CpdCapableLanguage; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguageModuleBase; import net.sourceforge.pmd.lang.LanguageProcessor; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.PmdCapableLanguage; -import net.sourceforge.pmd.lang.java.cpd.JavaTokenizer; +import net.sourceforge.pmd.lang.java.cpd.JavaCpdLexer; import net.sourceforge.pmd.lang.java.internal.JavaLanguageProcessor; import net.sourceforge.pmd.lang.java.internal.JavaLanguageProperties; @@ -62,7 +62,7 @@ public class JavaLanguageModule extends LanguageModuleBase implements PmdCapable } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new JavaTokenizer((JavaLanguageProperties) bundle); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new JavaCpdLexer((JavaLanguageProperties) bundle); } } diff --git a/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaTokenizer.java b/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaCpdLexer.java similarity index 97% rename from pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaTokenizer.java rename to pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaCpdLexer.java index 8331feff13..1cf9ae4a5c 100644 --- a/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaTokenizer.java +++ b/pmd-java/src/main/java/net/sourceforge/pmd/lang/java/cpd/JavaCpdLexer.java @@ -11,7 +11,7 @@ import net.sourceforge.pmd.cpd.CpdLanguageProperties; import net.sourceforge.pmd.cpd.TokenEntry; import net.sourceforge.pmd.cpd.TokenFactory; import net.sourceforge.pmd.cpd.impl.JavaCCTokenFilter; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; @@ -21,7 +21,10 @@ import net.sourceforge.pmd.lang.java.ast.JavaTokenKinds; import net.sourceforge.pmd.lang.java.ast.SyntacticJavaTokenizerFactory; import net.sourceforge.pmd.lang.java.internal.JavaLanguageProperties; -public class JavaTokenizer extends JavaCCTokenizer { +/** + *

    Note: This class has been called JavaTokenizer in PMD 6

    . + */ +public class JavaCpdLexer extends JavaccCpdLexer { private static final String CPD_START = "\"CPD-START\""; private static final String CPD_END = "\"CPD-END\""; @@ -32,7 +35,7 @@ public class JavaTokenizer extends JavaCCTokenizer { private final ConstructorDetector constructorDetector; - public JavaTokenizer(JavaLanguageProperties properties) { + public JavaCpdLexer(JavaLanguageProperties properties) { ignoreAnnotations = properties.getProperty(CpdLanguageProperties.CPD_IGNORE_METADATA); ignoreLiterals = properties.getProperty(CpdLanguageProperties.CPD_ANONYMIZE_LITERALS); ignoreIdentifiers = properties.getProperty(CpdLanguageProperties.CPD_ANONYMIZE_IDENTIFIERS); diff --git a/pmd-java/src/test/java/net/sourceforge/pmd/lang/java/cpd/JavaTokenizerTest.java b/pmd-java/src/test/java/net/sourceforge/pmd/lang/java/cpd/JavaCpdLexerTest.java similarity index 93% rename from pmd-java/src/test/java/net/sourceforge/pmd/lang/java/cpd/JavaTokenizerTest.java rename to pmd-java/src/test/java/net/sourceforge/pmd/lang/java/cpd/JavaCpdLexerTest.java index 17a873fc1c..edc0449546 100644 --- a/pmd-java/src/test/java/net/sourceforge/pmd/lang/java/cpd/JavaTokenizerTest.java +++ b/pmd-java/src/test/java/net/sourceforge/pmd/lang/java/cpd/JavaCpdLexerTest.java @@ -12,19 +12,19 @@ import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.CpdLanguageProperties; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.cpd.Tokens; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.cpd.test.LanguagePropertyConfig; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; import net.sourceforge.pmd.lang.document.FileId; import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.java.JavaLanguageModule; // TODO - enable tests -class JavaTokenizerTest extends CpdTextComparisonTest { +class JavaCpdLexerTest extends CpdTextComparisonTest { - JavaTokenizerTest() { + JavaCpdLexerTest() { super(JavaLanguageModule.getInstance(), ".java"); } @@ -40,10 +40,10 @@ class JavaTokenizerTest extends CpdTextComparisonTest { @Test void testLexExceptionLocation() { - Tokenizer tokenizer = newTokenizer(defaultProperties()); + CpdLexer cpdLexer = newCpdLexer(defaultProperties()); Tokens tokens = new Tokens(); - TokenMgrError lexException = assertThrows(TokenMgrError.class, () -> - Tokenizer.tokenize(tokenizer, + LexException lexException = assertThrows(LexException.class, () -> + CpdLexer.tokenize(cpdLexer, // note: the source deliberately contains an unbalanced quote, unterminated string literal TextDocument.readOnlyString("class F {\n String s=\"abc\";\"\n}\n", FileId.UNKNOWN, getLanguage().getDefaultVersion()), tokens) diff --git a/pmd-java/src/test/kotlin/net/sourceforge/pmd/lang/java/ast/KotlinTestingDsl.kt b/pmd-java/src/test/kotlin/net/sourceforge/pmd/lang/java/ast/KotlinTestingDsl.kt index b6f0c65ff0..791df7c2a3 100644 --- a/pmd-java/src/test/kotlin/net/sourceforge/pmd/lang/java/ast/KotlinTestingDsl.kt +++ b/pmd-java/src/test/kotlin/net/sourceforge/pmd/lang/java/ast/KotlinTestingDsl.kt @@ -263,7 +263,7 @@ open class ParserTestCtx(testScope: TestScope, Pair(true, null) } catch (e: ParseException) { Pair(false, e) - } catch (e: TokenMgrError) { + } catch (e: LexException) { Pair(false, e) } diff --git a/pmd-javascript/etc/grammar/Ecmascript5.jj b/pmd-javascript/etc/grammar/Ecmascript5.jj index cc1ada0481..85c9d786c1 100644 --- a/pmd-javascript/etc/grammar/Ecmascript5.jj +++ b/pmd-javascript/etc/grammar/Ecmascript5.jj @@ -15,9 +15,6 @@ options { PARSER_BEGIN(Ecmascript5ParserImpl) package net.sourceforge.pmd.lang.ecmascript5.ast; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; - public class Ecmascript5ParserImpl { } diff --git a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/EcmascriptLanguageModule.java b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/EcmascriptLanguageModule.java index 6fd4c2764c..31bc4af4aa 100644 --- a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/EcmascriptLanguageModule.java +++ b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/EcmascriptLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.ecmascript; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.ecmascript.ast.EcmascriptParser; -import net.sourceforge.pmd.lang.ecmascript.cpd.EcmascriptTokenizer; +import net.sourceforge.pmd.lang.ecmascript.cpd.EcmascriptCpdLexer; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; /** @@ -34,7 +34,7 @@ public class EcmascriptLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new EcmascriptTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new EcmascriptCpdLexer(); } } diff --git a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptTokenizer.java b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptCpdLexer.java similarity index 85% rename from pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptTokenizer.java rename to pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptCpdLexer.java index 34a0d42f30..2f8ddf9c61 100644 --- a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptTokenizer.java +++ b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptCpdLexer.java @@ -4,7 +4,7 @@ package net.sourceforge.pmd.lang.ecmascript.cpd; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; @@ -13,8 +13,10 @@ import net.sourceforge.pmd.lang.ecmascript5.ast.Ecmascript5TokenKinds; /** * The Ecmascript Tokenizer + * + *

    Note: This class has been called EcmascriptTokenizer in PMD 6

    . */ -public class EcmascriptTokenizer extends JavaCCTokenizer { +public class EcmascriptCpdLexer extends JavaccCpdLexer { @Override protected TokenManager makeLexerImpl(TextDocument doc) { diff --git a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/TsLanguageModule.java b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/TsLanguageModule.java index 827d42abe8..d2071565c6 100644 --- a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/TsLanguageModule.java +++ b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/TsLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.typescript; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; -import net.sourceforge.pmd.lang.typescript.cpd.TypeScriptTokenizer; +import net.sourceforge.pmd.lang.typescript.cpd.TypeScriptCpdLexer; /** * @author pguyot@kallisys.net @@ -27,7 +27,7 @@ public class TsLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new TypeScriptTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new TypeScriptCpdLexer(); } } diff --git a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptTokenizer.java b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptCpdLexer.java similarity index 78% rename from pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptTokenizer.java rename to pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptCpdLexer.java index f38c7f9509..aa18c60d5b 100644 --- a/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptTokenizer.java +++ b/pmd-javascript/src/main/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptCpdLexer.java @@ -7,10 +7,10 @@ package net.sourceforge.pmd.lang.typescript.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.typescript.ast.TypeScriptLexer; -public class TypeScriptTokenizer extends AntlrTokenizer { +public class TypeScriptCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { return new TypeScriptLexer(charStream); diff --git a/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/AnyTokenizerForTypescriptTest.java b/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/AnyCpdLexerForTypescriptTest.java similarity index 83% rename from pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/AnyTokenizerForTypescriptTest.java rename to pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/AnyCpdLexerForTypescriptTest.java index 4f5ecf87f8..c159efd6e9 100644 --- a/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/AnyTokenizerForTypescriptTest.java +++ b/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/AnyCpdLexerForTypescriptTest.java @@ -12,9 +12,9 @@ import net.sourceforge.pmd.lang.ecmascript.EcmascriptLanguageModule; /** * */ -class AnyTokenizerForTypescriptTest extends CpdTextComparisonTest { +class AnyCpdLexerForTypescriptTest extends CpdTextComparisonTest { - AnyTokenizerForTypescriptTest() { + AnyCpdLexerForTypescriptTest() { super(EcmascriptLanguageModule.getInstance(), ".ts"); } diff --git a/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptTokenizerTest.java b/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptCpdLexerTest.java similarity index 92% rename from pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptTokenizerTest.java rename to pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptCpdLexerTest.java index 32c5c769c0..16b176bf43 100644 --- a/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptTokenizerTest.java +++ b/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/ecmascript/cpd/EcmascriptCpdLexerTest.java @@ -11,9 +11,9 @@ import net.sourceforge.pmd.lang.ecmascript.EcmascriptLanguageModule; -class EcmascriptTokenizerTest extends CpdTextComparisonTest { +class EcmascriptCpdLexerTest extends CpdTextComparisonTest { - EcmascriptTokenizerTest() { + EcmascriptCpdLexerTest() { super(EcmascriptLanguageModule.getInstance(), ".js"); } diff --git a/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptTokenizerTest.java b/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptCpdLexerTest.java similarity index 84% rename from pmd-javascript/src/test/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptTokenizerTest.java rename to pmd-javascript/src/test/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptCpdLexerTest.java index 27ada122a6..aa2d7f84df 100644 --- a/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptTokenizerTest.java +++ b/pmd-javascript/src/test/java/net/sourceforge/pmd/lang/typescript/cpd/TypeScriptCpdLexerTest.java @@ -9,9 +9,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.typescript.TsLanguageModule; -class TypeScriptTokenizerTest extends CpdTextComparisonTest { +class TypeScriptCpdLexerTest extends CpdTextComparisonTest { - TypeScriptTokenizerTest() { + TypeScriptCpdLexerTest() { super(TsLanguageModule.getInstance(), ".ts"); } diff --git a/pmd-jsp/etc/grammar/Jsp.jjt b/pmd-jsp/etc/grammar/Jsp.jjt index 4afb6902af..1f74b3dcf7 100644 --- a/pmd-jsp/etc/grammar/Jsp.jjt +++ b/pmd-jsp/etc/grammar/Jsp.jjt @@ -30,10 +30,6 @@ options { PARSER_BEGIN(JspParserImpl) package net.sourceforge.pmd.lang.jsp.ast; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; -import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; - /** * JSP Parser for PMD. * @author Pieter, Application Engineers NV/SA, http://www.ae.be diff --git a/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/JspLanguageModule.java b/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/JspLanguageModule.java index b7b61443e1..e2f89845ba 100644 --- a/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/JspLanguageModule.java +++ b/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/JspLanguageModule.java @@ -5,11 +5,11 @@ package net.sourceforge.pmd.lang.jsp; import net.sourceforge.pmd.cpd.CpdCapableLanguage; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.jsp.cpd.JSPTokenizer; +import net.sourceforge.pmd.lang.jsp.cpd.JspCpdLexer; /** * Created by christoferdutz on 20.09.14. @@ -31,7 +31,7 @@ public class JspLanguageModule extends SimpleLanguageModuleBase implements CpdCa } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new JSPTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new JspCpdLexer(); } } diff --git a/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/cpd/JSPTokenizer.java b/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/cpd/JspCpdLexer.java similarity index 78% rename from pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/cpd/JSPTokenizer.java rename to pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/cpd/JspCpdLexer.java index 1a6b8044ab..56deb35676 100644 --- a/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/cpd/JSPTokenizer.java +++ b/pmd-jsp/src/main/java/net/sourceforge/pmd/lang/jsp/cpd/JspCpdLexer.java @@ -4,7 +4,7 @@ package net.sourceforge.pmd.lang.jsp.cpd; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; @@ -12,7 +12,10 @@ import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.jsp.ast.JspParser; import net.sourceforge.pmd.lang.jsp.ast.JspTokenKinds; -public class JSPTokenizer extends JavaCCTokenizer { +/** + *

    Note: This class has been called JSPTokenizer in PMD 6

    . + */ +public class JspCpdLexer extends JavaccCpdLexer { @Override protected TokenManager makeLexerImpl(TextDocument doc) { diff --git a/pmd-jsp/src/test/java/net/sourceforge/pmd/lang/jsp/cpd/JSPTokenizerTest.java b/pmd-jsp/src/test/java/net/sourceforge/pmd/lang/jsp/cpd/JspCpdLexerTest.java similarity index 84% rename from pmd-jsp/src/test/java/net/sourceforge/pmd/lang/jsp/cpd/JSPTokenizerTest.java rename to pmd-jsp/src/test/java/net/sourceforge/pmd/lang/jsp/cpd/JspCpdLexerTest.java index d03b31e07f..8eeb08725b 100644 --- a/pmd-jsp/src/test/java/net/sourceforge/pmd/lang/jsp/cpd/JSPTokenizerTest.java +++ b/pmd-jsp/src/test/java/net/sourceforge/pmd/lang/jsp/cpd/JspCpdLexerTest.java @@ -10,9 +10,9 @@ import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.jsp.JspLanguageModule; -class JSPTokenizerTest extends CpdTextComparisonTest { +class JspCpdLexerTest extends CpdTextComparisonTest { - JSPTokenizerTest() { + JspCpdLexerTest() { super(JspLanguageModule.getInstance(), ".jsp"); } diff --git a/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/JuliaLanguageModule.java b/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/JuliaLanguageModule.java index eb02d0eac9..e85b6e0ba1 100644 --- a/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/JuliaLanguageModule.java +++ b/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/JuliaLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.julia; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; -import net.sourceforge.pmd.lang.julia.cpd.JuliaTokenizer; +import net.sourceforge.pmd.lang.julia.cpd.JuliaCpdLexer; /** * Language implementation for Julia. @@ -28,7 +28,7 @@ public class JuliaLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new JuliaTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new JuliaCpdLexer(); } } diff --git a/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/cpd/JuliaTokenizer.java b/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/cpd/JuliaCpdLexer.java similarity index 71% rename from pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/cpd/JuliaTokenizer.java rename to pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/cpd/JuliaCpdLexer.java index 9a38293d5e..b9c3684bfe 100644 --- a/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/cpd/JuliaTokenizer.java +++ b/pmd-julia/src/main/java/net/sourceforge/pmd/lang/julia/cpd/JuliaCpdLexer.java @@ -7,13 +7,15 @@ package net.sourceforge.pmd.lang.julia.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.julia.ast.JuliaLexer; /** * The Julia Tokenizer. + * + *

    Note: This class has been called JuliaTokenizer in PMD 6

    . */ -public class JuliaTokenizer extends AntlrTokenizer { +public class JuliaCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { return new JuliaLexer(charStream); diff --git a/pmd-julia/src/test/java/net/sourceforge/pmd/lang/julia/cpd/JuliaTokenizerTest.java b/pmd-julia/src/test/java/net/sourceforge/pmd/lang/julia/cpd/JuliaCpdLexerTest.java similarity index 83% rename from pmd-julia/src/test/java/net/sourceforge/pmd/lang/julia/cpd/JuliaTokenizerTest.java rename to pmd-julia/src/test/java/net/sourceforge/pmd/lang/julia/cpd/JuliaCpdLexerTest.java index 3c493a5544..3fe59ee2d0 100644 --- a/pmd-julia/src/test/java/net/sourceforge/pmd/lang/julia/cpd/JuliaTokenizerTest.java +++ b/pmd-julia/src/test/java/net/sourceforge/pmd/lang/julia/cpd/JuliaCpdLexerTest.java @@ -9,8 +9,8 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.julia.JuliaLanguageModule; -class JuliaTokenizerTest extends CpdTextComparisonTest { - JuliaTokenizerTest() { +class JuliaCpdLexerTest extends CpdTextComparisonTest { + JuliaCpdLexerTest() { super(JuliaLanguageModule.getInstance(), ".jl"); } diff --git a/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/KotlinLanguageModule.java b/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/KotlinLanguageModule.java index 7a3474d982..f47a870efc 100644 --- a/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/KotlinLanguageModule.java +++ b/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/KotlinLanguageModule.java @@ -5,11 +5,11 @@ package net.sourceforge.pmd.lang.kotlin; import net.sourceforge.pmd.annotation.Experimental; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.kotlin.cpd.KotlinTokenizer; +import net.sourceforge.pmd.lang.kotlin.cpd.KotlinCpdLexer; /** * Language Module for Kotlin @@ -35,7 +35,7 @@ public class KotlinLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new KotlinTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new KotlinCpdLexer(); } } diff --git a/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinTokenizer.java b/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinCpdLexer.java similarity index 92% rename from pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinTokenizer.java rename to pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinCpdLexer.java index 149f7c5940..d1bc7c6416 100644 --- a/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinTokenizer.java +++ b/pmd-kotlin/src/main/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinCpdLexer.java @@ -7,16 +7,18 @@ package net.sourceforge.pmd.lang.kotlin.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.cpd.impl.AntlrTokenFilter; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.antlr4.AntlrToken; import net.sourceforge.pmd.lang.kotlin.ast.KotlinLexer; /** * The Kotlin Tokenizer + * + *

    Note: This class has been called KotlinTokenizer in PMD 6

    . */ -public class KotlinTokenizer extends AntlrTokenizer { +public class KotlinCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { diff --git a/pmd-kotlin/src/test/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinTokenizerTest.java b/pmd-kotlin/src/test/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinCpdLexerTest.java similarity index 86% rename from pmd-kotlin/src/test/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinTokenizerTest.java rename to pmd-kotlin/src/test/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinCpdLexerTest.java index ea3dd542bb..4016548186 100644 --- a/pmd-kotlin/src/test/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinTokenizerTest.java +++ b/pmd-kotlin/src/test/java/net/sourceforge/pmd/lang/kotlin/cpd/KotlinCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class KotlinTokenizerTest extends CpdTextComparisonTest { +class KotlinCpdLexerTest extends CpdTextComparisonTest { - KotlinTokenizerTest() { + KotlinCpdLexerTest() { super("kotlin", ".kt"); } diff --git a/pmd-lang-test/src/main/kotlin/net/sourceforge/pmd/cpd/test/CpdTextComparisonTest.kt b/pmd-lang-test/src/main/kotlin/net/sourceforge/pmd/cpd/test/CpdTextComparisonTest.kt index 5c1b1f1c12..1e15b1ba06 100644 --- a/pmd-lang-test/src/main/kotlin/net/sourceforge/pmd/cpd/test/CpdTextComparisonTest.kt +++ b/pmd-lang-test/src/main/kotlin/net/sourceforge/pmd/cpd/test/CpdTextComparisonTest.kt @@ -6,16 +6,13 @@ package net.sourceforge.pmd.cpd.test import io.kotest.assertions.throwables.shouldThrow import net.sourceforge.pmd.cpd.* -import net.sourceforge.pmd.lang.Language import net.sourceforge.pmd.lang.LanguagePropertyBundle import net.sourceforge.pmd.lang.LanguageRegistry -import net.sourceforge.pmd.lang.ast.TokenMgrError +import net.sourceforge.pmd.lang.ast.LexException import net.sourceforge.pmd.lang.document.TextDocument -import net.sourceforge.pmd.lang.document.TextFile import net.sourceforge.pmd.lang.document.FileId import net.sourceforge.pmd.test.BaseTextComparisonTest import org.apache.commons.lang3.StringUtils -import java.util.* /** * CPD test comparing a dump of a file against a saved baseline. @@ -34,9 +31,9 @@ abstract class CpdTextComparisonTest( extensionIncludingDot ) - fun newTokenizer(config: LanguagePropertyConfig): Tokenizer { + fun newCpdLexer(config: LanguagePropertyConfig): CpdLexer { val properties = language.newPropertyBundle().also { config.setProperties(it) } - return language.createCpdTokenizer(properties) + return language.createCpdLexer(properties) } override val resourceLoader: Class<*> @@ -67,26 +64,26 @@ abstract class CpdTextComparisonTest( config: LanguagePropertyConfig = defaultProperties() ) { super.doTest(fileBaseName, expectedSuffix) { fdata -> - val tokens = tokenize(newTokenizer(config), fdata) + val tokens = tokenize(newCpdLexer(config), fdata) buildString { format(tokens) } } } @JvmOverloads - fun expectTokenMgrError( + fun expectLexException( source: String, fileName: FileId = FileId.UNKNOWN, properties: LanguagePropertyConfig = defaultProperties() - ): TokenMgrError = - expectTokenMgrError(FileData(fileName, source), properties) + ): LexException = + expectLexException(FileData(fileName, source), properties) @JvmOverloads - fun expectTokenMgrError( + fun expectLexException( fileData: FileData, config: LanguagePropertyConfig = defaultProperties() - ): TokenMgrError = + ): LexException = shouldThrow { - tokenize(newTokenizer(config), fileData) + tokenize(newCpdLexer(config), fileData) } @@ -172,10 +169,10 @@ abstract class CpdTextComparisonTest( fun sourceCodeOf(text: String, fileName: FileId = FileId.UNKNOWN): FileData = FileData(fileName = fileName, fileText = text) - fun tokenize(tokenizer: Tokenizer, fileData: FileData): Tokens = + fun tokenize(cpdLexer: CpdLexer, fileData: FileData): Tokens = Tokens().also { tokens -> val source = sourceCodeOf(fileData) - Tokenizer.tokenize(tokenizer, source, tokens) + CpdLexer.tokenize(cpdLexer, source, tokens) } private companion object { diff --git a/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/LuaLanguageModule.java b/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/LuaLanguageModule.java index f28a706003..023a9a01b8 100644 --- a/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/LuaLanguageModule.java +++ b/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/LuaLanguageModule.java @@ -5,11 +5,11 @@ package net.sourceforge.pmd.lang.lua; import net.sourceforge.pmd.cpd.CpdLanguageProperties; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; -import net.sourceforge.pmd.lang.lua.cpd.LuaTokenizer; +import net.sourceforge.pmd.lang.lua.cpd.LuaCpdLexer; /** * @author Clément Fournier @@ -33,7 +33,7 @@ public class LuaLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new LuaTokenizer(bundle); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new LuaCpdLexer(bundle); } } diff --git a/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/cpd/LuaTokenizer.java b/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/cpd/LuaCpdLexer.java similarity index 96% rename from pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/cpd/LuaTokenizer.java rename to pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/cpd/LuaCpdLexer.java index 3222ce1789..fe99fa56d7 100644 --- a/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/cpd/LuaTokenizer.java +++ b/pmd-lua/src/main/java/net/sourceforge/pmd/lang/lua/cpd/LuaCpdLexer.java @@ -8,8 +8,8 @@ import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; import net.sourceforge.pmd.cpd.CpdLanguageProperties; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.cpd.impl.AntlrTokenFilter; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.antlr4.AntlrToken; @@ -17,12 +17,14 @@ import net.sourceforge.pmd.lang.lua.ast.LuaLexer; /** * The Lua Tokenizer + * + *

    Note: This class has been called LuaTokenizer in PMD 6

    . */ -public class LuaTokenizer extends AntlrTokenizer { +public class LuaCpdLexer extends AntlrCpdLexer { private final boolean ignoreLiteralSequences; - public LuaTokenizer(LanguagePropertyBundle bundle) { + public LuaCpdLexer(LanguagePropertyBundle bundle) { ignoreLiteralSequences = bundle.getProperty(CpdLanguageProperties.CPD_IGNORE_LITERAL_SEQUENCES); } diff --git a/pmd-lua/src/test/java/net/sourceforge/pmd/lang/lua/cpd/LuaTokenizerTest.java b/pmd-lua/src/test/java/net/sourceforge/pmd/lang/lua/cpd/LuaCpdLexerTest.java similarity index 88% rename from pmd-lua/src/test/java/net/sourceforge/pmd/lang/lua/cpd/LuaTokenizerTest.java rename to pmd-lua/src/test/java/net/sourceforge/pmd/lang/lua/cpd/LuaCpdLexerTest.java index a550f93818..e41f68035d 100644 --- a/pmd-lua/src/test/java/net/sourceforge/pmd/lang/lua/cpd/LuaTokenizerTest.java +++ b/pmd-lua/src/test/java/net/sourceforge/pmd/lang/lua/cpd/LuaCpdLexerTest.java @@ -8,8 +8,8 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class LuaTokenizerTest extends CpdTextComparisonTest { - LuaTokenizerTest() { +class LuaCpdLexerTest extends CpdTextComparisonTest { + LuaCpdLexerTest() { super("lua", ".lua"); } diff --git a/pmd-matlab/etc/grammar/Matlab.jj b/pmd-matlab/etc/grammar/Matlab.jj index 3ff1c8b27c..d83368fc73 100644 --- a/pmd-matlab/etc/grammar/Matlab.jj +++ b/pmd-matlab/etc/grammar/Matlab.jj @@ -21,9 +21,6 @@ options { PARSER_BEGIN(MatlabParserImpl) package net.sourceforge.pmd.lang.matlab.ast; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; - public class MatlabParserImpl { } diff --git a/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/MatlabLanguageModule.java b/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/MatlabLanguageModule.java index f59ec0c510..74b3d06985 100644 --- a/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/MatlabLanguageModule.java +++ b/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/MatlabLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.matlab; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; -import net.sourceforge.pmd.lang.matlab.cpd.MatlabTokenizer; +import net.sourceforge.pmd.lang.matlab.cpd.MatlabCpdLexer; /** * Defines the Language module for Matlab @@ -29,7 +29,7 @@ public class MatlabLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new MatlabTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new MatlabCpdLexer(); } } diff --git a/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabTokenizer.java b/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabCpdLexer.java similarity index 77% rename from pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabTokenizer.java rename to pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabCpdLexer.java index 269deed52b..cb9c4bfd26 100644 --- a/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabTokenizer.java +++ b/pmd-matlab/src/main/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabCpdLexer.java @@ -4,7 +4,7 @@ package net.sourceforge.pmd.lang.matlab.cpd; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; @@ -13,8 +13,10 @@ import net.sourceforge.pmd.lang.matlab.ast.MatlabTokenKinds; /** * The Matlab Tokenizer. + * + *

    Note: This class has been called MatlabTokenizer in PMD 6

    . */ -public class MatlabTokenizer extends JavaCCTokenizer { +public class MatlabCpdLexer extends JavaccCpdLexer { @Override protected TokenManager makeLexerImpl(TextDocument doc) { diff --git a/pmd-matlab/src/test/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabTokenizerTest.java b/pmd-matlab/src/test/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabCpdLexerTest.java similarity index 90% rename from pmd-matlab/src/test/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabTokenizerTest.java rename to pmd-matlab/src/test/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabCpdLexerTest.java index 68ed5fb0db..be9b58c6f6 100644 --- a/pmd-matlab/src/test/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabTokenizerTest.java +++ b/pmd-matlab/src/test/java/net/sourceforge/pmd/lang/matlab/cpd/MatlabCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class MatlabTokenizerTest extends CpdTextComparisonTest { +class MatlabCpdLexerTest extends CpdTextComparisonTest { - MatlabTokenizerTest() { + MatlabCpdLexerTest() { super("matlab", ".m"); } diff --git a/pmd-modelica/etc/grammar/Modelica.jjt b/pmd-modelica/etc/grammar/Modelica.jjt index d66718dd79..96ecbf0ead 100644 --- a/pmd-modelica/etc/grammar/Modelica.jjt +++ b/pmd-modelica/etc/grammar/Modelica.jjt @@ -49,9 +49,6 @@ options { PARSER_BEGIN(ModelicaParserImpl) package net.sourceforge.pmd.lang.modelica.ast; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; - class ModelicaParserImpl { } diff --git a/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/ModelicaLanguageModule.java b/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/ModelicaLanguageModule.java index 5a3ffd92a3..6baba2eae2 100644 --- a/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/ModelicaLanguageModule.java +++ b/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/ModelicaLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.modelica; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.modelica.cpd.ModelicaTokenizer; +import net.sourceforge.pmd.lang.modelica.cpd.ModelicaCpdLexer; public class ModelicaLanguageModule extends SimpleLanguageModuleBase { private static final String ID = "modelica"; @@ -26,7 +26,7 @@ public class ModelicaLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new ModelicaTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new ModelicaCpdLexer(); } } diff --git a/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/cpd/ModelicaTokenizer.java b/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/cpd/ModelicaCpdLexer.java similarity index 92% rename from pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/cpd/ModelicaTokenizer.java rename to pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/cpd/ModelicaCpdLexer.java index 83d3d3ad9b..48aa2c36cd 100644 --- a/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/cpd/ModelicaTokenizer.java +++ b/pmd-modelica/src/main/java/net/sourceforge/pmd/lang/modelica/cpd/ModelicaCpdLexer.java @@ -5,7 +5,7 @@ package net.sourceforge.pmd.lang.modelica.cpd; import net.sourceforge.pmd.cpd.impl.JavaCCTokenFilter; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; @@ -13,7 +13,10 @@ import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.modelica.ast.ModelicaTokenKinds; -public class ModelicaTokenizer extends JavaCCTokenizer { +/** + *

    Note: This class has been called MatlabTokenizer in PMD 6

    . + */ +public class ModelicaCpdLexer extends JavaccCpdLexer { @Override protected TokenManager makeLexerImpl(TextDocument doc) { diff --git a/pmd-objectivec/etc/grammar/ObjectiveC.jj b/pmd-objectivec/etc/grammar/ObjectiveC.jj index f37b9fe7d2..0826111bb9 100644 --- a/pmd-objectivec/etc/grammar/ObjectiveC.jj +++ b/pmd-objectivec/etc/grammar/ObjectiveC.jj @@ -18,12 +18,6 @@ options { PARSER_BEGIN(ObjectiveCParserImpl) package net.sourceforge.pmd.lang.objectivec.ast; -import java.io.*; -import java.util.*; - -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; - /** * Grammar to parse ObjectiveC 2.0 * @author Michael Hall - Based on Parnel SableCC ObjectiveC grammar and javacc C grammar diff --git a/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/ObjectiveCLanguageModule.java b/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/ObjectiveCLanguageModule.java index ec33354839..0053a51eb0 100644 --- a/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/ObjectiveCLanguageModule.java +++ b/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/ObjectiveCLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.objectivec; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; -import net.sourceforge.pmd.lang.objectivec.cpd.ObjectiveCTokenizer; +import net.sourceforge.pmd.lang.objectivec.cpd.ObjectiveCCpdLexer; /** * Defines the Language module for Objective-C @@ -29,7 +29,7 @@ public class ObjectiveCLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new ObjectiveCTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new ObjectiveCCpdLexer(); } } diff --git a/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCTokenizer.java b/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCCpdLexer.java similarity index 77% rename from pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCTokenizer.java rename to pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCCpdLexer.java index fb17635fce..c88140e02e 100644 --- a/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCTokenizer.java +++ b/pmd-objectivec/src/main/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCCpdLexer.java @@ -4,7 +4,7 @@ package net.sourceforge.pmd.lang.objectivec.cpd; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; @@ -13,8 +13,10 @@ import net.sourceforge.pmd.lang.objectivec.ast.ObjectiveCTokenKinds; /** * The Objective-C Tokenizer + * + *

    Note: This class has been called ObjectiveCTokenizer in PMD 6

    . */ -public class ObjectiveCTokenizer extends JavaCCTokenizer { +public class ObjectiveCCpdLexer extends JavaccCpdLexer { @Override protected TokenManager makeLexerImpl(TextDocument doc) { diff --git a/pmd-objectivec/src/test/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCTokenizerTest.java b/pmd-objectivec/src/test/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCCpdLexerTest.java similarity index 86% rename from pmd-objectivec/src/test/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCTokenizerTest.java rename to pmd-objectivec/src/test/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCCpdLexerTest.java index 33dffcf64b..5f8a923cea 100644 --- a/pmd-objectivec/src/test/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCTokenizerTest.java +++ b/pmd-objectivec/src/test/java/net/sourceforge/pmd/lang/objectivec/cpd/ObjectiveCCpdLexerTest.java @@ -8,10 +8,10 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class ObjectiveCTokenizerTest extends CpdTextComparisonTest { +class ObjectiveCCpdLexerTest extends CpdTextComparisonTest { - ObjectiveCTokenizerTest() { + ObjectiveCCpdLexerTest() { super("objectivec", ".m"); } diff --git a/pmd-perl/src/main/java/net/sourceforge/pmd/lang/perl/PerlLanguageModule.java b/pmd-perl/src/main/java/net/sourceforge/pmd/lang/perl/PerlLanguageModule.java index 0bcdc2919a..0ea67d6e14 100644 --- a/pmd-perl/src/main/java/net/sourceforge/pmd/lang/perl/PerlLanguageModule.java +++ b/pmd-perl/src/main/java/net/sourceforge/pmd/lang/perl/PerlLanguageModule.java @@ -4,8 +4,8 @@ package net.sourceforge.pmd.lang.perl; -import net.sourceforge.pmd.cpd.AnyTokenizer; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.AnyCpdLexer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; @@ -22,7 +22,7 @@ public class PerlLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new AnyTokenizer("#"); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new AnyCpdLexer("#"); } } diff --git a/pmd-perl/src/test/java/net/sourceforge/pmd/lang/perl/cpd/PerlTokenizerTest.java b/pmd-perl/src/test/java/net/sourceforge/pmd/lang/perl/cpd/PerlCpdLexerTest.java similarity index 80% rename from pmd-perl/src/test/java/net/sourceforge/pmd/lang/perl/cpd/PerlTokenizerTest.java rename to pmd-perl/src/test/java/net/sourceforge/pmd/lang/perl/cpd/PerlCpdLexerTest.java index 1c42757a11..bb39dcdc3f 100644 --- a/pmd-perl/src/test/java/net/sourceforge/pmd/lang/perl/cpd/PerlTokenizerTest.java +++ b/pmd-perl/src/test/java/net/sourceforge/pmd/lang/perl/cpd/PerlCpdLexerTest.java @@ -11,9 +11,9 @@ import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; /** * */ -class PerlTokenizerTest extends CpdTextComparisonTest { +class PerlCpdLexerTest extends CpdTextComparisonTest { - PerlTokenizerTest() { + PerlCpdLexerTest() { super("perl", ".pl"); } diff --git a/pmd-php/src/main/java/net/sourceforge/pmd/lang/php/PhpLanguageModule.java b/pmd-php/src/main/java/net/sourceforge/pmd/lang/php/PhpLanguageModule.java index 4abd4f1e4c..a0ec8430fd 100644 --- a/pmd-php/src/main/java/net/sourceforge/pmd/lang/php/PhpLanguageModule.java +++ b/pmd-php/src/main/java/net/sourceforge/pmd/lang/php/PhpLanguageModule.java @@ -4,8 +4,8 @@ package net.sourceforge.pmd.lang.php; -import net.sourceforge.pmd.cpd.AnyTokenizer; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.AnyCpdLexer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; @@ -25,7 +25,7 @@ public class PhpLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new AnyTokenizer("#|//"); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new AnyCpdLexer("#|//"); } } diff --git a/pmd-plsql/etc/grammar/PLSQL.jjt b/pmd-plsql/etc/grammar/PLSQL.jjt index 7aaf681e52..51be215856 100644 --- a/pmd-plsql/etc/grammar/PLSQL.jjt +++ b/pmd-plsql/etc/grammar/PLSQL.jjt @@ -157,13 +157,11 @@ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA package net.sourceforge.pmd.lang.plsql.ast; -import java.util.List; -import java.util.ArrayList; -import net.sourceforge.pmd.lang.ast.Node; import net.sourceforge.pmd.lang.document.Chars; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.plsql.ast.internal.ParsingExclusion; -import net.sourceforge.pmd.lang.ast.TokenMgrError; + +import java.util.ArrayList; +import java.util.List; public class PLSQLParserImpl { diff --git a/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/PLSQLLanguageModule.java b/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/PLSQLLanguageModule.java index 85f7203367..f5c51bc3d8 100644 --- a/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/PLSQLLanguageModule.java +++ b/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/PLSQLLanguageModule.java @@ -5,11 +5,11 @@ package net.sourceforge.pmd.lang.plsql; import net.sourceforge.pmd.cpd.CpdLanguageProperties; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.plsql.cpd.PLSQLTokenizer; +import net.sourceforge.pmd.lang.plsql.cpd.PLSQLCpdLexer; /** * Created by christoferdutz on 20.09.14. @@ -56,7 +56,7 @@ public class PLSQLLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new PLSQLTokenizer(bundle); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new PLSQLCpdLexer(bundle); } } diff --git a/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLTokenizer.java b/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLCpdLexer.java similarity index 89% rename from pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLTokenizer.java rename to pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLCpdLexer.java index ab67934112..c31a11d644 100644 --- a/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLTokenizer.java +++ b/pmd-plsql/src/main/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLCpdLexer.java @@ -5,7 +5,7 @@ package net.sourceforge.pmd.lang.plsql.cpd; import net.sourceforge.pmd.cpd.CpdLanguageProperties; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; @@ -13,12 +13,15 @@ import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.plsql.ast.PLSQLTokenKinds; -public class PLSQLTokenizer extends JavaCCTokenizer { +/** + *

    Note: This class has been called PLSQLTokenizer in PMD 6

    . + */ +public class PLSQLCpdLexer extends JavaccCpdLexer { private final boolean ignoreIdentifiers; private final boolean ignoreLiterals; - public PLSQLTokenizer(LanguagePropertyBundle properties) { + public PLSQLCpdLexer(LanguagePropertyBundle properties) { /* * The Tokenizer is derived from PLDoc, in which comments are very * important When looking for duplication, we are probably not diff --git a/pmd-plsql/src/test/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLTokenizerTest.java b/pmd-plsql/src/test/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLCpdLexerTest.java similarity index 87% rename from pmd-plsql/src/test/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLTokenizerTest.java rename to pmd-plsql/src/test/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLCpdLexerTest.java index 8247c98abd..60015818b2 100644 --- a/pmd-plsql/src/test/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLTokenizerTest.java +++ b/pmd-plsql/src/test/java/net/sourceforge/pmd/lang/plsql/cpd/PLSQLCpdLexerTest.java @@ -9,9 +9,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.plsql.PLSQLLanguageModule; -class PLSQLTokenizerTest extends CpdTextComparisonTest { +class PLSQLCpdLexerTest extends CpdTextComparisonTest { - PLSQLTokenizerTest() { + PLSQLCpdLexerTest() { super(PLSQLLanguageModule.getInstance(), ".sql"); } diff --git a/pmd-python/etc/grammar/Python.jj b/pmd-python/etc/grammar/Python.jj index d68de80c23..bbb0788d9b 100644 --- a/pmd-python/etc/grammar/Python.jj +++ b/pmd-python/etc/grammar/Python.jj @@ -17,9 +17,6 @@ PARSER_BEGIN(PythonParserImpl) package net.sourceforge.pmd.lang.python.ast; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; - public class PythonParserImpl { } diff --git a/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/PythonLanguageModule.java b/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/PythonLanguageModule.java index 0049b86b2b..997d5a33b7 100644 --- a/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/PythonLanguageModule.java +++ b/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/PythonLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.python; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; -import net.sourceforge.pmd.lang.python.cpd.PythonTokenizer; +import net.sourceforge.pmd.lang.python.cpd.PythonCpdLexer; /** * Defines the Language module for Python @@ -25,7 +25,7 @@ public class PythonLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new PythonTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new PythonCpdLexer(); } } diff --git a/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/cpd/PythonTokenizer.java b/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/cpd/PythonCpdLexer.java similarity index 89% rename from pmd-python/src/main/java/net/sourceforge/pmd/lang/python/cpd/PythonTokenizer.java rename to pmd-python/src/main/java/net/sourceforge/pmd/lang/python/cpd/PythonCpdLexer.java index 49c301a128..aa72ea27a8 100644 --- a/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/cpd/PythonTokenizer.java +++ b/pmd-python/src/main/java/net/sourceforge/pmd/lang/python/cpd/PythonCpdLexer.java @@ -6,7 +6,7 @@ package net.sourceforge.pmd.lang.python.cpd; import java.util.regex.Pattern; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; @@ -16,8 +16,10 @@ import net.sourceforge.pmd.lang.python.ast.PythonTokenKinds; /** * The Python tokenizer. + * + *

    Note: This class has been called PythonTokenizer in PMD 6

    . */ -public class PythonTokenizer extends JavaCCTokenizer { +public class PythonCpdLexer extends JavaccCpdLexer { private static final Pattern STRING_NL_ESCAPE = Pattern.compile("\\\\\\r?\\n"); diff --git a/pmd-python/src/test/java/net/sourceforge/pmd/lang/python/cpd/PythonTokenizerTest.java b/pmd-python/src/test/java/net/sourceforge/pmd/lang/python/cpd/PythonCpdLexerTest.java similarity index 89% rename from pmd-python/src/test/java/net/sourceforge/pmd/lang/python/cpd/PythonTokenizerTest.java rename to pmd-python/src/test/java/net/sourceforge/pmd/lang/python/cpd/PythonCpdLexerTest.java index 61357c8b1c..9dc1305c2f 100644 --- a/pmd-python/src/test/java/net/sourceforge/pmd/lang/python/cpd/PythonTokenizerTest.java +++ b/pmd-python/src/test/java/net/sourceforge/pmd/lang/python/cpd/PythonCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class PythonTokenizerTest extends CpdTextComparisonTest { +class PythonCpdLexerTest extends CpdTextComparisonTest { - PythonTokenizerTest() { + PythonCpdLexerTest() { super("python", ".py"); } diff --git a/pmd-ruby/src/main/java/net/sourceforge/pmd/lang/ruby/RubyLanguageModule.java b/pmd-ruby/src/main/java/net/sourceforge/pmd/lang/ruby/RubyLanguageModule.java index df8202101a..e9f9858b3f 100644 --- a/pmd-ruby/src/main/java/net/sourceforge/pmd/lang/ruby/RubyLanguageModule.java +++ b/pmd-ruby/src/main/java/net/sourceforge/pmd/lang/ruby/RubyLanguageModule.java @@ -4,8 +4,8 @@ package net.sourceforge.pmd.lang.ruby; -import net.sourceforge.pmd.cpd.AnyTokenizer; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.AnyCpdLexer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; @@ -27,7 +27,7 @@ public class RubyLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new AnyTokenizer("#"); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new AnyCpdLexer("#"); } } diff --git a/pmd-ruby/src/test/java/net/sourceforge/pmd/lang/ruby/cpd/RubyTokenizerTest.java b/pmd-ruby/src/test/java/net/sourceforge/pmd/lang/ruby/cpd/RubyCpdLexerTest.java similarity index 83% rename from pmd-ruby/src/test/java/net/sourceforge/pmd/lang/ruby/cpd/RubyTokenizerTest.java rename to pmd-ruby/src/test/java/net/sourceforge/pmd/lang/ruby/cpd/RubyCpdLexerTest.java index 9f34d472fd..871dcbb0c2 100644 --- a/pmd-ruby/src/test/java/net/sourceforge/pmd/lang/ruby/cpd/RubyTokenizerTest.java +++ b/pmd-ruby/src/test/java/net/sourceforge/pmd/lang/ruby/cpd/RubyCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class RubyTokenizerTest extends CpdTextComparisonTest { +class RubyCpdLexerTest extends CpdTextComparisonTest { - RubyTokenizerTest() { + RubyCpdLexerTest() { super("ruby", ".rb"); } diff --git a/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/cpd/ScalaTokenizer.java b/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/cpd/ScalaCpdLexer.java similarity index 96% rename from pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/cpd/ScalaTokenizer.java rename to pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/cpd/ScalaCpdLexer.java index d2be64ec60..ff442e45e3 100644 --- a/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/cpd/ScalaTokenizer.java +++ b/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/cpd/ScalaCpdLexer.java @@ -23,15 +23,17 @@ import scala.meta.tokens.Token; /** * Scala Tokenizer class. Uses the Scala Meta Tokenizer, but adapts it for use with generic filtering + * + *

    Note: This class has been called ScalaTokenizer in PMD 6

    . */ -public class ScalaTokenizer implements Tokenizer { +public class ScalaCpdLexer implements CpdLexer { private final Dialect dialect; /** * Create the Tokenizer using properties from the system environment. */ - public ScalaTokenizer(LanguagePropertyBundle bundle) { + public ScalaCpdLexer(LanguagePropertyBundle bundle) { LanguageVersion langVer = bundle.getLanguageVersion(); dialect = ScalaLanguageModule.dialectOf(langVer); } diff --git a/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/lang/scala/ScalaLanguageModule.java b/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/lang/scala/ScalaLanguageModule.java index 0488df2644..d2fc7105bf 100644 --- a/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/lang/scala/ScalaLanguageModule.java +++ b/pmd-scala-modules/pmd-scala-common/src/main/java/net/sourceforge/pmd/lang/scala/ScalaLanguageModule.java @@ -7,8 +7,8 @@ package net.sourceforge.pmd.lang.scala; import org.checkerframework.checker.nullness.qual.NonNull; import net.sourceforge.pmd.annotation.InternalApi; -import net.sourceforge.pmd.cpd.ScalaTokenizer; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; +import net.sourceforge.pmd.cpd.ScalaCpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.LanguageVersion; @@ -52,7 +52,7 @@ public class ScalaLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new ScalaTokenizer(bundle); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new ScalaCpdLexer(bundle); } } diff --git a/pmd-scala-modules/pmd-scala-common/src/test/java/net/sourceforge/pmd/cpd/ScalaTokenizerTest.java b/pmd-scala-modules/pmd-scala-common/src/test/java/net/sourceforge/pmd/cpd/ScalaCpdLexerTest.java similarity index 79% rename from pmd-scala-modules/pmd-scala-common/src/test/java/net/sourceforge/pmd/cpd/ScalaTokenizerTest.java rename to pmd-scala-modules/pmd-scala-common/src/test/java/net/sourceforge/pmd/cpd/ScalaCpdLexerTest.java index fc48791566..b4ac2eff13 100644 --- a/pmd-scala-modules/pmd-scala-common/src/test/java/net/sourceforge/pmd/cpd/ScalaTokenizerTest.java +++ b/pmd-scala-modules/pmd-scala-common/src/test/java/net/sourceforge/pmd/cpd/ScalaCpdLexerTest.java @@ -9,12 +9,12 @@ import static org.junit.jupiter.api.Assertions.assertThrows; import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -import net.sourceforge.pmd.lang.ast.TokenMgrError; +import net.sourceforge.pmd.lang.ast.LexException; import net.sourceforge.pmd.lang.scala.ScalaLanguageModule; -class ScalaTokenizerTest extends CpdTextComparisonTest { +class ScalaCpdLexerTest extends CpdTextComparisonTest { - ScalaTokenizerTest() { + ScalaCpdLexerTest() { super(ScalaLanguageModule.getInstance(), ".scala"); } @@ -35,7 +35,7 @@ class ScalaTokenizerTest extends CpdTextComparisonTest { @Test void tokenizeFailTest() { - assertThrows(TokenMgrError.class, () -> doTest("unlexable_sample")); + assertThrows(LexException.class, () -> doTest("unlexable_sample")); } @Test diff --git a/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/SwiftLanguageModule.java b/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/SwiftLanguageModule.java index bda47e0ba5..b62613ba66 100644 --- a/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/SwiftLanguageModule.java +++ b/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/SwiftLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.swift; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.swift.cpd.SwiftTokenizer; +import net.sourceforge.pmd.lang.swift.cpd.SwiftCpdLexer; /** * Language Module for Swift @@ -41,7 +41,7 @@ public class SwiftLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new SwiftTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new SwiftCpdLexer(); } } diff --git a/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftTokenizer.java b/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftCpdLexer.java similarity index 71% rename from pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftTokenizer.java rename to pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftCpdLexer.java index ae4ed9d5b0..ee255ad0ee 100644 --- a/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftTokenizer.java +++ b/pmd-swift/src/main/java/net/sourceforge/pmd/lang/swift/cpd/SwiftCpdLexer.java @@ -7,13 +7,15 @@ package net.sourceforge.pmd.lang.swift.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.swift.ast.SwiftLexer; /** * SwiftTokenizer + * + *

    Note: This class has been called SwiftTokenizer in PMD 6

    . */ -public class SwiftTokenizer extends AntlrTokenizer { +public class SwiftCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(final CharStream charStream) { diff --git a/pmd-swift/src/test/java/net/sourceforge/pmd/lang/swift/cpd/SwiftTokenizerTest.java b/pmd-swift/src/test/java/net/sourceforge/pmd/lang/swift/cpd/SwiftCpdLexerTest.java similarity index 92% rename from pmd-swift/src/test/java/net/sourceforge/pmd/lang/swift/cpd/SwiftTokenizerTest.java rename to pmd-swift/src/test/java/net/sourceforge/pmd/lang/swift/cpd/SwiftCpdLexerTest.java index e3a3c19ab8..414cc3273d 100644 --- a/pmd-swift/src/test/java/net/sourceforge/pmd/lang/swift/cpd/SwiftTokenizerTest.java +++ b/pmd-swift/src/test/java/net/sourceforge/pmd/lang/swift/cpd/SwiftCpdLexerTest.java @@ -9,9 +9,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.swift.SwiftLanguageModule; -class SwiftTokenizerTest extends CpdTextComparisonTest { +class SwiftCpdLexerTest extends CpdTextComparisonTest { - SwiftTokenizerTest() { + SwiftCpdLexerTest() { super(SwiftLanguageModule.getInstance(), ".swift"); } diff --git a/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/TSqlLanguageModule.java b/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/TSqlLanguageModule.java index 5dbd3e88ef..944394f8b4 100644 --- a/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/TSqlLanguageModule.java +++ b/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/TSqlLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.tsql; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.CpdOnlyLanguageModuleBase; -import net.sourceforge.pmd.lang.tsql.cpd.TSqlTokenizer; +import net.sourceforge.pmd.lang.tsql.cpd.TSqlCpdLexer; /** * @author pguyot@kallisys.net @@ -25,7 +25,7 @@ public class TSqlLanguageModule extends CpdOnlyLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new TSqlTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new TSqlCpdLexer(); } } diff --git a/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlTokenizer.java b/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlCpdLexer.java similarity index 70% rename from pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlTokenizer.java rename to pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlCpdLexer.java index 9676b37f7c..8d9b403f85 100644 --- a/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlTokenizer.java +++ b/pmd-tsql/src/main/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlCpdLexer.java @@ -7,10 +7,13 @@ package net.sourceforge.pmd.lang.tsql.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.tsql.ast.TSqlLexer; -public class TSqlTokenizer extends AntlrTokenizer { +/** + *

    Note: This class has been called TSqlTokenizer in PMD 6

    . + */ +public class TSqlCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { diff --git a/pmd-tsql/src/test/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlTokenizerTest.java b/pmd-tsql/src/test/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlCpdLexerTest.java similarity index 83% rename from pmd-tsql/src/test/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlTokenizerTest.java rename to pmd-tsql/src/test/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlCpdLexerTest.java index 79f3d42e41..3c2f19a322 100644 --- a/pmd-tsql/src/test/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlTokenizerTest.java +++ b/pmd-tsql/src/test/java/net/sourceforge/pmd/lang/tsql/cpd/TSqlCpdLexerTest.java @@ -8,9 +8,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class TSqlTokenizerTest extends CpdTextComparisonTest { +class TSqlCpdLexerTest extends CpdTextComparisonTest { - TSqlTokenizerTest() { + TSqlCpdLexerTest() { super("tsql", ".sql"); } diff --git a/pmd-visualforce/etc/grammar/Vf.jjt b/pmd-visualforce/etc/grammar/Vf.jjt index b15e8a8f03..09ce8bb1b3 100644 --- a/pmd-visualforce/etc/grammar/Vf.jjt +++ b/pmd-visualforce/etc/grammar/Vf.jjt @@ -13,9 +13,6 @@ options { PARSER_BEGIN(VfParserImpl) package net.sourceforge.pmd.lang.vf.ast; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; -import net.sourceforge.pmd.lang.ast.TokenMgrError; - public class VfParserImpl { diff --git a/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/VfLanguageModule.java b/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/VfLanguageModule.java index f596902c2b..7baeea127c 100644 --- a/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/VfLanguageModule.java +++ b/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/VfLanguageModule.java @@ -5,12 +5,12 @@ package net.sourceforge.pmd.lang.vf; import net.sourceforge.pmd.cpd.CpdCapableLanguage; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.apex.ApexLanguageModule; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.vf.cpd.VfTokenizer; +import net.sourceforge.pmd.lang.vf.cpd.VfCpdLexer; /** * @author sergey.gorbaty @@ -32,8 +32,8 @@ public class VfLanguageModule extends SimpleLanguageModuleBase implements CpdCap } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new VfTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new VfCpdLexer(); } @Override diff --git a/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/cpd/VfTokenizer.java b/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/cpd/VfCpdLexer.java similarity index 88% rename from pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/cpd/VfTokenizer.java rename to pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/cpd/VfCpdLexer.java index f7c669333b..4fc1a99109 100644 --- a/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/cpd/VfTokenizer.java +++ b/pmd-visualforce/src/main/java/net/sourceforge/pmd/lang/vf/cpd/VfCpdLexer.java @@ -4,7 +4,7 @@ package net.sourceforge.pmd.lang.vf.cpd; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaEscapeTranslator; @@ -16,9 +16,10 @@ import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.vf.ast.VfTokenKinds; /** + *

    Note: This class has been called VfTokenizer in PMD 6

    . * @author sergey.gorbaty */ -public class VfTokenizer extends JavaCCTokenizer { +public class VfCpdLexer extends JavaccCpdLexer { @Override protected TokenManager makeLexerImpl(TextDocument doc) { diff --git a/pmd-visualforce/src/test/java/net/sourceforge/pmd/lang/vf/cpd/VfTokenizerTest.java b/pmd-visualforce/src/test/java/net/sourceforge/pmd/lang/vf/cpd/VfCpdLexerTest.java similarity index 81% rename from pmd-visualforce/src/test/java/net/sourceforge/pmd/lang/vf/cpd/VfTokenizerTest.java rename to pmd-visualforce/src/test/java/net/sourceforge/pmd/lang/vf/cpd/VfCpdLexerTest.java index 0e67555e1e..3c42acc6c9 100644 --- a/pmd-visualforce/src/test/java/net/sourceforge/pmd/lang/vf/cpd/VfTokenizerTest.java +++ b/pmd-visualforce/src/test/java/net/sourceforge/pmd/lang/vf/cpd/VfCpdLexerTest.java @@ -9,9 +9,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; -class VfTokenizerTest extends CpdTextComparisonTest { +class VfCpdLexerTest extends CpdTextComparisonTest { - VfTokenizerTest() { + VfCpdLexerTest() { super("vf", ".page"); } diff --git a/pmd-vm/etc/grammar/Vm.jjt b/pmd-vm/etc/grammar/Vm.jjt index 5492f2b98e..aee65bcc95 100644 --- a/pmd-vm/etc/grammar/Vm.jjt +++ b/pmd-vm/etc/grammar/Vm.jjt @@ -37,16 +37,11 @@ PARSER_BEGIN(VmParserImpl) package net.sourceforge.pmd.lang.vm.ast; import java.io.IOException; -import java.io.Reader; import java.util.ArrayList; import java.util.List; -import java.util.HashMap; -import java.util.Map; import org.apache.commons.lang3.StringUtils; -import net.sourceforge.pmd.lang.ast.TokenMgrError; -import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; diff --git a/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/VmLanguageModule.java b/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/VmLanguageModule.java index b441d472ae..7f1ddd8544 100644 --- a/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/VmLanguageModule.java +++ b/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/VmLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.vm; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.vm.cpd.VmTokenizer; +import net.sourceforge.pmd.lang.vm.cpd.VmCpdLexer; /** * Created by christoferdutz on 20.09.14. @@ -32,7 +32,7 @@ public class VmLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new VmTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new VmCpdLexer(); } } diff --git a/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/cpd/VmTokenizer.java b/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/cpd/VmCpdLexer.java similarity index 78% rename from pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/cpd/VmTokenizer.java rename to pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/cpd/VmCpdLexer.java index d8d29ff173..9a2b0a78ba 100644 --- a/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/cpd/VmTokenizer.java +++ b/pmd-vm/src/main/java/net/sourceforge/pmd/lang/vm/cpd/VmCpdLexer.java @@ -6,14 +6,17 @@ package net.sourceforge.pmd.lang.vm.cpd; import java.io.IOException; -import net.sourceforge.pmd.cpd.impl.JavaCCTokenizer; +import net.sourceforge.pmd.cpd.impl.JavaccCpdLexer; import net.sourceforge.pmd.lang.TokenManager; import net.sourceforge.pmd.lang.ast.impl.javacc.CharStream; import net.sourceforge.pmd.lang.ast.impl.javacc.JavaccToken; import net.sourceforge.pmd.lang.document.TextDocument; import net.sourceforge.pmd.lang.vm.ast.VmTokenKinds; -public class VmTokenizer extends JavaCCTokenizer { +/** + *

    Note: This class has been called VmTokenizer in PMD 6

    . + */ +public class VmCpdLexer extends JavaccCpdLexer { @Override protected TokenManager makeLexerImpl(TextDocument doc) throws IOException { return VmTokenKinds.newTokenManager(CharStream.create(doc)); diff --git a/pmd-vm/src/test/java/net/sourceforge/pmd/lang/vm/cpd/VmTokenizerTest.java b/pmd-vm/src/test/java/net/sourceforge/pmd/lang/vm/cpd/VmCpdLexerTest.java similarity index 84% rename from pmd-vm/src/test/java/net/sourceforge/pmd/lang/vm/cpd/VmTokenizerTest.java rename to pmd-vm/src/test/java/net/sourceforge/pmd/lang/vm/cpd/VmCpdLexerTest.java index 98e018586c..3c7eebefaf 100644 --- a/pmd-vm/src/test/java/net/sourceforge/pmd/lang/vm/cpd/VmTokenizerTest.java +++ b/pmd-vm/src/test/java/net/sourceforge/pmd/lang/vm/cpd/VmCpdLexerTest.java @@ -9,8 +9,8 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.vm.VmLanguageModule; -class VmTokenizerTest extends CpdTextComparisonTest { - VmTokenizerTest() { +class VmCpdLexerTest extends CpdTextComparisonTest { + VmCpdLexerTest() { super(VmLanguageModule.getInstance(), ".vm"); } diff --git a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/pom/PomLanguageModule.java b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/pom/PomLanguageModule.java index 315a61efe3..8031b73394 100644 --- a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/pom/PomLanguageModule.java +++ b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/pom/PomLanguageModule.java @@ -4,12 +4,12 @@ package net.sourceforge.pmd.lang.pom; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; import net.sourceforge.pmd.lang.xml.XmlHandler; -import net.sourceforge.pmd.lang.xml.cpd.XmlTokenizer; +import net.sourceforge.pmd.lang.xml.cpd.XmlCpdLexer; public class PomLanguageModule extends SimpleLanguageModuleBase { private static final String ID = "pom"; @@ -26,7 +26,7 @@ public class PomLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new XmlTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new XmlCpdLexer(); } } diff --git a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/wsdl/WsdlLanguageModule.java b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/wsdl/WsdlLanguageModule.java index aa617d1dd4..1ae01b9bf9 100644 --- a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/wsdl/WsdlLanguageModule.java +++ b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/wsdl/WsdlLanguageModule.java @@ -4,12 +4,12 @@ package net.sourceforge.pmd.lang.wsdl; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; import net.sourceforge.pmd.lang.xml.XmlHandler; -import net.sourceforge.pmd.lang.xml.cpd.XmlTokenizer; +import net.sourceforge.pmd.lang.xml.cpd.XmlCpdLexer; /** * Created by bernardo-macedo on 24.06.15. @@ -30,7 +30,7 @@ public class WsdlLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new XmlTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new XmlCpdLexer(); } } diff --git a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/XmlLanguageModule.java b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/XmlLanguageModule.java index 83e81136d3..4b3e30929c 100644 --- a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/XmlLanguageModule.java +++ b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/XmlLanguageModule.java @@ -4,11 +4,11 @@ package net.sourceforge.pmd.lang.xml; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; -import net.sourceforge.pmd.lang.xml.cpd.XmlTokenizer; +import net.sourceforge.pmd.lang.xml.cpd.XmlCpdLexer; /** * Created by christoferdutz on 20.09.14. @@ -29,7 +29,7 @@ public class XmlLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new XmlTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new XmlCpdLexer(); } } diff --git a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/cpd/XmlTokenizer.java b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/cpd/XmlCpdLexer.java similarity index 69% rename from pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/cpd/XmlTokenizer.java rename to pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/cpd/XmlCpdLexer.java index 5b24f1ef1f..62b077ab4b 100644 --- a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/cpd/XmlTokenizer.java +++ b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xml/cpd/XmlCpdLexer.java @@ -7,10 +7,13 @@ package net.sourceforge.pmd.lang.xml.cpd; import org.antlr.v4.runtime.CharStream; import org.antlr.v4.runtime.Lexer; -import net.sourceforge.pmd.cpd.impl.AntlrTokenizer; +import net.sourceforge.pmd.cpd.impl.AntlrCpdLexer; import net.sourceforge.pmd.lang.xml.antlr4.XMLLexer; -public class XmlTokenizer extends AntlrTokenizer { +/** + *

    Note: This class has been called XmlTokenizer in PMD 6

    . + */ +public class XmlCpdLexer extends AntlrCpdLexer { @Override protected Lexer getLexerForSource(CharStream charStream) { diff --git a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xsl/XslLanguageModule.java b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xsl/XslLanguageModule.java index 13884bd243..6bbdbed637 100644 --- a/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xsl/XslLanguageModule.java +++ b/pmd-xml/src/main/java/net/sourceforge/pmd/lang/xsl/XslLanguageModule.java @@ -4,12 +4,12 @@ package net.sourceforge.pmd.lang.xsl; -import net.sourceforge.pmd.cpd.Tokenizer; +import net.sourceforge.pmd.cpd.CpdLexer; import net.sourceforge.pmd.lang.LanguagePropertyBundle; import net.sourceforge.pmd.lang.LanguageRegistry; import net.sourceforge.pmd.lang.impl.SimpleLanguageModuleBase; import net.sourceforge.pmd.lang.xml.XmlHandler; -import net.sourceforge.pmd.lang.xml.cpd.XmlTokenizer; +import net.sourceforge.pmd.lang.xml.cpd.XmlCpdLexer; /** * Created by christoferdutz on 20.09.14. @@ -31,7 +31,7 @@ public class XslLanguageModule extends SimpleLanguageModuleBase { } @Override - public Tokenizer createCpdTokenizer(LanguagePropertyBundle bundle) { - return new XmlTokenizer(); + public CpdLexer createCpdLexer(LanguagePropertyBundle bundle) { + return new XmlCpdLexer(); } } diff --git a/pmd-xml/src/test/java/net/sourceforge/pmd/lang/xml/cpd/XmlCPDTokenizerTest.java b/pmd-xml/src/test/java/net/sourceforge/pmd/lang/xml/cpd/XmlCPDCpdLexerTest.java similarity index 82% rename from pmd-xml/src/test/java/net/sourceforge/pmd/lang/xml/cpd/XmlCPDTokenizerTest.java rename to pmd-xml/src/test/java/net/sourceforge/pmd/lang/xml/cpd/XmlCPDCpdLexerTest.java index 5c4e25aa1d..fa94b5f390 100644 --- a/pmd-xml/src/test/java/net/sourceforge/pmd/lang/xml/cpd/XmlCPDTokenizerTest.java +++ b/pmd-xml/src/test/java/net/sourceforge/pmd/lang/xml/cpd/XmlCPDCpdLexerTest.java @@ -9,9 +9,9 @@ import org.junit.jupiter.api.Test; import net.sourceforge.pmd.cpd.test.CpdTextComparisonTest; import net.sourceforge.pmd.lang.xml.XmlLanguageModule; -class XmlCPDTokenizerTest extends CpdTextComparisonTest { +class XmlCPDCpdLexerTest extends CpdTextComparisonTest { - XmlCPDTokenizerTest() { + XmlCPDCpdLexerTest() { super(XmlLanguageModule.getInstance(), ".xml"); }