There was already an existing unit test for issue #1559. This unit test broke with this change, because the code it's attempting to tokenize is now valid. This test was slightly modified to still give an error, and a new test was added to verify that the code is now correctly tokenized, even in a more difficult situation.
Standardizing with StandardCharsets.UTF_8, given that it was used in
several places of code already, and it provides compile-time checking
(as opposed to "UTF-8")
Due to `skipBlocks = true` and `skipBlocksStart` being uninitialized in `CPPTokenizer`, `maybeSkipBlocks` would always throw a NullPointerException as it is assuming that `setProperties` is always called after the constructor.
Changed `CPPLanguage` to always invoke `setProperties`, similarly to `CsLanguage`.
This only happens in programmatic invocations.
Fixes#431
Note: the tokens separate by such continuation characters are not
merged and appear as two separate tokens, which could lead to false
negatives in duplication detection.
ContinuationReader is deleted, since skipping characters in the stream
makes it difficult to provide accurate line numbers/positions.
The following 2 code snippets could not be tokenized:
1: if (*pbuf == '\0x05'), the problem is the '\0' in the character
literal '\0x05'.
2: szPath = m_sdcacheDir + _T("\ oMedia");, the problem is the '\ '
in the string literal "\ oMedia".
I relaxed the lexical grammar so a '\' (backslash) can escape any
character inside a string or character literal. We can relax the grammar
because CPD only needs the tokens, so it is no problem to accept
'invalid' string / character literals. (according to the ANSI C
standard). Failing too fast because the tokenizer is too strict is
annoying because then we can't check the files for duplicated code.
Both snippets were taken from existing projects and be successfully
compiled, so for some C / C++ compilers it is valid code.
This commit fixes a lexical error when a C++ file contains an empty
character literal L''. The following code could not be tokenized:
std::wstring wsMessage( sMessage.length(), L'');
It triggers the following error:
net.sourceforge.pmd.lang.ast.TokenMgrError: Lexical error in file
'foo.cpp' at line 1, column 46. Encountered: "\'" (39), after : "\'"
This commit fixes a lexical eror when a preprocessor directive is
followed by a single line comment that starts with "//*".
The following code could not be tokenized:
#define LSTFVLES_CPP //*
It triggers the following error:
net.sourceforge.pmd.lang.ast.TokenMgrError: Lexical error in file
'foo.cpp' at line 2, column 0. Encountered: <EOF> after : ""
This commit fixes a lexical error when a C++ file contains ASM with the
'@' character. The following code cannot be tokenized and triggers a
lexical error:
asm void eSPI_boot()
{
// setup stack pointer
lis r1, _stack_addr@h
ori r1, r1, _stack_addr@l
}
The error that occurs:
net.sourceforge.pmd.lang.ast.TokenMgrError: Lexical error in file
CODE_LOADED_FROM_STRING at line 4, column 22. Encountered: "@" (64),
after : ""
