The '--files' command line option of CPD lets you specify which
directories and files should be scanned for duplicated code.
Unfortunately it didn't work when you specified files instead of
directories, for example: '--files foo.c bar.c'. In this example CPD
executed successful, but the files 'foo.c' and 'bar.c' are completely
ignored.
The following 2 code snippets could not be tokenized:
1: if (*pbuf == '\0x05'), the problem is the '\0' in the character
literal '\0x05'.
2: szPath = m_sdcacheDir + _T("\ oMedia");, the problem is the '\ '
in the string literal "\ oMedia".
I relaxed the lexical grammar so a '\' (backslash) can escape any
character inside a string or character literal. We can relax the grammar
because CPD only needs the tokens, so it is no problem to accept
'invalid' string / character literals. (according to the ANSI C
standard). Failing too fast because the tokenizer is too strict is
annoying because then we can't check the files for duplicated code.
Both snippets were taken from existing projects and be successfully
compiled, so for some C / C++ compilers it is valid code.
This commit fixes a lexical error when a C++ file contains an empty
character literal L''. The following code could not be tokenized:
std::wstring wsMessage( sMessage.length(), L'');
It triggers the following error:
net.sourceforge.pmd.lang.ast.TokenMgrError: Lexical error in file
'foo.cpp' at line 1, column 46. Encountered: "\'" (39), after : "\'"
This commit fixes a lexical eror when a preprocessor directive is
followed by a single line comment that starts with "//*".
The following code could not be tokenized:
#define LSTFVLES_CPP //*
It triggers the following error:
net.sourceforge.pmd.lang.ast.TokenMgrError: Lexical error in file
'foo.cpp' at line 2, column 0. Encountered: <EOF> after : ""
This commit fixes a lexical error when a C++ file contains ASM with the
'@' character. The following code cannot be tokenized and triggers a
lexical error:
asm void eSPI_boot()
{
// setup stack pointer
lis r1, _stack_addr@h
ori r1, r1, _stack_addr@l
}
The error that occurs:
net.sourceforge.pmd.lang.ast.TokenMgrError: Lexical error in file
CODE_LOADED_FROM_STRING at line 4, column 22. Encountered: "@" (64),
after : ""
The Scala tokenizer was extracted from the Sonar Scala plugin.
(https://github.com/SonarCommunity/sonar-scala) I copied the code
because the Sonar Scala plugin has a lot of unwanted dependencies. I
tried to keep the changes to the Scala Sonar plugin as minimal as
possible.
To tokenize the source files the official Scala compiler is used.
