Jan van Nunen c92dc9706c Relaxed C++ tokenizer so a '\'(backslash) can be use to escape any character inside a string or character. ...

The following 2 code snippets could not be tokenized:

1:  if (*pbuf == '\0x05'), the problem is the '\0' in the character
literal '\0x05'.
2:  szPath = m_sdcacheDir + _T("\    oMedia");, the problem is the '\ '
in the string literal "\    oMedia".

I relaxed the lexical grammar so a '\' (backslash) can escape any
character inside a string or character literal. We can relax the grammar
because CPD only needs the tokens, so it is no problem to accept
'invalid' string / character literals. (according to the ANSI C
standard).  Failing too fast because the tokenizer is too strict is
annoying because then we can't check the files for duplicated code.

Both snippets were taken from existing projects and be successfully
compiled, so for some C / C++ compilers it is valid code.

2015-01-13 11:36:54 +01:00

..

cpp.jj

Relaxed C++ tokenizer so a '\'(backslash) can be use to escape any character inside a string or character.

2015-01-13 11:36:54 +01:00