Lexing of `protected/`endprotected sections can never complete
`protected `endprotected sections are supported by a text callback in the Parser, however these may be VERY large in practice.
The lexer (in both preprocessing and parsing) uses yymore() to concatenate the data inside a `protected block to a single token. However once you have ~3Mb+ of encrypted data between `protected and `endprotected statements, I believe the repeated concatenations to the single token begin to degrade performance, it's not impossible to have enough encrypted data (10's-100's of Mb) that the preprocessing will never complete.
I think the solution would be to not concatenate the entire section into a VP_TEXT token and instead create many VP_TEXT tokens and ignore them incrementally if they come in. But I think this has consequences for the rest of the parser that become quite ugly (and would ultimately change how Verilog::Parser callbacks operate).
As a workaround I just remove these sections prior to handing text to the Preprocessor/Parser/etc.. but for other use-cases discarding this data would be incorrect.
#2 Updated by Wilson Snyder almost 5 years ago
- Subject changed from Lexing(?) of `protected/`endprotected sections can never complete to Lexing of `protected/`endprotected sections can never complete
- Status changed from New to Resolved
- Assignee set to Wilson Snyder
Fixed in git towards 3.413.
The preprocessor had a similar issue, for that returning it line by line is fine.
For the parser I added a use_protected argument, which is 1 for backward compatibility, but vhier and the SigParser don't need protected regions so set use_protected = 0.
Also available in: Atom