Stark, Tokens specification and Tokenizer implementation

added by DotNetKicks
2/7/2017 5:29:37 PM

1 Kicks, 370 Views

In this post, we are going to go through the implementation of the language tokenizer, which is the first, most basic and simplest part of a compiler pipeline, yet fundamental. I couldn't resist also to include some benchmarks between a handwritten tokenizer and the equivalent ANTLR generated lexer, so that this post will not just look plain theoretical.