Storing tokens for bootstrapping #45
Description
When the compiler is bootstrapping, it has to regenerate all of the tokens each time it iterates. This could be avoided if the tokens were collected at first, then in each iteration the tokens were reused. This would improve the compilation time.
However, there should also be an option to make the compiler recreate the tokens every time. This is important because if the lexer was changed that would require the tokens to be recollected at every iteration which would otherwise create problems.