M. J. Fromberger 169060e1cd Add a test that tokenization does not modify the input. ...

At present this test fails, since the tokenizer replaces text in shared slices
of the input. A subsequent commit will fix that.

Signed-off-by: M. J. Fromberger <michael.j.fromberger@gmail.com>

2019-01-29 10:03:09 -08:00

..

code-generator

Refactor Oniguruma integration

2018-08-29 18:01:13 +03:00

tokenizer

Add a test that tokenization does not modify the input.

2019-01-29 10:03:09 -08:00