tartrazine/internal/tokenizer
M. J. Fromberger 169060e1cd Add a test that tokenization does not modify the input.
At present this test fails, since the tokenizer replaces text in shared slices
of the input. A subsequent commit will fix that.

Signed-off-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
2019-01-29 10:03:09 -08:00
..
tokenize_test.go Add a test that tokenization does not modify the input. 2019-01-29 10:03:09 -08:00
tokenize.go Refactor Oniguruma integration 2018-08-29 18:01:13 +03:00