M. J. Fromberger 7d277b11de Copy the tokenizer input to avoid modifying the caller's copy. ... Addresses #196. Several of the tokenizer's processing steps wind up editing the source, and we don't want those changes to be observed by the caller, which may use the source for other purposes afterward. Signed-off-by: M. J. Fromberger <michael.j.fromberger@gmail.com>		2019-01-29 10:12:33 -08:00
..
tokenize_test.go	Add a test that tokenization does not modify the input.	2019-01-29 10:03:09 -08:00
tokenize.go	Copy the tokenizer input to avoid modifying the caller's copy.	2019-01-29 10:12:33 -08:00