4 Commits

Author	SHA1	Message	Date
Alexander Bezzubov	78d8f43a88	tokenizer: hide flex-based impl, avoid build failures on win ... TestPlan: - go test -run TestTokenize ./internal/tokenizer - go test -tags flex -run TestTokenize ./internal/tokenizer (shold fail as default fixtures are from regex-based tokenizer)	2020-03-19 19:58:48 +01:00
Alexander Bezzubov	ada6f15c93	address review feedback ... Signed-off-by: Alexander Bezzubov <bzz@apache.org>	2019-04-16 19:38:48 +02:00
Alexander Bezzubov	7929933eb5	tokenizer: cleanup & attributions ... Signed-off-by: Alexander Bezzubov <bzz@apache.org>	2019-04-14 21:38:16 +02:00
Alexander Bezzubov	8756fbdcb4	refactor to build tags ... Signed-off-by: Alexander Bezzubov <bzz@apache.org>	2019-04-14 21:38:16 +02:00