ralsina/tartrazine
1
0
Fork 0
mirror of https://github.com/ralsina/tartrazine.git synced 2025-08-15 15:00:17 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
Files
b41b4e14fe454144d7d4fc43996698e8f6797dc7
tartrazine/internal/tokenizer
History
Alexander Bezzubov 78d8f43a88 tokenizer: hide flex-based impl, avoid build failures on win 
TestPlan:
 - go test -run TestTokenize ./internal/tokenizer
 - go test -tags flex -run TestTokenize ./internal/tokenizer
   (shold fail as default fixtures are from regex-based tokenizer)
2020-03-19 19:58:48 +01:00
..
flex
tokenizer: hide flex-based impl, avoid build failures on win
2020-03-19 19:58:48 +01:00
common.go
address review feedback
2019-04-16 19:38:48 +02:00
tokenize_c.go
*: module rename to go-enry/go-enry/v4
2020-03-19 17:31:29 +01:00
tokenize_test.go
tokenizer: hide flex-based impl, avoid build failures on win
2020-03-19 19:58:48 +01:00
tokenize.go
tokenizer: fix a bug and regenerate the code \w latest Go
2020-03-19 19:08:21 +01:00