ralsina/tartrazine
1
0
Fork 0
mirror of https://github.com/ralsina/tartrazine.git synced 2025-06-19 14:43:05 -03:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

tokenizer: cleanup & attributions

Browse Source 
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
This commit is contained in:
Alexander Bezzubov
2019-04-08 16:52:26 +02:00
parent 8756fbdcb4
commit 7929933eb5
4 changed files with 4 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
internal/tokenizer/tokenize_test.go
View File

@ -132,9 +132,3 @@ func BenchmarkTokenizer(b *testing.B) {
}
}
}
//TODO(bzz): introduce tokenizer benchmark suit
// baseline - just read the files
// RE2
// oniguruma
// cgo to flex-based impl