ralsina/tartrazine
1
0
Fork 0
mirror of https://github.com/ralsina/tartrazine.git synced 2025-07-12 04:09:48 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

tokenizer: move flex-based to modules

Browse Source 
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
This commit is contained in:
Alexander Bezzubov
2019-04-17 13:54:34 +02:00
parent ae43e1a91f
commit 278eaf1c22
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
internal/tokenizer/tokenize_c.go
View File

@ -2,7 +2,7 @@
package tokenizer package tokenizer
import "gopkg.in/src-d/enry.v1/internal/tokenizer/flex" import "github.com/src-d/enry/v2/internal/tokenizer/flex"
// Tokenize returns lexical tokens from content. The tokens returned match what // Tokenize returns lexical tokens from content. The tokens returned match what
// the Linguist library returns. At most the first ByteLimit bytes of content are tokenized. // the Linguist library returns. At most the first ByteLimit bytes of content are tokenized.