ralsina/tartrazine
1
0
Fork 0
mirror of https://github.com/ralsina/tartrazine.git synced 2025-06-19 06:33:06 -03:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

address review feedback

Browse Source 
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
This commit is contained in:
Alexander Bezzubov
2019-04-14 22:15:18 +02:00
parent 7929933eb5
commit ada6f15c93
5 changed files with 22 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
internal/tokenizer/common.go
View File

@ -3,4 +3,5 @@
// be imported by other packages.
package tokenizer
const byteLimit = 100000
// ByteLimit defines the maximum prefix of an input text that will be tokenized.
const ByteLimit = 100000