The Linguist-defined language IDs are important to our use case because they are
used as database identifiers. This adds a new generator to extract the language
IDs into a map and uses that to implement GetLanguageID.
Because one language has the ID 0, there is no way to tell if a language name is
found or not. If desired, we could add this by returning (string, bool) from
GetLanguageID. But none of the other functions that take language names do this,
so I didn't want to introduce it here.
As Git on win does not support symlinks [1], we have to hard-code
the paths to fils under ./samples/ in Linguist codebase that are
known to be a symlink.
1. https://github.com/git-for-windows/git/wiki/Symbolic-Links
TestPlan:
- go test ./internal/code-generator/generator -run Test_GeneratorTestSuite
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
It seems that reading ./samples/ from Linguist consumes
a different number of files from filesystem on different OSes.
This change adds ENRY_DEBUG env var to print some debug output
about calculations of token stats from samples.
TestPlan:
- ENRY_DEBUG=1 go test -v ./internal/code-generator/generator \
-run Test_GeneratorTestSuite -testify.m TestGenerationFiles
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
On Win `make code-generate` produces unreasonable
Bayesian classifier weights from Linguist samples
silently, failing only the final classification tests.
TestPlan:
- go test ./internal/code-generator/... \
-run Test_GeneratorTestSuite -testify.m TestGenerationFiles
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
Reason is that gofmt can change between versions e.g
see https://go-review.googlesource.com/c/go/+/122295/
and this would avoid breaking tests and edit wars
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
Sync \w Github Linguist v7.2.0
Includes new way of handling `heuristics.yml` and
all `./data/*` re-generated using Github Linguist [v7.2.0](https://github.com/github/linguist/releases/tag/v7.2.0)
release tag.
- many new languages
- better vendoring detection
- update doc on update&known issues.
Instead of use a command to change imports before build, using a build tag to generate the correct binary.
This will allow applications to compile enry using oniguruma with less troubles.
Signed-off-by: Antonio Jesus Navarro Perez <antnavper@gmail.com>
This allows to use a cached directory with linguist instead of cloning and speeds up the tests by -10s on my local machine.
Signed-off-by: Vadim Markovtsev <vadim@sourced.tech>
renamed tmpLinguist to repoLinguist and SimpleLinguistTestSuite to EnryTestSuit in common_test.go
changed receiver's name for TestSuites to 's'
fixed comments
