As discussed in https://github.com/go-enry/go-enry/issues/54, this provides an
API for accessing a LanguageInfo struct which is populated with all the data
from the Linguist YAML source file. Functions are provided to access the
LanguageInfo by name or ID.
The other top-level functions like GetLanguageExtensions, GetLanguageGroup, etc.
could in principle be implemented using this structure, which would simplify the
code generation. But that would be a big change so I didn't do any of that.
Perhaps in the next major version something like that would make sense.
The Linguist-defined language IDs are important to our use case because they are
used as database identifiers. This adds a new generator to extract the language
IDs into a map and uses that to implement GetLanguageID.
Because one language has the ID 0, there is no way to tell if a language name is
found or not. If desired, we could add this by returning (string, bool) from
GetLanguageID. But none of the other functions that take language names do this,
so I didn't want to introduce it here.
As Git on win does not support symlinks [1], we have to hard-code
the paths to fils under ./samples/ in Linguist codebase that are
known to be a symlink.
1. https://github.com/git-for-windows/git/wiki/Symbolic-Links
TestPlan:
- go test ./internal/code-generator/generator -run Test_GeneratorTestSuite
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
It seems that reading ./samples/ from Linguist consumes
a different number of files from filesystem on different OSes.
This change adds ENRY_DEBUG env var to print some debug output
about calculations of token stats from samples.
TestPlan:
- ENRY_DEBUG=1 go test -v ./internal/code-generator/generator \
-run Test_GeneratorTestSuite -testify.m TestGenerationFiles
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
On Win `make code-generate` produces unreasonable
Bayesian classifier weights from Linguist samples
silently, failing only the final classification tests.
TestPlan:
- go test ./internal/code-generator/... \
-run Test_GeneratorTestSuite -testify.m TestGenerationFiles
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
TestPlan:
- go test -run TestTokenize ./internal/tokenizer
- go test -tags flex -run TestTokenize ./internal/tokenizer
(shold fail as default fixtures are from regex-based tokenizer)
Reason is that gofmt can change between versions e.g
see https://go-review.googlesource.com/c/go/+/122295/
and this would avoid breaking tests and edit wars
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
Sync \w Github Linguist v7.2.0
Includes new way of handling `heuristics.yml` and
all `./data/*` re-generated using Github Linguist [v7.2.0](https://github.com/github/linguist/releases/tag/v7.2.0)
release tag.
- many new languages
- better vendoring detection
- update doc on update&known issues.
Although this package is internal, it still exports an API and deserves some
comments. Serves in partial satisfaction of #195.
Signed-off-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
Addresses #196. Several of the tokenizer's processing steps wind up editing the
source, and we don't want those changes to be observed by the caller, which may
use the source for other purposes afterward.
Signed-off-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
At present this test fails, since the tokenizer replaces text in shared slices
of the input. A subsequent commit will fix that.
Signed-off-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
Instead of use a command to change imports before build, using a build tag to generate the correct binary.
This will allow applications to compile enry using oniguruma with less troubles.
Signed-off-by: Antonio Jesus Navarro Perez <antnavper@gmail.com>
This change names the dependency like its called. The link to the
package was correct, but all other references were renamed where I could
find time with git grep.
Signed-off-by: Zeger-Jan van de Weg <git@zjvandeweg.nl>
This allows to use a cached directory with linguist instead of cloning and speeds up the tests by -10s on my local machine.
Signed-off-by: Vadim Markovtsev <vadim@sourced.tech>
renamed tmpLinguist to repoLinguist and SimpleLinguistTestSuite to EnryTestSuit in common_test.go
changed receiver's name for TestSuites to 's'
fixed comments
fixed autogenerated comment
changed constant types names
GetLanguageByShebang doesn't print errors
languageInfo struct change to have only necessary fields
GetLanguageByShebang has a comment now
