Closes#17
Implements the IsGenerated helper function to filter out generated
files using the rules and matchers in:
- https://github.com/github/linguist/blob/master/lib/linguist/generated.rb
Since the vast majority of matchers have very different logic, it cannot
be autogenerated directly from linguist like other logics in enry, so it's
translated by hand.
There are three different types of matchers in this implementation:
- By extension, which mark as generated based only in the extension. These
are the fastest matchers, so they're done first.
- By file name, which matches patterns against the filename. These
are performed in second place. Unlike linguist, we try to use string
functions instead of regexps as much as possible.
- Finally, the rest of the matchers, which go into the content and try
to identify if they're generated or not based on the content. Unlike
linguist, we try to only read the content we need and not split it
all unless it's necessary and use byte functions instead of regexps
as much as possible.
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
This mitigates the problem that tokenizer uses regex
that matches platform-specific line endings
TestPlan:
- go test ./internal/code-generator/generator \
-run Test_GeneratorTestSuite -testify.m TestTokenizerOnATS
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
As Git on win does not support symlinks [1], we have to hard-code
the paths to fils under ./samples/ in Linguist codebase that are
known to be a symlink.
1. https://github.com/git-for-windows/git/wiki/Symbolic-Links
TestPlan:
- go test ./internal/code-generator/generator -run Test_GeneratorTestSuite
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
It seems that reading ./samples/ from Linguist consumes
a different number of files from filesystem on different OSes.
This change adds ENRY_DEBUG env var to print some debug output
about calculations of token stats from samples.
TestPlan:
- ENRY_DEBUG=1 go test -v ./internal/code-generator/generator \
-run Test_GeneratorTestSuite -testify.m TestGenerationFiles
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
On Win `make code-generate` produces unreasonable
Bayesian classifier weights from Linguist samples
silently, failing only the final classification tests.
TestPlan:
- go test ./internal/code-generator/... \
-run Test_GeneratorTestSuite -testify.m TestGenerationFiles
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
TestPlan:
- go test -run TestTokenize ./internal/tokenizer
- go test -tags flex -run TestTokenize ./internal/tokenizer
(shold fail as default fixtures are from regex-based tokenizer)
Even more reduces public API surface by
hiding un-used Classifier API for providing
a pre-trained classifier weights.
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
