The Linguist-defined language IDs are important to our use case because they are
used as database identifiers. This adds a new generator to extract the language
IDs into a map and uses that to implement GetLanguageID.
Because one language has the ID 0, there is no way to tell if a language name is
found or not. If desired, we could add this by returning (string, bool) from
GetLanguageID. But none of the other functions that take language names do this,
so I didn't want to introduce it here.
Closes#17
Implements the IsGenerated helper function to filter out generated
files using the rules and matchers in:
- https://github.com/github/linguist/blob/master/lib/linguist/generated.rb
Since the vast majority of matchers have very different logic, it cannot
be autogenerated directly from linguist like other logics in enry, so it's
translated by hand.
There are three different types of matchers in this implementation:
- By extension, which mark as generated based only in the extension. These
are the fastest matchers, so they're done first.
- By file name, which matches patterns against the filename. These
are performed in second place. Unlike linguist, we try to use string
functions instead of regexps as much as possible.
- Finally, the rest of the matchers, which go into the content and try
to identify if they're generated or not based on the content. Unlike
linguist, we try to only read the content we need and not split it
all unless it's necessary and use byte functions instead of regexps
as much as possible.
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Even more reduces public API surface by
hiding un-used Classifier API for providing
a pre-trained classifier weights.
Signed-off-by: Alexander Bezzubov <bzz@apache.org>
Sync \w Github Linguist v7.2.0
Includes new way of handling `heuristics.yml` and
all `./data/*` re-generated using Github Linguist [v7.2.0](https://github.com/github/linguist/releases/tag/v7.2.0)
release tag.
- many new languages
- better vendoring detection
- update doc on update&known issues.
Refactoring, consisting of
- remove unused method `isAuxiliaryLanguage` and `FileCountList`
in order to reduce public API surfaces (go/java)
- add GoDoc to public APIs
- ci: java profile use latest go src
It also now mimics https://docs.travis-ci.com/user/languages/go/#go-import-path
for non-go build image, as code relies on internal imports.
TEST PLAN:
- make test
Instead of use a command to change imports before build, using a build tag to generate the correct binary.
This will allow applications to compile enry using oniguruma with less troubles.
Signed-off-by: Antonio Jesus Navarro Perez <antnavper@gmail.com>
renamed tmpLinguist to repoLinguist and SimpleLinguistTestSuite to EnryTestSuit in common_test.go
changed receiver's name for TestSuites to 's'
fixed comments
