The same optimization still happens during package initialization
at runtime, but an effort was made to make it more transparent and
self-documented.
Both the test & the benchmark were updated.
Old version (usefull for benchmark) was
```go
func IsVendor(filename string) bool {
for _, matcher := range data.VendorMatchers {
if matcher.MatchString(filename) {
return true
}
}
return false
}
```
Test plan:
* go test -run ^TestIsVendor$ github.com/go-enry/go-enry/v2
The regex generation introduced by
20726a1de3
had a subtle but important bug in it. In particular the ^ and ^|/
prefixes were only applied to the first regex in the OR'd groupings and
not every regex in the grouping.
We fix this by wrapping each of those OR'd groups in a new non capturing
group to ensure the prefix applies to every entry in the OR'd listing.
Additionally we add three new quick tests of IsVendor to guard against
this regression in the future.
This fixes https://github.com/go-enry/go-enry/issues/135
Although iterating across the regexps is quicker than naively concatenating them,
it is still quite slow.
This PR proposes a slightly cleverer solution.
First instead of just concatenating with groups this PR uses non-capturing groups.
This speeds up the regexp processing.
Secondly we group the regexps in to 3 groups - those that have to be at the start,
those that are segments or at the start and the rest. This makes a considerable speed
improvement.
Thirdly the regexps are sorted within those groups - which also speeds things up.
All in all for a non-vendored file this makes IsVendor around twice as fast.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Closes#17
Implements the IsGenerated helper function to filter out generated
files using the rules and matchers in:
- https://github.com/github/linguist/blob/master/lib/linguist/generated.rb
Since the vast majority of matchers have very different logic, it cannot
be autogenerated directly from linguist like other logics in enry, so it's
translated by hand.
There are three different types of matchers in this implementation:
- By extension, which mark as generated based only in the extension. These
are the fastest matchers, so they're done first.
- By file name, which matches patterns against the filename. These
are performed in second place. Unlike linguist, we try to use string
functions instead of regexps as much as possible.
- Finally, the rest of the matchers, which go into the content and try
to identify if they're generated or not based on the content. Unlike
linguist, we try to only read the content we need and not split it
all unless it's necessary and use byte functions instead of regexps
as much as possible.
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Refactoring, consisting of
- remove unused method `isAuxiliaryLanguage` and `FileCountList`
in order to reduce public API surfaces (go/java)
- add GoDoc to public APIs
- ci: java profile use latest go src
It also now mimics https://docs.travis-ci.com/user/languages/go/#go-import-path
for non-go build image, as code relies on internal imports.
TEST PLAN:
- make test
Adds `FileCount` and `FileCountList` types for storing language file and
their count which can be sorted based on the value of count.
Signed-off-by: Sunny <me@darkowlzz.space>
