Although iterating across the regexps is quicker than naively concatenating them,
it is still quite slow.
This PR proposes a slightly cleverer solution.
First instead of just concatenating with groups this PR uses non-capturing groups.
This speeds up the regexp processing.
Secondly we group the regexps in to 3 groups - those that have to be at the start,
those that are segments or at the start and the rest. This makes a considerable speed
improvement.
Thirdly the regexps are sorted within those groups - which also speeds things up.
All in all for a non-vendored file this makes IsVendor around twice as fast.
Signed-off-by: Andrew Thornton <art27@cantab.net>
This file doesn't appear to be used any more, since the builds are run using
GitHub Actions.
This file is affected by the recent Codecov Bash Uploader exploit[1], but since it
hasn't been running, I don't think the project is affected.
[1] https://about.codecov.io/security-update/
Prior to this change, GetLanguages collected all candidate languages from each
strategy to pass to the next strategy (without de-duplicating them). Linguist
only uses the previous strategy's candidates for the next strategy. Also, it
would overwrite languages with nil if a strategy returned that, so you could get
into a situation where you go from multiple languages to no language.
See the Ruby code for details: aad49acc06/lib/linguist.rb (L14-L49)
This addresses https://github.com/src-d/enry/issues/207 because GetLanguages
should not return all candidates detected, otherwise it would work differently
than Linguist.
The Linguist-defined language IDs are important to our use case because they are
used as database identifiers. This adds a new generator to extract the language
IDs into a map and uses that to implement GetLanguageID.
Because one language has the ID 0, there is no way to tell if a language name is
found or not. If desired, we could add this by returning (string, bool) from
GetLanguageID. But none of the other functions that take language names do this,
so I didn't want to introduce it here.
Closes#17
Implements the IsGenerated helper function to filter out generated
files using the rules and matchers in:
- https://github.com/github/linguist/blob/master/lib/linguist/generated.rb
Since the vast majority of matchers have very different logic, it cannot
be autogenerated directly from linguist like other logics in enry, so it's
translated by hand.
There are three different types of matchers in this implementation:
- By extension, which mark as generated based only in the extension. These
are the fastest matchers, so they're done first.
- By file name, which matches patterns against the filename. These
are performed in second place. Unlike linguist, we try to use string
functions instead of regexps as much as possible.
- Finally, the rest of the matchers, which go into the content and try
to identify if they're generated or not based on the content. Unlike
linguist, we try to only read the content we need and not split it
all unless it's necessary and use byte functions instead of regexps
as much as possible.
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
