ralsina/tartrazine
1
0
Fork 0
mirror of https://github.com/ralsina/tartrazine.git synced 2025-06-19 06:33:06 -03:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Sync to linguist 7.2.0: heuristics.yml support (#189)

Browse Source 
Sync \w Github Linguist v7.2.0

Includes new way of handling `heuristics.yml` and
all `./data/*` re-generated using Github Linguist [v7.2.0](https://github.com/github/linguist/releases/tag/v7.2.0)
release tag.

 - many new languages
 - better vendoring detection
 - update doc on update&known issues.
This commit is contained in:
Alexander
2019-02-14 12:47:45 +01:00
committed by GitHub
parent 13d3d66d37
commit 3499750785
45 changed files with 105155 additions and 74316 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,61 @@
# source{d} Contributing Guidelines
source{d} projects accept contributions via GitHub pull requests.
This document outlines some of the
conventions on development workflow, commit message formatting, contact points,
and other resources to make it easier to get your contribution accepted.
## Certificate of Origin
By contributing to this project, you agree to the [Developer Certificate of
Origin (DCO)](DCO). This document was created by the Linux Kernel community and is a
simple statement that you, as a contributor, have the legal right to make the
contribution.
In order to show your agreement with the DCO you should include at the end of the commit message,
the following line: `Signed-off-by: John Doe <john.doe@example.com>`, using your real name.
This can be done easily using the [`-s`](https://github.com/git/git/blob/b2c150d3aa82f6583b9aadfecc5f8fa1c74aca09/Documentation/git-commit.txt#L154-L161) flag on the `git commit`.
If you find yourself pushed a few commits without `Signed-off-by`, you can still add it afterwards. We wrote a manual which can help: [fix-DCO.md](https://github.com/src-d/guide/blob/master/developer-community/fix-DCO.md).
## Support Channels
The official support channels, for both users and contributors, are:
- GitHub issues: each repository has its own list of issues.
- Slack: join the [source{d} Slack](https://join.slack.com/t/sourced-community/shared_invite/enQtMjc4Njk5MzEyNzM2LTFjNzY4NjEwZGEwMzRiNTM4MzRlMzQ4MmIzZjkwZmZlM2NjODUxZmJjNDI1OTcxNDAyMmZlNmFjODZlNTg0YWM) community.
*Before opening a new issue or submitting a new pull request, it's helpful to
search the project - it's likely that another user has already reported the
issue you're facing, or it's a known issue that we're already aware of.
## How to Contribute
Pull Requests (PRs) are the main and exclusive way to contribute code to source{d} projects.
In order for a PR to be accepted it needs to pass this list of requirements:
- The contribution must be correctly explained with natural language and providing a minimum working example that reproduces it.
- All PRs must be written idiomaticly:
- for Go: formatted according to [gofmt](https://golang.org/cmd/gofmt/), and without any warnings from [go lint](https://github.com/golang/lint) nor [go vet](https://golang.org/cmd/vet/)
- for other languages, similar constraints apply.
- They should in general include tests, and those shall pass.
- If the PR is a bug fix, it has to include a new unit test that fails before the patch is merged.
- If the PR is a new feature, it has to come with a suite of unit tests, that tests the new functionality.
- In any case, all the PRs have to pass the personal evaluation of at least one of the [maintainers](MAINTAINERS) of the project.
### Format of the commit message
Every commit message should describe what was changed, under which context and, if applicable, the GitHub issue it relates to:
```
plumbing: packp, Skip argument validations for unknown capabilities. Fixes #623
```
The format can be described more formally as follows:
```
<package>: <subpackage>, <what changed>. [Fixes #<issue-number>]
```

5
Makefile
View File

@ -46,6 +46,11 @@ clean: clean-linguist clean-shared
code-generate: $(LINGUIST_PATH) code-generate: $(LINGUIST_PATH)
mkdir -p data && \ mkdir -p data && \
go run internal/code-generator/main.go go run internal/code-generator/main.go
ENRY_TEST_REPO="$${PWD}/.linguist" go test -v \
-run Test_GeneratorTestSuite \
./internal/code-generator/generator \
-testify.m TestUpdateGeneratorTestSuiteGold \
-update_gold
benchmarks: $(LINGUIST_PATH) benchmarks: $(LINGUIST_PATH)
go test -run=NONE -bench=. && \ go test -run=NONE -bench=. && \

21
README.md
View File

@ -154,14 +154,17 @@ Generated Java bindings using a C-shared library and JNI are located under [`jav
Development Development
------------ ------------
*enry* re-uses parts of original [linguist](https://github.com/github/linguist) to generate internal data structures. In order to update to the latest upstream and generate the necessary code you must run: *enry* re-uses parts of original [linguist](https://github.com/github/linguist) to generate internal data structures. In order to update to the latest upstream and generate all the necessary code you must run:
git clone https://github.com/github/linguist.git .linguist
# update commit in generator_test.go (to re-generate .gold fixtures)
# https://github.com/src-d/enry/blob/13d3d66d37a87f23a013246a1b0678c9ee3d524b/internal/code-generator/generator/generator_test.go#L18
go generate go generate
We update enry when changes are done in linguist's master branch on the following files: We update enry when changes are done in linguist's master branch on the following files:
* [languages.yml](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml) * [languages.yml](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml)
* [heuristics.rb](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb) * [heuristics.yml](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.yml)
* [vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml) * [vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml)
* [documentation.yml](https://github.com/github/linguist/blob/master/lib/linguist/documentation.yml) * [documentation.yml](https://github.com/github/linguist/blob/master/lib/linguist/documentation.yml)
@ -183,17 +186,11 @@ Divergences from linguist
Using [linguist/samples](https://github.com/github/linguist/tree/master/samples) Using [linguist/samples](https://github.com/github/linguist/tree/master/samples)
as a set for the tests, the following issues were found: as a set for the tests, the following issues were found:
* With [hello.ms](https://github.com/github/linguist/blob/master/samples/Unix%20Assembly/hello.ms) we can't detect the language (Unix Assembly) because we don't have a matcher in contentMatchers (content.go) for Unix Assembly. Linguist uses this [regexp](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb#L300) in its code, * [Heuristics for ".es" extension](https://github.com/github/linguist/blob/e761f9b013e5b61161481fcb898b59721ee40e3d/lib/linguist/heuristics.yml#L103) in JavaScript could not be parsed, due to unsupported backreference in RE2 regexp engine
`elsif /(?<!\S)\.(include|globa?l)\s/.match(data) || /(?<!\/\*)(\A|\n)\s*\.[A-Za-z][_A-Za-z0-9]*:/.match(data.gsub(/"([^\\"]|\\.)*"|'([^\\']|\\.)*'|\\\s*(?:--.*)?\n/, ""))` * As of (Linguist v5.3.2)[https://github.com/github/linguist/releases/tag/v5.3.2] it is using [flex-based scanner in C for tokenization](https://github.com/github/linguist/pull/3846). Enry stil uses [extract_token](https://github.com/github/linguist/pull/3846/files#diff-d5179df0b71620e3fac4535cd1368d15L60) regex-based algorithm. Tracked under https://github.com/src-d/enry/issues/193
which we can't port. * Bayesian classifier cann't distinguish "SQL" vs "PLpgSQL". Tracked under https://github.com/src-d/enry/issues/194
* All files for the SQL language fall to the classifier because we don't parse
this [disambiguator
expression](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb#L433)
for `*.sql` files right. This expression doesn't comply with the pattern for the
rest in [heuristics.rb](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb).
`enry` [CLI tool](#cli) does not require a full Git repository to be present in filesystem in order to report languages. `enry` [CLI tool](#cli) does not require a full Git repository to be present in filesystem in order to report languages.
@ -232,7 +229,7 @@ As benchmarks depend on Ruby and Github-Linguist gem make sure you have:
If you want to reproduce the same benchmarks as reported above: If you want to reproduce the same benchmarks as reported above:
- Make sure all [dependencies](#benchmark-dependencies) are installed - Make sure all [dependencies](#benchmark-dependencies) are installed
- Install [gnuplot](http://gnuplot.info) (in order to plot the histogram) - Install [gnuplot](http://gnuplot.info) (in order to plot the histogram)
- Run `ENRY_TEST_REPO=.linguist benchmarks/run.sh` (takes ~15h) - Run `ENRY_TEST_REPO="$PWD/.linguist" benchmarks/run.sh` (takes ~15h)
It will run the benchmarks for enry and linguist, parse the output, create csv files and plot the histogram. This takes some time. It will run the benchmarks for enry and linguist, parse the output, create csv files and plot the histogram. This takes some time.

5
benchmark_test.go
View File

@ -28,9 +28,6 @@ var (
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
var exitCode int
defer os.Exit(exitCode)
flag.BoolVar(&slow, "slow", false, "run benchmarks per sample for strategies too") flag.BoolVar(&slow, "slow", false, "run benchmarks per sample for strategies too")
flag.Parse() flag.Parse()
@ -47,7 +44,7 @@ func TestMain(m *testing.M) {
log.Fatal(err) log.Fatal(err)
} }
exitCode = m.Run() os.Exit(m.Run())
} }
func cloneLinguist(linguistURL string) error { func cloneLinguist(linguistURL string) error {

11
common.go
View File

@ -16,7 +16,7 @@ const OtherLanguage = ""
// Strategy type fix the signature for the functions that can be used as a strategy. // Strategy type fix the signature for the functions that can be used as a strategy.
type Strategy func(filename string, content []byte, candidates []string) (languages []string) type Strategy func(filename string, content []byte, candidates []string) (languages []string)
// DefaultStrategies is the strategies' sequence GetLanguage uses to detect languages. // DefaultStrategies is a sequence of strategies used by GetLanguage to detect languages.
var DefaultStrategies = []Strategy{ var DefaultStrategies = []Strategy{
GetLanguagesByModeline, GetLanguagesByModeline,
GetLanguagesByFilename, GetLanguagesByFilename,
@ -397,12 +397,13 @@ func GetLanguagesByContent(filename string, content []byte, _ []string) []string
} }
ext := strings.ToLower(filepath.Ext(filename)) ext := strings.ToLower(filepath.Ext(filename))
fnMatcher, ok := data.ContentMatchers[ext]
heuristic, ok := data.ContentHeuristics[ext]
if !ok { if !ok {
return nil return nil
} }
return fnMatcher(content) return heuristic.Match(content)
} }
// GetLanguagesByClassifier uses DefaultClassifier as a Classifier and returns a sorted slice of possible languages ordered by // GetLanguagesByClassifier uses DefaultClassifier as a Classifier and returns a sorted slice of possible languages ordered by
@ -455,9 +456,7 @@ func GetLanguageType(language string) (langType Type) {
// GetLanguageByAlias returns either the language related to the given alias and ok set to true // GetLanguageByAlias returns either the language related to the given alias and ok set to true
// or Otherlanguage and ok set to false if the alias is not recognized. // or Otherlanguage and ok set to false if the alias is not recognized.
func GetLanguageByAlias(alias string) (lang string, ok bool) { func GetLanguageByAlias(alias string) (lang string, ok bool) {
a := strings.Split(alias, `,`)[0] lang, ok = data.LanguageByAlias(alias)
a = strings.ToLower(a)
lang, ok = data.LanguagesByAlias[a]
if !ok { if !ok {
lang = OtherLanguage lang = OtherLanguage
} }

75
common_test.go
View File

@ -11,6 +11,7 @@ import (
"gopkg.in/src-d/enry.v1/data" "gopkg.in/src-d/enry.v1/data"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite" "github.com/stretchr/testify/suite"
) )
@ -19,9 +20,36 @@ const linguistClonedEnvVar = "ENRY_TEST_REPO"
type EnryTestSuite struct { type EnryTestSuite struct {
suite.Suite suite.Suite
repoLinguist string tmpLinguist string
samplesDir string needToClone bool
cloned bool samplesDir string
}
func (s *EnryTestSuite) TestRegexpEdgeCases() {
var regexpEdgeCases = []struct {
lang string
filename string
}{
{lang: "ActionScript", filename: "FooBar.as"},
{lang: "Forth", filename: "asm.fr"},
{lang: "X PixMap", filename: "cc-public_domain_mark_white.pm"},
//{lang: "SQL", filename: "drop_stuff.sql"}, // https://github.com/src-d/enry/issues/194
{lang: "Fstar", filename: "Hacl.Spec.Bignum.Fmul.fst"},
{lang: "C++", filename: "Types.h"},
}
for _, r := range regexpEdgeCases {
filename := fmt.Sprintf("%s/samples/%s/%s", s.tmpLinguist, r.lang, r.filename)
content, err := ioutil.ReadFile(filename)
require.NoError(s.T(), err)
lang := GetLanguage(r.filename, content)
s.T().Logf("File:%s, lang:%s", filename, lang)
expLang, _ := data.LanguageByAlias(r.lang)
require.EqualValues(s.T(), expLang, lang)
}
} }
func Test_EnryTestSuite(t *testing.T) { func Test_EnryTestSuite(t *testing.T) {
@ -30,25 +58,24 @@ func Test_EnryTestSuite(t *testing.T) {
func (s *EnryTestSuite) SetupSuite() { func (s *EnryTestSuite) SetupSuite() {
var err error var err error
s.repoLinguist = os.Getenv(linguistClonedEnvVar) s.tmpLinguist = os.Getenv(linguistClonedEnvVar)
s.cloned = s.repoLinguist == "" s.needToClone = s.tmpLinguist == ""
if s.cloned { if s.needToClone {
s.repoLinguist, err = ioutil.TempDir("", "linguist-") s.tmpLinguist, err = ioutil.TempDir("", "linguist-")
assert.NoError(s.T(), err) require.NoError(s.T(), err)
} s.T().Logf("Cloning Linguist repo to '%s' as %s was not set\n",
s.tmpLinguist, linguistClonedEnvVar)
s.samplesDir = filepath.Join(s.repoLinguist, "samples") cmd := exec.Command("git", "clone", linguistURL, s.tmpLinguist)
if s.cloned {
cmd := exec.Command("git", "clone", linguistURL, s.repoLinguist)
err = cmd.Run() err = cmd.Run()
assert.NoError(s.T(), err) require.NoError(s.T(), err)
} }
s.samplesDir = filepath.Join(s.tmpLinguist, "samples")
s.T().Logf("using samples from %s", s.samplesDir)
cwd, err := os.Getwd() cwd, err := os.Getwd()
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
err = os.Chdir(s.repoLinguist) err = os.Chdir(s.tmpLinguist)
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
cmd := exec.Command("git", "checkout", data.LinguistCommit) cmd := exec.Command("git", "checkout", data.LinguistCommit)
@ -60,8 +87,8 @@ func (s *EnryTestSuite) SetupSuite() {
} }
func (s *EnryTestSuite) TearDownSuite() { func (s *EnryTestSuite) TearDownSuite() {
if s.cloned { if s.needToClone {
err := os.RemoveAll(s.repoLinguist) err := os.RemoveAll(s.tmpLinguist)
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
} }
} }
@ -88,7 +115,7 @@ func (s *EnryTestSuite) TestGetLanguage() {
} }
func (s *EnryTestSuite) TestGetLanguagesByModelineLinguist() { func (s *EnryTestSuite) TestGetLanguagesByModelineLinguist() {
var modelinesDir = filepath.Join(s.repoLinguist, "test/fixtures/Data/Modelines") var modelinesDir = filepath.Join(s.tmpLinguist, "test/fixtures/Data/Modelines")
tests := []struct { tests := []struct {
name string name string
@ -400,7 +427,8 @@ func (s *EnryTestSuite) TestGetLanguageByAlias() {
func (s *EnryTestSuite) TestLinguistCorpus() { func (s *EnryTestSuite) TestLinguistCorpus() {
const filenamesDir = "filenames" const filenamesDir = "filenames"
var cornerCases = map[string]bool{ var cornerCases = map[string]bool{
"hello.ms": true, "drop_stuff.sql": true, // https://github.com/src-d/enry/issues/194
// .es and .ice fail heuristics parsing, but do not fail any tests
} }
var total, failed, ok, other int var total, failed, ok, other int
@ -408,7 +436,7 @@ func (s *EnryTestSuite) TestLinguistCorpus() {
filepath.Walk(s.samplesDir, func(path string, f os.FileInfo, err error) error { filepath.Walk(s.samplesDir, func(path string, f os.FileInfo, err error) error {
if f.IsDir() { if f.IsDir() {
if f.Name() != filenamesDir { if f.Name() != filenamesDir {
expected = f.Name() expected, _ = data.LanguageByAlias(f.Name())
} }
return nil return nil
@ -431,11 +459,10 @@ func (s *EnryTestSuite) TestLinguistCorpus() {
} else { } else {
status = "failed" status = "failed"
failed++ failed++
} }
if _, ok := cornerCases[filename]; ok { if _, ok := cornerCases[filename]; ok {
fmt.Printf("\t\t[considered corner case] %s\texpected: %s\tobtained: %s\tstatus: %s\n", filename, expected, obtained, status) s.T().Logf("\t\t[considered corner case] %s\texpected: %s\tobtained: %s\tstatus: %s\n", filename, expected, obtained, status)
} else { } else {
assert.Equal(s.T(), expected, obtained, fmt.Sprintf("%s\texpected: %s\tobtained: %s\tstatus: %s\n", filename, expected, obtained, status)) assert.Equal(s.T(), expected, obtained, fmt.Sprintf("%s\texpected: %s\tobtained: %s\tstatus: %s\n", filename, expected, obtained, status))
} }
@ -443,5 +470,5 @@ func (s *EnryTestSuite) TestLinguistCorpus() {
return nil return nil
}) })
fmt.Printf("\t\ttotal files: %d, ok: %d, failed: %d, other: %d\n", total, ok, failed, other) s.T().Logf("\t\ttotal files: %d, ok: %d, failed: %d, other: %d\n", total, ok, failed, other)
} }

776
data/alias.go
View File

@ -1,11 +1,13 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
// LanguagesByAlias keeps alias for different languages and use the name of the languages as an alias too. import "strings"
// LanguageByAliasMap keeps alias for different languages and use the name of the languages as an alias too.
// All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores. // All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores.
var LanguagesByAlias = map[string]string{ var LanguageByAliasMap = map[string]string{
"1c_enterprise": "1C Enterprise", "1c_enterprise": "1C Enterprise",
"abap": "ABAP", "abap": "ABAP",
"abl": "OpenEdge ABL", "abl": "OpenEdge ABL",
@ -32,6 +34,7 @@ var LanguagesByAlias = map[string]string{
"alpine_abuild": "Alpine Abuild", "alpine_abuild": "Alpine Abuild",
"amfm": "Adobe Font Metrics", "amfm": "Adobe Font Metrics",
"ampl": "AMPL", "ampl": "AMPL",
"angelscript": "AngelScript",
"ant_build_system": "Ant Build System", "ant_build_system": "Ant Build System",
"antlr": "ANTLR", "antlr": "ANTLR",
"apache": "ApacheConf", "apache": "ApacheConf",
@ -40,79 +43,81 @@ var LanguagesByAlias = map[string]string{
"api_blueprint": "API Blueprint", "api_blueprint": "API Blueprint",
"apkbuild": "Alpine Abuild", "apkbuild": "Alpine Abuild",
"apl": "APL", "apl": "APL",
"apollo_guidance_computer": "Apollo Guidance Computer", "apollo_guidance_computer": "Apollo Guidance Computer",
"applescript": "AppleScript", "applescript": "AppleScript",
"arc": "Arc", "arc": "Arc",
"arduino": "Arduino", "arexx": "REXX",
"arexx": "REXX", "as3": "ActionScript",
"as3": "ActionScript", "asciidoc": "AsciiDoc",
"asciidoc": "AsciiDoc", "asm": "Assembly",
"asn.1": "ASN.1", "asn.1": "ASN.1",
"asp": "ASP", "asp": "ASP",
"aspectj": "AspectJ", "aspectj": "AspectJ",
"aspx": "ASP", "aspx": "ASP",
"aspx-vb": "ASP", "aspx-vb": "ASP",
"assembly": "Assembly", "assembly": "Assembly",
"ats": "ATS", "asymptote": "Asymptote",
"ats2": "ATS", "ats": "ATS",
"au3": "AutoIt", "ats2": "ATS",
"augeas": "Augeas", "au3": "AutoIt",
"autoconf": "M4Sugar", "augeas": "Augeas",
"autohotkey": "AutoHotkey", "autoconf": "M4Sugar",
"autoit": "AutoIt", "autohotkey": "AutoHotkey",
"autoit3": "AutoIt", "autoit": "AutoIt",
"autoitscript": "AutoIt", "autoit3": "AutoIt",
"awk": "Awk", "autoitscript": "AutoIt",
"b3d": "BlitzBasic", "awk": "Awk",
"ballerina": "Ballerina", "b3d": "BlitzBasic",
"bash": "Shell", "ballerina": "Ballerina",
"bash_session": "ShellSession", "bash": "Shell",
"bat": "Batchfile", "bash_session": "ShellSession",
"batch": "Batchfile", "bat": "Batchfile",
"batchfile": "Batchfile", "batch": "Batchfile",
"befunge": "Befunge", "batchfile": "Batchfile",
"bison": "Bison", "befunge": "Befunge",
"bitbake": "BitBake", "bison": "Bison",
"blade": "Blade", "bitbake": "BitBake",
"blitz3d": "BlitzBasic", "blade": "Blade",
"blitzbasic": "BlitzBasic", "blitz3d": "BlitzBasic",
"blitzmax": "BlitzMax", "blitzbasic": "BlitzBasic",
"blitzplus": "BlitzBasic", "blitzmax": "BlitzMax",
"bluespec": "Bluespec", "blitzplus": "BlitzBasic",
"bmax": "BlitzMax", "bluespec": "Bluespec",
"boo": "Boo", "bmax": "BlitzMax",
"bplus": "BlitzBasic", "boo": "Boo",
"brainfuck": "Brainfuck", "bplus": "BlitzBasic",
"brightscript": "Brightscript", "brainfuck": "Brainfuck",
"bro": "Bro", "brightscript": "Brightscript",
"bsdmake": "Makefile", "bro": "Bro",
"byond": "DM", "bsdmake": "Makefile",
"c": "C", "byond": "DM",
"c#": "C#", "c": "C",
"c++": "C++", "c#": "C#",
"c++-objdump": "Cpp-ObjDump", "c++": "C++",
"c-objdump": "C-ObjDump", "c++-objdump": "Cpp-ObjDump",
"c2hs": "C2hs Haskell", "c-objdump": "C-ObjDump",
"c2hs_haskell": "C2hs Haskell", "c2hs": "C2hs Haskell",
"cap'n_proto": "Cap'n Proto", "c2hs_haskell": "C2hs Haskell",
"carto": "CartoCSS", "cap'n_proto": "Cap'n Proto",
"cartocss": "CartoCSS", "carto": "CartoCSS",
"ceylon": "Ceylon", "cartocss": "CartoCSS",
"cfc": "ColdFusion CFC", "ceylon": "Ceylon",
"cfm": "ColdFusion", "cfc": "ColdFusion CFC",
"cfml": "ColdFusion", "cfm": "ColdFusion",
"chapel": "Chapel", "cfml": "ColdFusion",
"charity": "Charity", "chapel": "Chapel",
"chpl": "Chapel", "charity": "Charity",
"chuck": "ChucK", "chpl": "Chapel",
"cirru": "Cirru", "chuck": "ChucK",
"clarion": "Clarion", "cirru": "Cirru",
"clean": "Clean", "clarion": "Clarion",
"click": "Click", "clean": "Clean",
"clipper": "xBase", "click": "Click",
"clips": "CLIPS", "clipper": "xBase",
"clojure": "Clojure", "clips": "CLIPS",
"closure_templates": "Closure Templates", "clojure": "Clojure",
"closure_templates": "Closure Templates",
"cloud_firestore_security_rules": "Cloud Firestore Security Rules",
"cmake": "CMake", "cmake": "CMake",
"cobol": "COBOL", "cobol": "COBOL",
"coffee": "CoffeeScript", "coffee": "CoffeeScript",
@ -123,10 +128,15 @@ var LanguagesByAlias = map[string]string{
"coldfusion_html": "ColdFusion", "coldfusion_html": "ColdFusion",
"collada": "COLLADA", "collada": "COLLADA",
"common_lisp": "Common Lisp", "common_lisp": "Common Lisp",
"common_workflow_language": "Common Workflow Language",
"component_pascal": "Component Pascal", "component_pascal": "Component Pascal",
"conll": "CoNLL-U",
"conll-u": "CoNLL-U",
"conll-x": "CoNLL-U",
"console": "ShellSession", "console": "ShellSession",
"cool": "Cool", "cool": "Cool",
"coq": "Coq", "coq": "Coq",
"cperl": "Perl",
"cpp": "C++", "cpp": "C++",
"cpp-objdump": "Cpp-ObjDump", "cpp-objdump": "Cpp-ObjDump",
"creole": "Creole", "creole": "Creole",
@ -144,6 +154,7 @@ var LanguagesByAlias = map[string]string{
"cucumber": "Gherkin", "cucumber": "Gherkin",
"cuda": "Cuda", "cuda": "Cuda",
"cweb": "CWeb", "cweb": "CWeb",
"cwl": "Common Workflow Language",
"cycript": "Cycript", "cycript": "Cycript",
"cython": "Cython", "cython": "Cython",
"d": "D", "d": "D",
@ -176,55 +187,69 @@ var LanguagesByAlias = map[string]string{
"ecl": "ECL", "ecl": "ECL",
"eclipse": "ECLiPSe", "eclipse": "ECLiPSe",
"ecr": "HTML+ECR", "ecr": "HTML+ECR",
"edn": "edn", "edje_data_collection": "Edje Data Collection",
"eeschema_schematic": "KiCad Schematic", "edn": "edn",
"eex": "HTML+EEX", "eeschema_schematic": "KiCad Schematic",
"eiffel": "Eiffel", "eex": "HTML+EEX",
"ejs": "EJS", "eiffel": "Eiffel",
"elisp": "Emacs Lisp", "ejs": "EJS",
"elixir": "Elixir", "elisp": "Emacs Lisp",
"elm": "Elm", "elixir": "Elixir",
"emacs": "Emacs Lisp", "elm": "Elm",
"emacs_lisp": "Emacs Lisp", "emacs": "Emacs Lisp",
"emberscript": "EmberScript", "emacs_lisp": "Emacs Lisp",
"eq": "EQ", "emberscript": "EmberScript",
"erb": "HTML+ERB", "eml": "EML",
"erlang": "Erlang", "eq": "EQ",
"f#": "F#", "erb": "HTML+ERB",
"factor": "Factor", "erlang": "Erlang",
"fancy": "Fancy", "f#": "F#",
"fantom": "Fantom", "f*": "F*",
"filebench_wml": "Filebench WML", "factor": "Factor",
"filterscript": "Filterscript", "fancy": "Fancy",
"fish": "fish", "fantom": "Fantom",
"flex": "Lex", "figfont": "FIGlet Font",
"flux": "FLUX", "figlet_font": "FIGlet Font",
"formatted": "Formatted", "filebench_wml": "Filebench WML",
"forth": "Forth", "filterscript": "Filterscript",
"fortran": "Fortran", "fish": "fish",
"foxpro": "xBase", "flex": "Lex",
"freemarker": "FreeMarker", "flux": "FLUX",
"frege": "Frege", "formatted": "Formatted",
"fsharp": "F#", "forth": "Forth",
"ftl": "FreeMarker", "fortran": "Fortran",
"fundamental": "Text", "foxpro": "xBase",
"g-code": "G-code", "freemarker": "FreeMarker",
"game_maker_language": "Game Maker Language", "frege": "Frege",
"gams": "GAMS", "fsharp": "F#",
"gap": "GAP", "fstar": "F*",
"ftl": "FreeMarker",
"fundamental": "Text",
"g-code": "G-code",
"game_maker_language": "Game Maker Language",
"gams": "GAMS",
"gap": "GAP",
"gcc_machine_description": "GCC Machine Description", "gcc_machine_description": "GCC Machine Description",
"gdb": "GDB", "gdb": "GDB",
"gdscript": "GDScript", "gdscript": "GDScript",
"genie": "Genie", "genie": "Genie",
"genshi": "Genshi", "genshi": "Genshi",
"gentoo_ebuild": "Gentoo Ebuild", "gentoo_ebuild": "Gentoo Ebuild",
"gentoo_eclass": "Gentoo Eclass", "gentoo_eclass": "Gentoo Eclass",
"gerber_image": "Gerber Image", "gerber_image": "Gerber Image",
"gettext_catalog": "Gettext Catalog", "gettext_catalog": "Gettext Catalog",
"gf": "Grammatical Framework", "gf": "Grammatical Framework",
"gherkin": "Gherkin", "gherkin": "Gherkin",
"glsl": "GLSL", "git-ignore": "Ignore List",
"glyph": "Glyph", "git_attributes": "Git Attributes",
"git_config": "Git Config",
"gitattributes": "Git Attributes",
"gitconfig": "Git Config",
"gitignore": "Ignore List",
"gitmodules": "Git Config",
"glsl": "GLSL",
"glyph": "Glyph",
"glyph_bitmap_distribution_format": "Glyph Bitmap Distribution Format",
"gn": "GN", "gn": "GN",
"gnuplot": "Gnuplot", "gnuplot": "Gnuplot",
"go": "Go", "go": "Go",
@ -237,17 +262,20 @@ var LanguagesByAlias = map[string]string{
"graph_modeling_language": "Graph Modeling Language", "graph_modeling_language": "Graph Modeling Language",
"graphql": "GraphQL", "graphql": "GraphQL",
"graphviz_(dot)": "Graphviz (DOT)", "graphviz_(dot)": "Graphviz (DOT)",
"groff": "Roff",
"groovy": "Groovy", "groovy": "Groovy",
"groovy_server_pages": "Groovy Server Pages", "groovy_server_pages": "Groovy Server Pages",
"gsp": "Groovy Server Pages", "gsp": "Groovy Server Pages",
"hack": "Hack", "hack": "Hack",
"haml": "Haml", "haml": "Haml",
"handlebars": "Handlebars", "handlebars": "Handlebars",
"haproxy": "HAProxy",
"harbour": "Harbour", "harbour": "Harbour",
"haskell": "Haskell", "haskell": "Haskell",
"haxe": "Haxe", "haxe": "Haxe",
"hbs": "Handlebars", "hbs": "Handlebars",
"hcl": "HCL", "hcl": "HCL",
"hiveql": "HiveQL",
"hlsl": "HLSL", "hlsl": "HLSL",
"html": "HTML", "html": "HTML",
"html+django": "HTML+Django", "html+django": "HTML+Django",
@ -257,16 +285,20 @@ var LanguagesByAlias = map[string]string{
"html+erb": "HTML+ERB", "html+erb": "HTML+ERB",
"html+jinja": "HTML+Django", "html+jinja": "HTML+Django",
"html+php": "HTML+PHP", "html+php": "HTML+PHP",
"html+razor": "HTML+Razor",
"html+ruby": "RHTML", "html+ruby": "RHTML",
"htmlbars": "Handlebars", "htmlbars": "Handlebars",
"htmldjango": "HTML+Django", "htmldjango": "HTML+Django",
"http": "HTTP", "http": "HTTP",
"hxml": "HXML",
"hy": "Hy", "hy": "Hy",
"hylang": "Hy", "hylang": "Hy",
"hyphy": "HyPhy", "hyphy": "HyPhy",
"i7": "Inform 7", "i7": "Inform 7",
"idl": "IDL", "idl": "IDL",
"idris": "Idris", "idris": "Idris",
"ignore": "Ignore List",
"ignore_list": "Ignore List",
"igor": "IGOR Pro", "igor": "IGOR Pro",
"igor_pro": "IGOR Pro", "igor_pro": "IGOR Pro",
"igorpro": "IGOR Pro", "igorpro": "IGOR Pro",
@ -286,6 +318,7 @@ var LanguagesByAlias = map[string]string{
"j": "J", "j": "J",
"jasmin": "Jasmin", "jasmin": "Jasmin",
"java": "Java", "java": "Java",
"java_properties": "Java Properties",
"java_server_page": "Groovy Server Pages", "java_server_page": "Groovy Server Pages",
"java_server_pages": "Java Server Pages", "java_server_pages": "Java Server Pages",
"javascript": "JavaScript", "javascript": "JavaScript",
@ -297,6 +330,8 @@ var LanguagesByAlias = map[string]string{
"js": "JavaScript", "js": "JavaScript",
"json": "JSON", "json": "JSON",
"json5": "JSON5", "json5": "JSON5",
"json_with_comments": "JSON with Comments",
"jsonc": "JSON with Comments",
"jsoniq": "JSONiq", "jsoniq": "JSONiq",
"jsonld": "JSONLD", "jsonld": "JSONLD",
"jsp": "Java Server Pages", "jsp": "Java Server Pages",
@ -340,6 +375,7 @@ var LanguagesByAlias = map[string]string{
"loomscript": "LoomScript", "loomscript": "LoomScript",
"ls": "LiveScript", "ls": "LiveScript",
"lsl": "LSL", "lsl": "LSL",
"ltspice_symbol": "LTspice Symbol",
"lua": "Lua", "lua": "Lua",
"m": "M", "m": "M",
"m4": "M4", "m4": "M4",
@ -348,17 +384,22 @@ var LanguagesByAlias = map[string]string{
"make": "Makefile", "make": "Makefile",
"makefile": "Makefile", "makefile": "Makefile",
"mako": "Mako", "mako": "Mako",
"man": "Roff",
"man-page": "Roff",
"man_page": "Roff",
"manpage": "Roff",
"markdown": "Markdown", "markdown": "Markdown",
"marko": "Marko", "marko": "Marko",
"markojs": "Marko", "markojs": "Marko",
"mask": "Mask", "mask": "Mask",
"mathematica": "Mathematica", "mathematica": "Mathematica",
"matlab": "Matlab", "matlab": "MATLAB",
"maven_pom": "Maven POM", "maven_pom": "Maven POM",
"max": "Max", "max": "Max",
"max/msp": "Max", "max/msp": "Max",
"maxmsp": "Max", "maxmsp": "Max",
"maxscript": "MAXScript", "maxscript": "MAXScript",
"mdoc": "Roff",
"mediawiki": "MediaWiki", "mediawiki": "MediaWiki",
"mercury": "Mercury", "mercury": "Mercury",
"meson": "Meson", "meson": "Meson",
@ -369,6 +410,7 @@ var LanguagesByAlias = map[string]string{
"mma": "Mathematica", "mma": "Mathematica",
"modelica": "Modelica", "modelica": "Modelica",
"modula-2": "Modula-2", "modula-2": "Modula-2",
"modula-3": "Modula-3",
"module_management_system": "Module Management System", "module_management_system": "Module Management System",
"monkey": "Monkey", "monkey": "Monkey",
"moocode": "Moocode", "moocode": "Moocode",
@ -380,6 +422,7 @@ var LanguagesByAlias = map[string]string{
"mumps": "M", "mumps": "M",
"mupad": "mupad", "mupad": "mupad",
"myghty": "Myghty", "myghty": "Myghty",
"nanorc": "nanorc",
"nasm": "Assembly", "nasm": "Assembly",
"ncl": "NCL", "ncl": "NCL",
"nearley": "Nearley", "nearley": "Nearley",
@ -389,6 +432,7 @@ var LanguagesByAlias = map[string]string{
"netlinx+erb": "NetLinx+ERB", "netlinx+erb": "NetLinx+ERB",
"netlogo": "NetLogo", "netlogo": "NetLogo",
"newlisp": "NewLisp", "newlisp": "NewLisp",
"nextflow": "Nextflow",
"nginx": "Nginx", "nginx": "Nginx",
"nginx_configuration_file": "Nginx", "nginx_configuration_file": "Nginx",
"nim": "Nim", "nim": "Nim",
@ -421,8 +465,9 @@ var LanguagesByAlias = map[string]string{
"objectpascal": "Component Pascal", "objectpascal": "Component Pascal",
"objj": "Objective-J", "objj": "Objective-J",
"ocaml": "OCaml", "ocaml": "OCaml",
"octave": "Matlab", "octave": "MATLAB",
"omgrofl": "Omgrofl", "omgrofl": "Omgrofl",
"oncrpc": "RPC",
"ooc": "ooc", "ooc": "ooc",
"opa": "Opa", "opa": "Opa",
"opal": "Opal", "opal": "Opal",
@ -445,219 +490,270 @@ var LanguagesByAlias = map[string]string{
"parrot": "Parrot", "parrot": "Parrot",
"parrot_assembly": "Parrot Assembly", "parrot_assembly": "Parrot Assembly",
"parrot_internal_representation": "Parrot Internal Representation", "parrot_internal_representation": "Parrot Internal Representation",
"pascal": "Pascal", "pascal": "Pascal",
"pasm": "Parrot Assembly", "pasm": "Parrot Assembly",
"pawn": "PAWN", "pawn": "Pawn",
"pcbnew": "KiCad Layout", "pcbnew": "KiCad Layout",
"pep8": "Pep8", "pep8": "Pep8",
"perl": "Perl", "perl": "Perl",
"perl_6": "Perl 6", "perl6": "Perl 6",
"php": "PHP", "perl_6": "Perl 6",
"pic": "Pic", "php": "PHP",
"pickle": "Pickle", "pic": "Pic",
"picolisp": "PicoLisp", "pickle": "Pickle",
"piglatin": "PigLatin", "picolisp": "PicoLisp",
"pike": "Pike", "piglatin": "PigLatin",
"pir": "Parrot Internal Representation", "pike": "Pike",
"plpgsql": "PLpgSQL", "pir": "Parrot Internal Representation",
"plsql": "PLSQL", "plpgsql": "PLpgSQL",
"pod": "Pod", "plsql": "PLSQL",
"pogoscript": "PogoScript", "pod": "Pod",
"pony": "Pony", "pod_6": "Pod 6",
"posh": "PowerShell", "pogoscript": "PogoScript",
"postscr": "PostScript", "pony": "Pony",
"postscript": "PostScript", "posh": "PowerShell",
"pot": "Gettext Catalog", "postcss": "PostCSS",
"pov-ray": "POV-Ray SDL", "postscr": "PostScript",
"pov-ray_sdl": "POV-Ray SDL", "postscript": "PostScript",
"povray": "POV-Ray SDL", "pot": "Gettext Catalog",
"powerbuilder": "PowerBuilder", "pov-ray": "POV-Ray SDL",
"powershell": "PowerShell", "pov-ray_sdl": "POV-Ray SDL",
"processing": "Processing", "povray": "POV-Ray SDL",
"progress": "OpenEdge ABL", "powerbuilder": "PowerBuilder",
"prolog": "Prolog", "powershell": "PowerShell",
"propeller_spin": "Propeller Spin", "processing": "Processing",
"protobuf": "Protocol Buffer", "progress": "OpenEdge ABL",
"protocol_buffer": "Protocol Buffer", "prolog": "Prolog",
"protocol_buffers": "Protocol Buffer", "propeller_spin": "Propeller Spin",
"public_key": "Public Key", "protobuf": "Protocol Buffer",
"pug": "Pug", "protocol_buffer": "Protocol Buffer",
"puppet": "Puppet", "protocol_buffers": "Protocol Buffer",
"pure_data": "Pure Data", "public_key": "Public Key",
"purebasic": "PureBasic", "pug": "Pug",
"purescript": "PureScript", "puppet": "Puppet",
"pycon": "Python console", "pure_data": "Pure Data",
"pyrex": "Cython", "purebasic": "PureBasic",
"python": "Python", "purescript": "PureScript",
"python_console": "Python console", "pwsh": "PowerShell",
"python_traceback": "Python traceback", "pycon": "Python console",
"qmake": "QMake", "pyrex": "Cython",
"qml": "QML", "python": "Python",
"r": "R", "python3": "Python",
"racket": "Racket", "python_console": "Python console",
"ragel": "Ragel", "python_traceback": "Python traceback",
"ragel-rb": "Ragel", "q": "q",
"ragel-ruby": "Ragel", "qmake": "QMake",
"rake": "Ruby", "qml": "QML",
"raml": "RAML", "quake": "Quake",
"rascal": "Rascal", "r": "R",
"raw": "Raw token data", "racket": "Racket",
"raw_token_data": "Raw token data", "ragel": "Ragel",
"rb": "Ruby", "ragel-rb": "Ragel",
"rbx": "Ruby", "ragel-ruby": "Ragel",
"rdoc": "RDoc", "rake": "Ruby",
"realbasic": "REALbasic", "raml": "RAML",
"reason": "Reason", "rascal": "Rascal",
"rebol": "Rebol", "raw": "Raw token data",
"red": "Red", "raw_token_data": "Raw token data",
"red/system": "Red", "razor": "HTML+Razor",
"redcode": "Redcode", "rb": "Ruby",
"regex": "Regular Expression", "rbx": "Ruby",
"regexp": "Regular Expression", "rdoc": "RDoc",
"regular_expression": "Regular Expression", "realbasic": "REALbasic",
"ren'py": "Ren'Py", "reason": "Reason",
"renderscript": "RenderScript", "rebol": "Rebol",
"renpy": "Ren'Py", "red": "Red",
"restructuredtext": "reStructuredText", "red/system": "Red",
"rexx": "REXX", "redcode": "Redcode",
"rhtml": "RHTML", "regex": "Regular Expression",
"ring": "Ring", "regexp": "Regular Expression",
"rmarkdown": "RMarkdown", "regular_expression": "Regular Expression",
"robotframework": "RobotFramework", "ren'py": "Ren'Py",
"roff": "Roff", "renderscript": "RenderScript",
"rouge": "Rouge", "renpy": "Ren'Py",
"rpm_spec": "RPM Spec", "restructuredtext": "reStructuredText",
"rs-274x": "Gerber Image", "rexx": "REXX",
"rscript": "R", "rhtml": "RHTML",
"rss": "XML", "ring": "Ring",
"rst": "reStructuredText", "rmarkdown": "RMarkdown",
"ruby": "Ruby", "robotframework": "RobotFramework",
"runoff": "RUNOFF", "roff": "Roff",
"rust": "Rust", "roff_manpage": "Roff Manpage",
"rusthon": "Python", "rouge": "Rouge",
"sage": "Sage", "rpc": "RPC",
"salt": "SaltStack", "rpcgen": "RPC",
"saltstack": "SaltStack", "rpm_spec": "RPM Spec",
"saltstate": "SaltStack", "rs-274x": "Gerber Image",
"sas": "SAS", "rscript": "R",
"sass": "Sass", "rss": "XML",
"scala": "Scala", "rst": "reStructuredText",
"scaml": "Scaml", "ruby": "Ruby",
"scheme": "Scheme", "runoff": "RUNOFF",
"scilab": "Scilab", "rust": "Rust",
"scss": "SCSS", "rusthon": "Python",
"self": "Self", "sage": "Sage",
"sh": "Shell", "salt": "SaltStack",
"shaderlab": "ShaderLab", "saltstack": "SaltStack",
"shell": "Shell", "saltstate": "SaltStack",
"shell-script": "Shell", "sas": "SAS",
"shellsession": "ShellSession", "sass": "Sass",
"shen": "Shen", "scala": "Scala",
"slash": "Slash", "scaml": "Scaml",
"slim": "Slim", "scheme": "Scheme",
"smali": "Smali", "scilab": "Scilab",
"smalltalk": "Smalltalk", "scss": "SCSS",
"smarty": "Smarty", "sed": "sed",
"sml": "Standard ML", "self": "Self",
"smt": "SMT", "sh": "Shell",
"sourcemod": "SourcePawn", "shaderlab": "ShaderLab",
"sourcepawn": "SourcePawn", "shell": "Shell",
"sparql": "SPARQL", "shell-script": "Shell",
"specfile": "RPM Spec", "shellsession": "ShellSession",
"spline_font_database": "Spline Font Database", "shen": "Shen",
"splus": "R", "slash": "Slash",
"sqf": "SQF", "slice": "Slice",
"sql": "SQL", "slim": "Slim",
"sqlpl": "SQLPL", "smali": "Smali",
"squeak": "Smalltalk", "smalltalk": "Smalltalk",
"squirrel": "Squirrel", "smarty": "Smarty",
"srecode_template": "SRecode Template", "sml": "Standard ML",
"stan": "Stan", "smt": "SMT",
"standard_ml": "Standard ML", "snippet": "YASnippet",
"stata": "Stata", "solidity": "Solidity",
"ston": "STON", "sourcemod": "SourcePawn",
"stylus": "Stylus", "sourcepawn": "SourcePawn",
"sublime_text_config": "Sublime Text Config", "soy": "Closure Templates",
"subrip_text": "SubRip Text", "sparql": "SPARQL",
"supercollider": "SuperCollider", "specfile": "RPM Spec",
"svg": "SVG", "spline_font_database": "Spline Font Database",
"swift": "Swift", "splus": "R",
"systemverilog": "SystemVerilog", "sqf": "SQF",
"tcl": "Tcl", "sql": "SQL",
"tcsh": "Tcsh", "sqlpl": "SQLPL",
"tea": "Tea", "squeak": "Smalltalk",
"terra": "Terra", "squirrel": "Squirrel",
"tex": "TeX", "srecode_template": "SRecode Template",
"text": "Text", "stan": "Stan",
"textile": "Textile", "standard_ml": "Standard ML",
"thrift": "Thrift", "stata": "Stata",
"ti_program": "TI Program", "ston": "STON",
"tl": "Type Language", "stylus": "Stylus",
"tla": "TLA", "subrip_text": "SubRip Text",
"toml": "TOML", "sugarss": "SugarSS",
"ts": "TypeScript", "supercollider": "SuperCollider",
"turing": "Turing", "svg": "SVG",
"turtle": "Turtle", "swift": "Swift",
"twig": "Twig", "systemverilog": "SystemVerilog",
"txl": "TXL", "tcl": "Tcl",
"type_language": "Type Language", "tcsh": "Tcsh",
"typescript": "TypeScript", "tea": "Tea",
"udiff": "Diff", "terra": "Terra",
"unified_parallel_c": "Unified Parallel C", "terraform": "HCL",
"unity3d_asset": "Unity3D Asset", "tex": "TeX",
"unix_assembly": "Unix Assembly", "text": "Text",
"uno": "Uno", "textile": "Textile",
"unrealscript": "UnrealScript", "thrift": "Thrift",
"ur": "UrWeb", "ti_program": "TI Program",
"ur/web": "UrWeb", "tl": "Type Language",
"urweb": "UrWeb", "tla": "TLA",
"vala": "Vala", "toml": "TOML",
"vb.net": "Visual Basic", "troff": "Roff",
"vbnet": "Visual Basic", "ts": "TypeScript",
"vcl": "VCL", "turing": "Turing",
"verilog": "Verilog", "turtle": "Turtle",
"vhdl": "VHDL", "twig": "Twig",
"vim": "Vim script", "txl": "TXL",
"vim_script": "Vim script", "type_language": "Type Language",
"viml": "Vim script", "typescript": "TypeScript",
"visual_basic": "Visual Basic", "udiff": "Diff",
"volt": "Volt", "unified_parallel_c": "Unified Parallel C",
"vue": "Vue", "unity3d_asset": "Unity3D Asset",
"wasm": "WebAssembly", "unix_assembly": "Unix Assembly",
"wast": "WebAssembly", "uno": "Uno",
"wavefront_material": "Wavefront Material", "unrealscript": "UnrealScript",
"wavefront_object": "Wavefront Object", "ur": "UrWeb",
"web_ontology_language": "Web Ontology Language", "ur/web": "UrWeb",
"webassembly": "WebAssembly", "urweb": "UrWeb",
"webidl": "WebIDL", "vala": "Vala",
"winbatch": "Batchfile", "vb.net": "Visual Basic",
"wisp": "wisp", "vbnet": "Visual Basic",
"vcl": "VCL",
"verilog": "Verilog",
"vhdl": "VHDL",
"vim": "Vim script",
"vim_script": "Vim script",
"viml": "Vim script",
"visual_basic": "Visual Basic",
"volt": "Volt",
"vue": "Vue",
"wasm": "WebAssembly",
"wast": "WebAssembly",
"wavefront_material": "Wavefront Material",
"wavefront_object": "Wavefront Object",
"wdl": "wdl",
"web_ontology_language": "Web Ontology Language",
"webassembly": "WebAssembly",
"webidl": "WebIDL",
"winbatch": "Batchfile",
"windows_registry_entries": "Windows Registry Entries",
"wisp": "wisp",
"world_of_warcraft_addon_data": "World of Warcraft Addon Data", "world_of_warcraft_addon_data": "World of Warcraft Addon Data",
"wsdl": "XML", "wsdl": "XML",
"x10": "X10", "x10": "X10",
"xbase": "xBase", "x_bitmap": "X BitMap",
"xc": "XC", "x_font_directory_index": "X Font Directory Index",
"xcompose": "XCompose", "x_pixmap": "X PixMap",
"xhtml": "HTML", "xbase": "xBase",
"xml": "XML", "xbm": "X BitMap",
"xml+genshi": "Genshi", "xc": "XC",
"xml+kid": "Genshi", "xcompose": "XCompose",
"xojo": "Xojo", "xdr": "RPC",
"xpages": "XPages", "xhtml": "HTML",
"xpm": "XPM", "xml": "XML",
"xproc": "XProc", "xml+genshi": "Genshi",
"xquery": "XQuery", "xml+kid": "Genshi",
"xs": "XS", "xojo": "Xojo",
"xsd": "XML", "xpages": "XPages",
"xsl": "XSLT", "xpm": "X PixMap",
"xslt": "XSLT", "xproc": "XProc",
"xten": "X10", "xquery": "XQuery",
"xtend": "Xtend", "xs": "XS",
"yacc": "Yacc", "xsd": "XML",
"yaml": "YAML", "xsl": "XSLT",
"yang": "YANG", "xslt": "XSLT",
"yml": "YAML", "xten": "X10",
"zephir": "Zephir", "xtend": "Xtend",
"zimpl": "Zimpl", "yacc": "Yacc",
"zsh": "Shell", "yaml": "YAML",
"yang": "YANG",
"yara": "YARA",
"yas": "YASnippet",
"yasnippet": "YASnippet",
"yml": "YAML",
"zephir": "Zephir",
"zig": "Zig",
"zimpl": "Zimpl",
"zsh": "Shell",
}
// LanguageByAlias looks up the language name by it's alias or name.
// It mirrors the logic of github linguist and is needed e.g for heuristcs.yml
// that mixes names and aliases in a language field (see XPM example).
func LanguageByAlias(langOrAlias string) (lang string, ok bool) {
k := convertToAliasKey(langOrAlias)
lang, ok = LanguageByAliasMap[k]
return
}
// convertToAliasKey converts language name to a key in LanguageByAliasMap.
// Following
// - internal.code-generator.generator.convertToAliasKey()
// - GetLanguageByAlias()
// conventions.
// It is here to avoid dependency on "generate" and "enry" packages.
func convertToAliasKey(langName string) string {
ak := strings.SplitN(langName, `,`, 2)[0]
ak = strings.Replace(ak, ` `, `_`, -1)
ak = strings.ToLower(ak)
return ak
} }

4
data/commit.go
View File

@ -1,7 +1,7 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
// linguist's commit from which files were generated. // linguist's commit from which files were generated.
var LinguistCommit = "4cd558c37482e8d2c535d8107f2d11b49afbc5b5" var LinguistCommit = "e4560984058b4726010ca4b8f03ed9d0f8f464db"

1795
data/content.go
View File

File diff suppressed because it is too large Load Diff

3
data/doc.go Normal file
View File

@ -0,0 +1,3 @@
// Package data contains only auto-generated data-structures for all the language
// identification strategies from the Linguist project sources.
package data

4
data/documentation.go
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -8,10 +8,12 @@ import "gopkg.in/toqueteos/substring.v1"
var DocumentationMatchers = substring.Or( var DocumentationMatchers = substring.Or(
substring.Regexp(`^[Dd]ocs?/`), substring.Regexp(`^[Dd]ocs?/`),
substring.Regexp(`(^|/)[Dd]ocumentation/`), substring.Regexp(`(^|/)[Dd]ocumentation/`),
substring.Regexp(`(^|/)[Gg]roovydoc/`),
substring.Regexp(`(^|/)[Jj]avadoc/`), substring.Regexp(`(^|/)[Jj]avadoc/`),
substring.Regexp(`^[Mm]an/`), substring.Regexp(`^[Mm]an/`),
substring.Regexp(`^[Ee]xamples/`), substring.Regexp(`^[Ee]xamples/`),
substring.Regexp(`^[Dd]emos?/`), substring.Regexp(`^[Dd]emos?/`),
substring.Regexp(`(^|/)inst/doc/`),
substring.Regexp(`(^|/)CHANGE(S|LOG)?(\.|$)`), substring.Regexp(`(^|/)CHANGE(S|LOG)?(\.|$)`),
substring.Regexp(`(^|/)CONTRIBUTING(\.|$)`), substring.Regexp(`(^|/)CONTRIBUTING(\.|$)`),
substring.Regexp(`(^|/)COPYING(\.|$)`), substring.Regexp(`(^|/)COPYING(\.|$)`),

637
data/extension.go
View File

File diff suppressed because it is too large Load Diff

59
data/filename.go
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -8,41 +8,65 @@ var LanguagesByFilename = map[string][]string{
".XCompose": {"XCompose"}, ".XCompose": {"XCompose"},
".abbrev_defs": {"Emacs Lisp"}, ".abbrev_defs": {"Emacs Lisp"},
".arcconfig": {"JSON"}, ".arcconfig": {"JSON"},
".babelrc": {"JSON5"}, ".atomignore": {"Ignore List"},
".babelignore": {"Ignore List"},
".babelrc": {"JSON with Comments"},
".bash_aliases": {"Shell"},
".bash_history": {"Shell"}, ".bash_history": {"Shell"},
".bash_logout": {"Shell"}, ".bash_logout": {"Shell"},
".bash_profile": {"Shell"}, ".bash_profile": {"Shell"},
".bashrc": {"Shell"}, ".bashrc": {"Shell"},
".bzrignore": {"Ignore List"},
".clang-format": {"YAML"}, ".clang-format": {"YAML"},
".clang-tidy": {"YAML"}, ".clang-tidy": {"YAML"},
".classpath": {"XML"}, ".classpath": {"XML"},
".coffeelintignore": {"Ignore List"},
".cproject": {"XML"}, ".cproject": {"XML"},
".cshrc": {"Shell"}, ".cshrc": {"Shell"},
".cvsignore": {"Ignore List"},
".dockerignore": {"Ignore List"},
".editorconfig": {"INI"}, ".editorconfig": {"INI"},
".emacs": {"Emacs Lisp"}, ".emacs": {"Emacs Lisp"},
".emacs.desktop": {"Emacs Lisp"}, ".emacs.desktop": {"Emacs Lisp"},
".eslintignore": {"Ignore List"},
".eslintrc.json": {"JSON with Comments"},
".factor-boot-rc": {"Factor"}, ".factor-boot-rc": {"Factor"},
".factor-rc": {"Factor"}, ".factor-rc": {"Factor"},
".gclient": {"Python"}, ".gclient": {"Python"},
".gemrc": {"YAML"}, ".gemrc": {"YAML"},
".gitconfig": {"INI"}, ".gitattributes": {"Git Attributes"},
".gitconfig": {"Git Config"},
".gitignore": {"Ignore List"},
".gitmodules": {"Git Config"},
".gn": {"GN"}, ".gn": {"GN"},
".gnus": {"Emacs Lisp"}, ".gnus": {"Emacs Lisp"},
".gvimrc": {"Vim script"}, ".gvimrc": {"Vim script"},
".htaccess": {"ApacheConf"}, ".htaccess": {"ApacheConf"},
".htmlhintrc": {"JSON"},
".irbrc": {"Ruby"}, ".irbrc": {"Ruby"},
".jshintrc": {"JSON"}, ".jscsrc": {"JSON with Comments"},
".jshintrc": {"JSON with Comments"},
".jslintrc": {"JSON with Comments"},
".login": {"Shell"}, ".login": {"Shell"},
".nanorc": {"nanorc"},
".nodemonignore": {"Ignore List"},
".npmignore": {"Ignore List"},
".nvimrc": {"Vim script"}, ".nvimrc": {"Vim script"},
".php": {"PHP"}, ".php": {"PHP"},
".php_cs": {"PHP"}, ".php_cs": {"PHP"},
".php_cs.dist": {"PHP"}, ".php_cs.dist": {"PHP"},
".prettierignore": {"Ignore List"},
".profile": {"Shell"}, ".profile": {"Shell"},
".project": {"XML"}, ".project": {"XML"},
".pryrc": {"Ruby"}, ".pryrc": {"Ruby"},
".spacemacs": {"Emacs Lisp"}, ".spacemacs": {"Emacs Lisp"},
".stylelintignore": {"Ignore List"},
".tern-config": {"JSON"},
".tern-project": {"JSON"},
".vimrc": {"Vim script"}, ".vimrc": {"Vim script"},
".viper": {"Emacs Lisp"}, ".viper": {"Emacs Lisp"},
".vscodeignore": {"Ignore List"},
".watchmanconfig": {"JSON"},
".zlogin": {"Shell"}, ".zlogin": {"Shell"},
".zlogout": {"Shell"}, ".zlogout": {"Shell"},
".zprofile": {"Shell"}, ".zprofile": {"Shell"},
@ -55,6 +79,7 @@ var LanguagesByFilename = map[string][]string{
"BSDmakefile": {"Makefile"}, "BSDmakefile": {"Makefile"},
"BUCK": {"Python"}, "BUCK": {"Python"},
"BUILD": {"Python"}, "BUILD": {"Python"},
"BUILD.bazel": {"Python"},
"Berksfile": {"Ruby"}, "Berksfile": {"Ruby"},
"Brewfile": {"Ruby"}, "Brewfile": {"Ruby"},
"Buildfile": {"Ruby"}, "Buildfile": {"Ruby"},
@ -64,6 +89,7 @@ var LanguagesByFilename = map[string][]string{
"COPYRIGHT.regex": {"Text"}, "COPYRIGHT.regex": {"Text"},
"Cakefile": {"CoffeeScript"}, "Cakefile": {"CoffeeScript"},
"Capfile": {"Ruby"}, "Capfile": {"Ruby"},
"Cargo.lock": {"TOML"},
"Cask": {"Emacs Lisp"}, "Cask": {"Emacs Lisp"},
"Dangerfile": {"Ruby"}, "Dangerfile": {"Ruby"},
"Deliverfile": {"Ruby"}, "Deliverfile": {"Ruby"},
@ -75,6 +101,7 @@ var LanguagesByFilename = map[string][]string{
"GNUmakefile": {"Makefile"}, "GNUmakefile": {"Makefile"},
"Gemfile": {"Ruby"}, "Gemfile": {"Ruby"},
"Gemfile.lock": {"Ruby"}, "Gemfile.lock": {"Ruby"},
"Gopkg.lock": {"TOML"},
"Guardfile": {"Ruby"}, "Guardfile": {"Ruby"},
"INSTALL": {"Text"}, "INSTALL": {"Text"},
"INSTALL.mysql": {"Text"}, "INSTALL.mysql": {"Text"},
@ -85,6 +112,7 @@ var LanguagesByFilename = map[string][]string{
"LICENSE": {"Text"}, "LICENSE": {"Text"},
"LICENSE.mysql": {"Text"}, "LICENSE.mysql": {"Text"},
"Makefile": {"Makefile"}, "Makefile": {"Makefile"},
"Makefile.PL": {"Perl"},
"Makefile.am": {"Makefile"}, "Makefile.am": {"Makefile"},
"Makefile.boot": {"Makefile"}, "Makefile.boot": {"Makefile"},
"Makefile.frag": {"Makefile"}, "Makefile.frag": {"Makefile"},
@ -107,7 +135,7 @@ var LanguagesByFilename = map[string][]string{
"README.mysql": {"Text"}, "README.mysql": {"Text"},
"ROOT": {"Isabelle ROOT"}, "ROOT": {"Isabelle ROOT"},
"Rakefile": {"Ruby"}, "Rakefile": {"Ruby"},
"Rexfile": {"Perl 6"}, "Rexfile": {"Perl"},
"SConscript": {"Python"}, "SConscript": {"Python"},
"SConstruct": {"Python"}, "SConstruct": {"Python"},
"Settings.StyleCop": {"XML"}, "Settings.StyleCop": {"XML"},
@ -127,6 +155,7 @@ var LanguagesByFilename = map[string][]string{
"ack": {"Perl"}, "ack": {"Perl"},
"ant.xml": {"Ant Build System"}, "ant.xml": {"Ant Build System"},
"apache2.conf": {"ApacheConf"}, "apache2.conf": {"ApacheConf"},
"bash_aliases": {"Shell"},
"bash_logout": {"Shell"}, "bash_logout": {"Shell"},
"bash_profile": {"Shell"}, "bash_profile": {"Shell"},
"bashrc": {"Shell"}, "bashrc": {"Shell"},
@ -136,19 +165,34 @@ var LanguagesByFilename = map[string][]string{
"click.me": {"Text"}, "click.me": {"Text"},
"composer.lock": {"JSON"}, "composer.lock": {"JSON"},
"configure.ac": {"M4Sugar"}, "configure.ac": {"M4Sugar"},
"contents.lr": {"Markdown"},
"cpanfile": {"Perl"}, "cpanfile": {"Perl"},
"cshrc": {"Shell"}, "cshrc": {"Shell"},
"delete.me": {"Text"}, "delete.me": {"Text"},
"descrip.mmk": {"Module Management System"}, "descrip.mmk": {"Module Management System"},
"descrip.mms": {"Module Management System"}, "descrip.mms": {"Module Management System"},
"encodings.dir": {"X Font Directory Index"},
"expr-dist": {"R"}, "expr-dist": {"R"},
"firestore.rules": {"Cloud Firestore Security Rules"},
"fonts.alias": {"X Font Directory Index"},
"fonts.dir": {"X Font Directory Index"},
"fonts.scale": {"X Font Directory Index"},
"fp-lib-table": {"KiCad Layout"}, "fp-lib-table": {"KiCad Layout"},
"gitignore-global": {"Ignore List"},
"gitignore_global": {"Ignore List"},
"glide.lock": {"YAML"},
"go.mod": {"Text"},
"go.sum": {"Text"},
"gradlew": {"Shell"}, "gradlew": {"Shell"},
"gvimrc": {"Vim script"}, "gvimrc": {"Vim script"},
"haproxy.cfg": {"HAProxy"},
"httpd.conf": {"ApacheConf"}, "httpd.conf": {"ApacheConf"},
"jsconfig.json": {"JSON with Comments"},
"keep.me": {"Text"}, "keep.me": {"Text"},
"ld.script": {"Linker Script"}, "ld.script": {"Linker Script"},
"login": {"Shell"}, "login": {"Shell"},
"m3makefile": {"Quake"},
"m3overrides": {"Quake"},
"makefile": {"Makefile"}, "makefile": {"Makefile"},
"makefile.sco": {"Makefile"}, "makefile.sco": {"Makefile"},
"man": {"Shell"}, "man": {"Shell"},
@ -159,7 +203,10 @@ var LanguagesByFilename = map[string][]string{
"mkfile": {"Makefile"}, "mkfile": {"Makefile"},
"mmn": {"Roff"}, "mmn": {"Roff"},
"mmt": {"Roff"}, "mmt": {"Roff"},
"nanorc": {"nanorc"},
"nextflow.config": {"Nextflow"},
"nginx.conf": {"Nginx"}, "nginx.conf": {"Nginx"},
"nim.cfg": {"Nim"},
"nvimrc": {"Vim script"}, "nvimrc": {"Vim script"},
"owh": {"Tcl"}, "owh": {"Tcl"},
"packages.config": {"XML"}, "packages.config": {"XML"},
@ -171,9 +218,9 @@ var LanguagesByFilename = map[string][]string{
"rebar.config.lock": {"Erlang"}, "rebar.config.lock": {"Erlang"},
"rebar.lock": {"Erlang"}, "rebar.lock": {"Erlang"},
"riemann.config": {"Clojure"}, "riemann.config": {"Clojure"},
"script": {"C"},
"starfield": {"Tcl"}, "starfield": {"Tcl"},
"test.me": {"Text"}, "test.me": {"Text"},
"tsconfig.json": {"JSON with Comments"},
"vimrc": {"Vim script"}, "vimrc": {"Vim script"},
"wscript": {"Python"}, "wscript": {"Python"},
"xcompose": {"XCompose"}, "xcompose": {"XCompose"},

75101
data/frequencies.go
View File

File diff suppressed because it is too large Load Diff

35
data/heuristics.go Normal file
View File

@ -0,0 +1,35 @@
package data
import "gopkg.in/src-d/enry.v1/data/rule"
// Heuristics implements a rule-based content matching engine.
// Heuristics is a number of sequntially applied rule.Heuristic where a
// matching one disambiguages language(s) for a single file extension.
type Heuristics []rule.Heuristic
// Match returns languages identified by the matching rule of the heuristic.
func (hs Heuristics) Match(data []byte) []string {
var matchedLangs []string
for _, heuristic := range hs {
if heuristic.Match(data) {
for _, langOrAlias := range heuristic.Languages() {
lang, ok := LanguageByAlias(langOrAlias)
if !ok { // should never happen
// reaching here means language name/alias in heuristics.yml
// is not consistent with languages.yml
// but we do not surface any such error at the API
continue
}
matchedLangs = append(matchedLangs, lang)
}
break
}
}
return matchedLangs
}
// matchString is a convenience used only in tests.
func (hs *Heuristics) matchString(data string) []string {
return hs.Match([]byte(data))
}

61
data/heuristics_test.go Normal file
View File

@ -0,0 +1,61 @@
package data
import (
"regexp"
"testing"
"github.com/stretchr/testify/assert"
"gopkg.in/src-d/enry.v1/data/rule"
)
var testContentHeuristics = map[string]*Heuristics{
".md": &Heuristics{ // final pattern for parsed YAML rule
rule.Or(
rule.MatchingLanguages("Markdown"),
regexp.MustCompile(`(^[-A-Za-z0-9=#!\*\[|>])|<\/ | \A\z`),
),
rule.Or(
rule.MatchingLanguages("GCC Machine Description"),
regexp.MustCompile(`^(;;|\(define_)`),
),
rule.Always(
rule.MatchingLanguages("Markdown"),
),
},
".ms": &Heuristics{
// Order defines precedence: And, Or, Not, Named, Always
rule.And(
rule.MatchingLanguages("Unix Assembly"),
rule.Not(rule.MatchingLanguages(""), regexp.MustCompile(`/\*`)),
rule.Or(
rule.MatchingLanguages(""),
regexp.MustCompile(`^\s*\.(?:include\s|globa?l\s|[A-Za-z][_A-Za-z0-9]*:)`),
),
),
rule.Or(
rule.MatchingLanguages("Roff"),
regexp.MustCompile(`^[.''][A-Za-z]{2}(\s|$)`),
),
rule.Always(
rule.MatchingLanguages("MAXScript"),
),
},
}
func TestContentHeuristic_MatchingAlways(t *testing.T) {
lang := testContentHeuristics[".md"].matchString("")
assert.Equal(t, []string{"Markdown"}, lang)
lang = testContentHeuristics[".ms"].matchString("")
assert.Equal(t, []string{"MAXScript"}, lang)
}
func TestContentHeuristic_MatchingAnd(t *testing.T) {
lang := testContentHeuristics[".md"].matchString(";;")
assert.Equal(t, []string{"GCC Machine Description"}, lang)
}
func TestContentHeuristic_MatchingOr(t *testing.T) {
lang := testContentHeuristics[".ms"].matchString(" .include \"math.s\"")
assert.Equal(t, []string{"Unix Assembly"}, lang)
}

17
data/interpreter.go
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -8,6 +8,7 @@ var LanguagesByInterpreter = map[string][]string{
"apl": {"APL"}, "apl": {"APL"},
"aplx": {"APL"}, "aplx": {"APL"},
"ash": {"Shell"}, "ash": {"Shell"},
"asy": {"Asymptote"},
"awk": {"Awk"}, "awk": {"Awk"},
"bash": {"Shell"}, "bash": {"Shell"},
"bigloo": {"Scheme"}, "bigloo": {"Scheme"},
@ -16,9 +17,11 @@ var LanguagesByInterpreter = map[string][]string{
"chicken": {"Scheme"}, "chicken": {"Scheme"},
"clisp": {"Common Lisp"}, "clisp": {"Common Lisp"},
"coffee": {"CoffeeScript"}, "coffee": {"CoffeeScript"},
"cperl": {"Perl"},
"crystal": {"Crystal"}, "crystal": {"Crystal"},
"csi": {"Scheme"}, "csi": {"Scheme"},
"cvc4": {"SMT"}, "cvc4": {"SMT"},
"cwl-runner": {"Common Workflow Language"},
"dart": {"Dart"}, "dart": {"Dart"},
"dash": {"Shell"}, "dash": {"Shell"},
"dtrace": {"DTrace"}, "dtrace": {"DTrace"},
@ -34,7 +37,9 @@ var LanguagesByInterpreter = map[string][]string{
"gnuplot": {"Gnuplot"}, "gnuplot": {"Gnuplot"},
"gosh": {"Scheme"}, "gosh": {"Scheme"},
"groovy": {"Groovy"}, "groovy": {"Groovy"},
"gsed": {"sed"},
"guile": {"Scheme"}, "guile": {"Scheme"},
"hy": {"Hy"},
"instantfpc": {"Pascal"}, "instantfpc": {"Pascal"},
"io": {"Io"}, "io": {"Io"},
"ioke": {"Ioke"}, "ioke": {"Ioke"},
@ -50,12 +55,14 @@ var LanguagesByInterpreter = map[string][]string{
"make": {"Makefile"}, "make": {"Makefile"},
"mathsat5": {"SMT"}, "mathsat5": {"SMT"},
"mawk": {"Awk"}, "mawk": {"Awk"},
"minised": {"sed"},
"mksh": {"Shell"}, "mksh": {"Shell"},
"mmi": {"Mercury"}, "mmi": {"Mercury"},
"moon": {"MoonScript"}, "moon": {"MoonScript"},
"nawk": {"Awk"}, "nawk": {"Awk"},
"newlisp": {"NewLisp"}, "newlisp": {"NewLisp"},
"node": {"JavaScript"}, "nextflow": {"Nextflow"},
"node": {"JavaScript", "TypeScript"},
"nush": {"Nu"}, "nush": {"Nu"},
"ocaml": {"OCaml", "Reason"}, "ocaml": {"OCaml", "Reason"},
"ocamlrun": {"OCaml"}, "ocamlrun": {"OCaml"},
@ -66,11 +73,12 @@ var LanguagesByInterpreter = map[string][]string{
"parrot": {"Parrot Assembly", "Parrot Internal Representation"}, "parrot": {"Parrot Assembly", "Parrot Internal Representation"},
"pdksh": {"Shell"}, "pdksh": {"Shell"},
"perl": {"Perl", "Pod"}, "perl": {"Perl", "Pod"},
"perl6": {"Perl 6"}, "perl6": {"Perl 6", "Pod 6"},
"php": {"PHP"}, "php": {"PHP"},
"picolisp": {"PicoLisp"}, "picolisp": {"PicoLisp"},
"pike": {"Pike"}, "pike": {"Pike"},
"pil": {"PicoLisp"}, "pil": {"PicoLisp"},
"pwsh": {"PowerShell"},
"python": {"Python"}, "python": {"Python"},
"python2": {"Python"}, "python2": {"Python"},
"python3": {"Python"}, "python3": {"Python"},
@ -87,11 +95,14 @@ var LanguagesByInterpreter = map[string][]string{
"runhaskell": {"Haskell"}, "runhaskell": {"Haskell"},
"sbcl": {"Common Lisp"}, "sbcl": {"Common Lisp"},
"scala": {"Scala"}, "scala": {"Scala"},
"scheme": {"Scheme"},
"sclang": {"SuperCollider"}, "sclang": {"SuperCollider"},
"scsynth": {"SuperCollider"}, "scsynth": {"SuperCollider"},
"sed": {"sed"},
"sh": {"Shell"}, "sh": {"Shell"},
"smt-rat": {"SMT"}, "smt-rat": {"SMT"},
"smtinterpol": {"SMT"}, "smtinterpol": {"SMT"},
"ssed": {"sed"},
"stp": {"SMT"}, "stp": {"SMT"},
"swipl": {"Prolog"}, "swipl": {"Prolog"},
"tcc": {"C"}, "tcc": {"C"},

391
data/mimeType.go
View File

@ -1,204 +1,219 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
var LanguagesMime = map[string]string{ var LanguagesMime = map[string]string{
"AGS Script": "text/x-c++src", "AGS Script": "text/x-c++src",
"APL": "text/apl", "APL": "text/apl",
"ASN.1": "text/x-ttcn-asn", "ASN.1": "text/x-ttcn-asn",
"ASP": "application/x-aspx", "ASP": "application/x-aspx",
"Alpine Abuild": "text/x-sh", "Alpine Abuild": "text/x-sh",
"Ant Build System": "application/xml", "AngelScript": "text/x-c++src",
"Apex": "text/x-java", "Ant Build System": "application/xml",
"Arduino": "text/x-c++src", "Apex": "text/x-java",
"Brainfuck": "text/x-brainfuck", "Asymptote": "text/x-kotlin",
"C": "text/x-csrc", "Brainfuck": "text/x-brainfuck",
"C#": "text/x-csharp", "C": "text/x-csrc",
"C++": "text/x-c++src", "C#": "text/x-csharp",
"C2hs Haskell": "text/x-haskell", "C++": "text/x-c++src",
"CMake": "text/x-cmake", "C2hs Haskell": "text/x-haskell",
"COBOL": "text/x-cobol", "CMake": "text/x-cmake",
"COLLADA": "text/xml", "COBOL": "text/x-cobol",
"CSON": "text/x-coffeescript", "COLLADA": "text/xml",
"CSS": "text/css", "CSON": "text/x-coffeescript",
"ChucK": "text/x-java", "CSS": "text/css",
"Clojure": "text/x-clojure", "ChucK": "text/x-java",
"Closure Templates": "text/x-soy", "Clojure": "text/x-clojure",
"CoffeeScript": "text/x-coffeescript", "Closure Templates": "text/x-soy",
"Common Lisp": "text/x-common-lisp", "Cloud Firestore Security Rules": "text/css",
"Component Pascal": "text/x-pascal", "CoffeeScript": "text/x-coffeescript",
"Crystal": "text/x-crystal", "Common Lisp": "text/x-common-lisp",
"Cuda": "text/x-c++src", "Common Workflow Language": "text/x-yaml",
"Cycript": "text/javascript", "Component Pascal": "text/x-pascal",
"Cython": "text/x-cython", "Crystal": "text/x-crystal",
"D": "text/x-d", "Cuda": "text/x-c++src",
"DTrace": "text/x-csrc", "Cycript": "text/javascript",
"Dart": "application/dart", "Cython": "text/x-cython",
"Diff": "text/x-diff", "D": "text/x-d",
"Dockerfile": "text/x-dockerfile", "DTrace": "text/x-csrc",
"Dylan": "text/x-dylan", "Dart": "application/dart",
"EBNF": "text/x-ebnf", "Diff": "text/x-diff",
"ECL": "text/x-ecl", "Dockerfile": "text/x-dockerfile",
"EQ": "text/x-csharp", "Dylan": "text/x-dylan",
"Eagle": "text/xml", "EBNF": "text/x-ebnf",
"Easybuild": "text/x-python", "ECL": "text/x-ecl",
"Ecere Projects": "application/json", "EQ": "text/x-csharp",
"Eiffel": "text/x-eiffel", "Eagle": "text/xml",
"Elm": "text/x-elm", "Easybuild": "text/x-python",
"Emacs Lisp": "text/x-common-lisp", "Ecere Projects": "application/json",
"EmberScript": "text/x-coffeescript", "Edje Data Collection": "application/json",
"Erlang": "text/x-erlang", "Eiffel": "text/x-eiffel",
"F#": "text/x-fsharp", "Elm": "text/x-elm",
"Factor": "text/x-factor", "Emacs Lisp": "text/x-common-lisp",
"Forth": "text/x-forth", "EmberScript": "text/x-coffeescript",
"Fortran": "text/x-fortran", "Erlang": "text/x-erlang",
"GCC Machine Description": "text/x-common-lisp", "F#": "text/x-fsharp",
"Factor": "text/x-factor",
"Forth": "text/x-forth",
"Fortran": "text/x-fortran",
"GCC Machine Description": "text/x-common-lisp",
"GN": "text/x-python", "GN": "text/x-python",
"Game Maker Language": "text/x-c++src", "Game Maker Language": "text/x-c++src",
"Genshi": "text/xml", "Genshi": "text/xml",
"Gentoo Ebuild": "text/x-sh", "Gentoo Ebuild": "text/x-sh",
"Gentoo Eclass": "text/x-sh", "Gentoo Eclass": "text/x-sh",
"Git Attributes": "text/x-sh",
"Git Config": "text/x-properties",
"Glyph": "text/x-tcl", "Glyph": "text/x-tcl",
"Go": "text/x-go", "Go": "text/x-go",
"Grammatical Framework": "text/x-haskell", "Grammatical Framework": "text/x-haskell",
"Groovy": "text/x-groovy", "Groovy": "text/x-groovy",
"Groovy Server Pages": "application/x-jsp", "Groovy Server Pages": "application/x-jsp",
"HCL": "text/x-ruby", "HCL": "text/x-ruby",
"HTML": "text/html", "HTML": "text/html",
"HTML+Django": "text/x-django", "HTML+Django": "text/x-django",
"HTML+ECR": "text/html", "HTML+ECR": "text/html",
"HTML+EEX": "text/html", "HTML+EEX": "text/html",
"HTML+ERB": "application/x-erb", "HTML+ERB": "application/x-erb",
"HTML+PHP": "application/x-httpd-php", "HTML+PHP": "application/x-httpd-php",
"HTTP": "message/http", "HTML+Razor": "text/html",
"Hack": "application/x-httpd-php", "HTTP": "message/http",
"Haml": "text/x-haml", "Hack": "application/x-httpd-php",
"Haskell": "text/x-haskell", "Haml": "text/x-haml",
"Haxe": "text/x-haxe", "Haskell": "text/x-haskell",
"IDL": "text/x-idl", "Haxe": "text/x-haxe",
"INI": "text/x-properties", "IDL": "text/x-idl",
"IRC log": "text/mirc", "INI": "text/x-properties",
"JSON": "application/json", "IRC log": "text/mirc",
"JSON5": "application/json", "Ignore List": "text/x-sh",
"JSONiq": "application/json", "JSON": "application/json",
"JSX": "text/jsx", "JSON with Comments": "text/javascript",
"Java": "text/x-java", "JSON5": "application/json",
"Java Server Pages": "application/x-jsp", "JSONLD": "application/json",
"JavaScript": "text/javascript", "JSONiq": "application/json",
"Julia": "text/x-julia", "JSX": "text/jsx",
"Jupyter Notebook": "application/json", "Java": "text/x-java",
"KiCad Layout": "text/x-common-lisp", "Java Properties": "text/x-properties",
"Kit": "text/html", "Java Server Pages": "application/x-jsp",
"Kotlin": "text/x-kotlin", "JavaScript": "text/javascript",
"LFE": "text/x-common-lisp", "Julia": "text/x-julia",
"LabVIEW": "text/xml", "Jupyter Notebook": "application/json",
"Latte": "text/x-smarty", "KiCad Layout": "text/x-common-lisp",
"Less": "text/css", "Kit": "text/html",
"Literate Haskell": "text/x-literate-haskell", "Kotlin": "text/x-kotlin",
"LiveScript": "text/x-livescript", "LFE": "text/x-common-lisp",
"LookML": "text/x-yaml", "LTspice Symbol": "text/x-spreadsheet",
"Lua": "text/x-lua", "LabVIEW": "text/xml",
"M": "text/x-mumps", "Latte": "text/x-smarty",
"MTML": "text/html", "Less": "text/css",
"MUF": "text/x-forth", "Literate Haskell": "text/x-literate-haskell",
"Makefile": "text/x-cmake", "LiveScript": "text/x-livescript",
"Markdown": "text/x-gfm", "LookML": "text/x-yaml",
"Marko": "text/html", "Lua": "text/x-lua",
"Mathematica": "text/x-mathematica", "M": "text/x-mumps",
"Matlab": "text/x-octave", "MATLAB": "text/x-octave",
"Maven POM": "text/xml", "MTML": "text/html",
"Max": "application/json", "MUF": "text/x-forth",
"Metal": "text/x-c++src", "Makefile": "text/x-cmake",
"Mirah": "text/x-ruby", "Markdown": "text/x-gfm",
"Modelica": "text/x-modelica", "Marko": "text/html",
"NSIS": "text/x-nsis", "Mathematica": "text/x-mathematica",
"NetLogo": "text/x-common-lisp", "Maven POM": "text/xml",
"NewLisp": "text/x-common-lisp", "Max": "application/json",
"Nginx": "text/x-nginx-conf", "Metal": "text/x-c++src",
"Nu": "text/x-scheme", "Mirah": "text/x-ruby",
"NumPy": "text/x-python", "Modelica": "text/x-modelica",
"OCaml": "text/x-ocaml", "NSIS": "text/x-nsis",
"Objective-C": "text/x-objectivec", "NetLogo": "text/x-common-lisp",
"Objective-C++": "text/x-objectivec", "NewLisp": "text/x-common-lisp",
"OpenCL": "text/x-csrc", "Nginx": "text/x-nginx-conf",
"OpenRC runscript": "text/x-sh", "Nu": "text/x-scheme",
"Oz": "text/x-oz", "NumPy": "text/x-python",
"PHP": "application/x-httpd-php", "OCaml": "text/x-ocaml",
"PLSQL": "text/x-plsql", "Objective-C": "text/x-objectivec",
"PLpgSQL": "text/x-sql", "Objective-C++": "text/x-objectivec",
"Pascal": "text/x-pascal", "OpenCL": "text/x-csrc",
"Perl": "text/x-perl", "OpenRC runscript": "text/x-sh",
"Perl 6": "text/x-perl", "Oz": "text/x-oz",
"Pic": "text/troff", "PHP": "application/x-httpd-php",
"Pod": "text/x-perl", "PLSQL": "text/x-plsql",
"PowerShell": "application/x-powershell", "PLpgSQL": "text/x-sql",
"Protocol Buffer": "text/x-protobuf", "Pascal": "text/x-pascal",
"Public Key": "application/pgp", "Perl": "text/x-perl",
"Pug": "text/x-pug", "Perl 6": "text/x-perl",
"Puppet": "text/x-puppet", "Pic": "text/troff",
"PureScript": "text/x-haskell", "Pod": "text/x-perl",
"Python": "text/x-python", "PowerShell": "application/x-powershell",
"R": "text/x-rsrc", "Protocol Buffer": "text/x-protobuf",
"RAML": "text/x-yaml", "Public Key": "application/pgp",
"RHTML": "application/x-erb", "Pug": "text/x-pug",
"RMarkdown": "text/x-gfm", "Puppet": "text/x-puppet",
"RPM Spec": "text/x-rpm-spec", "PureScript": "text/x-haskell",
"Reason": "text/x-rustsrc", "Python": "text/x-python",
"Roff": "text/troff", "R": "text/x-rsrc",
"Rouge": "text/x-clojure", "RAML": "text/x-yaml",
"Ruby": "text/x-ruby", "RHTML": "application/x-erb",
"Rust": "text/x-rustsrc", "RMarkdown": "text/x-gfm",
"SAS": "text/x-sas", "RPM Spec": "text/x-rpm-spec",
"SCSS": "text/x-scss", "Reason": "text/x-rustsrc",
"SPARQL": "application/sparql-query", "Roff": "text/troff",
"SQL": "text/x-sql", "Roff Manpage": "text/troff",
"SQLPL": "text/x-sql", "Rouge": "text/x-clojure",
"SRecode Template": "text/x-common-lisp", "Ruby": "text/x-ruby",
"SVG": "text/xml", "Rust": "text/x-rustsrc",
"Sage": "text/x-python", "SAS": "text/x-sas",
"SaltStack": "text/x-yaml", "SCSS": "text/x-scss",
"Sass": "text/x-sass", "SPARQL": "application/sparql-query",
"Scala": "text/x-scala", "SQL": "text/x-sql",
"Scheme": "text/x-scheme", "SQLPL": "text/x-sql",
"Shell": "text/x-sh", "SRecode Template": "text/x-common-lisp",
"ShellSession": "text/x-sh", "SVG": "text/xml",
"Slim": "text/x-slim", "Sage": "text/x-python",
"Smalltalk": "text/x-stsrc", "SaltStack": "text/x-yaml",
"Smarty": "text/x-smarty", "Sass": "text/x-sass",
"Squirrel": "text/x-c++src", "Scala": "text/x-scala",
"Standard ML": "text/x-ocaml", "Scheme": "text/x-scheme",
"Sublime Text Config": "text/javascript", "Shell": "text/x-sh",
"Swift": "text/x-swift", "ShellSession": "text/x-sh",
"SystemVerilog": "text/x-systemverilog", "Slim": "text/x-slim",
"TOML": "text/x-toml", "Smalltalk": "text/x-stsrc",
"Tcl": "text/x-tcl", "Smarty": "text/x-smarty",
"Tcsh": "text/x-sh", "Squirrel": "text/x-c++src",
"TeX": "text/x-stex", "Standard ML": "text/x-ocaml",
"Terra": "text/x-lua", "Swift": "text/x-swift",
"Textile": "text/x-textile", "SystemVerilog": "text/x-systemverilog",
"Turtle": "text/turtle", "TOML": "text/x-toml",
"Twig": "text/x-twig", "Tcl": "text/x-tcl",
"TypeScript": "application/typescript", "Tcsh": "text/x-sh",
"Unified Parallel C": "text/x-csrc", "TeX": "text/x-stex",
"Unity3D Asset": "text/x-yaml", "Terra": "text/x-lua",
"Uno": "text/x-csharp", "Textile": "text/x-textile",
"UnrealScript": "text/x-java", "Turtle": "text/turtle",
"VHDL": "text/x-vhdl", "Twig": "text/x-twig",
"Verilog": "text/x-verilog", "TypeScript": "application/typescript",
"Visual Basic": "text/x-vb", "Unified Parallel C": "text/x-csrc",
"Volt": "text/x-d", "Unity3D Asset": "text/x-yaml",
"WebAssembly": "text/x-common-lisp", "Uno": "text/x-csharp",
"WebIDL": "text/x-webidl", "UnrealScript": "text/x-java",
"XC": "text/x-csrc", "VHDL": "text/x-vhdl",
"XML": "text/xml", "Verilog": "text/x-verilog",
"XPages": "text/xml", "Visual Basic": "text/x-vb",
"XProc": "text/xml", "Volt": "text/x-d",
"XQuery": "application/xquery", "WebAssembly": "text/x-common-lisp",
"XS": "text/x-csrc", "WebIDL": "text/x-webidl",
"XSLT": "text/xml", "Windows Registry Entries": "text/x-properties",
"YAML": "text/x-yaml", "X BitMap": "text/x-csrc",
"edn": "text/x-clojure", "X PixMap": "text/x-csrc",
"reStructuredText": "text/x-rst", "XC": "text/x-csrc",
"wisp": "text/x-clojure", "XML": "text/xml",
"XPages": "text/xml",
"XProc": "text/xml",
"XQuery": "application/xquery",
"XS": "text/x-csrc",
"XSLT": "text/xml",
"YAML": "text/x-yaml",
"edn": "text/x-clojure",
"reStructuredText": "text/x-rst",
"wisp": "text/x-clojure",
} }

109
data/rule/rule.go Normal file
View File

@ -0,0 +1,109 @@
// Package rule contains rule-based heuristic implementations.
// It is used in the generated code in content.go for disambiguation of languages
// with colliding extensions, based on regexps from Linguist data.
package rule
// Heuristic consist of (a number of) rules where each, if matches,
// identifes content as belonging to a programming language(s).
type Heuristic interface {
Matcher
Languages() []string
}
// Matcher checks if the data matches (number of) pattern.
// Every heuristic rule below implements this interface.
// A regexp.Regexp satisfies this interface and can be used instead.
type Matcher interface {
Match(data []byte) bool
}
// languages struct incapsulate data common to every Matcher: all languages
// that it identifies.
type languages struct {
langs []string
}
// Languages returns all languages, identified by this Matcher.
func (l languages) Languages() []string {
return l.langs
}
// MatchingLanguages is a helper to create new languages.
func MatchingLanguages(langs ...string) languages {
return languages{langs}
}
// Implements a Heuristic.
type or struct {
languages
pattern Matcher
}
// Or rule matches, if a single matching pattern exists.
// It recives only one pattern as it relies on compile-time optimization that
// represtes union with | inside a single regexp.
func Or(l languages, r Matcher) Heuristic {
return or{l, r}
}
// Match implements rule.Matcher.
func (r or) Match(data []byte) bool {
return r.pattern.Match(data)
}
// Implements a Heuristic.
type and struct {
languages
patterns []Matcher
}
// And rule matches, if each of the patterns does match.
func And(l languages, m ...Matcher) Heuristic {
return and{l, m}
}
// Match implements data.Matcher.
func (r and) Match(data []byte) bool {
for _, p := range r.patterns {
if !p.Match(data) {
return false
}
}
return true
}
// Implements a Heuristic.
type not struct {
languages
Patterns []Matcher
}
// Not rule matches if none of the patterns match.
func Not(l languages, r ...Matcher) Heuristic {
return not{l, r}
}
// Match implements data.Matcher.
func (r not) Match(data []byte) bool {
for _, p := range r.Patterns {
if p.Match(data) {
return false
}
}
return true
}
// Implements a Heuristic.
type always struct {
languages
}
// Always rule always matches. Often is used as a default fallback.
func Always(l languages) Heuristic {
return always{l}
}
// Match implements Matcher.
func (r always) Match(data []byte) bool {
return true
}

39
data/rule/rule_test.go Normal file
View File

@ -0,0 +1,39 @@
package rule
import (
"regexp"
"testing"
"github.com/stretchr/testify/assert"
)
const lang = "ActionScript"
var fixtures = []struct {
name string
rule Heuristic
numLangs int
matching string
noMatch string
}{
{"Always", Always(MatchingLanguages(lang)), 1, "a", ""},
{"Not", Not(MatchingLanguages(lang), regexp.MustCompile(`a`)), 1, "b", "a"},
{"And", And(MatchingLanguages(lang), regexp.MustCompile(`a`), regexp.MustCompile(`b`)), 1, "ab", "a"},
{"Or", Or(MatchingLanguages(lang), regexp.MustCompile(`a|b`)), 1, "ab", "c"},
}
func TestRules(t *testing.T) {
for _, f := range fixtures {
t.Run(f.name, func(t *testing.T) {
assert.NotNil(t, f.rule)
assert.NotNil(t, f.rule.Languages())
assert.Equal(t, f.numLangs, len(f.rule.Languages()))
assert.Truef(t, f.rule.Match([]byte(f.matching)),
"'%s' is expected to .Match() by rule %s%v", f.matching, f.name, f.rule)
if f.noMatch != "" {
assert.Falsef(t, f.rule.Match([]byte(f.noMatch)),
"'%s' is expected NOT to .Match() by rule %s%v", f.noMatch, f.name, f.rule)
}
})
}
}

622
data/type.go
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -21,282 +21,302 @@ var LanguagesType = map[string]int{
"Agda": 2, "Agda": 2,
"Alloy": 2, "Alloy": 2,
"Alpine Abuild": 2, "Alpine Abuild": 2,
"AngelScript": 2,
"Ant Build System": 1, "Ant Build System": 1,
"ApacheConf": 1, "ApacheConf": 1,
"Apex": 2, "Apex": 2,
"Apollo Guidance Computer": 2, "Apollo Guidance Computer": 2,
"AppleScript": 2, "AppleScript": 2,
"Arc": 2, "Arc": 2,
"Arduino": 2, "AsciiDoc": 4,
"AsciiDoc": 4, "AspectJ": 2,
"AspectJ": 2, "Assembly": 2,
"Assembly": 2, "Asymptote": 2,
"Augeas": 2, "Augeas": 2,
"AutoHotkey": 2, "AutoHotkey": 2,
"AutoIt": 2, "AutoIt": 2,
"Awk": 2, "Awk": 2,
"Ballerina": 2, "Ballerina": 2,
"Batchfile": 2, "Batchfile": 2,
"Befunge": 2, "Befunge": 2,
"Bison": 2, "Bison": 2,
"BitBake": 2, "BitBake": 2,
"Blade": 3, "Blade": 3,
"BlitzBasic": 2, "BlitzBasic": 2,
"BlitzMax": 2, "BlitzMax": 2,
"Bluespec": 2, "Bluespec": 2,
"Boo": 2, "Boo": 2,
"Brainfuck": 2, "Brainfuck": 2,
"Brightscript": 2, "Brightscript": 2,
"Bro": 2, "Bro": 2,
"C": 2, "C": 2,
"C#": 2, "C#": 2,
"C++": 2, "C++": 2,
"C-ObjDump": 1, "C-ObjDump": 1,
"C2hs Haskell": 2, "C2hs Haskell": 2,
"CLIPS": 2, "CLIPS": 2,
"CMake": 2, "CMake": 2,
"COBOL": 2, "COBOL": 2,
"COLLADA": 1, "COLLADA": 1,
"CSON": 1, "CSON": 1,
"CSS": 3, "CSS": 3,
"CSV": 1, "CSV": 1,
"CWeb": 2, "CWeb": 2,
"Cap'n Proto": 2, "Cap'n Proto": 2,
"CartoCSS": 2, "CartoCSS": 2,
"Ceylon": 2, "Ceylon": 2,
"Chapel": 2, "Chapel": 2,
"Charity": 2, "Charity": 2,
"ChucK": 2, "ChucK": 2,
"Cirru": 2, "Cirru": 2,
"Clarion": 2, "Clarion": 2,
"Clean": 2, "Clean": 2,
"Click": 2, "Click": 2,
"Clojure": 2, "Clojure": 2,
"Closure Templates": 3, "Closure Templates": 3,
"CoffeeScript": 2, "Cloud Firestore Security Rules": 1,
"ColdFusion": 2, "CoNLL-U": 1,
"ColdFusion CFC": 2, "CoffeeScript": 2,
"Common Lisp": 2, "ColdFusion": 2,
"Component Pascal": 2, "ColdFusion CFC": 2,
"Cool": 2, "Common Lisp": 2,
"Coq": 2, "Common Workflow Language": 2,
"Cpp-ObjDump": 1, "Component Pascal": 2,
"Creole": 4, "Cool": 2,
"Crystal": 2, "Coq": 2,
"Csound": 2, "Cpp-ObjDump": 1,
"Csound Document": 2, "Creole": 4,
"Csound Score": 2, "Crystal": 2,
"Cuda": 2, "Csound": 2,
"Cycript": 2, "Csound Document": 2,
"Cython": 2, "Csound Score": 2,
"D": 2, "Cuda": 2,
"D-ObjDump": 1, "Cycript": 2,
"DIGITAL Command Language": 2, "Cython": 2,
"DM": 2, "D": 2,
"DNS Zone": 1, "D-ObjDump": 1,
"DTrace": 2, "DIGITAL Command Language": 2,
"Darcs Patch": 1, "DM": 2,
"Dart": 2, "DNS Zone": 1,
"DataWeave": 2, "DTrace": 2,
"Diff": 1, "Darcs Patch": 1,
"Dockerfile": 1, "Dart": 2,
"Dogescript": 2, "DataWeave": 2,
"Dylan": 2, "Diff": 1,
"E": 2, "Dockerfile": 2,
"EBNF": 1, "Dogescript": 2,
"ECL": 2, "Dylan": 2,
"ECLiPSe": 2, "E": 2,
"EJS": 3, "EBNF": 1,
"EQ": 2, "ECL": 2,
"Eagle": 1, "ECLiPSe": 2,
"Easybuild": 1, "EJS": 3,
"Ecere Projects": 1, "EML": 1,
"Eiffel": 2, "EQ": 2,
"Elixir": 2, "Eagle": 1,
"Elm": 2, "Easybuild": 1,
"Emacs Lisp": 2, "Ecere Projects": 1,
"EmberScript": 2, "Edje Data Collection": 1,
"Erlang": 2, "Eiffel": 2,
"F#": 2, "Elixir": 2,
"FLUX": 2, "Elm": 2,
"Factor": 2, "Emacs Lisp": 2,
"Fancy": 2, "EmberScript": 2,
"Fantom": 2, "Erlang": 2,
"Filebench WML": 2, "F#": 2,
"Filterscript": 2, "F*": 2,
"Formatted": 1, "FIGlet Font": 1,
"Forth": 2, "FLUX": 2,
"Fortran": 2, "Factor": 2,
"FreeMarker": 2, "Fancy": 2,
"Frege": 2, "Fantom": 2,
"G-code": 1, "Filebench WML": 2,
"GAMS": 2, "Filterscript": 2,
"GAP": 2, "Formatted": 1,
"Forth": 2,
"Fortran": 2,
"FreeMarker": 2,
"Frege": 2,
"G-code": 1,
"GAMS": 2,
"GAP": 2,
"GCC Machine Description": 2, "GCC Machine Description": 2,
"GDB": 2, "GDB": 2,
"GDScript": 2, "GDScript": 2,
"GLSL": 2, "GLSL": 2,
"GN": 1, "GN": 1,
"Game Maker Language": 2, "Game Maker Language": 2,
"Genie": 2, "Genie": 2,
"Genshi": 2, "Genshi": 2,
"Gentoo Ebuild": 2, "Gentoo Ebuild": 2,
"Gentoo Eclass": 2, "Gentoo Eclass": 2,
"Gerber Image": 1, "Gerber Image": 1,
"Gettext Catalog": 4, "Gettext Catalog": 4,
"Gherkin": 2, "Gherkin": 2,
"Glyph": 2, "Git Attributes": 1,
"Gnuplot": 2, "Git Config": 1,
"Go": 2, "Glyph": 2,
"Golo": 2, "Glyph Bitmap Distribution Format": 1,
"Gosu": 2, "Gnuplot": 2,
"Grace": 2, "Go": 2,
"Gradle": 1, "Golo": 2,
"Grammatical Framework": 2, "Gosu": 2,
"Graph Modeling Language": 1, "Grace": 2,
"GraphQL": 1, "Gradle": 1,
"Graphviz (DOT)": 1, "Grammatical Framework": 2,
"Groovy": 2, "Graph Modeling Language": 1,
"Groovy Server Pages": 2, "GraphQL": 1,
"HCL": 2, "Graphviz (DOT)": 1,
"HLSL": 2, "Groovy": 2,
"HTML": 3, "Groovy Server Pages": 2,
"HTML+Django": 3, "HAProxy": 1,
"HTML+ECR": 3, "HCL": 2,
"HTML+EEX": 3, "HLSL": 2,
"HTML+ERB": 3, "HTML": 3,
"HTML+PHP": 3, "HTML+Django": 3,
"HTTP": 1, "HTML+ECR": 3,
"Hack": 2, "HTML+EEX": 3,
"Haml": 3, "HTML+ERB": 3,
"Handlebars": 3, "HTML+PHP": 3,
"Harbour": 2, "HTML+Razor": 3,
"Haskell": 2, "HTTP": 1,
"Haxe": 2, "HXML": 1,
"Hy": 2, "Hack": 2,
"HyPhy": 2, "Haml": 3,
"IDL": 2, "Handlebars": 3,
"IGOR Pro": 2, "Harbour": 2,
"INI": 1, "Haskell": 2,
"IRC log": 1, "Haxe": 2,
"Idris": 2, "HiveQL": 2,
"Inform 7": 2, "Hy": 2,
"Inno Setup": 2, "HyPhy": 2,
"Io": 2, "IDL": 2,
"Ioke": 2, "IGOR Pro": 2,
"Isabelle": 2, "INI": 1,
"Isabelle ROOT": 2, "IRC log": 1,
"J": 2, "Idris": 2,
"JFlex": 2, "Ignore List": 1,
"JSON": 1, "Inform 7": 2,
"JSON5": 1, "Inno Setup": 2,
"JSONLD": 1, "Io": 2,
"JSONiq": 2, "Ioke": 2,
"JSX": 2, "Isabelle": 2,
"Jasmin": 2, "Isabelle ROOT": 2,
"Java": 2, "J": 2,
"Java Server Pages": 2, "JFlex": 2,
"JavaScript": 2, "JSON": 1,
"Jison": 2, "JSON with Comments": 1,
"Jison Lex": 2, "JSON5": 1,
"Jolie": 2, "JSONLD": 1,
"Julia": 2, "JSONiq": 2,
"Jupyter Notebook": 3, "JSX": 2,
"KRL": 2, "Jasmin": 2,
"KiCad Layout": 1, "Java": 2,
"KiCad Legacy Layout": 1, "Java Properties": 1,
"KiCad Schematic": 1, "Java Server Pages": 2,
"Kit": 3, "JavaScript": 2,
"Kotlin": 2, "Jison": 2,
"LFE": 2, "Jison Lex": 2,
"LLVM": 2, "Jolie": 2,
"LOLCODE": 2, "Julia": 2,
"LSL": 2, "Jupyter Notebook": 3,
"LabVIEW": 2, "KRL": 2,
"Lasso": 2, "KiCad Layout": 1,
"Latte": 3, "KiCad Legacy Layout": 1,
"Lean": 2, "KiCad Schematic": 1,
"Less": 3, "Kit": 3,
"Lex": 2, "Kotlin": 2,
"LilyPond": 2, "LFE": 2,
"Limbo": 2, "LLVM": 2,
"Linker Script": 1, "LOLCODE": 2,
"Linux Kernel Module": 1, "LSL": 2,
"Liquid": 3, "LTspice Symbol": 1,
"Literate Agda": 2, "LabVIEW": 2,
"Literate CoffeeScript": 2, "Lasso": 2,
"Literate Haskell": 2, "Latte": 3,
"LiveScript": 2, "Lean": 2,
"Logos": 2, "Less": 3,
"Logtalk": 2, "Lex": 2,
"LookML": 2, "LilyPond": 2,
"LoomScript": 2, "Limbo": 2,
"Lua": 2, "Linker Script": 1,
"M": 2, "Linux Kernel Module": 1,
"M4": 2, "Liquid": 3,
"M4Sugar": 2, "Literate Agda": 2,
"MAXScript": 2, "Literate CoffeeScript": 2,
"MQL4": 2, "Literate Haskell": 2,
"MQL5": 2, "LiveScript": 2,
"MTML": 3, "Logos": 2,
"MUF": 2, "Logtalk": 2,
"Makefile": 2, "LookML": 2,
"Mako": 2, "LoomScript": 2,
"Markdown": 4, "Lua": 2,
"Marko": 3, "M": 2,
"Mask": 3, "M4": 2,
"Mathematica": 2, "M4Sugar": 2,
"Matlab": 2, "MATLAB": 2,
"Maven POM": 1, "MAXScript": 2,
"Max": 2, "MQL4": 2,
"MediaWiki": 4, "MQL5": 2,
"Mercury": 2, "MTML": 3,
"Meson": 2, "MUF": 2,
"Metal": 2, "Makefile": 2,
"MiniD": 2, "Mako": 2,
"Mirah": 2, "Markdown": 4,
"Modelica": 2, "Marko": 3,
"Modula-2": 2, "Mask": 3,
"Module Management System": 2, "Mathematica": 2,
"Monkey": 2, "Maven POM": 1,
"Moocode": 2, "Max": 2,
"MoonScript": 2, "MediaWiki": 4,
"Myghty": 2, "Mercury": 2,
"NCL": 2, "Meson": 2,
"NL": 1, "Metal": 2,
"NSIS": 2, "MiniD": 2,
"Nearley": 2, "Mirah": 2,
"Nemerle": 2, "Modelica": 2,
"NetLinx": 2, "Modula-2": 2,
"NetLinx+ERB": 2, "Modula-3": 2,
"NetLogo": 2, "Module Management System": 2,
"NewLisp": 2, "Monkey": 2,
"Nginx": 1, "Moocode": 2,
"Nim": 2, "MoonScript": 2,
"Ninja": 1, "Myghty": 2,
"Nit": 2, "NCL": 2,
"Nix": 2, "NL": 1,
"Nu": 2, "NSIS": 2,
"NumPy": 2, "Nearley": 2,
"OCaml": 2, "Nemerle": 2,
"ObjDump": 1, "NetLinx": 2,
"Objective-C": 2, "NetLinx+ERB": 2,
"Objective-C++": 2, "NetLogo": 2,
"Objective-J": 2, "NewLisp": 2,
"Omgrofl": 2, "Nextflow": 2,
"Opa": 2, "Nginx": 1,
"Opal": 2, "Nim": 2,
"OpenCL": 2, "Ninja": 1,
"OpenEdge ABL": 2, "Nit": 2,
"OpenRC runscript": 2, "Nix": 2,
"OpenSCAD": 2, "Nu": 2,
"OpenType Feature File": 1, "NumPy": 2,
"OCaml": 2,
"ObjDump": 1,
"Objective-C": 2,
"Objective-C++": 2,
"Objective-J": 2,
"Omgrofl": 2,
"Opa": 2,
"Opal": 2,
"OpenCL": 2,
"OpenEdge ABL": 2,
"OpenRC runscript": 2,
"OpenSCAD": 2,
"OpenType Feature File": 1,
"Org": 4, "Org": 4,
"Ox": 2, "Ox": 2,
"Oxygene": 2, "Oxygene": 2,
"Oz": 2, "Oz": 2,
"P4": 2, "P4": 2,
"PAWN": 2,
"PHP": 2, "PHP": 2,
"PLSQL": 2, "PLSQL": 2,
"PLpgSQL": 2, "PLpgSQL": 2,
@ -307,6 +327,7 @@ var LanguagesType = map[string]int{
"Parrot Assembly": 2, "Parrot Assembly": 2,
"Parrot Internal Representation": 2, "Parrot Internal Representation": 2,
"Pascal": 2, "Pascal": 2,
"Pawn": 2,
"Pep8": 2, "Pep8": 2,
"Perl": 2, "Perl": 2,
"Perl 6": 2, "Perl 6": 2,
@ -316,8 +337,10 @@ var LanguagesType = map[string]int{
"PigLatin": 2, "PigLatin": 2,
"Pike": 2, "Pike": 2,
"Pod": 4, "Pod": 4,
"Pod 6": 4,
"PogoScript": 2, "PogoScript": 2,
"Pony": 2, "Pony": 2,
"PostCSS": 3,
"PostScript": 3, "PostScript": 3,
"PowerBuilder": 2, "PowerBuilder": 2,
"PowerShell": 2, "PowerShell": 2,
@ -336,6 +359,7 @@ var LanguagesType = map[string]int{
"Python traceback": 1, "Python traceback": 1,
"QML": 2, "QML": 2,
"QMake": 2, "QMake": 2,
"Quake": 2,
"R": 2, "R": 2,
"RAML": 3, "RAML": 3,
"RDoc": 4, "RDoc": 4,
@ -343,6 +367,7 @@ var LanguagesType = map[string]int{
"REXX": 2, "REXX": 2,
"RHTML": 3, "RHTML": 3,
"RMarkdown": 4, "RMarkdown": 4,
"RPC": 2,
"RPM Spec": 1, "RPM Spec": 1,
"RUNOFF": 3, "RUNOFF": 3,
"Racket": 2, "Racket": 2,
@ -359,6 +384,7 @@ var LanguagesType = map[string]int{
"Ring": 2, "Ring": 2,
"RobotFramework": 2, "RobotFramework": 2,
"Roff": 3, "Roff": 3,
"Roff Manpage": 3,
"Rouge": 2, "Rouge": 2,
"Ruby": 2, "Ruby": 2,
"Rust": 2, "Rust": 2,
@ -385,10 +411,12 @@ var LanguagesType = map[string]int{
"ShellSession": 2, "ShellSession": 2,
"Shen": 2, "Shen": 2,
"Slash": 2, "Slash": 2,
"Slice": 2,
"Slim": 3, "Slim": 3,
"Smali": 2, "Smali": 2,
"Smalltalk": 2, "Smalltalk": 2,
"Smarty": 2, "Smarty": 2,
"Solidity": 2,
"SourcePawn": 2, "SourcePawn": 2,
"Spline Font Database": 1, "Spline Font Database": 1,
"Squirrel": 2, "Squirrel": 2,
@ -397,7 +425,7 @@ var LanguagesType = map[string]int{
"Stata": 2, "Stata": 2,
"Stylus": 3, "Stylus": 3,
"SubRip Text": 1, "SubRip Text": 1,
"Sublime Text Config": 1, "SugarSS": 3,
"SuperCollider": 2, "SuperCollider": 2,
"Swift": 2, "Swift": 2,
"SystemVerilog": 2, "SystemVerilog": 2,
@ -437,32 +465,42 @@ var LanguagesType = map[string]int{
"Web Ontology Language": 1, "Web Ontology Language": 1,
"WebAssembly": 2, "WebAssembly": 2,
"WebIDL": 2, "WebIDL": 2,
"Windows Registry Entries": 1,
"World of Warcraft Addon Data": 1, "World of Warcraft Addon Data": 1,
"X10": 2, "X BitMap": 1,
"XC": 2, "X Font Directory Index": 1,
"XCompose": 1, "X PixMap": 1,
"XML": 1, "X10": 2,
"XPM": 1, "XC": 2,
"XPages": 1, "XCompose": 1,
"XProc": 2, "XML": 1,
"XQuery": 2, "XPages": 1,
"XS": 2, "XProc": 2,
"XSLT": 2, "XQuery": 2,
"Xojo": 2, "XS": 2,
"Xtend": 2, "XSLT": 2,
"YAML": 1, "Xojo": 2,
"YANG": 1, "Xtend": 2,
"Yacc": 2, "YAML": 1,
"Zephir": 2, "YANG": 1,
"Zimpl": 2, "YARA": 2,
"desktop": 1, "YASnippet": 3,
"eC": 2, "Yacc": 2,
"edn": 1, "Zephir": 2,
"fish": 2, "Zig": 2,
"mupad": 2, "Zimpl": 2,
"nesC": 2, "desktop": 1,
"ooc": 2, "eC": 2,
"reStructuredText": 4, "edn": 1,
"wisp": 2, "fish": 2,
"xBase": 2, "mupad": 2,
"nanorc": 1,
"nesC": 2,
"ooc": 2,
"q": 2,
"reStructuredText": 4,
"sed": 2,
"wdl": 2,
"wisp": 2,
"xBase": 2,
} }

19
data/vendor.go
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -10,7 +10,6 @@ var VendorMatchers = substring.Or(
substring.Regexp(`^[Dd]ependencies/`), substring.Regexp(`^[Dd]ependencies/`),
substring.Regexp(`(^|/)dist/`), substring.Regexp(`(^|/)dist/`),
substring.Regexp(`^deps/`), substring.Regexp(`^deps/`),
substring.Regexp(`^tools/`),
substring.Regexp(`(^|/)configure$`), substring.Regexp(`(^|/)configure$`),
substring.Regexp(`(^|/)config.guess$`), substring.Regexp(`(^|/)config.guess$`),
substring.Regexp(`(^|/)config.sub$`), substring.Regexp(`(^|/)config.sub$`),
@ -32,13 +31,15 @@ var VendorMatchers = substring.Or(
substring.Regexp(`(^|/)bootstrap([^.]*)\.(js|css|less|scss|styl)$`), substring.Regexp(`(^|/)bootstrap([^.]*)\.(js|css|less|scss|styl)$`),
substring.Regexp(`(^|/)custom\.bootstrap([^\s]*)(js|css|less|scss|styl)$`), substring.Regexp(`(^|/)custom\.bootstrap([^\s]*)(js|css|less|scss|styl)$`),
substring.Regexp(`(^|/)font-awesome\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)font-awesome\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)font-awesome/.*\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)foundation\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)foundation\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)normalize\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)normalize\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)skeleton\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)skeleton\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)[Bb]ourbon/.*\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)[Bb]ourbon/.*\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)animate\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)animate\.(css|less|scss|styl)$`),
substring.Regexp(`third[-_]?party/`), substring.Regexp(`(^|/)materialize\.(css|less|scss|styl|js)$`),
substring.Regexp(`3rd[-_]?party/`), substring.Regexp(`(^|/)select2/.*\.(css|scss|js)$`),
substring.Regexp(`(3rd|[Tt]hird)[-_]?[Pp]arty/`),
substring.Regexp(`vendors?/`), substring.Regexp(`vendors?/`),
substring.Regexp(`extern(al)?/`), substring.Regexp(`extern(al)?/`),
substring.Regexp(`(^|/)[Vv]+endor/`), substring.Regexp(`(^|/)[Vv]+endor/`),
@ -53,6 +54,9 @@ var VendorMatchers = substring.Or(
substring.Regexp(`jquery.fancybox.(js|css)`), substring.Regexp(`jquery.fancybox.(js|css)`),
substring.Regexp(`fuelux.js`), substring.Regexp(`fuelux.js`),
substring.Regexp(`(^|/)jquery\.fileupload(-\w+)?\.js$`), substring.Regexp(`(^|/)jquery\.fileupload(-\w+)?\.js$`),
substring.Regexp(`jquery.dataTables.js`),
substring.Regexp(`bootbox.js`),
substring.Regexp(`pdf.worker.js`),
substring.Regexp(`(^|/)slick\.\w+.js$`), substring.Regexp(`(^|/)slick\.\w+.js$`),
substring.Regexp(`(^|/)Leaflet\.Coordinates-\d+\.\d+\.\d+\.src\.js$`), substring.Regexp(`(^|/)Leaflet\.Coordinates-\d+\.\d+\.\d+\.src\.js$`),
substring.Regexp(`leaflet.draw-src.js`), substring.Regexp(`leaflet.draw-src.js`),
@ -63,6 +67,7 @@ var VendorMatchers = substring.Or(
substring.Regexp(`wicket-leaflet.js`), substring.Regexp(`wicket-leaflet.js`),
substring.Regexp(`.sublime-project`), substring.Regexp(`.sublime-project`),
substring.Regexp(`.sublime-workspace`), substring.Regexp(`.sublime-workspace`),
substring.Regexp(`.vscode`),
substring.Regexp(`(^|/)prototype(.*)\.js$`), substring.Regexp(`(^|/)prototype(.*)\.js$`),
substring.Regexp(`(^|/)effects\.js$`), substring.Regexp(`(^|/)effects\.js$`),
substring.Regexp(`(^|/)controls\.js$`), substring.Regexp(`(^|/)controls\.js$`),
@ -99,8 +104,7 @@ var VendorMatchers = substring.Or(
substring.Regexp(`^.osx$`), substring.Regexp(`^.osx$`),
substring.Regexp(`\.xctemplate/`), substring.Regexp(`\.xctemplate/`),
substring.Regexp(`\.imageset/`), substring.Regexp(`\.imageset/`),
substring.Regexp(`^Carthage/`), substring.Regexp(`(^|/)Carthage/`),
substring.Regexp(`^Pods/`),
substring.Regexp(`(^|/)Sparkle/`), substring.Regexp(`(^|/)Sparkle/`),
substring.Regexp(`Crashlytics.framework/`), substring.Regexp(`Crashlytics.framework/`),
substring.Regexp(`Fabric.framework/`), substring.Regexp(`Fabric.framework/`),
@ -113,6 +117,9 @@ var VendorMatchers = substring.Or(
substring.Regexp(`(^|/)gradlew$`), substring.Regexp(`(^|/)gradlew$`),
substring.Regexp(`(^|/)gradlew\.bat$`), substring.Regexp(`(^|/)gradlew\.bat$`),
substring.Regexp(`(^|/)gradle/wrapper/`), substring.Regexp(`(^|/)gradle/wrapper/`),
substring.Regexp(`(^|/)mvnw$`),
substring.Regexp(`(^|/)mvnw\.cmd$`),
substring.Regexp(`(^|/)\.mvn/wrapper/`),
substring.Regexp(`-vsdoc\.js$`), substring.Regexp(`-vsdoc\.js$`),
substring.Regexp(`\.intellisense\.js$`), substring.Regexp(`\.intellisense\.js$`),
substring.Regexp(`(^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$`), substring.Regexp(`(^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$`),

31
internal/code-generator/assets/alias.go.tmpl
View File

@ -1,9 +1,34 @@
package data package data
// LanguagesByAlias keeps alias for different languages and use the name of the languages as an alias too. import "strings"
// LanguageByAliasMap keeps alias for different languages and use the name of the languages as an alias too.
// All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores. // All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores.
var LanguagesByAlias = map[string]string{ var LanguageByAliasMap = map[string]string{
{{range $alias, $language := . -}} {{range $alias, $language := . -}}
"{{ $alias }}": {{ printf "%q" $language -}}, "{{ $alias }}": {{ printf "%q" $language -}},
{{end -}} {{end -}}
} }
// LanguageByAlias looks up the language name by it's alias or name.
// It mirrors the logic of github linguist and is needed e.g for heuristcs.yml
// that mixes names and aliases in a language field (see XPM example).
func LanguageByAlias(langOrAlias string) (lang string, ok bool) {
k := convertToAliasKey(langOrAlias)
lang, ok = LanguageByAliasMap[k]
return
}
// convertToAliasKey converts language name to a key in LanguageByAliasMap.
// Following
// - internal.code-generator.generator.convertToAliasKey()
// - GetLanguageByAlias()
// conventions.
// It is here to avoid dependency on "generate" and "enry" packages.
func convertToAliasKey(langName string) string {
ak := strings.SplitN(langName, `,`, 2)[0]
ak = strings.Replace(ak, ` `, `_`, -1)
ak = strings.ToLower(ak)
return ak
}

70
internal/code-generator/assets/content.go.tmpl
View File

@ -1,33 +1,51 @@
package data package data
import "gopkg.in/toqueteos/substring.v1" import (
"regexp"
type languageMatcher func ([]byte) []string "gopkg.in/src-d/enry.v1/data/rule"
)
var ContentMatchers = map[string]languageMatcher{ var ContentHeuristics = map[string]*Heuristics{
{{ range $index, $disambiguator := . -}} {{ range $ext, $rules := . -}}
{{ printf "%q" $disambiguator.Extension }}: func(i []byte) []string { {{ printf "%q" $ext }}: &Heuristics{
{{ range $i, $language := $disambiguator.Languages -}} {{ range $rule := $rules -}}
{{template "Rule" $rule}}
{{- if not (avoidLanguage $language) }} {{ end -}}
{{- if gt (len $language.Heuristics) 0 }} },
{{- if gt $i 0 }} else {{ end -}}
if {{- range $j, $heuristic := $language.Heuristics }} {{ $heuristic.Name }}.Match(string(i))
{{- if lt $j (len $language.LogicRelations) }} {{index $language.LogicRelations $j}} {{- end -}} {{ end }} {
return []string{ {{- printf "%q" $language.Language -}} }
}
{{- end -}}
{{- end -}}
{{- end}}
return {{ returnLanguages $disambiguator.Languages | returnStringSlice }}
},
{{ end -}} {{ end -}}
} }
var ( {{ define "Rule" -}}
{{ range $index, $heuristic := getAllHeuristics . -}} {{ if eq .Op "And" -}}
{{ $heuristic.Name }} = substring.Regexp(`{{ $heuristic.Regexp }}`) rule.And(
{{ end -}} {{ template "Languages" .Langs -}}
) {{ range $rule := .Rules -}}
{{template "Rule" $rule}}
{{ end -}}
),
{{- else if eq .Op "Or" -}}
rule.Or(
{{ template "Languages" .Langs -}}
regexp.MustCompile(`{{ .Pattern }}`),
),
{{- else if eq .Op "Not" -}}
rule.Not(
{{ template "Languages" .Langs -}}
regexp.MustCompile(`{{ .Pattern }}`),
),
{{- else if eq .Op "Always" -}}
rule.Always(
{{ template "Languages" .Langs -}}
),
{{ end -}}
{{ end -}}
{{define "Languages" -}}
{{with . -}}
rule.MatchingLanguages( {{range .}} {{printf "\"%s\"" .}}, {{end}} ),
{{ else -}}
rule.MatchingLanguages(""),
{{end -}}
{{end}}

11
internal/code-generator/generator/aliases.go
View File

@ -2,10 +2,11 @@ package generator
import ( import (
"bytes" "bytes"
"gopkg.in/yaml.v2"
"io" "io"
"io/ioutil" "io/ioutil"
"strings" "strings"
"gopkg.in/yaml.v2"
) )
// Aliases reads from fileToParse and builds source file from tmplPath. It complies with type File signature. // Aliases reads from fileToParse and builds source file from tmplPath. It complies with type File signature.
@ -21,10 +22,10 @@ func Aliases(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string
} }
orderedLangList := getAlphabeticalOrderedKeys(languages) orderedLangList := getAlphabeticalOrderedKeys(languages)
languagesByAlias := buildAliasLanguageMap(languages, orderedLangList) languageByAlias := buildAliasLanguageMap(languages, orderedLangList)
buf := &bytes.Buffer{} buf := &bytes.Buffer{}
if err := executeAliasesTemplate(buf, languagesByAlias, tmplPath, tmplName, commit); err != nil { if err := executeAliasesTemplate(buf, languageByAlias, tmplPath, tmplName, commit); err != nil {
return err return err
} }
@ -52,6 +53,6 @@ func convertToAliasKey(s string) (key string) {
return return
} }
func executeAliasesTemplate(out io.Writer, languagesByAlias map[string]string, aliasesTmplPath, aliasesTmpl, commit string) error { func executeAliasesTemplate(out io.Writer, languageByAlias map[string]string, aliasesTmplPath, aliasesTmpl, commit string) error {
return executeTemplate(out, aliasesTmpl, aliasesTmplPath, commit, nil, languagesByAlias) return executeTemplate(out, aliasesTmpl, aliasesTmplPath, commit, nil, languageByAlias)
} }

18
internal/code-generator/generator/documentation.go
View File

@ -2,13 +2,14 @@ package generator
import ( import (
"bytes" "bytes"
"gopkg.in/yaml.v2"
"io"
"io/ioutil" "io/ioutil"
"gopkg.in/yaml.v2"
) )
// Documentation reads from fileToParse and builds source file from tmplPath. It complies with type File signature. // Documentation generates regex matchers in Go for documentation files/dirs.
func Documentation(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error { // It is of generator.File type.
func Documentation(fileToParse, _, outFile, tmplPath, tmplName, commit string) error {
data, err := ioutil.ReadFile(fileToParse) data, err := ioutil.ReadFile(fileToParse)
if err != nil { if err != nil {
return err return err
@ -20,13 +21,10 @@ func Documentation(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit
} }
buf := &bytes.Buffer{} buf := &bytes.Buffer{}
if err := executeDocumentationTemplate(buf, regexpList, tmplPath, tmplName, commit); err != nil { err = executeTemplate(buf, tmplName, tmplPath, commit, nil, regexpList)
if err != nil {
return err return err
} }
return formatedWrite(outPath, buf.Bytes()) return formatedWrite(outFile, buf.Bytes())
}
func executeDocumentationTemplate(out io.Writer, regexpList []string, tmplPath, tmplName, commit string) error {
return executeTemplate(out, tmplName, tmplPath, commit, nil, regexpList)
} }

11
internal/code-generator/generator/generator.go
View File

@ -1,3 +1,5 @@
// Package generator provides facilities to generate Go code for the
// package data in enry from YAML files describing supported languages in Linguist.
package generator package generator
import ( import (
@ -9,7 +11,10 @@ import (
"text/template" "text/template"
) )
// File is the function's type that generate source file from a file to be parsed, linguist's samples dir and a template. // File is a common type for all generator functions.
// It generates Go source code file based on template in tmplPath,
// by parsing the data in fileToParse and linguist's samplesDir
// saving results to an outFile.
type File func(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error type File func(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error
func formatedWrite(outPath string, source []byte) error { func formatedWrite(outPath string, source []byte) error {
@ -28,16 +33,14 @@ func executeTemplate(w io.Writer, name, path, commit string, fmap template.FuncM
return commit return commit
} }
buf := bytes.NewBuffer(nil)
const headerTmpl = "header.go.tmpl" const headerTmpl = "header.go.tmpl"
headerPath := filepath.Join(filepath.Dir(path), headerTmpl) headerPath := filepath.Join(filepath.Dir(path), headerTmpl)
h := template.Must(template.New(headerTmpl).Funcs(template.FuncMap{ h := template.Must(template.New(headerTmpl).Funcs(template.FuncMap{
"getCommit": getCommit, "getCommit": getCommit,
}).ParseFiles(headerPath)) }).ParseFiles(headerPath))
buf := bytes.NewBuffer(nil)
if err := h.Execute(buf, data); err != nil { if err := h.Execute(buf, data); err != nil {
return err return err
} }

91
internal/code-generator/generator/generator_test.go
View File

@ -1,6 +1,7 @@
package generator package generator
import ( import (
"flag"
"fmt" "fmt"
"io/ioutil" "io/ioutil"
"os" "os"
@ -15,7 +16,7 @@ import (
const ( const (
linguistURL = "https://github.com/github/linguist.git" linguistURL = "https://github.com/github/linguist.git"
linguistClonedEnvVar = "ENRY_TEST_REPO" linguistClonedEnvVar = "ENRY_TEST_REPO"
commit = "d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68" commit = "e4560984058b4726010ca4b8f03ed9d0f8f464db"
samplesDir = "samples" samplesDir = "samples"
languagesFile = "lib/linguist/languages.yml" languagesFile = "lib/linguist/languages.yml"
@ -28,7 +29,7 @@ const (
extensionTestTmplName = "extension.go.tmpl" extensionTestTmplName = "extension.go.tmpl"
// Heuristics test // Heuristics test
heuristicsTestFile = "lib/linguist/heuristics.rb" heuristicsTestFile = "lib/linguist/heuristics.yml"
contentGold = testDir + "/content.gold" contentGold = testDir + "/content.gold"
contentTestTmplPath = assetsDir + "/content.go.tmpl" contentTestTmplPath = assetsDir + "/content.go.tmpl"
contentTestTmplName = "content.go.tmpl" contentTestTmplName = "content.go.tmpl"
@ -85,13 +86,27 @@ type GeneratorTestSuite struct {
suite.Suite suite.Suite
tmpLinguist string tmpLinguist string
cloned bool cloned bool
testCases []testCase
} }
func TestGeneratorTestSuite(t *testing.T) { type testCase struct {
name string
fileToParse string
samplesDir string
tmplPath string
tmplName string
commit string
generate File
wantOut string
}
var updateGold = flag.Bool("update_gold", false, "Update golden test files")
func Test_GeneratorTestSuite(t *testing.T) {
suite.Run(t, new(GeneratorTestSuite)) suite.Run(t, new(GeneratorTestSuite))
} }
func (s *GeneratorTestSuite) SetupSuite() { func (s *GeneratorTestSuite) maybeCloneLinguist() {
var err error var err error
s.tmpLinguist = os.Getenv(linguistClonedEnvVar) s.tmpLinguist = os.Getenv(linguistClonedEnvVar)
s.cloned = s.tmpLinguist == "" s.cloned = s.tmpLinguist == ""
@ -101,40 +116,25 @@ func (s *GeneratorTestSuite) SetupSuite() {
cmd := exec.Command("git", "clone", linguistURL, s.tmpLinguist) cmd := exec.Command("git", "clone", linguistURL, s.tmpLinguist)
err = cmd.Run() err = cmd.Run()
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
}
cwd, err := os.Getwd() cwd, err := os.Getwd()
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
err = os.Chdir(s.tmpLinguist) err = os.Chdir(s.tmpLinguist)
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
cmd := exec.Command("git", "checkout", commit) cmd = exec.Command("git", "checkout", commit)
err = cmd.Run() err = cmd.Run()
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
err = os.Chdir(cwd) err = os.Chdir(cwd)
assert.NoError(s.T(), err)
}
func (s *GeneratorTestSuite) TearDownSuite() {
if s.cloned {
err := os.RemoveAll(s.tmpLinguist)
assert.NoError(s.T(), err) assert.NoError(s.T(), err)
} }
} }
func (s *GeneratorTestSuite) TestGenerationFiles() { func (s *GeneratorTestSuite) SetupSuite() {
tests := []struct { s.maybeCloneLinguist()
name string s.testCases = []testCase{
fileToParse string
samplesDir string
tmplPath string
tmplName string
commit string
generate File
wantOut string
}{
{ {
name: "Extensions()", name: "Extensions()",
fileToParse: filepath.Join(s.tmpLinguist, languagesFile), fileToParse: filepath.Join(s.tmpLinguist, languagesFile),
@ -152,7 +152,7 @@ func (s *GeneratorTestSuite) TestGenerationFiles() {
tmplPath: contentTestTmplPath, tmplPath: contentTestTmplPath,
tmplName: contentTestTmplName, tmplName: contentTestTmplName,
commit: commit, commit: commit,
generate: Heuristics, generate: GenHeuristics,
wantOut: contentGold, wantOut: contentGold,
}, },
{ {
@ -244,8 +244,35 @@ func (s *GeneratorTestSuite) TestGenerationFiles() {
wantOut: mimeTypeGold, wantOut: mimeTypeGold,
}, },
} }
}
for _, test := range tests { func (s *GeneratorTestSuite) TearDownSuite() {
if s.cloned {
err := os.RemoveAll(s.tmpLinguist)
if err != nil {
s.T().Logf("Failed to clean up %s after the test.\n", s.tmpLinguist)
}
}
}
// TestUpdateGeneratorTestSuiteGold is a Gold results generation automation.
// It should only be enabled&run manually on every new Linguist version
// to update *.gold files.
func (s *GeneratorTestSuite) TestUpdateGeneratorTestSuiteGold() {
if !*updateGold {
s.T().Skip()
}
s.T().Logf("Generating new *.gold test files")
for _, test := range s.testCases {
dst := test.wantOut
s.T().Logf("Generating %s from %s\n", dst, test.fileToParse)
err := test.generate(test.fileToParse, test.samplesDir, dst, test.tmplPath, test.tmplName, test.commit)
assert.NoError(s.T(), err)
}
}
func (s *GeneratorTestSuite) TestGenerationFiles() {
for _, test := range s.testCases {
gold, err := ioutil.ReadFile(test.wantOut) gold, err := ioutil.ReadFile(test.wantOut)
assert.NoError(s.T(), err) assert.NoError(s.T(), err)

577
internal/code-generator/generator/heuristics.go
View File

@ -1,483 +1,176 @@
package generator package generator
import ( import (
"bufio"
"bytes" "bytes"
"fmt" "fmt"
"io"
"io/ioutil" "io/ioutil"
"strconv" "log"
"strings" "strings"
"text/template"
"gopkg.in/src-d/enry.v1/regex" yaml "gopkg.in/yaml.v2"
) )
// Heuristics reads from fileToParse and builds source file from tmplPath. It complies with type File signature. const (
func Heuristics(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error { multilinePrefix = "(?m)"
data, err := ioutil.ReadFile(fileToParse) orPipe = "|"
)
// GenHeuristics generates language identification heuristics in Go.
// It is of generator.File type.
func GenHeuristics(fileToParse, _, outPath, tmplPath, tmplName, commit string) error {
heuristicsYaml, err := parseYaml(fileToParse)
if err != nil { if err != nil {
return err return err
} }
disambiguators, err := getDisambiguators(data) langPatterns, err := loadHeuristics(heuristicsYaml)
if err != nil { if err != nil {
return err return err
} }
buf := &bytes.Buffer{} buf := &bytes.Buffer{}
if err := executeContentTemplate(buf, disambiguators, tmplPath, tmplName, commit); err != nil { err = executeTemplate(buf, tmplName, tmplPath, commit, nil, langPatterns)
if err != nil {
return err return err
} }
return formatedWrite(outPath, buf.Bytes()) return formatedWrite(outPath, buf.Bytes())
} }
const ( // loadHeuristics transforms parsed YAML to map[".ext"]->IR for code generation.
unknownLanguage = "OtherLanguage" func loadHeuristics(yaml *Heuristics) (map[string][]*LanguagePattern, error) {
emptyFile = "^$" var patterns = make(map[string][]*LanguagePattern)
) for _, disambiguation := range yaml.Disambiguations {
var rules []*LanguagePattern
var ( for _, rule := range disambiguation.Rules {
disambLine = regex.MustCompile(`^(\s*)disambiguate`) langPattern := loadRule(yaml.NamedPatterns, rule)
definedRegs = make(map[string]string) if langPattern != nil {
illegalCharacter = map[string]string{ rules = append(rules, langPattern)
"#": "Sharp",
"+": "Plus",
"-": "Dash",
}
)
type disambiguator struct {
Extension string `json:"extension,omitempty"`
Languages []*languageHeuristics `json:"languages,omitempty"`
}
func (d *disambiguator) setHeuristicsNames() {
for _, lang := range d.Languages {
for i, heuristic := range lang.Heuristics {
name := buildName(d.Extension, lang.Language, i)
heuristic.Name = name
}
}
}
func buildName(extension, language string, id int) string {
extension = strings.TrimPrefix(extension, `.`)
language = strings.Join(strings.Fields(language), ``)
name := strings.Join([]string{extension, language, "Matcher", strconv.Itoa(id)}, `_`)
for k, v := range illegalCharacter {
if strings.Contains(name, k) {
name = strings.Replace(name, k, v, -1)
}
}
return name
}
type languageHeuristics struct {
Language string `json:"language,omitempty"`
Heuristics []*heuristic `json:"heuristics,omitempty"`
LogicRelations []string `json:"logic_relations,omitempty"`
}
func (l *languageHeuristics) clone() (*languageHeuristics, error) {
language := l.Language
logicRels := make([]string, len(l.LogicRelations))
if copy(logicRels, l.LogicRelations) != len(l.LogicRelations) {
return nil, fmt.Errorf("error copying logic relations")
}
heuristics := make([]*heuristic, 0, len(l.Heuristics))
for _, h := range l.Heuristics {
heuristic := *h
heuristics = append(heuristics, &heuristic)
}
clone := &languageHeuristics{
Language: language,
Heuristics: heuristics,
LogicRelations: logicRels,
}
return clone, nil
}
type heuristic struct {
Name string `json:"name,omitempty"`
Regexp string `json:"regexp,omitempty"`
}
// A disambiguate block looks like:
// disambiguate ".mod", ".extension" do |data|
// if data.include?('<!ENTITY ') && data.include?('patata')
// Language["XML"]
// elsif /^\s*MODULE [\w\.]+;/i.match(data) || /^\s*END [\w\.]+;/i.match(data) || data.empty?
// Language["Modula-2"]
// elsif (/^\s*import (scala|java)\./.match(data) || /^\s*val\s+\w+\s*=/.match(data) || /^\s*class\b/.match(data))
// Language["Scala"]
// elsif (data.include?("gap> "))
// Language["GAP"]
// else
// [Language["Linux Kernel Module"], Language["AMPL"]]
// end
// end
func getDisambiguators(heuristics []byte) ([]*disambiguator, error) {
seenExtensions := map[string]bool{}
buf := bufio.NewScanner(bytes.NewReader(heuristics))
disambiguators := make([]*disambiguator, 0, 50)
for buf.Scan() {
line := buf.Text()
if disambLine.MatchString(line) {
d, err := parseDisambiguators(line, buf, seenExtensions)
if err != nil {
return nil, err
}
disambiguators = append(disambiguators, d...)
}
lookForRegexpVariables(line)
}
if err := buf.Err(); err != nil {
return nil, err
}
return disambiguators, nil
}
func lookForRegexpVariables(line string) {
if strings.Contains(line, "ObjectiveCRegex = ") {
line = strings.TrimSpace(line)
reg := strings.TrimPrefix(line, "ObjectiveCRegex = ")
definedRegs["ObjectiveCRegex"] = reg
}
if strings.Contains(line, "fortran_rx = ") {
line = strings.TrimSpace(line)
reg := strings.TrimPrefix(line, "fortran_rx = ")
definedRegs["fortran_rx"] = reg
}
}
func parseDisambiguators(line string, buf *bufio.Scanner, seenExtensions map[string]bool) ([]*disambiguator, error) {
disambList := make([]*disambiguator, 0, 2)
splitted := strings.Fields(line)
for _, v := range splitted {
if strings.HasPrefix(v, `"`) {
extension := strings.Trim(v, `",`)
if _, ok := seenExtensions[extension]; !ok {
d := &disambiguator{Extension: extension}
disambList = append(disambList, d)
seenExtensions[extension] = true
} }
} }
// unroll to a single map
for _, ext := range disambiguation.Extensions {
if _, ok := patterns[ext]; ok {
return nil, fmt.Errorf("cannt add extension '%s', it already exists for %q", ext, patterns[ext])
}
patterns[ext] = rules
}
}
return patterns, nil
}
// loadRule transforms single rule from parsed YAML to IR for code generation.
// For OrPattern case, it always combines multiple patterns into a single one.
func loadRule(namedPatterns map[string]StringArray, rule *Rule) *LanguagePattern {
var result *LanguagePattern
if len(rule.And) != 0 { // AndPattern
var subPatterns []*LanguagePattern
for _, r := range rule.And {
subp := loadRule(namedPatterns, r)
subPatterns = append(subPatterns, subp)
}
result = &LanguagePattern{"And", rule.Languages, "", subPatterns}
} else if len(rule.Pattern) != 0 { // OrPattern
conjunction := strings.Join(rule.Pattern, orPipe)
pattern := convertToValidRegexp(conjunction)
result = &LanguagePattern{"Or", rule.Languages, pattern, nil}
} else if rule.NegativePattern != "" { // NotPattern
pattern := convertToValidRegexp(rule.NegativePattern)
result = &LanguagePattern{"Not", rule.Languages, pattern, nil}
} else if rule.NamedPattern != "" { // Named OrPattern
conjunction := strings.Join(namedPatterns[rule.NamedPattern], orPipe)
pattern := convertToValidRegexp(conjunction)
result = &LanguagePattern{"Or", rule.Languages, pattern, nil}
} else { // AlwaysPattern
result = &LanguagePattern{"Always", rule.Languages, "", nil}
} }
langsHeuristics, err := getLanguagesHeuristics(buf) if isUnsupportedRegexpSyntax(result.Pattern) {
log.Printf("skipping rule: language:'%q', rule:'%q'\n", rule.Languages, result.Pattern)
return nil
}
return result
}
// LanguagePattern is an IR of parsed Rule suitable for code generations.
// Strings are used as this is to be be consumed by text/template.
type LanguagePattern struct {
Op string
Langs []string
Pattern string
Rules []*LanguagePattern
}
type Heuristics struct {
Disambiguations []*Disambiguation
NamedPatterns map[string]StringArray `yaml:"named_patterns"`
}
type Disambiguation struct {
Extensions []string `yaml:"extensions,flow"`
Rules []*Rule `yaml:"rules"`
}
type Rule struct {
Patterns `yaml:",inline"`
Languages StringArray `yaml:"language"`
And []*Rule
}
type Patterns struct {
Pattern StringArray `yaml:"pattern,omitempty"`
NamedPattern string `yaml:"named_pattern,omitempty"`
NegativePattern string `yaml:"negative_pattern,omitempty"`
}
// StringArray is workaround for parsing named_pattern,
// wich is sometimes arry and sometimes not.
// See https://github.com/go-yaml/yaml/issues/100
type StringArray []string
// UnmarshalYAML allowes to parse element always as a []string
func (sa *StringArray) UnmarshalYAML(unmarshal func(interface{}) error) error {
var multi []string
if err := unmarshal(&multi); err != nil {
var single string
if err := unmarshal(&single); err != nil {
return err
}
*sa = []string{single}
} else {
*sa = multi
}
return nil
}
func parseYaml(file string) (*Heuristics, error) {
data, err := ioutil.ReadFile(file)
if err != nil { if err != nil {
return nil, err return nil, err
} }
for i, disamb := range disambList { h := &Heuristics{}
lh := langsHeuristics if err := yaml.Unmarshal(data, &h); err != nil {
if i != 0 {
lh = cloneLanguagesHeuristics(langsHeuristics)
}
disamb.Languages = lh
disamb.setHeuristicsNames()
}
return disambList, nil
}
func cloneLanguagesHeuristics(list []*languageHeuristics) []*languageHeuristics {
cloneList := make([]*languageHeuristics, 0, len(list))
for _, langHeu := range list {
clone, _ := langHeu.clone()
cloneList = append(cloneList, clone)
}
return cloneList
}
func getLanguagesHeuristics(buf *bufio.Scanner) ([]*languageHeuristics, error) {
langsList := make([][]string, 0, 2)
heuristicsList := make([][]*heuristic, 0, 1)
logicRelsList := make([][]string, 0, 1)
lastWasMatch := false
for buf.Scan() {
line := buf.Text()
if strings.TrimSpace(line) == "end" {
break
}
if hasRegExp(line) {
line := cleanRegExpLine(line)
logicRels := getLogicRelations(line)
heuristics := getHeuristics(line)
if lastWasMatch {
i := len(heuristicsList) - 1
heuristicsList[i] = append(heuristicsList[i], heuristics...)
i = len(logicRelsList) - 1
logicRelsList[i] = append(logicRelsList[i], logicRels...)
} else {
heuristicsList = append(heuristicsList, heuristics)
logicRelsList = append(logicRelsList, logicRels)
}
lastWasMatch = true
}
if strings.Contains(line, "Language") {
langs := getLanguages(line)
langsList = append(langsList, langs)
lastWasMatch = false
}
}
if err := buf.Err(); err != nil {
return nil, err return nil, err
} }
langsHeuristics := buildLanguagesHeuristics(langsList, heuristicsList, logicRelsList) return h, nil
return langsHeuristics, nil
} }
func hasRegExp(line string) bool { // isUnsupportedRegexpSyntax filters regexp syntax that is not supported by RE2.
return strings.Contains(line, ".match") || strings.Contains(line, ".include?") || strings.Contains(line, ".empty?") // In particular, we stumbled up on usage of next cases:
// - named & numbered capturing group/after text matching
// - backreference
// For referece on supported syntax see https://github.com/google/re2/wiki/Syntax
func isUnsupportedRegexpSyntax(reg string) bool {
return strings.Contains(reg, `(?<`) || strings.Contains(reg, `\1`) ||
// See https://github.com/github/linguist/pull/4243#discussion_r246105067
(strings.HasPrefix(reg, multilinePrefix+`/`) && strings.HasSuffix(reg, `/`))
} }
func cleanRegExpLine(line string) string { // convertToValidRegexp converts Ruby regexp syntaxt to RE2 equivalent.
if strings.Contains(line, "if ") { // Does not work with Ruby regexp literals.
line = line[strings.Index(line, `if `)+3:] func convertToValidRegexp(rubyRegexp string) string {
} return multilinePrefix + rubyRegexp
line = strings.TrimSpace(line)
line = strings.TrimPrefix(line, `(`)
if strings.Contains(line, "))") {
line = strings.TrimSuffix(line, `)`)
}
return line
}
func getLogicRelations(line string) []string {
rels := make([]string, 0)
splitted := strings.Split(line, "||")
for i, v := range splitted {
if strings.Contains(v, "&&") {
rels = append(rels, "&&")
}
if i < len(splitted)-1 {
rels = append(rels, "||")
}
}
if len(rels) == 0 {
rels = nil
}
return rels
}
func getHeuristics(line string) []*heuristic {
splitted := splitByLogicOps(line)
heuristics := make([]*heuristic, 0, len(splitted))
for _, v := range splitted {
v = strings.TrimSpace(v)
var reg string
if strings.Contains(v, ".match") {
reg = v[:strings.Index(v, ".match")]
reg = replaceRegexpVariables(reg)
}
if strings.Contains(v, ".include?") {
reg = includeToRegExp(v)
}
if strings.Contains(v, ".empty?") {
reg = emptyFile
}
if reg != "" {
reg = convertToValidRegexp(reg)
heuristics = append(heuristics, &heuristic{Regexp: reg})
}
}
return heuristics
}
func splitByLogicOps(line string) []string {
splitted := make([]string, 0, 1)
splitOr := strings.Split(line, "||")
for _, v := range splitOr {
splitAnd := strings.Split(v, "&&")
splitted = append(splitted, splitAnd...)
}
return splitted
}
func replaceRegexpVariables(reg string) string {
repl := reg
if v, ok := definedRegs[reg]; ok {
repl = v
}
return repl
}
func convertToValidRegexp(reg string) string {
// example: `/^(\s*)(<Project|<Import|<Property|<?xml|xmlns)/i``
// Ruby modifier "m" matches multiple lines, recognizing newlines as normal characters, Go use flag "s" for that.
const (
caseSensitive = "i"
matchEOL = "s"
rubyCaseSensitive = "i"
rubyMultiLine = "m"
)
if reg == emptyFile {
return reg
}
reg = strings.TrimPrefix(reg, `/`)
flags := "(?m"
lastSlash := strings.LastIndex(reg, `/`)
if lastSlash == -1 {
return flags + ")" + reg
}
specialChars := reg[lastSlash:]
reg = reg[:lastSlash]
if lastSlash == len(reg)-1 {
return flags + ")" + reg
}
if strings.Contains(specialChars, rubyCaseSensitive) {
flags = flags + caseSensitive
}
if strings.Contains(specialChars, rubyMultiLine) {
flags = flags + matchEOL
}
return flags + ")" + reg
}
func includeToRegExp(include string) string {
content := include[strings.Index(include, `(`)+1 : strings.Index(include, `)`)]
content = strings.Trim(content, `"'`)
return regex.QuoteMeta(content)
}
func getLanguages(line string) []string {
languages := make([]string, 0)
splitted := strings.Split(line, `,`)
for _, lang := range splitted {
lang = trimLanguage(lang)
languages = append(languages, lang)
}
return languages
}
func trimLanguage(enclosedLang string) string {
lang := strings.TrimSpace(enclosedLang)
lang = lang[strings.Index(lang, `"`)+1:]
lang = lang[:strings.Index(lang, `"`)]
return lang
}
func buildLanguagesHeuristics(langsList [][]string, heuristicsList [][]*heuristic, logicRelsList [][]string) []*languageHeuristics {
langsHeuristics := make([]*languageHeuristics, 0, len(langsList))
for i, langSlice := range langsList {
var heuristics []*heuristic
if i < len(heuristicsList) {
heuristics = heuristicsList[i]
}
var rels []string
if i < len(logicRelsList) {
rels = logicRelsList[i]
}
for _, lang := range langSlice {
lh := &languageHeuristics{
Language: lang,
Heuristics: heuristics,
LogicRelations: rels,
}
langsHeuristics = append(langsHeuristics, lh)
}
}
return langsHeuristics
}
func executeContentTemplate(out io.Writer, disambiguators []*disambiguator, tmplPath, tmplName, commit string) error {
fmap := template.FuncMap{
"getAllHeuristics": getAllHeuristics,
"returnStringSlice": func(slice []string) string {
if len(slice) == 0 {
return "nil"
}
return `[]string{` + strings.Join(slice, `, `) + `}`
},
"returnLanguages": returnLanguages,
"avoidLanguage": avoidLanguage,
}
return executeTemplate(out, tmplName, tmplPath, commit, fmap, disambiguators)
}
func getAllHeuristics(disambiguators []*disambiguator) []*heuristic {
heuristics := make([]*heuristic, 0)
for _, disamb := range disambiguators {
for _, lang := range disamb.Languages {
if !avoidLanguage(lang) {
heuristics = append(heuristics, lang.Heuristics...)
}
}
}
return heuristics
}
func avoidLanguage(lang *languageHeuristics) bool {
// necessary to avoid corner cases
for _, heuristic := range lang.Heuristics {
if containsInvalidRegexp(heuristic.Regexp) {
return true
}
}
return false
}
func containsInvalidRegexp(reg string) bool {
return strings.Contains(reg, `(?<`) || strings.Contains(reg, `\1`)
}
func returnLanguages(langsHeuristics []*languageHeuristics) []string {
langs := make([]string, 0)
for _, langHeu := range langsHeuristics {
if len(langHeu.Heuristics) == 0 {
langs = append(langs, `"`+langHeu.Language+`"`)
}
}
return langs
} }

123
internal/code-generator/generator/heuristics_test.go Normal file
View File

@ -0,0 +1,123 @@
package generator
import (
"bytes"
"fmt"
"go/format"
"testing"
"text/template"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestYAMLParsing(t *testing.T) {
heuristics, err := parseYaml("test_files/heuristics.yml")
require.NoError(t, err)
assert.NotNil(t, heuristics)
// extensions
require.NotNil(t, heuristics.Disambiguations)
assert.Equal(t, 4, len(heuristics.Disambiguations))
assert.Equal(t, 2, len(heuristics.Disambiguations[0].Extensions))
rules := heuristics.Disambiguations[0].Rules
assert.Equal(t, 2, len(rules))
require.Equal(t, "Objective-C", rules[0].Languages[0])
assert.Equal(t, 1, len(rules[0].Pattern))
rules = heuristics.Disambiguations[1].Rules
assert.Equal(t, 3, len(rules))
require.Equal(t, "Forth", rules[0].Languages[0])
require.Equal(t, 2, len(rules[0].Pattern))
rules = heuristics.Disambiguations[2].Rules
assert.Equal(t, 3, len(rules))
require.Equal(t, "Unix Assembly", rules[1].Languages[0])
require.NotNil(t, rules[1].And)
assert.Equal(t, 2, len(rules[1].And))
require.NotNil(t, rules[1].And[0].NegativePattern)
assert.Equal(t, "np", rules[1].And[0].NegativePattern)
rules = heuristics.Disambiguations[3].Rules
assert.Equal(t, 1, len(rules))
assert.Equal(t, "Linux Kernel Module", rules[0].Languages[0])
assert.Equal(t, "AMPL", rules[0].Languages[1])
// named_patterns
require.NotNil(t, heuristics.NamedPatterns)
assert.Equal(t, 2, len(heuristics.NamedPatterns))
assert.Equal(t, 1, len(heuristics.NamedPatterns["fortran"]))
assert.Equal(t, 2, len(heuristics.NamedPatterns["cpp"]))
}
func TestSingleRuleLoading(t *testing.T) {
namedPatterns := map[string]StringArray{"cpp": []string{"cpp_ptrn1", "cpp_ptrn2"}}
rules := []*Rule{
&Rule{Languages: []string{"a"}, Patterns: Patterns{NamedPattern: "cpp"}},
&Rule{Languages: []string{"b"}, And: []*Rule{}},
}
// named_pattern case
langPattern := loadRule(namedPatterns, rules[0])
require.Equal(t, "a", langPattern.Langs[0])
assert.NotEmpty(t, langPattern.Pattern)
// and case
langPattern = loadRule(namedPatterns, rules[1])
require.Equal(t, "b", langPattern.Langs[0])
}
func TestLoadingAllHeuristics(t *testing.T) {
parsedYaml, err := parseYaml("test_files/heuristics.yml")
require.NoError(t, err)
hs, err := loadHeuristics(parsedYaml)
// grep -Eo "extensions:\ (.*)" internal/code-generator/generator/test_files/heuristics.yml
assert.Equal(t, 5, len(hs))
}
func TestLoadingHeuristicsForSameExt(t *testing.T) {
parsedYaml := &Heuristics{
Disambiguations: []*Disambiguation{
&Disambiguation{
Extensions: []string{".a", ".b"},
Rules: []*Rule{&Rule{Languages: []string{"A"}}},
},
&Disambiguation{
Extensions: []string{".b"},
Rules: []*Rule{&Rule{Languages: []string{"B"}}},
},
},
}
_, err := loadHeuristics(parsedYaml)
require.Error(t, err)
}
func TestTemplateMatcherVars(t *testing.T) {
parsed, err := parseYaml("test_files/heuristics.yml")
require.NoError(t, err)
heuristics, err := loadHeuristics(parsed)
require.NoError(t, err)
// render a tmpl
const contentTmpl = "../assets/content.go.tmpl"
tmpl, err := template.ParseFiles(contentTmpl)
require.NoError(t, err)
buf := bytes.NewBuffer(nil)
err = tmpl.Execute(buf, heuristics)
require.NoError(t, err, fmt.Sprintf("%+v", tmpl))
require.NotEmpty(t, buf)
// TODO(bzz) add more advanced test using go/ast package, to verify the
// strucutre of generated code:
// - check key literal exists in map for each extension:
src, err := format.Source(buf.Bytes())
require.NoError(t, err, "\n%s\n", string(src))
}

5
internal/code-generator/generator/mimeType.go
View File

@ -2,11 +2,14 @@ package generator
import ( import (
"bytes" "bytes"
"gopkg.in/yaml.v2"
"io" "io"
"io/ioutil" "io/ioutil"
"gopkg.in/yaml.v2"
) )
// MimeType generates a map in Go with language name -> MIME string.
// It is of generator.File type.
func MimeType(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error { func MimeType(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error {
data, err := ioutil.ReadFile(fileToParse) data, err := ioutil.ReadFile(fileToParse)
if err != nil { if err != nil {

805
internal/code-generator/generator/test_files/alias.gold
View File

@ -1,23 +1,29 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
// LanguagesByAlias keeps alias for different languages and use the name of the languages as an alias too. import "strings"
// LanguageByAliasMap keeps alias for different languages and use the name of the languages as an alias too.
// All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores. // All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores.
var LanguagesByAlias = map[string]string{ var LanguageByAliasMap = map[string]string{
"1c_enterprise": "1C Enterprise", "1c_enterprise": "1C Enterprise",
"abap": "ABAP", "abap": "ABAP",
"abl": "OpenEdge ABL", "abl": "OpenEdge ABL",
"abnf": "ABNF", "abnf": "ABNF",
"abuild": "Alpine Abuild", "abuild": "Alpine Abuild",
"aconf": "ApacheConf", "acfm": "Adobe Font Metrics",
"actionscript": "ActionScript", "aconf": "ApacheConf",
"actionscript3": "ActionScript", "actionscript": "ActionScript",
"actionscript_3": "ActionScript", "actionscript3": "ActionScript",
"ada": "Ada", "actionscript_3": "ActionScript",
"ada2005": "Ada", "ada": "Ada",
"ada95": "Ada", "ada2005": "Ada",
"ada95": "Ada",
"adobe_composite_font_metrics": "Adobe Font Metrics",
"adobe_font_metrics": "Adobe Font Metrics",
"adobe_multiple_font_metrics": "Adobe Font Metrics",
"advpl": "xBase", "advpl": "xBase",
"afdko": "OpenType Feature File", "afdko": "OpenType Feature File",
"agda": "Agda", "agda": "Agda",
@ -26,7 +32,9 @@ var LanguagesByAlias = map[string]string{
"ahk": "AutoHotkey", "ahk": "AutoHotkey",
"alloy": "Alloy", "alloy": "Alloy",
"alpine_abuild": "Alpine Abuild", "alpine_abuild": "Alpine Abuild",
"amfm": "Adobe Font Metrics",
"ampl": "AMPL", "ampl": "AMPL",
"angelscript": "AngelScript",
"ant_build_system": "Ant Build System", "ant_build_system": "Ant Build System",
"antlr": "ANTLR", "antlr": "ANTLR",
"apache": "ApacheConf", "apache": "ApacheConf",
@ -35,78 +43,81 @@ var LanguagesByAlias = map[string]string{
"api_blueprint": "API Blueprint", "api_blueprint": "API Blueprint",
"apkbuild": "Alpine Abuild", "apkbuild": "Alpine Abuild",
"apl": "APL", "apl": "APL",
"apollo_guidance_computer": "Apollo Guidance Computer", "apollo_guidance_computer": "Apollo Guidance Computer",
"applescript": "AppleScript", "applescript": "AppleScript",
"arc": "Arc", "arc": "Arc",
"arduino": "Arduino", "arexx": "REXX",
"arexx": "REXX", "as3": "ActionScript",
"as3": "ActionScript", "asciidoc": "AsciiDoc",
"asciidoc": "AsciiDoc", "asm": "Assembly",
"asn.1": "ASN.1", "asn.1": "ASN.1",
"asp": "ASP", "asp": "ASP",
"aspectj": "AspectJ", "aspectj": "AspectJ",
"aspx": "ASP", "aspx": "ASP",
"aspx-vb": "ASP", "aspx-vb": "ASP",
"assembly": "Assembly", "assembly": "Assembly",
"ats": "ATS", "asymptote": "Asymptote",
"ats2": "ATS", "ats": "ATS",
"au3": "AutoIt", "ats2": "ATS",
"augeas": "Augeas", "au3": "AutoIt",
"autoconf": "M4Sugar", "augeas": "Augeas",
"autohotkey": "AutoHotkey", "autoconf": "M4Sugar",
"autoit": "AutoIt", "autohotkey": "AutoHotkey",
"autoit3": "AutoIt", "autoit": "AutoIt",
"autoitscript": "AutoIt", "autoit3": "AutoIt",
"awk": "Awk", "autoitscript": "AutoIt",
"b3d": "BlitzBasic", "awk": "Awk",
"bash": "Shell", "b3d": "BlitzBasic",
"bash_session": "ShellSession", "ballerina": "Ballerina",
"bat": "Batchfile", "bash": "Shell",
"batch": "Batchfile", "bash_session": "ShellSession",
"batchfile": "Batchfile", "bat": "Batchfile",
"befunge": "Befunge", "batch": "Batchfile",
"bison": "Bison", "batchfile": "Batchfile",
"bitbake": "BitBake", "befunge": "Befunge",
"blade": "Blade", "bison": "Bison",
"blitz3d": "BlitzBasic", "bitbake": "BitBake",
"blitzbasic": "BlitzBasic", "blade": "Blade",
"blitzmax": "BlitzMax", "blitz3d": "BlitzBasic",
"blitzplus": "BlitzBasic", "blitzbasic": "BlitzBasic",
"bluespec": "Bluespec", "blitzmax": "BlitzMax",
"bmax": "BlitzMax", "blitzplus": "BlitzBasic",
"boo": "Boo", "bluespec": "Bluespec",
"bplus": "BlitzBasic", "bmax": "BlitzMax",
"brainfuck": "Brainfuck", "boo": "Boo",
"brightscript": "Brightscript", "bplus": "BlitzBasic",
"bro": "Bro", "brainfuck": "Brainfuck",
"bsdmake": "Makefile", "brightscript": "Brightscript",
"byond": "DM", "bro": "Bro",
"c": "C", "bsdmake": "Makefile",
"c#": "C#", "byond": "DM",
"c++": "C++", "c": "C",
"c++-objdump": "Cpp-ObjDump", "c#": "C#",
"c-objdump": "C-ObjDump", "c++": "C++",
"c2hs": "C2hs Haskell", "c++-objdump": "Cpp-ObjDump",
"c2hs_haskell": "C2hs Haskell", "c-objdump": "C-ObjDump",
"cap'n_proto": "Cap'n Proto", "c2hs": "C2hs Haskell",
"carto": "CartoCSS", "c2hs_haskell": "C2hs Haskell",
"cartocss": "CartoCSS", "cap'n_proto": "Cap'n Proto",
"ceylon": "Ceylon", "carto": "CartoCSS",
"cfc": "ColdFusion CFC", "cartocss": "CartoCSS",
"cfm": "ColdFusion", "ceylon": "Ceylon",
"cfml": "ColdFusion", "cfc": "ColdFusion CFC",
"chapel": "Chapel", "cfm": "ColdFusion",
"charity": "Charity", "cfml": "ColdFusion",
"chpl": "Chapel", "chapel": "Chapel",
"chuck": "ChucK", "charity": "Charity",
"cirru": "Cirru", "chpl": "Chapel",
"clarion": "Clarion", "chuck": "ChucK",
"clean": "Clean", "cirru": "Cirru",
"click": "Click", "clarion": "Clarion",
"clipper": "xBase", "clean": "Clean",
"clips": "CLIPS", "click": "Click",
"clojure": "Clojure", "clipper": "xBase",
"closure_templates": "Closure Templates", "clips": "CLIPS",
"clojure": "Clojure",
"closure_templates": "Closure Templates",
"cloud_firestore_security_rules": "Cloud Firestore Security Rules",
"cmake": "CMake", "cmake": "CMake",
"cobol": "COBOL", "cobol": "COBOL",
"coffee": "CoffeeScript", "coffee": "CoffeeScript",
@ -117,10 +128,15 @@ var LanguagesByAlias = map[string]string{
"coldfusion_html": "ColdFusion", "coldfusion_html": "ColdFusion",
"collada": "COLLADA", "collada": "COLLADA",
"common_lisp": "Common Lisp", "common_lisp": "Common Lisp",
"common_workflow_language": "Common Workflow Language",
"component_pascal": "Component Pascal", "component_pascal": "Component Pascal",
"conll": "CoNLL-U",
"conll-u": "CoNLL-U",
"conll-x": "CoNLL-U",
"console": "ShellSession", "console": "ShellSession",
"cool": "Cool", "cool": "Cool",
"coq": "Coq", "coq": "Coq",
"cperl": "Perl",
"cpp": "C++", "cpp": "C++",
"cpp-objdump": "Cpp-ObjDump", "cpp-objdump": "Cpp-ObjDump",
"creole": "Creole", "creole": "Creole",
@ -138,12 +154,14 @@ var LanguagesByAlias = map[string]string{
"cucumber": "Gherkin", "cucumber": "Gherkin",
"cuda": "Cuda", "cuda": "Cuda",
"cweb": "CWeb", "cweb": "CWeb",
"cwl": "Common Workflow Language",
"cycript": "Cycript", "cycript": "Cycript",
"cython": "Cython", "cython": "Cython",
"d": "D", "d": "D",
"d-objdump": "D-ObjDump", "d-objdump": "D-ObjDump",
"darcs_patch": "Darcs Patch", "darcs_patch": "Darcs Patch",
"dart": "Dart", "dart": "Dart",
"dataweave": "DataWeave",
"dcl": "DIGITAL Command Language", "dcl": "DIGITAL Command Language",
"delphi": "Component Pascal", "delphi": "Component Pascal",
"desktop": "desktop", "desktop": "desktop",
@ -169,53 +187,69 @@ var LanguagesByAlias = map[string]string{
"ecl": "ECL", "ecl": "ECL",
"eclipse": "ECLiPSe", "eclipse": "ECLiPSe",
"ecr": "HTML+ECR", "ecr": "HTML+ECR",
"edn": "edn", "edje_data_collection": "Edje Data Collection",
"eex": "HTML+EEX", "edn": "edn",
"eiffel": "Eiffel", "eeschema_schematic": "KiCad Schematic",
"ejs": "EJS", "eex": "HTML+EEX",
"elisp": "Emacs Lisp", "eiffel": "Eiffel",
"elixir": "Elixir", "ejs": "EJS",
"elm": "Elm", "elisp": "Emacs Lisp",
"emacs": "Emacs Lisp", "elixir": "Elixir",
"emacs_lisp": "Emacs Lisp", "elm": "Elm",
"emberscript": "EmberScript", "emacs": "Emacs Lisp",
"eq": "EQ", "emacs_lisp": "Emacs Lisp",
"erb": "HTML+ERB", "emberscript": "EmberScript",
"erlang": "Erlang", "eml": "EML",
"f#": "F#", "eq": "EQ",
"factor": "Factor", "erb": "HTML+ERB",
"fancy": "Fancy", "erlang": "Erlang",
"fantom": "Fantom", "f#": "F#",
"filebench_wml": "Filebench WML", "f*": "F*",
"filterscript": "Filterscript", "factor": "Factor",
"fish": "fish", "fancy": "Fancy",
"flex": "Lex", "fantom": "Fantom",
"flux": "FLUX", "figfont": "FIGlet Font",
"formatted": "Formatted", "figlet_font": "FIGlet Font",
"forth": "Forth", "filebench_wml": "Filebench WML",
"fortran": "Fortran", "filterscript": "Filterscript",
"foxpro": "xBase", "fish": "fish",
"freemarker": "FreeMarker", "flex": "Lex",
"frege": "Frege", "flux": "FLUX",
"fsharp": "F#", "formatted": "Formatted",
"ftl": "FreeMarker", "forth": "Forth",
"fundamental": "Text", "fortran": "Fortran",
"g-code": "G-code", "foxpro": "xBase",
"game_maker_language": "Game Maker Language", "freemarker": "FreeMarker",
"gams": "GAMS", "frege": "Frege",
"gap": "GAP", "fsharp": "F#",
"fstar": "F*",
"ftl": "FreeMarker",
"fundamental": "Text",
"g-code": "G-code",
"game_maker_language": "Game Maker Language",
"gams": "GAMS",
"gap": "GAP",
"gcc_machine_description": "GCC Machine Description", "gcc_machine_description": "GCC Machine Description",
"gdb": "GDB", "gdb": "GDB",
"gdscript": "GDScript", "gdscript": "GDScript",
"genie": "Genie", "genie": "Genie",
"genshi": "Genshi", "genshi": "Genshi",
"gentoo_ebuild": "Gentoo Ebuild", "gentoo_ebuild": "Gentoo Ebuild",
"gentoo_eclass": "Gentoo Eclass", "gentoo_eclass": "Gentoo Eclass",
"gettext_catalog": "Gettext Catalog", "gerber_image": "Gerber Image",
"gf": "Grammatical Framework", "gettext_catalog": "Gettext Catalog",
"gherkin": "Gherkin", "gf": "Grammatical Framework",
"glsl": "GLSL", "gherkin": "Gherkin",
"glyph": "Glyph", "git-ignore": "Ignore List",
"git_attributes": "Git Attributes",
"git_config": "Git Config",
"gitattributes": "Git Attributes",
"gitconfig": "Git Config",
"gitignore": "Ignore List",
"gitmodules": "Git Config",
"glsl": "GLSL",
"glyph": "Glyph",
"glyph_bitmap_distribution_format": "Glyph Bitmap Distribution Format",
"gn": "GN", "gn": "GN",
"gnuplot": "Gnuplot", "gnuplot": "Gnuplot",
"go": "Go", "go": "Go",
@ -228,17 +262,20 @@ var LanguagesByAlias = map[string]string{
"graph_modeling_language": "Graph Modeling Language", "graph_modeling_language": "Graph Modeling Language",
"graphql": "GraphQL", "graphql": "GraphQL",
"graphviz_(dot)": "Graphviz (DOT)", "graphviz_(dot)": "Graphviz (DOT)",
"groff": "Roff",
"groovy": "Groovy", "groovy": "Groovy",
"groovy_server_pages": "Groovy Server Pages", "groovy_server_pages": "Groovy Server Pages",
"gsp": "Groovy Server Pages", "gsp": "Groovy Server Pages",
"hack": "Hack", "hack": "Hack",
"haml": "Haml", "haml": "Haml",
"handlebars": "Handlebars", "handlebars": "Handlebars",
"haproxy": "HAProxy",
"harbour": "Harbour", "harbour": "Harbour",
"haskell": "Haskell", "haskell": "Haskell",
"haxe": "Haxe", "haxe": "Haxe",
"hbs": "Handlebars", "hbs": "Handlebars",
"hcl": "HCL", "hcl": "HCL",
"hiveql": "HiveQL",
"hlsl": "HLSL", "hlsl": "HLSL",
"html": "HTML", "html": "HTML",
"html+django": "HTML+Django", "html+django": "HTML+Django",
@ -248,16 +285,20 @@ var LanguagesByAlias = map[string]string{
"html+erb": "HTML+ERB", "html+erb": "HTML+ERB",
"html+jinja": "HTML+Django", "html+jinja": "HTML+Django",
"html+php": "HTML+PHP", "html+php": "HTML+PHP",
"html+razor": "HTML+Razor",
"html+ruby": "RHTML", "html+ruby": "RHTML",
"htmlbars": "Handlebars", "htmlbars": "Handlebars",
"htmldjango": "HTML+Django", "htmldjango": "HTML+Django",
"http": "HTTP", "http": "HTTP",
"hxml": "HXML",
"hy": "Hy", "hy": "Hy",
"hylang": "Hy", "hylang": "Hy",
"hyphy": "HyPhy", "hyphy": "HyPhy",
"i7": "Inform 7", "i7": "Inform 7",
"idl": "IDL", "idl": "IDL",
"idris": "Idris", "idris": "Idris",
"ignore": "Ignore List",
"ignore_list": "Ignore List",
"igor": "IGOR Pro", "igor": "IGOR Pro",
"igor_pro": "IGOR Pro", "igor_pro": "IGOR Pro",
"igorpro": "IGOR Pro", "igorpro": "IGOR Pro",
@ -277,6 +318,7 @@ var LanguagesByAlias = map[string]string{
"j": "J", "j": "J",
"jasmin": "Jasmin", "jasmin": "Jasmin",
"java": "Java", "java": "Java",
"java_properties": "Java Properties",
"java_server_page": "Groovy Server Pages", "java_server_page": "Groovy Server Pages",
"java_server_pages": "Java Server Pages", "java_server_pages": "Java Server Pages",
"javascript": "JavaScript", "javascript": "JavaScript",
@ -288,13 +330,17 @@ var LanguagesByAlias = map[string]string{
"js": "JavaScript", "js": "JavaScript",
"json": "JSON", "json": "JSON",
"json5": "JSON5", "json5": "JSON5",
"json_with_comments": "JSON with Comments",
"jsonc": "JSON with Comments",
"jsoniq": "JSONiq", "jsoniq": "JSONiq",
"jsonld": "JSONLD", "jsonld": "JSONLD",
"jsp": "Java Server Pages", "jsp": "Java Server Pages",
"jsx": "JSX", "jsx": "JSX",
"julia": "Julia", "julia": "Julia",
"jupyter_notebook": "Jupyter Notebook", "jupyter_notebook": "Jupyter Notebook",
"kicad": "KiCad", "kicad_layout": "KiCad Layout",
"kicad_legacy_layout": "KiCad Legacy Layout",
"kicad_schematic": "KiCad Schematic",
"kit": "Kit", "kit": "Kit",
"kotlin": "Kotlin", "kotlin": "Kotlin",
"krl": "KRL", "krl": "KRL",
@ -329,6 +375,7 @@ var LanguagesByAlias = map[string]string{
"loomscript": "LoomScript", "loomscript": "LoomScript",
"ls": "LiveScript", "ls": "LiveScript",
"lsl": "LSL", "lsl": "LSL",
"ltspice_symbol": "LTspice Symbol",
"lua": "Lua", "lua": "Lua",
"m": "M", "m": "M",
"m4": "M4", "m4": "M4",
@ -337,17 +384,22 @@ var LanguagesByAlias = map[string]string{
"make": "Makefile", "make": "Makefile",
"makefile": "Makefile", "makefile": "Makefile",
"mako": "Mako", "mako": "Mako",
"man": "Roff",
"man-page": "Roff",
"man_page": "Roff",
"manpage": "Roff",
"markdown": "Markdown", "markdown": "Markdown",
"marko": "Marko", "marko": "Marko",
"markojs": "Marko", "markojs": "Marko",
"mask": "Mask", "mask": "Mask",
"mathematica": "Mathematica", "mathematica": "Mathematica",
"matlab": "Matlab", "matlab": "MATLAB",
"maven_pom": "Maven POM", "maven_pom": "Maven POM",
"max": "Max", "max": "Max",
"max/msp": "Max", "max/msp": "Max",
"maxmsp": "Max", "maxmsp": "Max",
"maxscript": "MAXScript", "maxscript": "MAXScript",
"mdoc": "Roff",
"mediawiki": "MediaWiki", "mediawiki": "MediaWiki",
"mercury": "Mercury", "mercury": "Mercury",
"meson": "Meson", "meson": "Meson",
@ -358,6 +410,7 @@ var LanguagesByAlias = map[string]string{
"mma": "Mathematica", "mma": "Mathematica",
"modelica": "Modelica", "modelica": "Modelica",
"modula-2": "Modula-2", "modula-2": "Modula-2",
"modula-3": "Modula-3",
"module_management_system": "Module Management System", "module_management_system": "Module Management System",
"monkey": "Monkey", "monkey": "Monkey",
"moocode": "Moocode", "moocode": "Moocode",
@ -369,14 +422,17 @@ var LanguagesByAlias = map[string]string{
"mumps": "M", "mumps": "M",
"mupad": "mupad", "mupad": "mupad",
"myghty": "Myghty", "myghty": "Myghty",
"nanorc": "nanorc",
"nasm": "Assembly", "nasm": "Assembly",
"ncl": "NCL", "ncl": "NCL",
"nearley": "Nearley",
"nemerle": "Nemerle", "nemerle": "Nemerle",
"nesc": "nesC", "nesc": "nesC",
"netlinx": "NetLinx", "netlinx": "NetLinx",
"netlinx+erb": "NetLinx+ERB", "netlinx+erb": "NetLinx+ERB",
"netlogo": "NetLogo", "netlogo": "NetLogo",
"newlisp": "NewLisp", "newlisp": "NewLisp",
"nextflow": "Nextflow",
"nginx": "Nginx", "nginx": "Nginx",
"nginx_configuration_file": "Nginx", "nginx_configuration_file": "Nginx",
"nim": "Nim", "nim": "Nim",
@ -409,8 +465,9 @@ var LanguagesByAlias = map[string]string{
"objectpascal": "Component Pascal", "objectpascal": "Component Pascal",
"objj": "Objective-J", "objj": "Objective-J",
"ocaml": "OCaml", "ocaml": "OCaml",
"octave": "Matlab", "octave": "MATLAB",
"omgrofl": "Omgrofl", "omgrofl": "Omgrofl",
"oncrpc": "RPC",
"ooc": "ooc", "ooc": "ooc",
"opa": "Opa", "opa": "Opa",
"opal": "Opal", "opal": "Opal",
@ -433,216 +490,270 @@ var LanguagesByAlias = map[string]string{
"parrot": "Parrot", "parrot": "Parrot",
"parrot_assembly": "Parrot Assembly", "parrot_assembly": "Parrot Assembly",
"parrot_internal_representation": "Parrot Internal Representation", "parrot_internal_representation": "Parrot Internal Representation",
"pascal": "Pascal", "pascal": "Pascal",
"pasm": "Parrot Assembly", "pasm": "Parrot Assembly",
"pawn": "PAWN", "pawn": "Pawn",
"pep8": "Pep8", "pcbnew": "KiCad Layout",
"perl": "Perl", "pep8": "Pep8",
"perl_6": "Perl 6", "perl": "Perl",
"php": "PHP", "perl6": "Perl 6",
"pic": "Pic", "perl_6": "Perl 6",
"pickle": "Pickle", "php": "PHP",
"picolisp": "PicoLisp", "pic": "Pic",
"piglatin": "PigLatin", "pickle": "Pickle",
"pike": "Pike", "picolisp": "PicoLisp",
"pir": "Parrot Internal Representation", "piglatin": "PigLatin",
"plpgsql": "PLpgSQL", "pike": "Pike",
"plsql": "PLSQL", "pir": "Parrot Internal Representation",
"pod": "Pod", "plpgsql": "PLpgSQL",
"pogoscript": "PogoScript", "plsql": "PLSQL",
"pony": "Pony", "pod": "Pod",
"posh": "PowerShell", "pod_6": "Pod 6",
"postscr": "PostScript", "pogoscript": "PogoScript",
"postscript": "PostScript", "pony": "Pony",
"pot": "Gettext Catalog", "posh": "PowerShell",
"pov-ray": "POV-Ray SDL", "postcss": "PostCSS",
"pov-ray_sdl": "POV-Ray SDL", "postscr": "PostScript",
"povray": "POV-Ray SDL", "postscript": "PostScript",
"powerbuilder": "PowerBuilder", "pot": "Gettext Catalog",
"powershell": "PowerShell", "pov-ray": "POV-Ray SDL",
"processing": "Processing", "pov-ray_sdl": "POV-Ray SDL",
"progress": "OpenEdge ABL", "povray": "POV-Ray SDL",
"prolog": "Prolog", "powerbuilder": "PowerBuilder",
"propeller_spin": "Propeller Spin", "powershell": "PowerShell",
"protobuf": "Protocol Buffer", "processing": "Processing",
"protocol_buffer": "Protocol Buffer", "progress": "OpenEdge ABL",
"protocol_buffers": "Protocol Buffer", "prolog": "Prolog",
"public_key": "Public Key", "propeller_spin": "Propeller Spin",
"pug": "Pug", "protobuf": "Protocol Buffer",
"puppet": "Puppet", "protocol_buffer": "Protocol Buffer",
"pure_data": "Pure Data", "protocol_buffers": "Protocol Buffer",
"purebasic": "PureBasic", "public_key": "Public Key",
"purescript": "PureScript", "pug": "Pug",
"pycon": "Python console", "puppet": "Puppet",
"pyrex": "Cython", "pure_data": "Pure Data",
"python": "Python", "purebasic": "PureBasic",
"python_console": "Python console", "purescript": "PureScript",
"python_traceback": "Python traceback", "pwsh": "PowerShell",
"qmake": "QMake", "pycon": "Python console",
"qml": "QML", "pyrex": "Cython",
"r": "R", "python": "Python",
"racket": "Racket", "python3": "Python",
"ragel": "Ragel", "python_console": "Python console",
"ragel-rb": "Ragel", "python_traceback": "Python traceback",
"ragel-ruby": "Ragel", "q": "q",
"rake": "Ruby", "qmake": "QMake",
"raml": "RAML", "qml": "QML",
"rascal": "Rascal", "quake": "Quake",
"raw": "Raw token data", "r": "R",
"raw_token_data": "Raw token data", "racket": "Racket",
"rb": "Ruby", "ragel": "Ragel",
"rbx": "Ruby", "ragel-rb": "Ragel",
"rdoc": "RDoc", "ragel-ruby": "Ragel",
"realbasic": "REALbasic", "rake": "Ruby",
"reason": "Reason", "raml": "RAML",
"rebol": "Rebol", "rascal": "Rascal",
"red": "Red", "raw": "Raw token data",
"red/system": "Red", "raw_token_data": "Raw token data",
"redcode": "Redcode", "razor": "HTML+Razor",
"regex": "Regular Expression", "rb": "Ruby",
"regexp": "Regular Expression", "rbx": "Ruby",
"regular_expression": "Regular Expression", "rdoc": "RDoc",
"ren'py": "Ren'Py", "realbasic": "REALbasic",
"renderscript": "RenderScript", "reason": "Reason",
"renpy": "Ren'Py", "rebol": "Rebol",
"restructuredtext": "reStructuredText", "red": "Red",
"rexx": "REXX", "red/system": "Red",
"rhtml": "RHTML", "redcode": "Redcode",
"ring": "Ring", "regex": "Regular Expression",
"rmarkdown": "RMarkdown", "regexp": "Regular Expression",
"robotframework": "RobotFramework", "regular_expression": "Regular Expression",
"roff": "Roff", "ren'py": "Ren'Py",
"rouge": "Rouge", "renderscript": "RenderScript",
"rpm_spec": "RPM Spec", "renpy": "Ren'Py",
"rscript": "R", "restructuredtext": "reStructuredText",
"rss": "XML", "rexx": "REXX",
"rst": "reStructuredText", "rhtml": "RHTML",
"ruby": "Ruby", "ring": "Ring",
"runoff": "RUNOFF", "rmarkdown": "RMarkdown",
"rust": "Rust", "robotframework": "RobotFramework",
"rusthon": "Python", "roff": "Roff",
"sage": "Sage", "roff_manpage": "Roff Manpage",
"salt": "SaltStack", "rouge": "Rouge",
"saltstack": "SaltStack", "rpc": "RPC",
"saltstate": "SaltStack", "rpcgen": "RPC",
"sas": "SAS", "rpm_spec": "RPM Spec",
"sass": "Sass", "rs-274x": "Gerber Image",
"scala": "Scala", "rscript": "R",
"scaml": "Scaml", "rss": "XML",
"scheme": "Scheme", "rst": "reStructuredText",
"scilab": "Scilab", "ruby": "Ruby",
"scss": "SCSS", "runoff": "RUNOFF",
"self": "Self", "rust": "Rust",
"sh": "Shell", "rusthon": "Python",
"shaderlab": "ShaderLab", "sage": "Sage",
"shell": "Shell", "salt": "SaltStack",
"shell-script": "Shell", "saltstack": "SaltStack",
"shellsession": "ShellSession", "saltstate": "SaltStack",
"shen": "Shen", "sas": "SAS",
"slash": "Slash", "sass": "Sass",
"slim": "Slim", "scala": "Scala",
"smali": "Smali", "scaml": "Scaml",
"smalltalk": "Smalltalk", "scheme": "Scheme",
"smarty": "Smarty", "scilab": "Scilab",
"sml": "Standard ML", "scss": "SCSS",
"smt": "SMT", "sed": "sed",
"sourcemod": "SourcePawn", "self": "Self",
"sourcepawn": "SourcePawn", "sh": "Shell",
"sparql": "SPARQL", "shaderlab": "ShaderLab",
"specfile": "RPM Spec", "shell": "Shell",
"spline_font_database": "Spline Font Database", "shell-script": "Shell",
"splus": "R", "shellsession": "ShellSession",
"sqf": "SQF", "shen": "Shen",
"sql": "SQL", "slash": "Slash",
"sqlpl": "SQLPL", "slice": "Slice",
"squeak": "Smalltalk", "slim": "Slim",
"squirrel": "Squirrel", "smali": "Smali",
"srecode_template": "SRecode Template", "smalltalk": "Smalltalk",
"stan": "Stan", "smarty": "Smarty",
"standard_ml": "Standard ML", "sml": "Standard ML",
"stata": "Stata", "smt": "SMT",
"ston": "STON", "snippet": "YASnippet",
"stylus": "Stylus", "solidity": "Solidity",
"sublime_text_config": "Sublime Text Config", "sourcemod": "SourcePawn",
"subrip_text": "SubRip Text", "sourcepawn": "SourcePawn",
"supercollider": "SuperCollider", "soy": "Closure Templates",
"svg": "SVG", "sparql": "SPARQL",
"swift": "Swift", "specfile": "RPM Spec",
"systemverilog": "SystemVerilog", "spline_font_database": "Spline Font Database",
"tcl": "Tcl", "splus": "R",
"tcsh": "Tcsh", "sqf": "SQF",
"tea": "Tea", "sql": "SQL",
"terra": "Terra", "sqlpl": "SQLPL",
"tex": "TeX", "squeak": "Smalltalk",
"text": "Text", "squirrel": "Squirrel",
"textile": "Textile", "srecode_template": "SRecode Template",
"thrift": "Thrift", "stan": "Stan",
"ti_program": "TI Program", "standard_ml": "Standard ML",
"tl": "Type Language", "stata": "Stata",
"tla": "TLA", "ston": "STON",
"toml": "TOML", "stylus": "Stylus",
"ts": "TypeScript", "subrip_text": "SubRip Text",
"turing": "Turing", "sugarss": "SugarSS",
"turtle": "Turtle", "supercollider": "SuperCollider",
"twig": "Twig", "svg": "SVG",
"txl": "TXL", "swift": "Swift",
"type_language": "Type Language", "systemverilog": "SystemVerilog",
"typescript": "TypeScript", "tcl": "Tcl",
"udiff": "Diff", "tcsh": "Tcsh",
"unified_parallel_c": "Unified Parallel C", "tea": "Tea",
"unity3d_asset": "Unity3D Asset", "terra": "Terra",
"unix_assembly": "Unix Assembly", "terraform": "HCL",
"uno": "Uno", "tex": "TeX",
"unrealscript": "UnrealScript", "text": "Text",
"ur": "UrWeb", "textile": "Textile",
"ur/web": "UrWeb", "thrift": "Thrift",
"urweb": "UrWeb", "ti_program": "TI Program",
"vala": "Vala", "tl": "Type Language",
"vb.net": "Visual Basic", "tla": "TLA",
"vbnet": "Visual Basic", "toml": "TOML",
"vcl": "VCL", "troff": "Roff",
"verilog": "Verilog", "ts": "TypeScript",
"vhdl": "VHDL", "turing": "Turing",
"vim": "Vim script", "turtle": "Turtle",
"vim_script": "Vim script", "twig": "Twig",
"viml": "Vim script", "txl": "TXL",
"visual_basic": "Visual Basic", "type_language": "Type Language",
"volt": "Volt", "typescript": "TypeScript",
"vue": "Vue", "udiff": "Diff",
"wasm": "WebAssembly", "unified_parallel_c": "Unified Parallel C",
"wast": "WebAssembly", "unity3d_asset": "Unity3D Asset",
"wavefront_material": "Wavefront Material", "unix_assembly": "Unix Assembly",
"wavefront_object": "Wavefront Object", "uno": "Uno",
"web_ontology_language": "Web Ontology Language", "unrealscript": "UnrealScript",
"webassembly": "WebAssembly", "ur": "UrWeb",
"webidl": "WebIDL", "ur/web": "UrWeb",
"winbatch": "Batchfile", "urweb": "UrWeb",
"wisp": "wisp", "vala": "Vala",
"vb.net": "Visual Basic",
"vbnet": "Visual Basic",
"vcl": "VCL",
"verilog": "Verilog",
"vhdl": "VHDL",
"vim": "Vim script",
"vim_script": "Vim script",
"viml": "Vim script",
"visual_basic": "Visual Basic",
"volt": "Volt",
"vue": "Vue",
"wasm": "WebAssembly",
"wast": "WebAssembly",
"wavefront_material": "Wavefront Material",
"wavefront_object": "Wavefront Object",
"wdl": "wdl",
"web_ontology_language": "Web Ontology Language",
"webassembly": "WebAssembly",
"webidl": "WebIDL",
"winbatch": "Batchfile",
"windows_registry_entries": "Windows Registry Entries",
"wisp": "wisp",
"world_of_warcraft_addon_data": "World of Warcraft Addon Data", "world_of_warcraft_addon_data": "World of Warcraft Addon Data",
"wsdl": "XML", "wsdl": "XML",
"x10": "X10", "x10": "X10",
"xbase": "xBase", "x_bitmap": "X BitMap",
"xc": "XC", "x_font_directory_index": "X Font Directory Index",
"xcompose": "XCompose", "x_pixmap": "X PixMap",
"xhtml": "HTML", "xbase": "xBase",
"xml": "XML", "xbm": "X BitMap",
"xml+genshi": "Genshi", "xc": "XC",
"xml+kid": "Genshi", "xcompose": "XCompose",
"xojo": "Xojo", "xdr": "RPC",
"xpages": "XPages", "xhtml": "HTML",
"xproc": "XProc", "xml": "XML",
"xquery": "XQuery", "xml+genshi": "Genshi",
"xs": "XS", "xml+kid": "Genshi",
"xsd": "XML", "xojo": "Xojo",
"xsl": "XSLT", "xpages": "XPages",
"xslt": "XSLT", "xpm": "X PixMap",
"xten": "X10", "xproc": "XProc",
"xtend": "Xtend", "xquery": "XQuery",
"yacc": "Yacc", "xs": "XS",
"yaml": "YAML", "xsd": "XML",
"yang": "YANG", "xsl": "XSLT",
"yml": "YAML", "xslt": "XSLT",
"zephir": "Zephir", "xten": "X10",
"zimpl": "Zimpl", "xtend": "Xtend",
"zsh": "Shell", "yacc": "Yacc",
"yaml": "YAML",
"yang": "YANG",
"yara": "YARA",
"yas": "YASnippet",
"yasnippet": "YASnippet",
"yml": "YAML",
"zephir": "Zephir",
"zig": "Zig",
"zimpl": "Zimpl",
"zsh": "Shell",
}
// LanguageByAlias looks up the language name by it's alias or name.
// It mirrors the logic of github linguist and is needed e.g for heuristcs.yml
// that mixes names and aliases in a language field (see XPM example).
func LanguageByAlias(langOrAlias string) (lang string, ok bool) {
k := convertToAliasKey(langOrAlias)
lang, ok = LanguageByAliasMap[k]
return
}
// convertToAliasKey converts language name to a key in LanguageByAliasMap.
// Following
// - internal.code-generator.generator.convertToAliasKey()
// - GetLanguageByAlias()
// conventions.
// It is here to avoid dependency on "generate" and "enry" packages.
func convertToAliasKey(langName string) string {
ak := strings.SplitN(langName, `,`, 2)[0]
ak = strings.Replace(ak, ` `, `_`, -1)
ak = strings.ToLower(ak)
return ak
} }

4
internal/code-generator/generator/test_files/commit.gold
View File

@ -1,7 +1,7 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
// linguist's commit from which files were generated. // linguist's commit from which files were generated.
var LinguistCommit = "d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68" var LinguistCommit = "e4560984058b4726010ca4b8f03ed9d0f8f464db"

1780
internal/code-generator/generator/test_files/content.gold
View File

File diff suppressed because it is too large Load Diff

4
internal/code-generator/generator/test_files/documentation.gold
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -8,10 +8,12 @@ import "gopkg.in/toqueteos/substring.v1"
var DocumentationMatchers = substring.Or( var DocumentationMatchers = substring.Or(
substring.Regexp(`^[Dd]ocs?/`), substring.Regexp(`^[Dd]ocs?/`),
substring.Regexp(`(^|/)[Dd]ocumentation/`), substring.Regexp(`(^|/)[Dd]ocumentation/`),
substring.Regexp(`(^|/)[Gg]roovydoc/`),
substring.Regexp(`(^|/)[Jj]avadoc/`), substring.Regexp(`(^|/)[Jj]avadoc/`),
substring.Regexp(`^[Mm]an/`), substring.Regexp(`^[Mm]an/`),
substring.Regexp(`^[Ee]xamples/`), substring.Regexp(`^[Ee]xamples/`),
substring.Regexp(`^[Dd]emos?/`), substring.Regexp(`^[Dd]emos?/`),
substring.Regexp(`(^|/)inst/doc/`),
substring.Regexp(`(^|/)CHANGE(S|LOG)?(\.|$)`), substring.Regexp(`(^|/)CHANGE(S|LOG)?(\.|$)`),
substring.Regexp(`(^|/)CONTRIBUTING(\.|$)`), substring.Regexp(`(^|/)CONTRIBUTING(\.|$)`),
substring.Regexp(`(^|/)COPYING(\.|$)`), substring.Regexp(`(^|/)COPYING(\.|$)`),

708
internal/code-generator/generator/test_files/extension.gold
View File

File diff suppressed because it is too large Load Diff

63
internal/code-generator/generator/test_files/filename.gold
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -8,40 +8,65 @@ var LanguagesByFilename = map[string][]string{
".XCompose": {"XCompose"}, ".XCompose": {"XCompose"},
".abbrev_defs": {"Emacs Lisp"}, ".abbrev_defs": {"Emacs Lisp"},
".arcconfig": {"JSON"}, ".arcconfig": {"JSON"},
".babelrc": {"JSON5"}, ".atomignore": {"Ignore List"},
".babelignore": {"Ignore List"},
".babelrc": {"JSON with Comments"},
".bash_aliases": {"Shell"},
".bash_history": {"Shell"}, ".bash_history": {"Shell"},
".bash_logout": {"Shell"}, ".bash_logout": {"Shell"},
".bash_profile": {"Shell"}, ".bash_profile": {"Shell"},
".bashrc": {"Shell"}, ".bashrc": {"Shell"},
".bzrignore": {"Ignore List"},
".clang-format": {"YAML"}, ".clang-format": {"YAML"},
".clang-tidy": {"YAML"},
".classpath": {"XML"}, ".classpath": {"XML"},
".coffeelintignore": {"Ignore List"},
".cproject": {"XML"}, ".cproject": {"XML"},
".cshrc": {"Shell"}, ".cshrc": {"Shell"},
".cvsignore": {"Ignore List"},
".dockerignore": {"Ignore List"},
".editorconfig": {"INI"}, ".editorconfig": {"INI"},
".emacs": {"Emacs Lisp"}, ".emacs": {"Emacs Lisp"},
".emacs.desktop": {"Emacs Lisp"}, ".emacs.desktop": {"Emacs Lisp"},
".eslintignore": {"Ignore List"},
".eslintrc.json": {"JSON with Comments"},
".factor-boot-rc": {"Factor"}, ".factor-boot-rc": {"Factor"},
".factor-rc": {"Factor"}, ".factor-rc": {"Factor"},
".gclient": {"Python"}, ".gclient": {"Python"},
".gemrc": {"YAML"}, ".gemrc": {"YAML"},
".gitconfig": {"INI"}, ".gitattributes": {"Git Attributes"},
".gitconfig": {"Git Config"},
".gitignore": {"Ignore List"},
".gitmodules": {"Git Config"},
".gn": {"GN"}, ".gn": {"GN"},
".gnus": {"Emacs Lisp"}, ".gnus": {"Emacs Lisp"},
".gvimrc": {"Vim script"}, ".gvimrc": {"Vim script"},
".htaccess": {"ApacheConf"}, ".htaccess": {"ApacheConf"},
".htmlhintrc": {"JSON"},
".irbrc": {"Ruby"}, ".irbrc": {"Ruby"},
".jshintrc": {"JSON"}, ".jscsrc": {"JSON with Comments"},
".jshintrc": {"JSON with Comments"},
".jslintrc": {"JSON with Comments"},
".login": {"Shell"}, ".login": {"Shell"},
".nanorc": {"nanorc"},
".nodemonignore": {"Ignore List"},
".npmignore": {"Ignore List"},
".nvimrc": {"Vim script"}, ".nvimrc": {"Vim script"},
".php": {"PHP"}, ".php": {"PHP"},
".php_cs": {"PHP"}, ".php_cs": {"PHP"},
".php_cs.dist": {"PHP"}, ".php_cs.dist": {"PHP"},
".prettierignore": {"Ignore List"},
".profile": {"Shell"}, ".profile": {"Shell"},
".project": {"XML"}, ".project": {"XML"},
".pryrc": {"Ruby"}, ".pryrc": {"Ruby"},
".spacemacs": {"Emacs Lisp"}, ".spacemacs": {"Emacs Lisp"},
".stylelintignore": {"Ignore List"},
".tern-config": {"JSON"},
".tern-project": {"JSON"},
".vimrc": {"Vim script"}, ".vimrc": {"Vim script"},
".viper": {"Emacs Lisp"}, ".viper": {"Emacs Lisp"},
".vscodeignore": {"Ignore List"},
".watchmanconfig": {"JSON"},
".zlogin": {"Shell"}, ".zlogin": {"Shell"},
".zlogout": {"Shell"}, ".zlogout": {"Shell"},
".zprofile": {"Shell"}, ".zprofile": {"Shell"},
@ -54,6 +79,7 @@ var LanguagesByFilename = map[string][]string{
"BSDmakefile": {"Makefile"}, "BSDmakefile": {"Makefile"},
"BUCK": {"Python"}, "BUCK": {"Python"},
"BUILD": {"Python"}, "BUILD": {"Python"},
"BUILD.bazel": {"Python"},
"Berksfile": {"Ruby"}, "Berksfile": {"Ruby"},
"Brewfile": {"Ruby"}, "Brewfile": {"Ruby"},
"Buildfile": {"Ruby"}, "Buildfile": {"Ruby"},
@ -63,6 +89,7 @@ var LanguagesByFilename = map[string][]string{
"COPYRIGHT.regex": {"Text"}, "COPYRIGHT.regex": {"Text"},
"Cakefile": {"CoffeeScript"}, "Cakefile": {"CoffeeScript"},
"Capfile": {"Ruby"}, "Capfile": {"Ruby"},
"Cargo.lock": {"TOML"},
"Cask": {"Emacs Lisp"}, "Cask": {"Emacs Lisp"},
"Dangerfile": {"Ruby"}, "Dangerfile": {"Ruby"},
"Deliverfile": {"Ruby"}, "Deliverfile": {"Ruby"},
@ -74,6 +101,7 @@ var LanguagesByFilename = map[string][]string{
"GNUmakefile": {"Makefile"}, "GNUmakefile": {"Makefile"},
"Gemfile": {"Ruby"}, "Gemfile": {"Ruby"},
"Gemfile.lock": {"Ruby"}, "Gemfile.lock": {"Ruby"},
"Gopkg.lock": {"TOML"},
"Guardfile": {"Ruby"}, "Guardfile": {"Ruby"},
"INSTALL": {"Text"}, "INSTALL": {"Text"},
"INSTALL.mysql": {"Text"}, "INSTALL.mysql": {"Text"},
@ -84,6 +112,7 @@ var LanguagesByFilename = map[string][]string{
"LICENSE": {"Text"}, "LICENSE": {"Text"},
"LICENSE.mysql": {"Text"}, "LICENSE.mysql": {"Text"},
"Makefile": {"Makefile"}, "Makefile": {"Makefile"},
"Makefile.PL": {"Perl"},
"Makefile.am": {"Makefile"}, "Makefile.am": {"Makefile"},
"Makefile.boot": {"Makefile"}, "Makefile.boot": {"Makefile"},
"Makefile.frag": {"Makefile"}, "Makefile.frag": {"Makefile"},
@ -106,7 +135,7 @@ var LanguagesByFilename = map[string][]string{
"README.mysql": {"Text"}, "README.mysql": {"Text"},
"ROOT": {"Isabelle ROOT"}, "ROOT": {"Isabelle ROOT"},
"Rakefile": {"Ruby"}, "Rakefile": {"Ruby"},
"Rexfile": {"Perl 6"}, "Rexfile": {"Perl"},
"SConscript": {"Python"}, "SConscript": {"Python"},
"SConstruct": {"Python"}, "SConstruct": {"Python"},
"Settings.StyleCop": {"XML"}, "Settings.StyleCop": {"XML"},
@ -126,25 +155,44 @@ var LanguagesByFilename = map[string][]string{
"ack": {"Perl"}, "ack": {"Perl"},
"ant.xml": {"Ant Build System"}, "ant.xml": {"Ant Build System"},
"apache2.conf": {"ApacheConf"}, "apache2.conf": {"ApacheConf"},
"bash_aliases": {"Shell"},
"bash_logout": {"Shell"}, "bash_logout": {"Shell"},
"bash_profile": {"Shell"}, "bash_profile": {"Shell"},
"bashrc": {"Shell"}, "bashrc": {"Shell"},
"build.xml": {"Ant Build System"}, "build.xml": {"Ant Build System"},
"buildfile": {"Ruby"}, "buildfile": {"Ruby"},
"buildozer.spec": {"INI"},
"click.me": {"Text"}, "click.me": {"Text"},
"composer.lock": {"JSON"}, "composer.lock": {"JSON"},
"configure.ac": {"M4Sugar"}, "configure.ac": {"M4Sugar"},
"contents.lr": {"Markdown"},
"cpanfile": {"Perl"},
"cshrc": {"Shell"}, "cshrc": {"Shell"},
"delete.me": {"Text"}, "delete.me": {"Text"},
"descrip.mmk": {"Module Management System"}, "descrip.mmk": {"Module Management System"},
"descrip.mms": {"Module Management System"}, "descrip.mms": {"Module Management System"},
"encodings.dir": {"X Font Directory Index"},
"expr-dist": {"R"}, "expr-dist": {"R"},
"firestore.rules": {"Cloud Firestore Security Rules"},
"fonts.alias": {"X Font Directory Index"},
"fonts.dir": {"X Font Directory Index"},
"fonts.scale": {"X Font Directory Index"},
"fp-lib-table": {"KiCad Layout"},
"gitignore-global": {"Ignore List"},
"gitignore_global": {"Ignore List"},
"glide.lock": {"YAML"},
"go.mod": {"Text"},
"go.sum": {"Text"},
"gradlew": {"Shell"}, "gradlew": {"Shell"},
"gvimrc": {"Vim script"}, "gvimrc": {"Vim script"},
"haproxy.cfg": {"HAProxy"},
"httpd.conf": {"ApacheConf"}, "httpd.conf": {"ApacheConf"},
"jsconfig.json": {"JSON with Comments"},
"keep.me": {"Text"}, "keep.me": {"Text"},
"ld.script": {"Linker Script"}, "ld.script": {"Linker Script"},
"login": {"Shell"}, "login": {"Shell"},
"m3makefile": {"Quake"},
"m3overrides": {"Quake"},
"makefile": {"Makefile"}, "makefile": {"Makefile"},
"makefile.sco": {"Makefile"}, "makefile.sco": {"Makefile"},
"man": {"Shell"}, "man": {"Shell"},
@ -155,7 +203,10 @@ var LanguagesByFilename = map[string][]string{
"mkfile": {"Makefile"}, "mkfile": {"Makefile"},
"mmn": {"Roff"}, "mmn": {"Roff"},
"mmt": {"Roff"}, "mmt": {"Roff"},
"nanorc": {"nanorc"},
"nextflow.config": {"Nextflow"},
"nginx.conf": {"Nginx"}, "nginx.conf": {"Nginx"},
"nim.cfg": {"Nim"},
"nvimrc": {"Vim script"}, "nvimrc": {"Vim script"},
"owh": {"Tcl"}, "owh": {"Tcl"},
"packages.config": {"XML"}, "packages.config": {"XML"},
@ -167,9 +218,9 @@ var LanguagesByFilename = map[string][]string{
"rebar.config.lock": {"Erlang"}, "rebar.config.lock": {"Erlang"},
"rebar.lock": {"Erlang"}, "rebar.lock": {"Erlang"},
"riemann.config": {"Clojure"}, "riemann.config": {"Clojure"},
"script": {"C"},
"starfield": {"Tcl"}, "starfield": {"Tcl"},
"test.me": {"Text"}, "test.me": {"Text"},
"tsconfig.json": {"JSON with Comments"},
"vimrc": {"Vim script"}, "vimrc": {"Vim script"},
"wscript": {"Python"}, "wscript": {"Python"},
"xcompose": {"XCompose"}, "xcompose": {"XCompose"},

94174
internal/code-generator/generator/test_files/frequencies.gold
View File

File diff suppressed because it is too large Load Diff

40
internal/code-generator/generator/test_files/heuristics.yml Normal file
View File

@ -0,0 +1,40 @@
# Tests care about number and order of heuristics in this fixture
disambiguations:
- extensions: ['.h', '.hh']
rules:
- language: Objective-C
pattern: 'objc'
- language: C++
named_pattern: cpp
- extensions: ['.f']
rules:
- language: Forth
pattern: #as in .md
- 'f'
- 'f1'
- language: Filebench WML
pattern: 'f2'
- language: Fortran
named_pattern: fortran
- extensions: ['.ms']
rules:
- language: Roff
pattern: 'rp'
- language: Unix Assembly
and:
- negative_pattern: 'np'
- pattern: 'p'
- language: MAXScript
- extensions: ['.mod']
rules:
- language: [Linux Kernel Module, AMPL]
named_patterns:
cpp:
- 'regex1'
- 'regex2'
fortran: 'regex3'

26
internal/code-generator/generator/test_files/interpreter.gold
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -7,6 +7,8 @@ var LanguagesByInterpreter = map[string][]string{
"Rscript": {"R"}, "Rscript": {"R"},
"apl": {"APL"}, "apl": {"APL"},
"aplx": {"APL"}, "aplx": {"APL"},
"ash": {"Shell"},
"asy": {"Asymptote"},
"awk": {"Awk"}, "awk": {"Awk"},
"bash": {"Shell"}, "bash": {"Shell"},
"bigloo": {"Scheme"}, "bigloo": {"Scheme"},
@ -15,10 +17,13 @@ var LanguagesByInterpreter = map[string][]string{
"chicken": {"Scheme"}, "chicken": {"Scheme"},
"clisp": {"Common Lisp"}, "clisp": {"Common Lisp"},
"coffee": {"CoffeeScript"}, "coffee": {"CoffeeScript"},
"cperl": {"Perl"},
"crystal": {"Crystal"}, "crystal": {"Crystal"},
"csi": {"Scheme"}, "csi": {"Scheme"},
"cvc4": {"SMT"}, "cvc4": {"SMT"},
"cwl-runner": {"Common Workflow Language"},
"dart": {"Dart"}, "dart": {"Dart"},
"dash": {"Shell"},
"dtrace": {"DTrace"}, "dtrace": {"DTrace"},
"dyalog": {"APL"}, "dyalog": {"APL"},
"ecl": {"Common Lisp"}, "ecl": {"Common Lisp"},
@ -26,11 +31,15 @@ var LanguagesByInterpreter = map[string][]string{
"escript": {"Erlang"}, "escript": {"Erlang"},
"fish": {"fish"}, "fish": {"fish"},
"gawk": {"Awk"}, "gawk": {"Awk"},
"gerbv": {"Gerber Image"},
"gerbview": {"Gerber Image"},
"gn": {"GN"}, "gn": {"GN"},
"gnuplot": {"Gnuplot"}, "gnuplot": {"Gnuplot"},
"gosh": {"Scheme"}, "gosh": {"Scheme"},
"groovy": {"Groovy"}, "groovy": {"Groovy"},
"gsed": {"sed"},
"guile": {"Scheme"}, "guile": {"Scheme"},
"hy": {"Hy"},
"instantfpc": {"Pascal"}, "instantfpc": {"Pascal"},
"io": {"Io"}, "io": {"Io"},
"ioke": {"Ioke"}, "ioke": {"Ioke"},
@ -38,6 +47,7 @@ var LanguagesByInterpreter = map[string][]string{
"jolie": {"Jolie"}, "jolie": {"Jolie"},
"jruby": {"Ruby"}, "jruby": {"Ruby"},
"julia": {"Julia"}, "julia": {"Julia"},
"ksh": {"Shell"},
"lisp": {"Common Lisp"}, "lisp": {"Common Lisp"},
"lsl": {"LSL"}, "lsl": {"LSL"},
"lua": {"Lua", "Terra"}, "lua": {"Lua", "Terra"},
@ -45,11 +55,14 @@ var LanguagesByInterpreter = map[string][]string{
"make": {"Makefile"}, "make": {"Makefile"},
"mathsat5": {"SMT"}, "mathsat5": {"SMT"},
"mawk": {"Awk"}, "mawk": {"Awk"},
"minised": {"sed"},
"mksh": {"Shell"},
"mmi": {"Mercury"}, "mmi": {"Mercury"},
"moon": {"MoonScript"}, "moon": {"MoonScript"},
"nawk": {"Awk"}, "nawk": {"Awk"},
"newlisp": {"NewLisp"}, "newlisp": {"NewLisp"},
"node": {"JavaScript"}, "nextflow": {"Nextflow"},
"node": {"JavaScript", "TypeScript"},
"nush": {"Nu"}, "nush": {"Nu"},
"ocaml": {"OCaml", "Reason"}, "ocaml": {"OCaml", "Reason"},
"ocamlrun": {"OCaml"}, "ocamlrun": {"OCaml"},
@ -58,12 +71,14 @@ var LanguagesByInterpreter = map[string][]string{
"opensmt": {"SMT"}, "opensmt": {"SMT"},
"osascript": {"AppleScript"}, "osascript": {"AppleScript"},
"parrot": {"Parrot Assembly", "Parrot Internal Representation"}, "parrot": {"Parrot Assembly", "Parrot Internal Representation"},
"perl": {"Perl"}, "pdksh": {"Shell"},
"perl6": {"Perl 6"}, "perl": {"Perl", "Pod"},
"perl6": {"Perl 6", "Pod 6"},
"php": {"PHP"}, "php": {"PHP"},
"picolisp": {"PicoLisp"}, "picolisp": {"PicoLisp"},
"pike": {"Pike"}, "pike": {"Pike"},
"pil": {"PicoLisp"}, "pil": {"PicoLisp"},
"pwsh": {"PowerShell"},
"python": {"Python"}, "python": {"Python"},
"python2": {"Python"}, "python2": {"Python"},
"python3": {"Python"}, "python3": {"Python"},
@ -80,11 +95,14 @@ var LanguagesByInterpreter = map[string][]string{
"runhaskell": {"Haskell"}, "runhaskell": {"Haskell"},
"sbcl": {"Common Lisp"}, "sbcl": {"Common Lisp"},
"scala": {"Scala"}, "scala": {"Scala"},
"scheme": {"Scheme"},
"sclang": {"SuperCollider"}, "sclang": {"SuperCollider"},
"scsynth": {"SuperCollider"}, "scsynth": {"SuperCollider"},
"sed": {"sed"},
"sh": {"Shell"}, "sh": {"Shell"},
"smt-rat": {"SMT"}, "smt-rat": {"SMT"},
"smtinterpol": {"SMT"}, "smtinterpol": {"SMT"},
"ssed": {"sed"},
"stp": {"SMT"}, "stp": {"SMT"},
"swipl": {"Prolog"}, "swipl": {"Prolog"},
"tcc": {"C"}, "tcc": {"C"},

390
internal/code-generator/generator/test_files/mimeType.gold
View File

@ -1,203 +1,219 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
var LanguagesMime = map[string]string{ var LanguagesMime = map[string]string{
"AGS Script": "text/x-c++src", "AGS Script": "text/x-c++src",
"APL": "text/apl", "APL": "text/apl",
"ASN.1": "text/x-ttcn-asn", "ASN.1": "text/x-ttcn-asn",
"ASP": "application/x-aspx", "ASP": "application/x-aspx",
"Alpine Abuild": "text/x-sh", "Alpine Abuild": "text/x-sh",
"Ant Build System": "application/xml", "AngelScript": "text/x-c++src",
"Apex": "text/x-java", "Ant Build System": "application/xml",
"Arduino": "text/x-c++src", "Apex": "text/x-java",
"Brainfuck": "text/x-brainfuck", "Asymptote": "text/x-kotlin",
"C": "text/x-csrc", "Brainfuck": "text/x-brainfuck",
"C#": "text/x-csharp", "C": "text/x-csrc",
"C++": "text/x-c++src", "C#": "text/x-csharp",
"C2hs Haskell": "text/x-haskell", "C++": "text/x-c++src",
"CMake": "text/x-cmake", "C2hs Haskell": "text/x-haskell",
"COBOL": "text/x-cobol", "CMake": "text/x-cmake",
"COLLADA": "text/xml", "COBOL": "text/x-cobol",
"CSON": "text/x-coffeescript", "COLLADA": "text/xml",
"CSS": "text/css", "CSON": "text/x-coffeescript",
"ChucK": "text/x-java", "CSS": "text/css",
"Clojure": "text/x-clojure", "ChucK": "text/x-java",
"Closure Templates": "text/x-soy", "Clojure": "text/x-clojure",
"CoffeeScript": "text/x-coffeescript", "Closure Templates": "text/x-soy",
"Common Lisp": "text/x-common-lisp", "Cloud Firestore Security Rules": "text/css",
"Component Pascal": "text/x-pascal", "CoffeeScript": "text/x-coffeescript",
"Crystal": "text/x-crystal", "Common Lisp": "text/x-common-lisp",
"Cuda": "text/x-c++src", "Common Workflow Language": "text/x-yaml",
"Cycript": "text/javascript", "Component Pascal": "text/x-pascal",
"Cython": "text/x-cython", "Crystal": "text/x-crystal",
"D": "text/x-d", "Cuda": "text/x-c++src",
"DTrace": "text/x-csrc", "Cycript": "text/javascript",
"Dart": "application/dart", "Cython": "text/x-cython",
"Diff": "text/x-diff", "D": "text/x-d",
"Dockerfile": "text/x-dockerfile", "DTrace": "text/x-csrc",
"Dylan": "text/x-dylan", "Dart": "application/dart",
"EBNF": "text/x-ebnf", "Diff": "text/x-diff",
"ECL": "text/x-ecl", "Dockerfile": "text/x-dockerfile",
"EQ": "text/x-csharp", "Dylan": "text/x-dylan",
"Eagle": "text/xml", "EBNF": "text/x-ebnf",
"Easybuild": "text/x-python", "ECL": "text/x-ecl",
"Ecere Projects": "application/json", "EQ": "text/x-csharp",
"Eiffel": "text/x-eiffel", "Eagle": "text/xml",
"Elm": "text/x-elm", "Easybuild": "text/x-python",
"Emacs Lisp": "text/x-common-lisp", "Ecere Projects": "application/json",
"EmberScript": "text/x-coffeescript", "Edje Data Collection": "application/json",
"Erlang": "text/x-erlang", "Eiffel": "text/x-eiffel",
"F#": "text/x-fsharp", "Elm": "text/x-elm",
"Factor": "text/x-factor", "Emacs Lisp": "text/x-common-lisp",
"Forth": "text/x-forth", "EmberScript": "text/x-coffeescript",
"Fortran": "text/x-fortran", "Erlang": "text/x-erlang",
"GCC Machine Description": "text/x-common-lisp", "F#": "text/x-fsharp",
"Factor": "text/x-factor",
"Forth": "text/x-forth",
"Fortran": "text/x-fortran",
"GCC Machine Description": "text/x-common-lisp",
"GN": "text/x-python", "GN": "text/x-python",
"Game Maker Language": "text/x-c++src", "Game Maker Language": "text/x-c++src",
"Genshi": "text/xml", "Genshi": "text/xml",
"Gentoo Ebuild": "text/x-sh", "Gentoo Ebuild": "text/x-sh",
"Gentoo Eclass": "text/x-sh", "Gentoo Eclass": "text/x-sh",
"Git Attributes": "text/x-sh",
"Git Config": "text/x-properties",
"Glyph": "text/x-tcl", "Glyph": "text/x-tcl",
"Go": "text/x-go", "Go": "text/x-go",
"Grammatical Framework": "text/x-haskell", "Grammatical Framework": "text/x-haskell",
"Groovy": "text/x-groovy", "Groovy": "text/x-groovy",
"Groovy Server Pages": "application/x-jsp", "Groovy Server Pages": "application/x-jsp",
"HCL": "text/x-ruby", "HCL": "text/x-ruby",
"HTML": "text/html", "HTML": "text/html",
"HTML+Django": "text/x-django", "HTML+Django": "text/x-django",
"HTML+ECR": "text/html", "HTML+ECR": "text/html",
"HTML+EEX": "text/html", "HTML+EEX": "text/html",
"HTML+ERB": "application/x-erb", "HTML+ERB": "application/x-erb",
"HTML+PHP": "application/x-httpd-php", "HTML+PHP": "application/x-httpd-php",
"HTTP": "message/http", "HTML+Razor": "text/html",
"Hack": "application/x-httpd-php", "HTTP": "message/http",
"Haml": "text/x-haml", "Hack": "application/x-httpd-php",
"Haskell": "text/x-haskell", "Haml": "text/x-haml",
"Haxe": "text/x-haxe", "Haskell": "text/x-haskell",
"IDL": "text/x-idl", "Haxe": "text/x-haxe",
"INI": "text/x-properties", "IDL": "text/x-idl",
"IRC log": "text/mirc", "INI": "text/x-properties",
"JSON": "application/json", "IRC log": "text/mirc",
"JSON5": "application/json", "Ignore List": "text/x-sh",
"JSONiq": "application/json", "JSON": "application/json",
"JSX": "text/jsx", "JSON with Comments": "text/javascript",
"Java": "text/x-java", "JSON5": "application/json",
"Java Server Pages": "application/x-jsp", "JSONLD": "application/json",
"JavaScript": "text/javascript", "JSONiq": "application/json",
"Julia": "text/x-julia", "JSX": "text/jsx",
"Jupyter Notebook": "application/json", "Java": "text/x-java",
"Kit": "text/html", "Java Properties": "text/x-properties",
"Kotlin": "text/x-kotlin", "Java Server Pages": "application/x-jsp",
"LFE": "text/x-common-lisp", "JavaScript": "text/javascript",
"LabVIEW": "text/xml", "Julia": "text/x-julia",
"Latte": "text/x-smarty", "Jupyter Notebook": "application/json",
"Less": "text/css", "KiCad Layout": "text/x-common-lisp",
"Literate Haskell": "text/x-literate-haskell", "Kit": "text/html",
"LiveScript": "text/x-livescript", "Kotlin": "text/x-kotlin",
"LookML": "text/x-yaml", "LFE": "text/x-common-lisp",
"Lua": "text/x-lua", "LTspice Symbol": "text/x-spreadsheet",
"M": "text/x-mumps", "LabVIEW": "text/xml",
"MTML": "text/html", "Latte": "text/x-smarty",
"MUF": "text/x-forth", "Less": "text/css",
"Makefile": "text/x-cmake", "Literate Haskell": "text/x-literate-haskell",
"Markdown": "text/x-gfm", "LiveScript": "text/x-livescript",
"Marko": "text/html", "LookML": "text/x-yaml",
"Mathematica": "text/x-mathematica", "Lua": "text/x-lua",
"Matlab": "text/x-octave", "M": "text/x-mumps",
"Maven POM": "text/xml", "MATLAB": "text/x-octave",
"Max": "application/json", "MTML": "text/html",
"Metal": "text/x-c++src", "MUF": "text/x-forth",
"Mirah": "text/x-ruby", "Makefile": "text/x-cmake",
"Modelica": "text/x-modelica", "Markdown": "text/x-gfm",
"NSIS": "text/x-nsis", "Marko": "text/html",
"NetLogo": "text/x-common-lisp", "Mathematica": "text/x-mathematica",
"NewLisp": "text/x-common-lisp", "Maven POM": "text/xml",
"Nginx": "text/x-nginx-conf", "Max": "application/json",
"Nu": "text/x-scheme", "Metal": "text/x-c++src",
"NumPy": "text/x-python", "Mirah": "text/x-ruby",
"OCaml": "text/x-ocaml", "Modelica": "text/x-modelica",
"Objective-C": "text/x-objectivec", "NSIS": "text/x-nsis",
"Objective-C++": "text/x-objectivec", "NetLogo": "text/x-common-lisp",
"OpenCL": "text/x-csrc", "NewLisp": "text/x-common-lisp",
"OpenRC runscript": "text/x-sh", "Nginx": "text/x-nginx-conf",
"Oz": "text/x-oz", "Nu": "text/x-scheme",
"PHP": "application/x-httpd-php", "NumPy": "text/x-python",
"PLSQL": "text/x-plsql", "OCaml": "text/x-ocaml",
"PLpgSQL": "text/x-sql", "Objective-C": "text/x-objectivec",
"Pascal": "text/x-pascal", "Objective-C++": "text/x-objectivec",
"Perl": "text/x-perl", "OpenCL": "text/x-csrc",
"Perl 6": "text/x-perl", "OpenRC runscript": "text/x-sh",
"Pic": "text/troff", "Oz": "text/x-oz",
"Pod": "text/x-perl", "PHP": "application/x-httpd-php",
"PowerShell": "application/x-powershell", "PLSQL": "text/x-plsql",
"Protocol Buffer": "text/x-protobuf", "PLpgSQL": "text/x-sql",
"Public Key": "application/pgp", "Pascal": "text/x-pascal",
"Pug": "text/x-pug", "Perl": "text/x-perl",
"Puppet": "text/x-puppet", "Perl 6": "text/x-perl",
"PureScript": "text/x-haskell", "Pic": "text/troff",
"Python": "text/x-python", "Pod": "text/x-perl",
"R": "text/x-rsrc", "PowerShell": "application/x-powershell",
"RAML": "text/x-yaml", "Protocol Buffer": "text/x-protobuf",
"RHTML": "application/x-erb", "Public Key": "application/pgp",
"RMarkdown": "text/x-gfm", "Pug": "text/x-pug",
"RPM Spec": "text/x-rpm-spec", "Puppet": "text/x-puppet",
"Reason": "text/x-rustsrc", "PureScript": "text/x-haskell",
"Roff": "text/troff", "Python": "text/x-python",
"Rouge": "text/x-clojure", "R": "text/x-rsrc",
"Ruby": "text/x-ruby", "RAML": "text/x-yaml",
"Rust": "text/x-rustsrc", "RHTML": "application/x-erb",
"SAS": "text/x-sas", "RMarkdown": "text/x-gfm",
"SCSS": "text/x-scss", "RPM Spec": "text/x-rpm-spec",
"SPARQL": "application/sparql-query", "Reason": "text/x-rustsrc",
"SQL": "text/x-sql", "Roff": "text/troff",
"SQLPL": "text/x-sql", "Roff Manpage": "text/troff",
"SRecode Template": "text/x-common-lisp", "Rouge": "text/x-clojure",
"SVG": "text/xml", "Ruby": "text/x-ruby",
"Sage": "text/x-python", "Rust": "text/x-rustsrc",
"SaltStack": "text/x-yaml", "SAS": "text/x-sas",
"Sass": "text/x-sass", "SCSS": "text/x-scss",
"Scala": "text/x-scala", "SPARQL": "application/sparql-query",
"Scheme": "text/x-scheme", "SQL": "text/x-sql",
"Shell": "text/x-sh", "SQLPL": "text/x-sql",
"ShellSession": "text/x-sh", "SRecode Template": "text/x-common-lisp",
"Slim": "text/x-slim", "SVG": "text/xml",
"Smalltalk": "text/x-stsrc", "Sage": "text/x-python",
"Smarty": "text/x-smarty", "SaltStack": "text/x-yaml",
"Squirrel": "text/x-c++src", "Sass": "text/x-sass",
"Standard ML": "text/x-ocaml", "Scala": "text/x-scala",
"Sublime Text Config": "text/javascript", "Scheme": "text/x-scheme",
"Swift": "text/x-swift", "Shell": "text/x-sh",
"SystemVerilog": "text/x-systemverilog", "ShellSession": "text/x-sh",
"TOML": "text/x-toml", "Slim": "text/x-slim",
"Tcl": "text/x-tcl", "Smalltalk": "text/x-stsrc",
"Tcsh": "text/x-sh", "Smarty": "text/x-smarty",
"TeX": "text/x-stex", "Squirrel": "text/x-c++src",
"Terra": "text/x-lua", "Standard ML": "text/x-ocaml",
"Textile": "text/x-textile", "Swift": "text/x-swift",
"Turtle": "text/turtle", "SystemVerilog": "text/x-systemverilog",
"Twig": "text/x-twig", "TOML": "text/x-toml",
"TypeScript": "application/typescript", "Tcl": "text/x-tcl",
"Unified Parallel C": "text/x-csrc", "Tcsh": "text/x-sh",
"Unity3D Asset": "text/x-yaml", "TeX": "text/x-stex",
"Uno": "text/x-csharp", "Terra": "text/x-lua",
"UnrealScript": "text/x-java", "Textile": "text/x-textile",
"VHDL": "text/x-vhdl", "Turtle": "text/turtle",
"Verilog": "text/x-verilog", "Twig": "text/x-twig",
"Visual Basic": "text/x-vb", "TypeScript": "application/typescript",
"Volt": "text/x-d", "Unified Parallel C": "text/x-csrc",
"WebAssembly": "text/x-common-lisp", "Unity3D Asset": "text/x-yaml",
"WebIDL": "text/x-webidl", "Uno": "text/x-csharp",
"XC": "text/x-csrc", "UnrealScript": "text/x-java",
"XML": "text/xml", "VHDL": "text/x-vhdl",
"XPages": "text/xml", "Verilog": "text/x-verilog",
"XProc": "text/xml", "Visual Basic": "text/x-vb",
"XQuery": "application/xquery", "Volt": "text/x-d",
"XS": "text/x-csrc", "WebAssembly": "text/x-common-lisp",
"XSLT": "text/xml", "WebIDL": "text/x-webidl",
"YAML": "text/x-yaml", "Windows Registry Entries": "text/x-properties",
"edn": "text/x-clojure", "X BitMap": "text/x-csrc",
"reStructuredText": "text/x-rst", "X PixMap": "text/x-csrc",
"wisp": "text/x-clojure", "XC": "text/x-csrc",
"XML": "text/xml",
"XPages": "text/xml",
"XProc": "text/xml",
"XQuery": "application/xquery",
"XS": "text/x-csrc",
"XSLT": "text/xml",
"YAML": "text/x-yaml",
"edn": "text/x-clojure",
"reStructuredText": "text/x-rst",
"wisp": "text/x-clojure",
} }

660
internal/code-generator/generator/test_files/type.gold
View File

@ -1,295 +1,322 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
var LanguagesType = map[string]int{ var LanguagesType = map[string]int{
"1C Enterprise": 2, "1C Enterprise": 2,
"ABAP": 2, "ABAP": 2,
"ABNF": 1, "ABNF": 1,
"AGS Script": 2, "AGS Script": 2,
"AMPL": 2, "AMPL": 2,
"ANTLR": 2, "ANTLR": 2,
"API Blueprint": 3, "API Blueprint": 3,
"APL": 2, "APL": 2,
"ASN.1": 1, "ASN.1": 1,
"ASP": 2, "ASP": 2,
"ATS": 2, "ATS": 2,
"ActionScript": 2, "ActionScript": 2,
"Ada": 2, "Ada": 2,
"Agda": 2, "Adobe Font Metrics": 1,
"Alloy": 2, "Agda": 2,
"Alpine Abuild": 2, "Alloy": 2,
"Ant Build System": 1, "Alpine Abuild": 2,
"ApacheConf": 3, "AngelScript": 2,
"Apex": 2, "Ant Build System": 1,
"Apollo Guidance Computer": 2, "ApacheConf": 1,
"AppleScript": 2, "Apex": 2,
"Arc": 2, "Apollo Guidance Computer": 2,
"Arduino": 2, "AppleScript": 2,
"AsciiDoc": 4, "Arc": 2,
"AspectJ": 2, "AsciiDoc": 4,
"Assembly": 2, "AspectJ": 2,
"Augeas": 2, "Assembly": 2,
"AutoHotkey": 2, "Asymptote": 2,
"AutoIt": 2, "Augeas": 2,
"Awk": 2, "AutoHotkey": 2,
"Batchfile": 2, "AutoIt": 2,
"Befunge": 2, "Awk": 2,
"Bison": 2, "Ballerina": 2,
"BitBake": 2, "Batchfile": 2,
"Blade": 3, "Befunge": 2,
"BlitzBasic": 2, "Bison": 2,
"BlitzMax": 2, "BitBake": 2,
"Bluespec": 2, "Blade": 3,
"Boo": 2, "BlitzBasic": 2,
"Brainfuck": 2, "BlitzMax": 2,
"Brightscript": 2, "Bluespec": 2,
"Bro": 2, "Boo": 2,
"C": 2, "Brainfuck": 2,
"C#": 2, "Brightscript": 2,
"C++": 2, "Bro": 2,
"C-ObjDump": 1, "C": 2,
"C2hs Haskell": 2, "C#": 2,
"CLIPS": 2, "C++": 2,
"CMake": 2, "C-ObjDump": 1,
"COBOL": 2, "C2hs Haskell": 2,
"COLLADA": 1, "CLIPS": 2,
"CSON": 1, "CMake": 2,
"CSS": 3, "COBOL": 2,
"CSV": 1, "COLLADA": 1,
"CWeb": 2, "CSON": 1,
"Cap'n Proto": 2, "CSS": 3,
"CartoCSS": 2, "CSV": 1,
"Ceylon": 2, "CWeb": 2,
"Chapel": 2, "Cap'n Proto": 2,
"Charity": 2, "CartoCSS": 2,
"ChucK": 2, "Ceylon": 2,
"Cirru": 2, "Chapel": 2,
"Clarion": 2, "Charity": 2,
"Clean": 2, "ChucK": 2,
"Click": 2, "Cirru": 2,
"Clojure": 2, "Clarion": 2,
"Closure Templates": 3, "Clean": 2,
"CoffeeScript": 2, "Click": 2,
"ColdFusion": 2, "Clojure": 2,
"ColdFusion CFC": 2, "Closure Templates": 3,
"Common Lisp": 2, "Cloud Firestore Security Rules": 1,
"Component Pascal": 2, "CoNLL-U": 1,
"Cool": 2, "CoffeeScript": 2,
"Coq": 2, "ColdFusion": 2,
"Cpp-ObjDump": 1, "ColdFusion CFC": 2,
"Creole": 4, "Common Lisp": 2,
"Crystal": 2, "Common Workflow Language": 2,
"Csound": 2, "Component Pascal": 2,
"Csound Document": 2, "Cool": 2,
"Csound Score": 2, "Coq": 2,
"Cuda": 2, "Cpp-ObjDump": 1,
"Cycript": 2, "Creole": 4,
"Cython": 2, "Crystal": 2,
"D": 2, "Csound": 2,
"D-ObjDump": 1, "Csound Document": 2,
"DIGITAL Command Language": 2, "Csound Score": 2,
"DM": 2, "Cuda": 2,
"DNS Zone": 1, "Cycript": 2,
"DTrace": 2, "Cython": 2,
"Darcs Patch": 1, "D": 2,
"Dart": 2, "D-ObjDump": 1,
"Diff": 1, "DIGITAL Command Language": 2,
"Dockerfile": 1, "DM": 2,
"Dogescript": 2, "DNS Zone": 1,
"Dylan": 2, "DTrace": 2,
"E": 2, "Darcs Patch": 1,
"EBNF": 1, "Dart": 2,
"ECL": 2, "DataWeave": 2,
"ECLiPSe": 2, "Diff": 1,
"EJS": 3, "Dockerfile": 2,
"EQ": 2, "Dogescript": 2,
"Eagle": 3, "Dylan": 2,
"Easybuild": 1, "E": 2,
"Ecere Projects": 1, "EBNF": 1,
"Eiffel": 2, "ECL": 2,
"Elixir": 2, "ECLiPSe": 2,
"Elm": 2, "EJS": 3,
"Emacs Lisp": 2, "EML": 1,
"EmberScript": 2, "EQ": 2,
"Erlang": 2, "Eagle": 1,
"F#": 2, "Easybuild": 1,
"FLUX": 2, "Ecere Projects": 1,
"Factor": 2, "Edje Data Collection": 1,
"Fancy": 2, "Eiffel": 2,
"Fantom": 2, "Elixir": 2,
"Filebench WML": 2, "Elm": 2,
"Filterscript": 2, "Emacs Lisp": 2,
"Formatted": 1, "EmberScript": 2,
"Forth": 2, "Erlang": 2,
"Fortran": 2, "F#": 2,
"FreeMarker": 2, "F*": 2,
"Frege": 2, "FIGlet Font": 1,
"G-code": 1, "FLUX": 2,
"GAMS": 2, "Factor": 2,
"GAP": 2, "Fancy": 2,
"Fantom": 2,
"Filebench WML": 2,
"Filterscript": 2,
"Formatted": 1,
"Forth": 2,
"Fortran": 2,
"FreeMarker": 2,
"Frege": 2,
"G-code": 1,
"GAMS": 2,
"GAP": 2,
"GCC Machine Description": 2, "GCC Machine Description": 2,
"GDB": 2, "GDB": 2,
"GDScript": 2, "GDScript": 2,
"GLSL": 2, "GLSL": 2,
"GN": 1, "GN": 1,
"Game Maker Language": 2, "Game Maker Language": 2,
"Genie": 2, "Genie": 2,
"Genshi": 2, "Genshi": 2,
"Gentoo Ebuild": 2, "Gentoo Ebuild": 2,
"Gentoo Eclass": 2, "Gentoo Eclass": 2,
"Gettext Catalog": 4, "Gerber Image": 1,
"Gherkin": 2, "Gettext Catalog": 4,
"Glyph": 2, "Gherkin": 2,
"Gnuplot": 2, "Git Attributes": 1,
"Go": 2, "Git Config": 1,
"Golo": 2, "Glyph": 2,
"Gosu": 2, "Glyph Bitmap Distribution Format": 1,
"Grace": 2, "Gnuplot": 2,
"Gradle": 1, "Go": 2,
"Grammatical Framework": 2, "Golo": 2,
"Graph Modeling Language": 1, "Gosu": 2,
"GraphQL": 1, "Grace": 2,
"Graphviz (DOT)": 1, "Gradle": 1,
"Groovy": 2, "Grammatical Framework": 2,
"Groovy Server Pages": 2, "Graph Modeling Language": 1,
"HCL": 2, "GraphQL": 1,
"HLSL": 2, "Graphviz (DOT)": 1,
"HTML": 3, "Groovy": 2,
"HTML+Django": 3, "Groovy Server Pages": 2,
"HTML+ECR": 3, "HAProxy": 1,
"HTML+EEX": 3, "HCL": 2,
"HTML+ERB": 3, "HLSL": 2,
"HTML+PHP": 3, "HTML": 3,
"HTTP": 1, "HTML+Django": 3,
"Hack": 2, "HTML+ECR": 3,
"Haml": 3, "HTML+EEX": 3,
"Handlebars": 3, "HTML+ERB": 3,
"Harbour": 2, "HTML+PHP": 3,
"Haskell": 2, "HTML+Razor": 3,
"Haxe": 2, "HTTP": 1,
"Hy": 2, "HXML": 1,
"HyPhy": 2, "Hack": 2,
"IDL": 2, "Haml": 3,
"IGOR Pro": 2, "Handlebars": 3,
"INI": 1, "Harbour": 2,
"IRC log": 1, "Haskell": 2,
"Idris": 2, "Haxe": 2,
"Inform 7": 2, "HiveQL": 2,
"Inno Setup": 2, "Hy": 2,
"Io": 2, "HyPhy": 2,
"Ioke": 2, "IDL": 2,
"Isabelle": 2, "IGOR Pro": 2,
"Isabelle ROOT": 2, "INI": 1,
"J": 2, "IRC log": 1,
"JFlex": 2, "Idris": 2,
"JSON": 1, "Ignore List": 1,
"JSON5": 1, "Inform 7": 2,
"JSONLD": 1, "Inno Setup": 2,
"JSONiq": 2, "Io": 2,
"JSX": 2, "Ioke": 2,
"Jasmin": 2, "Isabelle": 2,
"Java": 2, "Isabelle ROOT": 2,
"Java Server Pages": 2, "J": 2,
"JavaScript": 2, "JFlex": 2,
"Jison": 2, "JSON": 1,
"Jison Lex": 2, "JSON with Comments": 1,
"Jolie": 2, "JSON5": 1,
"Julia": 2, "JSONLD": 1,
"Jupyter Notebook": 3, "JSONiq": 2,
"KRL": 2, "JSX": 2,
"KiCad": 2, "Jasmin": 2,
"Kit": 3, "Java": 2,
"Kotlin": 2, "Java Properties": 1,
"LFE": 2, "Java Server Pages": 2,
"LLVM": 2, "JavaScript": 2,
"LOLCODE": 2, "Jison": 2,
"LSL": 2, "Jison Lex": 2,
"LabVIEW": 2, "Jolie": 2,
"Lasso": 2, "Julia": 2,
"Latte": 3, "Jupyter Notebook": 3,
"Lean": 2, "KRL": 2,
"Less": 3, "KiCad Layout": 1,
"Lex": 2, "KiCad Legacy Layout": 1,
"LilyPond": 2, "KiCad Schematic": 1,
"Limbo": 2, "Kit": 3,
"Linker Script": 1, "Kotlin": 2,
"Linux Kernel Module": 1, "LFE": 2,
"Liquid": 3, "LLVM": 2,
"Literate Agda": 2, "LOLCODE": 2,
"Literate CoffeeScript": 2, "LSL": 2,
"Literate Haskell": 2, "LTspice Symbol": 1,
"LiveScript": 2, "LabVIEW": 2,
"Logos": 2, "Lasso": 2,
"Logtalk": 2, "Latte": 3,
"LookML": 2, "Lean": 2,
"LoomScript": 2, "Less": 3,
"Lua": 2, "Lex": 2,
"M": 2, "LilyPond": 2,
"M4": 2, "Limbo": 2,
"M4Sugar": 2, "Linker Script": 1,
"MAXScript": 2, "Linux Kernel Module": 1,
"MQL4": 2, "Liquid": 3,
"MQL5": 2, "Literate Agda": 2,
"MTML": 3, "Literate CoffeeScript": 2,
"MUF": 2, "Literate Haskell": 2,
"Makefile": 2, "LiveScript": 2,
"Mako": 2, "Logos": 2,
"Markdown": 4, "Logtalk": 2,
"Marko": 3, "LookML": 2,
"Mask": 3, "LoomScript": 2,
"Mathematica": 2, "Lua": 2,
"Matlab": 2, "M": 2,
"Maven POM": 1, "M4": 2,
"Max": 2, "M4Sugar": 2,
"MediaWiki": 4, "MATLAB": 2,
"Mercury": 2, "MAXScript": 2,
"Meson": 2, "MQL4": 2,
"Metal": 2, "MQL5": 2,
"MiniD": 2, "MTML": 3,
"Mirah": 2, "MUF": 2,
"Modelica": 2, "Makefile": 2,
"Modula-2": 2, "Mako": 2,
"Module Management System": 2, "Markdown": 4,
"Monkey": 2, "Marko": 3,
"Moocode": 2, "Mask": 3,
"MoonScript": 2, "Mathematica": 2,
"Myghty": 2, "Maven POM": 1,
"NCL": 2, "Max": 2,
"NL": 1, "MediaWiki": 4,
"NSIS": 2, "Mercury": 2,
"Nemerle": 2, "Meson": 2,
"NetLinx": 2, "Metal": 2,
"NetLinx+ERB": 2, "MiniD": 2,
"NetLogo": 2, "Mirah": 2,
"NewLisp": 2, "Modelica": 2,
"Nginx": 3, "Modula-2": 2,
"Nim": 2, "Modula-3": 2,
"Ninja": 1, "Module Management System": 2,
"Nit": 2, "Monkey": 2,
"Nix": 2, "Moocode": 2,
"Nu": 2, "MoonScript": 2,
"NumPy": 2, "Myghty": 2,
"OCaml": 2, "NCL": 2,
"ObjDump": 1, "NL": 1,
"Objective-C": 2, "NSIS": 2,
"Objective-C++": 2, "Nearley": 2,
"Objective-J": 2, "Nemerle": 2,
"Omgrofl": 2, "NetLinx": 2,
"Opa": 2, "NetLinx+ERB": 2,
"Opal": 2, "NetLogo": 2,
"OpenCL": 2, "NewLisp": 2,
"OpenEdge ABL": 2, "Nextflow": 2,
"OpenRC runscript": 2, "Nginx": 1,
"OpenSCAD": 2, "Nim": 2,
"OpenType Feature File": 1, "Ninja": 1,
"Nit": 2,
"Nix": 2,
"Nu": 2,
"NumPy": 2,
"OCaml": 2,
"ObjDump": 1,
"Objective-C": 2,
"Objective-C++": 2,
"Objective-J": 2,
"Omgrofl": 2,
"Opa": 2,
"Opal": 2,
"OpenCL": 2,
"OpenEdge ABL": 2,
"OpenRC runscript": 2,
"OpenSCAD": 2,
"OpenType Feature File": 1,
"Org": 4, "Org": 4,
"Ox": 2, "Ox": 2,
"Oxygene": 2, "Oxygene": 2,
"Oz": 2, "Oz": 2,
"P4": 2, "P4": 2,
"PAWN": 2,
"PHP": 2, "PHP": 2,
"PLSQL": 2, "PLSQL": 2,
"PLpgSQL": 2, "PLpgSQL": 2,
@ -300,6 +327,7 @@ var LanguagesType = map[string]int{
"Parrot Assembly": 2, "Parrot Assembly": 2,
"Parrot Internal Representation": 2, "Parrot Internal Representation": 2,
"Pascal": 2, "Pascal": 2,
"Pawn": 2,
"Pep8": 2, "Pep8": 2,
"Perl": 2, "Perl": 2,
"Perl 6": 2, "Perl 6": 2,
@ -309,19 +337,21 @@ var LanguagesType = map[string]int{
"PigLatin": 2, "PigLatin": 2,
"Pike": 2, "Pike": 2,
"Pod": 4, "Pod": 4,
"Pod 6": 4,
"PogoScript": 2, "PogoScript": 2,
"Pony": 2, "Pony": 2,
"PostCSS": 3,
"PostScript": 3, "PostScript": 3,
"PowerBuilder": 2, "PowerBuilder": 2,
"PowerShell": 2, "PowerShell": 2,
"Processing": 2, "Processing": 2,
"Prolog": 2, "Prolog": 2,
"Propeller Spin": 2, "Propeller Spin": 2,
"Protocol Buffer": 3, "Protocol Buffer": 1,
"Public Key": 1, "Public Key": 1,
"Pug": 3, "Pug": 3,
"Puppet": 2, "Puppet": 2,
"Pure Data": 2, "Pure Data": 1,
"PureBasic": 2, "PureBasic": 2,
"PureScript": 2, "PureScript": 2,
"Python": 2, "Python": 2,
@ -329,6 +359,7 @@ var LanguagesType = map[string]int{
"Python traceback": 1, "Python traceback": 1,
"QML": 2, "QML": 2,
"QMake": 2, "QMake": 2,
"Quake": 2,
"R": 2, "R": 2,
"RAML": 3, "RAML": 3,
"RDoc": 4, "RDoc": 4,
@ -336,6 +367,7 @@ var LanguagesType = map[string]int{
"REXX": 2, "REXX": 2,
"RHTML": 3, "RHTML": 3,
"RMarkdown": 4, "RMarkdown": 4,
"RPC": 2,
"RPM Spec": 1, "RPM Spec": 1,
"RUNOFF": 3, "RUNOFF": 3,
"Racket": 2, "Racket": 2,
@ -352,6 +384,7 @@ var LanguagesType = map[string]int{
"Ring": 2, "Ring": 2,
"RobotFramework": 2, "RobotFramework": 2,
"Roff": 3, "Roff": 3,
"Roff Manpage": 3,
"Rouge": 2, "Rouge": 2,
"Ruby": 2, "Ruby": 2,
"Rust": 2, "Rust": 2,
@ -378,10 +411,12 @@ var LanguagesType = map[string]int{
"ShellSession": 2, "ShellSession": 2,
"Shen": 2, "Shen": 2,
"Slash": 2, "Slash": 2,
"Slice": 2,
"Slim": 3, "Slim": 3,
"Smali": 2, "Smali": 2,
"Smalltalk": 2, "Smalltalk": 2,
"Smarty": 2, "Smarty": 2,
"Solidity": 2,
"SourcePawn": 2, "SourcePawn": 2,
"Spline Font Database": 1, "Spline Font Database": 1,
"Squirrel": 2, "Squirrel": 2,
@ -390,7 +425,7 @@ var LanguagesType = map[string]int{
"Stata": 2, "Stata": 2,
"Stylus": 3, "Stylus": 3,
"SubRip Text": 1, "SubRip Text": 1,
"Sublime Text Config": 1, "SugarSS": 3,
"SuperCollider": 2, "SuperCollider": 2,
"Swift": 2, "Swift": 2,
"SystemVerilog": 2, "SystemVerilog": 2,
@ -427,34 +462,45 @@ var LanguagesType = map[string]int{
"Vue": 3, "Vue": 3,
"Wavefront Material": 1, "Wavefront Material": 1,
"Wavefront Object": 1, "Wavefront Object": 1,
"Web Ontology Language": 3, "Web Ontology Language": 1,
"WebAssembly": 2, "WebAssembly": 2,
"WebIDL": 2, "WebIDL": 2,
"Windows Registry Entries": 1,
"World of Warcraft Addon Data": 1, "World of Warcraft Addon Data": 1,
"X10": 2, "X BitMap": 1,
"XC": 2, "X Font Directory Index": 1,
"XCompose": 1, "X PixMap": 1,
"XML": 1, "X10": 2,
"XPages": 2, "XC": 2,
"XProc": 2, "XCompose": 1,
"XQuery": 2, "XML": 1,
"XS": 2, "XPages": 1,
"XSLT": 2, "XProc": 2,
"Xojo": 2, "XQuery": 2,
"Xtend": 2, "XS": 2,
"YAML": 1, "XSLT": 2,
"YANG": 1, "Xojo": 2,
"Yacc": 2, "Xtend": 2,
"Zephir": 2, "YAML": 1,
"Zimpl": 2, "YANG": 1,
"desktop": 1, "YARA": 2,
"eC": 2, "YASnippet": 3,
"edn": 1, "Yacc": 2,
"fish": 2, "Zephir": 2,
"mupad": 2, "Zig": 2,
"nesC": 2, "Zimpl": 2,
"ooc": 2, "desktop": 1,
"reStructuredText": 4, "eC": 2,
"wisp": 2, "edn": 1,
"xBase": 2, "fish": 2,
"mupad": 2,
"nanorc": 1,
"nesC": 2,
"ooc": 2,
"q": 2,
"reStructuredText": 4,
"sed": 2,
"wdl": 2,
"wisp": 2,
"xBase": 2,
} }

20
internal/code-generator/generator/test_files/vendor.gold
View File

@ -1,5 +1,5 @@
// Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT. // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68 // Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
package data package data
@ -10,7 +10,6 @@ var VendorMatchers = substring.Or(
substring.Regexp(`^[Dd]ependencies/`), substring.Regexp(`^[Dd]ependencies/`),
substring.Regexp(`(^|/)dist/`), substring.Regexp(`(^|/)dist/`),
substring.Regexp(`^deps/`), substring.Regexp(`^deps/`),
substring.Regexp(`^tools/`),
substring.Regexp(`(^|/)configure$`), substring.Regexp(`(^|/)configure$`),
substring.Regexp(`(^|/)config.guess$`), substring.Regexp(`(^|/)config.guess$`),
substring.Regexp(`(^|/)config.sub$`), substring.Regexp(`(^|/)config.sub$`),
@ -32,13 +31,15 @@ var VendorMatchers = substring.Or(
substring.Regexp(`(^|/)bootstrap([^.]*)\.(js|css|less|scss|styl)$`), substring.Regexp(`(^|/)bootstrap([^.]*)\.(js|css|less|scss|styl)$`),
substring.Regexp(`(^|/)custom\.bootstrap([^\s]*)(js|css|less|scss|styl)$`), substring.Regexp(`(^|/)custom\.bootstrap([^\s]*)(js|css|less|scss|styl)$`),
substring.Regexp(`(^|/)font-awesome\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)font-awesome\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)font-awesome/.*\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)foundation\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)foundation\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)normalize\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)normalize\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)skeleton\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)skeleton\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)[Bb]ourbon/.*\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)[Bb]ourbon/.*\.(css|less|scss|styl)$`),
substring.Regexp(`(^|/)animate\.(css|less|scss|styl)$`), substring.Regexp(`(^|/)animate\.(css|less|scss|styl)$`),
substring.Regexp(`third[-_]?party/`), substring.Regexp(`(^|/)materialize\.(css|less|scss|styl|js)$`),
substring.Regexp(`3rd[-_]?party/`), substring.Regexp(`(^|/)select2/.*\.(css|scss|js)$`),
substring.Regexp(`(3rd|[Tt]hird)[-_]?[Pp]arty/`),
substring.Regexp(`vendors?/`), substring.Regexp(`vendors?/`),
substring.Regexp(`extern(al)?/`), substring.Regexp(`extern(al)?/`),
substring.Regexp(`(^|/)[Vv]+endor/`), substring.Regexp(`(^|/)[Vv]+endor/`),
@ -53,6 +54,9 @@ var VendorMatchers = substring.Or(
substring.Regexp(`jquery.fancybox.(js|css)`), substring.Regexp(`jquery.fancybox.(js|css)`),
substring.Regexp(`fuelux.js`), substring.Regexp(`fuelux.js`),
substring.Regexp(`(^|/)jquery\.fileupload(-\w+)?\.js$`), substring.Regexp(`(^|/)jquery\.fileupload(-\w+)?\.js$`),
substring.Regexp(`jquery.dataTables.js`),
substring.Regexp(`bootbox.js`),
substring.Regexp(`pdf.worker.js`),
substring.Regexp(`(^|/)slick\.\w+.js$`), substring.Regexp(`(^|/)slick\.\w+.js$`),
substring.Regexp(`(^|/)Leaflet\.Coordinates-\d+\.\d+\.\d+\.src\.js$`), substring.Regexp(`(^|/)Leaflet\.Coordinates-\d+\.\d+\.\d+\.src\.js$`),
substring.Regexp(`leaflet.draw-src.js`), substring.Regexp(`leaflet.draw-src.js`),
@ -63,6 +67,7 @@ var VendorMatchers = substring.Or(
substring.Regexp(`wicket-leaflet.js`), substring.Regexp(`wicket-leaflet.js`),
substring.Regexp(`.sublime-project`), substring.Regexp(`.sublime-project`),
substring.Regexp(`.sublime-workspace`), substring.Regexp(`.sublime-workspace`),
substring.Regexp(`.vscode`),
substring.Regexp(`(^|/)prototype(.*)\.js$`), substring.Regexp(`(^|/)prototype(.*)\.js$`),
substring.Regexp(`(^|/)effects\.js$`), substring.Regexp(`(^|/)effects\.js$`),
substring.Regexp(`(^|/)controls\.js$`), substring.Regexp(`(^|/)controls\.js$`),
@ -87,6 +92,7 @@ var VendorMatchers = substring.Or(
substring.Regexp(`(^|/)angular([^.]*)\.js$`), substring.Regexp(`(^|/)angular([^.]*)\.js$`),
substring.Regexp(`(^|\/)d3(\.v\d+)?([^.]*)\.js$`), substring.Regexp(`(^|\/)d3(\.v\d+)?([^.]*)\.js$`),
substring.Regexp(`(^|/)react(-[^.]*)?\.js$`), substring.Regexp(`(^|/)react(-[^.]*)?\.js$`),
substring.Regexp(`(^|/)flow-typed/.*\.js$`),
substring.Regexp(`(^|/)modernizr\-\d\.\d+(\.\d+)?\.js$`), substring.Regexp(`(^|/)modernizr\-\d\.\d+(\.\d+)?\.js$`),
substring.Regexp(`(^|/)modernizr\.custom\.\d+\.js$`), substring.Regexp(`(^|/)modernizr\.custom\.\d+\.js$`),
substring.Regexp(`(^|/)knockout-(\d+\.){3}(debug\.)?js$`), substring.Regexp(`(^|/)knockout-(\d+\.){3}(debug\.)?js$`),
@ -98,8 +104,7 @@ var VendorMatchers = substring.Or(
substring.Regexp(`^.osx$`), substring.Regexp(`^.osx$`),
substring.Regexp(`\.xctemplate/`), substring.Regexp(`\.xctemplate/`),
substring.Regexp(`\.imageset/`), substring.Regexp(`\.imageset/`),
substring.Regexp(`^Carthage/`), substring.Regexp(`(^|/)Carthage/`),
substring.Regexp(`^Pods/`),
substring.Regexp(`(^|/)Sparkle/`), substring.Regexp(`(^|/)Sparkle/`),
substring.Regexp(`Crashlytics.framework/`), substring.Regexp(`Crashlytics.framework/`),
substring.Regexp(`Fabric.framework/`), substring.Regexp(`Fabric.framework/`),
@ -112,6 +117,9 @@ var VendorMatchers = substring.Or(
substring.Regexp(`(^|/)gradlew$`), substring.Regexp(`(^|/)gradlew$`),
substring.Regexp(`(^|/)gradlew\.bat$`), substring.Regexp(`(^|/)gradlew\.bat$`),
substring.Regexp(`(^|/)gradle/wrapper/`), substring.Regexp(`(^|/)gradle/wrapper/`),
substring.Regexp(`(^|/)mvnw$`),
substring.Regexp(`(^|/)mvnw\.cmd$`),
substring.Regexp(`(^|/)\.mvn/wrapper/`),
substring.Regexp(`-vsdoc\.js$`), substring.Regexp(`-vsdoc\.js$`),
substring.Regexp(`\.intellisense\.js$`), substring.Regexp(`\.intellisense\.js$`),
substring.Regexp(`(^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$`), substring.Regexp(`(^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$`),

6
internal/code-generator/generator/vendor.go
View File

@ -2,12 +2,14 @@ package generator
import ( import (
"bytes" "bytes"
"gopkg.in/yaml.v2"
"io" "io"
"io/ioutil" "io/ioutil"
"gopkg.in/yaml.v2"
) )
// Vendor reads from fileToParse and builds source file from tmplPath. It complies with type File signature. // Vendor generates regex matchers in Go for vendoring files/dirs.
// It is of generator.File type.
func Vendor(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error { func Vendor(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error {
data, err := ioutil.ReadFile(fileToParse) data, err := ioutil.ReadFile(fileToParse)
if err != nil { if err != nil {

4
internal/code-generator/main.go
View File

@ -20,7 +20,7 @@ const (
extensionsTmpl = "extension.go.tmpl" extensionsTmpl = "extension.go.tmpl"
// content.go generation // content.go generation
heuristicsRuby = ".linguist/lib/linguist/heuristics.rb" heuristicsYAML = ".linguist/lib/linguist/heuristics.yml"
contentFile = "data/content.go" contentFile = "data/content.go"
contentTmplPath = "internal/code-generator/assets/content.go.tmpl" contentTmplPath = "internal/code-generator/assets/content.go.tmpl"
contentTmpl = "content.go.tmpl" contentTmpl = "content.go.tmpl"
@ -93,7 +93,7 @@ func main() {
fileList := []*generatorFiles{ fileList := []*generatorFiles{
{generator.Extensions, languagesYAML, "", extensionsFile, extensionsTmplPath, extensionsTmpl, commit}, {generator.Extensions, languagesYAML, "", extensionsFile, extensionsTmplPath, extensionsTmpl, commit},
{generator.Heuristics, heuristicsRuby, "", contentFile, contentTmplPath, contentTmpl, commit}, {generator.GenHeuristics, heuristicsYAML, "", contentFile, contentTmplPath, contentTmpl, commit},
{generator.Vendor, vendorYAML, "", vendorFile, vendorTmplPath, vendorTmpl, commit}, {generator.Vendor, vendorYAML, "", vendorFile, vendorTmplPath, vendorTmpl, commit},
{generator.Documentation, documentationYAML, "", documentationFile, documentationTmplPath, documentationTmpl, commit}, {generator.Documentation, documentationYAML, "", documentationFile, documentationTmplPath, documentationTmpl, commit},
{generator.Types, languagesYAML, "", typeFile, typeTmplPath, typeTmpl, commit}, {generator.Types, languagesYAML, "", typeFile, typeTmplPath, typeTmpl, commit},