Sync to linguist 7.2.0: heuristics.yml support (#189)

Sync \w Github Linguist v7.2.0

Includes new way of handling `heuristics.yml` and
all `./data/*` re-generated using Github Linguist [v7.2.0](https://github.com/github/linguist/releases/tag/v7.2.0)
release tag.

 - many new languages
 - better vendoring detection
 - update doc on update&known issues.

CONTRIBUTING.md

@@ -0,0 +1,61 @@
+# source{d} Contributing Guidelines
+
+source{d} projects accept contributions via GitHub pull requests.
+This document outlines some of the
+conventions on development workflow, commit message formatting, contact points,
+and other resources to make it easier to get your contribution accepted.
+
+## Certificate of Origin
+
+By contributing to this project, you agree to the [Developer Certificate of
+Origin (DCO)](DCO). This document was created by the Linux Kernel community and is a
+simple statement that you, as a contributor, have the legal right to make the
+contribution.
+
+In order to show your agreement with the DCO you should include at the end of the commit message,
+the following line: `Signed-off-by: John Doe <john.doe@example.com>`, using your real name.
+
+This can be done easily using the [`-s`](https://github.com/git/git/blob/b2c150d3aa82f6583b9aadfecc5f8fa1c74aca09/Documentation/git-commit.txt#L154-L161) flag on the `git commit`.
+
+If you find yourself pushed a few commits without `Signed-off-by`, you can still add it afterwards. We wrote a manual which can help: [fix-DCO.md](https://github.com/src-d/guide/blob/master/developer-community/fix-DCO.md).
+
+## Support Channels
+
+The official support channels, for both users and contributors, are:
+
+- GitHub issues: each repository has its own list of issues.
+- Slack: join the [source{d} Slack](https://join.slack.com/t/sourced-community/shared_invite/enQtMjc4Njk5MzEyNzM2LTFjNzY4NjEwZGEwMzRiNTM4MzRlMzQ4MmIzZjkwZmZlM2NjODUxZmJjNDI1OTcxNDAyMmZlNmFjODZlNTg0YWM) community.
+
+*Before opening a new issue or submitting a new pull request, it's helpful to
+search the project - it's likely that another user has already reported the
+issue you're facing, or it's a known issue that we're already aware of.
+
+
+## How to Contribute
+
+Pull Requests (PRs) are the main and exclusive way to contribute code to source{d} projects.
+In order for a PR to be accepted it needs to pass this list of requirements:
+
+- The contribution must be correctly explained with natural language and providing a minimum working example that reproduces it.
+- All PRs must be written idiomaticly:
+    - for Go: formatted according to [gofmt](https://golang.org/cmd/gofmt/), and without any warnings from [go lint](https://github.com/golang/lint) nor [go vet](https://golang.org/cmd/vet/)
+    - for other languages, similar constraints apply.
+- They should in general include tests, and those shall pass.
+    - If the PR is a bug fix, it has to include a new unit test that fails before the patch is merged.
+    - If the PR is a new feature, it has to come with a suite of unit tests, that tests the new functionality.
+    - In any case, all the PRs have to pass the personal evaluation of at least one of the [maintainers](MAINTAINERS) of the project.
+
+
+### Format of the commit message
+
+Every commit message should describe what was changed, under which context and, if applicable, the GitHub issue it relates to:
+
+```
+plumbing: packp, Skip argument validations for unknown capabilities. Fixes #623
+```
+
+The format can be described more formally as follows:
+
+```
+<package>: <subpackage>, <what changed>. [Fixes #<issue-number>]
+```

Makefile

@@ -46,6 +46,11 @@ clean: clean-linguist clean-shared
 code-generate: $(LINGUIST_PATH)
 	mkdir -p data && \
 	go run internal/code-generator/main.go
+	ENRY_TEST_REPO="$${PWD}/.linguist" go test  -v \
+		-run Test_GeneratorTestSuite \
+		./internal/code-generator/generator \
+		-testify.m TestUpdateGeneratorTestSuiteGold \
+		-update_gold
 
 benchmarks: $(LINGUIST_PATH)
 	go test -run=NONE -bench=. && \

README.md

@@ -154,14 +154,17 @@ Generated Java bindings using a C-shared library and JNI are located under [`jav
 Development
 ------------
 
-*enry* re-uses parts of original [linguist](https://github.com/github/linguist) to generate internal data structures. In order to update to the latest upstream and generate the necessary code you must run:
+*enry* re-uses parts of original [linguist](https://github.com/github/linguist) to generate internal data structures. In order to update to the latest upstream and generate all the necessary code you must run:
 
+    git clone https://github.com/github/linguist.git .linguist
+    # update commit in generator_test.go (to re-generate .gold fixtures)
+    # https://github.com/src-d/enry/blob/13d3d66d37a87f23a013246a1b0678c9ee3d524b/internal/code-generator/generator/generator_test.go#L18
     go generate
 
 We update enry when changes are done in linguist's master branch on the following files:
 
 * [languages.yml](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml)
-* [heuristics.rb](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb)
+* [heuristics.yml](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.yml)
 * [vendor.yml](https://github.com/github/linguist/blob/master/lib/linguist/vendor.yml)
 * [documentation.yml](https://github.com/github/linguist/blob/master/lib/linguist/documentation.yml)
 
@@ -183,17 +186,11 @@ Divergences from linguist
 Using [linguist/samples](https://github.com/github/linguist/tree/master/samples)
 as a set for the tests, the following issues were found:
 
-* With [hello.ms](https://github.com/github/linguist/blob/master/samples/Unix%20Assembly/hello.ms) we can't detect the language (Unix Assembly) because we don't have a matcher in contentMatchers (content.go) for Unix Assembly. Linguist uses this [regexp](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb#L300) in its code,
+* [Heuristics for ".es" extension](https://github.com/github/linguist/blob/e761f9b013e5b61161481fcb898b59721ee40e3d/lib/linguist/heuristics.yml#L103) in JavaScript could not be parsed, due to unsupported backreference in RE2 regexp engine
 
-    `elsif /(?<!\S)\.(include|globa?l)\s/.match(data) || /(?<!\/\*)(\A|\n)\s*\.[A-Za-z][_A-Za-z0-9]*:/.match(data.gsub(/"([^\\"]|\\.)*"|'([^\\']|\\.)*'|\\\s*(?:--.*)?\n/, ""))`
+* As of (Linguist v5.3.2)[https://github.com/github/linguist/releases/tag/v5.3.2] it is using [flex-based scanner in C for tokenization](https://github.com/github/linguist/pull/3846). Enry stil uses [extract_token](https://github.com/github/linguist/pull/3846/files#diff-d5179df0b71620e3fac4535cd1368d15L60) regex-based algorithm. Tracked under https://github.com/src-d/enry/issues/193
 
-    which we can't port.
+* Bayesian classifier cann't distinguish "SQL" vs "PLpgSQL". Tracked under https://github.com/src-d/enry/issues/194
-
-* All files for the SQL language fall to the classifier because we don't parse
-this [disambiguator
-expression](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb#L433)
-for `*.sql` files right. This expression doesn't comply with the pattern for the
-rest in [heuristics.rb](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.rb).
 
 `enry` [CLI tool](#cli) does not require a full Git repository to be present in filesystem in order to report languages.
 
@@ -232,7 +229,7 @@ As benchmarks depend on Ruby and Github-Linguist gem make sure you have:
 If you want to reproduce the same benchmarks as reported above:
  - Make sure all [dependencies](#benchmark-dependencies) are installed
  - Install [gnuplot](http://gnuplot.info) (in order to plot the histogram)
- - Run `ENRY_TEST_REPO=.linguist benchmarks/run.sh` (takes ~15h)
+ - Run `ENRY_TEST_REPO="$PWD/.linguist" benchmarks/run.sh` (takes ~15h)
 
 It will run the benchmarks for enry and linguist, parse the output, create csv files and plot the histogram. This takes some time.
 

benchmark_test.go

@@ -28,9 +28,6 @@ var (
 )
 
 func TestMain(m *testing.M) {
-	var exitCode int
-	defer os.Exit(exitCode)
-
 	flag.BoolVar(&slow, "slow", false, "run benchmarks per sample for strategies too")
 	flag.Parse()
 
@@ -47,7 +44,7 @@ func TestMain(m *testing.M) {
 		log.Fatal(err)
 	}
 
-	exitCode = m.Run()
+	os.Exit(m.Run())
 }
 
 func cloneLinguist(linguistURL string) error {

common.go

@@ -16,7 +16,7 @@ const OtherLanguage = ""
 // Strategy type fix the signature for the functions that can be used as a strategy.
 type Strategy func(filename string, content []byte, candidates []string) (languages []string)
 
-// DefaultStrategies is the strategies' sequence GetLanguage uses to detect languages.
+// DefaultStrategies is a sequence of strategies used by GetLanguage to detect languages.
 var DefaultStrategies = []Strategy{
 	GetLanguagesByModeline,
 	GetLanguagesByFilename,
@@ -397,12 +397,13 @@ func GetLanguagesByContent(filename string, content []byte, _ []string) []string
 	}
 
 	ext := strings.ToLower(filepath.Ext(filename))
-	fnMatcher, ok := data.ContentMatchers[ext]
+
+	heuristic, ok := data.ContentHeuristics[ext]
 	if !ok {
 		return nil
 	}
 
-	return fnMatcher(content)
+	return heuristic.Match(content)
 }
 
 // GetLanguagesByClassifier uses DefaultClassifier as a Classifier and returns a sorted slice of possible languages ordered by
@@ -455,9 +456,7 @@ func GetLanguageType(language string) (langType Type) {
 // GetLanguageByAlias returns either the language related to the given alias and ok set to true
 // or Otherlanguage and ok set to false if the alias is not recognized.
 func GetLanguageByAlias(alias string) (lang string, ok bool) {
-	a := strings.Split(alias, `,`)[0]
+	lang, ok = data.LanguageByAlias(alias)
-	a = strings.ToLower(a)
-	lang, ok = data.LanguagesByAlias[a]
 	if !ok {
 		lang = OtherLanguage
 	}

common_test.go

@@ -11,6 +11,7 @@ import (
 	"gopkg.in/src-d/enry.v1/data"
 
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 	"github.com/stretchr/testify/suite"
 )
 
@@ -19,9 +20,36 @@ const linguistClonedEnvVar = "ENRY_TEST_REPO"
 
 type EnryTestSuite struct {
 	suite.Suite
-	repoLinguist string
+	tmpLinguist string
+	needToClone bool
 	samplesDir  string
-	cloned       bool
+}
+
+func (s *EnryTestSuite) TestRegexpEdgeCases() {
+	var regexpEdgeCases = []struct {
+		lang     string
+		filename string
+	}{
+		{lang: "ActionScript", filename: "FooBar.as"},
+		{lang: "Forth", filename: "asm.fr"},
+		{lang: "X PixMap", filename: "cc-public_domain_mark_white.pm"},
+		//{lang: "SQL", filename: "drop_stuff.sql"}, // https://github.com/src-d/enry/issues/194
+		{lang: "Fstar", filename: "Hacl.Spec.Bignum.Fmul.fst"},
+		{lang: "C++", filename: "Types.h"},
+	}
+
+	for _, r := range regexpEdgeCases {
+		filename := fmt.Sprintf("%s/samples/%s/%s", s.tmpLinguist, r.lang, r.filename)
+
+		content, err := ioutil.ReadFile(filename)
+		require.NoError(s.T(), err)
+
+		lang := GetLanguage(r.filename, content)
+		s.T().Logf("File:%s, lang:%s", filename, lang)
+
+		expLang, _ := data.LanguageByAlias(r.lang)
+		require.EqualValues(s.T(), expLang, lang)
+	}
 }
 
 func Test_EnryTestSuite(t *testing.T) {
@@ -30,25 +58,24 @@ func Test_EnryTestSuite(t *testing.T) {
 
 func (s *EnryTestSuite) SetupSuite() {
 	var err error
-	s.repoLinguist = os.Getenv(linguistClonedEnvVar)
+	s.tmpLinguist = os.Getenv(linguistClonedEnvVar)
-	s.cloned = s.repoLinguist == ""
+	s.needToClone = s.tmpLinguist == ""
-	if s.cloned {
+	if s.needToClone {
-		s.repoLinguist, err = ioutil.TempDir("", "linguist-")
+		s.tmpLinguist, err = ioutil.TempDir("", "linguist-")
-		assert.NoError(s.T(), err)
+		require.NoError(s.T(), err)
-	}
+		s.T().Logf("Cloning Linguist repo to '%s' as %s was not set\n",
-
+			s.tmpLinguist, linguistClonedEnvVar)
-	s.samplesDir = filepath.Join(s.repoLinguist, "samples")
+		cmd := exec.Command("git", "clone", linguistURL, s.tmpLinguist)
-
-	if s.cloned {
-		cmd := exec.Command("git", "clone", linguistURL, s.repoLinguist)
 		err = cmd.Run()
-		assert.NoError(s.T(), err)
+		require.NoError(s.T(), err)
 	}
+	s.samplesDir = filepath.Join(s.tmpLinguist, "samples")
+	s.T().Logf("using samples from %s", s.samplesDir)
 
 	cwd, err := os.Getwd()
 	assert.NoError(s.T(), err)
 
-	err = os.Chdir(s.repoLinguist)
+	err = os.Chdir(s.tmpLinguist)
 	assert.NoError(s.T(), err)
 
 	cmd := exec.Command("git", "checkout", data.LinguistCommit)
@@ -60,8 +87,8 @@ func (s *EnryTestSuite) SetupSuite() {
 }
 
 func (s *EnryTestSuite) TearDownSuite() {
-	if s.cloned {
+	if s.needToClone {
-		err := os.RemoveAll(s.repoLinguist)
+		err := os.RemoveAll(s.tmpLinguist)
 		assert.NoError(s.T(), err)
 	}
 }
@@ -88,7 +115,7 @@ func (s *EnryTestSuite) TestGetLanguage() {
 }
 
 func (s *EnryTestSuite) TestGetLanguagesByModelineLinguist() {
-	var modelinesDir = filepath.Join(s.repoLinguist, "test/fixtures/Data/Modelines")
+	var modelinesDir = filepath.Join(s.tmpLinguist, "test/fixtures/Data/Modelines")
 
 	tests := []struct {
 		name       string
@@ -400,7 +427,8 @@ func (s *EnryTestSuite) TestGetLanguageByAlias() {
 func (s *EnryTestSuite) TestLinguistCorpus() {
 	const filenamesDir = "filenames"
 	var cornerCases = map[string]bool{
-		"hello.ms": true,
+		"drop_stuff.sql": true, // https://github.com/src-d/enry/issues/194
+		// .es and .ice fail heuristics parsing, but do not fail any tests
 	}
 
 	var total, failed, ok, other int
@@ -408,7 +436,7 @@ func (s *EnryTestSuite) TestLinguistCorpus() {
 	filepath.Walk(s.samplesDir, func(path string, f os.FileInfo, err error) error {
 		if f.IsDir() {
 			if f.Name() != filenamesDir {
-				expected = f.Name()
+				expected, _ = data.LanguageByAlias(f.Name())
 			}
 
 			return nil
@@ -431,11 +459,10 @@ func (s *EnryTestSuite) TestLinguistCorpus() {
 		} else {
 			status = "failed"
 			failed++
-
 		}
 
 		if _, ok := cornerCases[filename]; ok {
-			fmt.Printf("\t\t[considered corner case] %s\texpected: %s\tobtained: %s\tstatus: %s\n", filename, expected, obtained, status)
+			s.T().Logf("\t\t[considered corner case] %s\texpected: %s\tobtained: %s\tstatus: %s\n", filename, expected, obtained, status)
 		} else {
 			assert.Equal(s.T(), expected, obtained, fmt.Sprintf("%s\texpected: %s\tobtained: %s\tstatus: %s\n", filename, expected, obtained, status))
 		}
@@ -443,5 +470,5 @@ func (s *EnryTestSuite) TestLinguistCorpus() {
 		return nil
 	})
 
-	fmt.Printf("\t\ttotal files: %d, ok: %d, failed: %d, other: %d\n", total, ok, failed, other)
+	s.T().Logf("\t\ttotal files: %d, ok: %d, failed: %d, other: %d\n", total, ok, failed, other)
 }

data/alias.go

@@ -1,11 +1,13 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
-// LanguagesByAlias keeps alias for different languages and use the name of the languages as an alias too.
+import "strings"
+
+// LanguageByAliasMap keeps alias for different languages and use the name of the languages as an alias too.
 // All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores.
-var LanguagesByAlias = map[string]string{
+var LanguageByAliasMap = map[string]string{
 	"1c_enterprise":  "1C Enterprise",
 	"abap":           "ABAP",
 	"abl":            "OpenEdge ABL",
@@ -32,6 +34,7 @@ var LanguagesByAlias = map[string]string{
 	"alpine_abuild":    "Alpine Abuild",
 	"amfm":             "Adobe Font Metrics",
 	"ampl":             "AMPL",
+	"angelscript":      "AngelScript",
 	"ant_build_system": "Ant Build System",
 	"antlr":            "ANTLR",
 	"apache":           "ApacheConf",
@@ -43,16 +46,17 @@ var LanguagesByAlias = map[string]string{
 	"apollo_guidance_computer":       "Apollo Guidance Computer",
 	"applescript":                    "AppleScript",
 	"arc":                            "Arc",
-	"arduino":                  "Arduino",
 	"arexx":                          "REXX",
 	"as3":                            "ActionScript",
 	"asciidoc":                       "AsciiDoc",
+	"asm":                            "Assembly",
 	"asn.1":                          "ASN.1",
 	"asp":                            "ASP",
 	"aspectj":                        "AspectJ",
 	"aspx":                           "ASP",
 	"aspx-vb":                        "ASP",
 	"assembly":                       "Assembly",
+	"asymptote":                      "Asymptote",
 	"ats":                            "ATS",
 	"ats2":                           "ATS",
 	"au3":                            "AutoIt",
@@ -113,6 +117,7 @@ var LanguagesByAlias = map[string]string{
 	"clips":                          "CLIPS",
 	"clojure":                        "Clojure",
 	"closure_templates":              "Closure Templates",
+	"cloud_firestore_security_rules": "Cloud Firestore Security Rules",
 	"cmake":                    "CMake",
 	"cobol":                    "COBOL",
 	"coffee":                   "CoffeeScript",
@@ -123,10 +128,15 @@ var LanguagesByAlias = map[string]string{
 	"coldfusion_html":          "ColdFusion",
 	"collada":                  "COLLADA",
 	"common_lisp":              "Common Lisp",
+	"common_workflow_language": "Common Workflow Language",
 	"component_pascal":         "Component Pascal",
+	"conll":                    "CoNLL-U",
+	"conll-u":                  "CoNLL-U",
+	"conll-x":                  "CoNLL-U",
 	"console":                  "ShellSession",
 	"cool":                     "Cool",
 	"coq":                      "Coq",
+	"cperl":                    "Perl",
 	"cpp":                      "C++",
 	"cpp-objdump":              "Cpp-ObjDump",
 	"creole":                   "Creole",
@@ -144,6 +154,7 @@ var LanguagesByAlias = map[string]string{
 	"cucumber":                 "Gherkin",
 	"cuda":                     "Cuda",
 	"cweb":                     "CWeb",
+	"cwl":                      "Common Workflow Language",
 	"cycript":                  "Cycript",
 	"cython":                   "Cython",
 	"d":                        "D",
@@ -176,6 +187,7 @@ var LanguagesByAlias = map[string]string{
 	"ecl":                      "ECL",
 	"eclipse":                  "ECLiPSe",
 	"ecr":                      "HTML+ECR",
+	"edje_data_collection": "Edje Data Collection",
 	"edn":                 "edn",
 	"eeschema_schematic":  "KiCad Schematic",
 	"eex":                 "HTML+EEX",
@@ -187,13 +199,17 @@ var LanguagesByAlias = map[string]string{
 	"emacs":               "Emacs Lisp",
 	"emacs_lisp":          "Emacs Lisp",
 	"emberscript":         "EmberScript",
+	"eml":                 "EML",
 	"eq":                  "EQ",
 	"erb":                 "HTML+ERB",
 	"erlang":              "Erlang",
 	"f#":                  "F#",
+	"f*":                  "F*",
 	"factor":              "Factor",
 	"fancy":               "Fancy",
 	"fantom":              "Fantom",
+	"figfont":             "FIGlet Font",
+	"figlet_font":         "FIGlet Font",
 	"filebench_wml":       "Filebench WML",
 	"filterscript":        "Filterscript",
 	"fish":                "fish",
@@ -206,6 +222,7 @@ var LanguagesByAlias = map[string]string{
 	"freemarker":          "FreeMarker",
 	"frege":               "Frege",
 	"fsharp":              "F#",
+	"fstar":               "F*",
 	"ftl":                 "FreeMarker",
 	"fundamental":         "Text",
 	"g-code":              "G-code",
@@ -223,8 +240,16 @@ var LanguagesByAlias = map[string]string{
 	"gettext_catalog": "Gettext Catalog",
 	"gf":              "Grammatical Framework",
 	"gherkin":         "Gherkin",
+	"git-ignore":      "Ignore List",
+	"git_attributes":  "Git Attributes",
+	"git_config":      "Git Config",
+	"gitattributes":   "Git Attributes",
+	"gitconfig":       "Git Config",
+	"gitignore":       "Ignore List",
+	"gitmodules":      "Git Config",
 	"glsl":            "GLSL",
 	"glyph":           "Glyph",
+	"glyph_bitmap_distribution_format": "Glyph Bitmap Distribution Format",
 	"gn":                      "GN",
 	"gnuplot":                 "Gnuplot",
 	"go":                      "Go",
@@ -237,17 +262,20 @@ var LanguagesByAlias = map[string]string{
 	"graph_modeling_language": "Graph Modeling Language",
 	"graphql":                 "GraphQL",
 	"graphviz_(dot)":          "Graphviz (DOT)",
+	"groff":                   "Roff",
 	"groovy":                  "Groovy",
 	"groovy_server_pages":     "Groovy Server Pages",
 	"gsp":                      "Groovy Server Pages",
 	"hack":                     "Hack",
 	"haml":                     "Haml",
 	"handlebars":               "Handlebars",
+	"haproxy":                  "HAProxy",
 	"harbour":                  "Harbour",
 	"haskell":                  "Haskell",
 	"haxe":                     "Haxe",
 	"hbs":                      "Handlebars",
 	"hcl":                      "HCL",
+	"hiveql":                   "HiveQL",
 	"hlsl":                     "HLSL",
 	"html":                     "HTML",
 	"html+django":              "HTML+Django",
@@ -257,16 +285,20 @@ var LanguagesByAlias = map[string]string{
 	"html+erb":                 "HTML+ERB",
 	"html+jinja":               "HTML+Django",
 	"html+php":                 "HTML+PHP",
+	"html+razor":               "HTML+Razor",
 	"html+ruby":                "RHTML",
 	"htmlbars":                 "Handlebars",
 	"htmldjango":               "HTML+Django",
 	"http":                     "HTTP",
+	"hxml":                     "HXML",
 	"hy":                       "Hy",
 	"hylang":                   "Hy",
 	"hyphy":                    "HyPhy",
 	"i7":                       "Inform 7",
 	"idl":                      "IDL",
 	"idris":                    "Idris",
+	"ignore":                   "Ignore List",
+	"ignore_list":              "Ignore List",
 	"igor":                     "IGOR Pro",
 	"igor_pro":                 "IGOR Pro",
 	"igorpro":                  "IGOR Pro",
@@ -286,6 +318,7 @@ var LanguagesByAlias = map[string]string{
 	"j":                        "J",
 	"jasmin":                   "Jasmin",
 	"java":                     "Java",
+	"java_properties":          "Java Properties",
 	"java_server_page":         "Groovy Server Pages",
 	"java_server_pages":        "Java Server Pages",
 	"javascript":               "JavaScript",
@@ -297,6 +330,8 @@ var LanguagesByAlias = map[string]string{
 	"js":                       "JavaScript",
 	"json":                     "JSON",
 	"json5":                    "JSON5",
+	"json_with_comments":       "JSON with Comments",
+	"jsonc":                    "JSON with Comments",
 	"jsoniq":                   "JSONiq",
 	"jsonld":                   "JSONLD",
 	"jsp":                      "Java Server Pages",
@@ -340,6 +375,7 @@ var LanguagesByAlias = map[string]string{
 	"loomscript":               "LoomScript",
 	"ls":                       "LiveScript",
 	"lsl":                      "LSL",
+	"ltspice_symbol":           "LTspice Symbol",
 	"lua":                      "Lua",
 	"m":                        "M",
 	"m4":                       "M4",
@@ -348,17 +384,22 @@ var LanguagesByAlias = map[string]string{
 	"make":                     "Makefile",
 	"makefile":                 "Makefile",
 	"mako":                     "Mako",
+	"man":                      "Roff",
+	"man-page":                 "Roff",
+	"man_page":                 "Roff",
+	"manpage":                  "Roff",
 	"markdown":                 "Markdown",
 	"marko":                    "Marko",
 	"markojs":                  "Marko",
 	"mask":                     "Mask",
 	"mathematica":              "Mathematica",
-	"matlab":                   "Matlab",
+	"matlab":                   "MATLAB",
 	"maven_pom":                "Maven POM",
 	"max":                      "Max",
 	"max/msp":                  "Max",
 	"maxmsp":                   "Max",
 	"maxscript":                "MAXScript",
+	"mdoc":                     "Roff",
 	"mediawiki":                "MediaWiki",
 	"mercury":                  "Mercury",
 	"meson":                    "Meson",
@@ -369,6 +410,7 @@ var LanguagesByAlias = map[string]string{
 	"mma":                      "Mathematica",
 	"modelica":                 "Modelica",
 	"modula-2":                 "Modula-2",
+	"modula-3":                 "Modula-3",
 	"module_management_system": "Module Management System",
 	"monkey":                   "Monkey",
 	"moocode":                  "Moocode",
@@ -380,6 +422,7 @@ var LanguagesByAlias = map[string]string{
 	"mumps":                    "M",
 	"mupad":                    "mupad",
 	"myghty":                   "Myghty",
+	"nanorc":                   "nanorc",
 	"nasm":                     "Assembly",
 	"ncl":                      "NCL",
 	"nearley":                  "Nearley",
@@ -389,6 +432,7 @@ var LanguagesByAlias = map[string]string{
 	"netlinx+erb":              "NetLinx+ERB",
 	"netlogo":                  "NetLogo",
 	"newlisp":                  "NewLisp",
+	"nextflow":                 "Nextflow",
 	"nginx":                    "Nginx",
 	"nginx_configuration_file": "Nginx",
 	"nim":                   "Nim",
@@ -421,8 +465,9 @@ var LanguagesByAlias = map[string]string{
 	"objectpascal":          "Component Pascal",
 	"objj":                  "Objective-J",
 	"ocaml":                 "OCaml",
-	"octave":                "Matlab",
+	"octave":                "MATLAB",
 	"omgrofl":               "Omgrofl",
+	"oncrpc":                "RPC",
 	"ooc":                   "ooc",
 	"opa":                   "Opa",
 	"opal":                  "Opal",
@@ -447,10 +492,11 @@ var LanguagesByAlias = map[string]string{
 	"parrot_internal_representation": "Parrot Internal Representation",
 	"pascal":               "Pascal",
 	"pasm":                 "Parrot Assembly",
-	"pawn":                  "PAWN",
+	"pawn":                 "Pawn",
 	"pcbnew":               "KiCad Layout",
 	"pep8":                 "Pep8",
 	"perl":                 "Perl",
+	"perl6":                "Perl 6",
 	"perl_6":               "Perl 6",
 	"php":                  "PHP",
 	"pic":                  "Pic",
@@ -462,9 +508,11 @@ var LanguagesByAlias = map[string]string{
 	"plpgsql":              "PLpgSQL",
 	"plsql":                "PLSQL",
 	"pod":                  "Pod",
+	"pod_6":                "Pod 6",
 	"pogoscript":           "PogoScript",
 	"pony":                 "Pony",
 	"posh":                 "PowerShell",
+	"postcss":              "PostCSS",
 	"postscr":              "PostScript",
 	"postscript":           "PostScript",
 	"pot":                  "Gettext Catalog",
@@ -486,13 +534,17 @@ var LanguagesByAlias = map[string]string{
 	"pure_data":            "Pure Data",
 	"purebasic":            "PureBasic",
 	"purescript":           "PureScript",
+	"pwsh":                 "PowerShell",
 	"pycon":                "Python console",
 	"pyrex":                "Cython",
 	"python":               "Python",
+	"python3":              "Python",
 	"python_console":       "Python console",
 	"python_traceback":     "Python traceback",
+	"q":                    "q",
 	"qmake":                "QMake",
 	"qml":                  "QML",
+	"quake":                "Quake",
 	"r":                    "R",
 	"racket":               "Racket",
 	"ragel":                "Ragel",
@@ -503,6 +555,7 @@ var LanguagesByAlias = map[string]string{
 	"rascal":               "Rascal",
 	"raw":                  "Raw token data",
 	"raw_token_data":       "Raw token data",
+	"razor":                "HTML+Razor",
 	"rb":                   "Ruby",
 	"rbx":                  "Ruby",
 	"rdoc":                 "RDoc",
@@ -525,7 +578,10 @@ var LanguagesByAlias = map[string]string{
 	"rmarkdown":            "RMarkdown",
 	"robotframework":       "RobotFramework",
 	"roff":                 "Roff",
+	"roff_manpage":         "Roff Manpage",
 	"rouge":                "Rouge",
+	"rpc":                  "RPC",
+	"rpcgen":               "RPC",
 	"rpm_spec":             "RPM Spec",
 	"rs-274x":              "Gerber Image",
 	"rscript":              "R",
@@ -546,6 +602,7 @@ var LanguagesByAlias = map[string]string{
 	"scheme":               "Scheme",
 	"scilab":               "Scilab",
 	"scss":                 "SCSS",
+	"sed":                  "sed",
 	"self":                 "Self",
 	"sh":                   "Shell",
 	"shaderlab":            "ShaderLab",
@@ -554,14 +611,18 @@ var LanguagesByAlias = map[string]string{
 	"shellsession":         "ShellSession",
 	"shen":                 "Shen",
 	"slash":                "Slash",
+	"slice":                "Slice",
 	"slim":                 "Slim",
 	"smali":                "Smali",
 	"smalltalk":            "Smalltalk",
 	"smarty":               "Smarty",
 	"sml":                  "Standard ML",
 	"smt":                  "SMT",
+	"snippet":              "YASnippet",
+	"solidity":             "Solidity",
 	"sourcemod":            "SourcePawn",
 	"sourcepawn":           "SourcePawn",
+	"soy":                  "Closure Templates",
 	"sparql":               "SPARQL",
 	"specfile":             "RPM Spec",
 	"spline_font_database": "Spline Font Database",
@@ -577,8 +638,8 @@ var LanguagesByAlias = map[string]string{
 	"stata":                "Stata",
 	"ston":                 "STON",
 	"stylus":               "Stylus",
-	"sublime_text_config":   "Sublime Text Config",
 	"subrip_text":          "SubRip Text",
+	"sugarss":              "SugarSS",
 	"supercollider":        "SuperCollider",
 	"svg":                  "SVG",
 	"swift":                "Swift",
@@ -587,6 +648,7 @@ var LanguagesByAlias = map[string]string{
 	"tcsh":                 "Tcsh",
 	"tea":                  "Tea",
 	"terra":                "Terra",
+	"terraform":            "HCL",
 	"tex":                  "TeX",
 	"text":                 "Text",
 	"textile":              "Textile",
@@ -595,6 +657,7 @@ var LanguagesByAlias = map[string]string{
 	"tl":                   "Type Language",
 	"tla":                  "TLA",
 	"toml":                 "TOML",
+	"troff":                "Roff",
 	"ts":                   "TypeScript",
 	"turing":               "Turing",
 	"turtle":               "Turtle",
@@ -627,24 +690,31 @@ var LanguagesByAlias = map[string]string{
 	"wast":                 "WebAssembly",
 	"wavefront_material":   "Wavefront Material",
 	"wavefront_object":     "Wavefront Object",
+	"wdl":                  "wdl",
 	"web_ontology_language":    "Web Ontology Language",
 	"webassembly":              "WebAssembly",
 	"webidl":                   "WebIDL",
 	"winbatch":                 "Batchfile",
+	"windows_registry_entries": "Windows Registry Entries",
 	"wisp": "wisp",
 	"world_of_warcraft_addon_data": "World of Warcraft Addon Data",
 	"wsdl":                   "XML",
 	"x10":                    "X10",
+	"x_bitmap":               "X BitMap",
+	"x_font_directory_index": "X Font Directory Index",
+	"x_pixmap":               "X PixMap",
 	"xbase":                  "xBase",
+	"xbm":                    "X BitMap",
 	"xc":                     "XC",
 	"xcompose":               "XCompose",
+	"xdr":                    "RPC",
 	"xhtml":                  "HTML",
 	"xml":                    "XML",
 	"xml+genshi":             "Genshi",
 	"xml+kid":                "Genshi",
 	"xojo":                   "Xojo",
 	"xpages":                 "XPages",
-	"xpm":        "XPM",
+	"xpm":                    "X PixMap",
 	"xproc":                  "XProc",
 	"xquery":                 "XQuery",
 	"xs":                     "XS",
@@ -656,8 +726,34 @@ var LanguagesByAlias = map[string]string{
 	"yacc":                   "Yacc",
 	"yaml":                   "YAML",
 	"yang":                   "YANG",
+	"yara":                   "YARA",
+	"yas":                    "YASnippet",
+	"yasnippet":              "YASnippet",
 	"yml":                    "YAML",
 	"zephir":                 "Zephir",
+	"zig":                    "Zig",
 	"zimpl":                  "Zimpl",
 	"zsh":                    "Shell",
 }
+
+// LanguageByAlias looks up the language name by it's alias or name.
+// It mirrors the logic of github linguist and is needed e.g for heuristcs.yml
+// that mixes names and aliases in a language field (see XPM example).
+func LanguageByAlias(langOrAlias string) (lang string, ok bool) {
+	k := convertToAliasKey(langOrAlias)
+	lang, ok = LanguageByAliasMap[k]
+	return
+}
+
+// convertToAliasKey converts language name to a key in LanguageByAliasMap.
+// Following
+//  - internal.code-generator.generator.convertToAliasKey()
+//  - GetLanguageByAlias()
+// conventions.
+// It is here to avoid dependency on "generate" and "enry" packages.
+func convertToAliasKey(langName string) string {
+	ak := strings.SplitN(langName, `,`, 2)[0]
+	ak = strings.Replace(ak, ` `, `_`, -1)
+	ak = strings.ToLower(ak)
+	return ak
+}

data/commit.go

@@ -1,7 +1,7 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
 // linguist's commit from which files were generated.
-var LinguistCommit = "4cd558c37482e8d2c535d8107f2d11b49afbc5b5"
+var LinguistCommit = "e4560984058b4726010ca4b8f03ed9d0f8f464db"

data/doc.go

@@ -0,0 +1,3 @@
+// Package data contains only auto-generated data-structures for all the language
+// identification strategies from the Linguist project sources.
+package data

data/documentation.go

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -8,10 +8,12 @@ import "gopkg.in/toqueteos/substring.v1"
 var DocumentationMatchers = substring.Or(
 	substring.Regexp(`^[Dd]ocs?/`),
 	substring.Regexp(`(^|/)[Dd]ocumentation/`),
+	substring.Regexp(`(^|/)[Gg]roovydoc/`),
 	substring.Regexp(`(^|/)[Jj]avadoc/`),
 	substring.Regexp(`^[Mm]an/`),
 	substring.Regexp(`^[Ee]xamples/`),
 	substring.Regexp(`^[Dd]emos?/`),
+	substring.Regexp(`(^|/)inst/doc/`),
 	substring.Regexp(`(^|/)CHANGE(S|LOG)?(\.|$)`),
 	substring.Regexp(`(^|/)CONTRIBUTING(\.|$)`),
 	substring.Regexp(`(^|/)COPYING(\.|$)`),

data/extension.go

@@ -1,32 +1,34 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
 var LanguagesByExtension = map[string][]string{
-	".1":                   {"Roff"},
+	".1":                   {"Roff", "Roff Manpage"},
-	".1in":                 {"Roff"},
+	".1in":                 {"Roff", "Roff Manpage"},
-	".1m":                  {"Roff"},
+	".1m":                  {"Roff", "Roff Manpage"},
-	".1x":                  {"Roff"},
+	".1x":                  {"Roff", "Roff Manpage"},
-	".2":                   {"Roff"},
+	".2":                   {"Roff", "Roff Manpage"},
-	".3":                   {"Roff"},
+	".3":                   {"Roff", "Roff Manpage"},
-	".3in":                 {"Roff"},
+	".3in":                 {"Roff", "Roff Manpage"},
-	".3m":                  {"Roff"},
+	".3m":                  {"Roff", "Roff Manpage"},
-	".3qt":                 {"Roff"},
+	".3p":                  {"Roff", "Roff Manpage"},
-	".3x":                  {"Roff"},
+	".3pm":                 {"Roff", "Roff Manpage"},
-	".4":                   {"Roff"},
+	".3qt":                 {"Roff", "Roff Manpage"},
+	".3x":                  {"Roff", "Roff Manpage"},
+	".4":                   {"Roff", "Roff Manpage"},
 	".4th":                 {"Forth"},
-	".5":                   {"Roff"},
+	".5":                   {"Roff", "Roff Manpage"},
-	".6":                   {"Roff"},
+	".6":                   {"Roff", "Roff Manpage"},
 	".6pl":                 {"Perl 6"},
 	".6pm":                 {"Perl 6"},
-	".7":                   {"Roff"},
+	".7":                   {"Roff", "Roff Manpage"},
-	".8":                   {"Roff"},
+	".8":                   {"Roff", "Roff Manpage"},
 	".8xk":                 {"TI Program"},
 	".8xk.txt":             {"TI Program"},
 	".8xp":                 {"TI Program"},
 	".8xp.txt":             {"TI Program"},
-	".9":                   {"Roff"},
+	".9":                   {"Roff", "Roff Manpage"},
 	"._coffee":             {"CoffeeScript"},
 	"._js":                 {"JavaScript"},
 	"._ls":                 {"LiveScript"},
@@ -50,6 +52,7 @@ var LanguagesByExtension = map[string][]string{
 	".al":                  {"Perl"},
 	".als":                 {"Alloy"},
 	".ampl":                {"AMPL"},
+	".angelscript":         {"AngelScript"},
 	".anim":                {"Unity3D Asset"},
 	".ant":                 {"XML"},
 	".apacheconf":          {"ApacheConf"},
@@ -59,7 +62,7 @@ var LanguagesByExtension = map[string][]string{
 	".applescript":         {"AppleScript"},
 	".arc":                 {"Arc"},
 	".arpa":                {"DNS Zone"},
-	".as":                  {"ActionScript"},
+	".as":                  {"ActionScript", "AngelScript"},
 	".asax":                {"ASP"},
 	".asc":                 {"AGS Script", "AsciiDoc", "Public Key"},
 	".asciidoc":            {"AsciiDoc"},
@@ -74,10 +77,12 @@ var LanguagesByExtension = map[string][]string{
 	".asp":                 {"ASP"},
 	".aspx":                {"ASP"},
 	".asset":               {"Unity3D Asset"},
+	".asy":                 {"Asymptote", "LTspice Symbol"},
 	".au3":                 {"AutoIt"},
 	".aug":                 {"Augeas"},
 	".auk":                 {"Awk"},
 	".aux":                 {"TeX"},
+	".avsc":                {"JSON"},
 	".aw":                  {"PHP"},
 	".awk":                 {"Awk"},
 	".axd":                 {"ASP"},
@@ -94,6 +99,7 @@ var LanguagesByExtension = map[string][]string{
 	".bats":                {"Shell"},
 	".bb":                  {"BitBake", "BlitzBasic"},
 	".bbx":                 {"TeX"},
+	".bdf":                 {"Glyph Bitmap Distribution Format"},
 	".bdy":                 {"PLSQL"},
 	".befunge":             {"Befunge"},
 	".bf":                  {"Brainfuck", "HyPhy"},
@@ -130,7 +136,7 @@ var LanguagesByExtension = map[string][]string{
 	".cdf":                 {"Mathematica"},
 	".ceylon":              {"Ceylon"},
 	".cfc":                 {"ColdFusion CFC"},
-	".cfg":                 {"INI"},
+	".cfg":                 {"HAProxy", "INI"},
 	".cfm":                 {"ColdFusion"},
 	".cfml":                {"ColdFusion"},
 	".cgi":                 {"Perl", "Python", "Shell"},
@@ -163,6 +169,8 @@ var LanguagesByExtension = map[string][]string{
 	".coffee":              {"CoffeeScript"},
 	".com":                 {"DIGITAL Command Language"},
 	".command":             {"Shell"},
+	".conll":               {"CoNLL-U"},
+	".conllu":              {"CoNLL-U"},
 	".coq":                 {"Coq"},
 	".cp":                  {"C++", "Component Pascal"},
 	".cpp":                 {"C++"},
@@ -179,7 +187,7 @@ var LanguagesByExtension = map[string][]string{
 	".csd":                 {"Csound Document"},
 	".csdef":               {"XML"},
 	".csh":                 {"Tcsh"},
-	".cshtml":              {"C#"},
+	".cshtml":              {"HTML+Razor"},
 	".csl":                 {"XML"},
 	".cson":                {"CSON"},
 	".csproj":              {"XML"},
@@ -191,6 +199,7 @@ var LanguagesByExtension = map[string][]string{
 	".cu":                  {"Cuda"},
 	".cuh":                 {"Cuda"},
 	".cw":                  {"Redcode"},
+	".cwl":                 {"Common Workflow Language"},
 	".cxx":                 {"C++"},
 	".cxx-objdump":         {"Cpp-ObjDump"},
 	".cy":                  {"Cycript"},
@@ -241,6 +250,7 @@ var LanguagesByExtension = map[string][]string{
 	".eclass":              {"Gentoo Eclass"},
 	".eclxml":              {"ECL"},
 	".ecr":                 {"HTML+ECR"},
+	".edc":                 {"Edje Data Collection"},
 	".edn":                 {"edn"},
 	".eex":                 {"HTML+EEX"},
 	".eh":                  {"eC"},
@@ -253,6 +263,7 @@ var LanguagesByExtension = map[string][]string{
 	".emacs":               {"Emacs Lisp"},
 	".emacs.desktop":       {"Emacs Lisp"},
 	".emberscript":         {"EmberScript"},
+	".eml":                 {"EML"},
 	".epj":                 {"Ecere Projects"},
 	".eps":                 {"PostScript"},
 	".eq":                  {"EQ"},
@@ -280,6 +291,7 @@ var LanguagesByExtension = map[string][]string{
 	".filters":             {"XML"},
 	".fish":                {"fish"},
 	".flex":                {"JFlex"},
+	".flf":                 {"FIGlet Font"},
 	".flux":                {"FLUX"},
 	".fnc":                 {"PLSQL"},
 	".for":                 {"Formatted", "Forth", "Fortran"},
@@ -297,6 +309,7 @@ var LanguagesByExtension = map[string][]string{
 	".fshader":             {"GLSL"},
 	".fsi":                 {"F#"},
 	".fsproj":              {"XML"},
+	".fst":                 {"F*"},
 	".fsx":                 {"F#"},
 	".fth":                 {"Forth"},
 	".ftl":                 {"FreeMarker"},
@@ -325,13 +338,17 @@ var LanguagesByExtension = map[string][]string{
 	".geom":                {"GLSL"},
 	".gf":                  {"Grammatical Framework"},
 	".gi":                  {"GAP"},
+	".gitconfig":           {"Git Config"},
+	".gitignore":           {"Ignore List"},
 	".gko":                 {"Gerber Image"},
 	".glade":               {"XML"},
 	".glf":                 {"Glyph"},
 	".glsl":                {"GLSL"},
 	".glslv":               {"GLSL"},
+	".gltf":                {"JSON"},
 	".gml":                 {"Game Maker Language", "Graph Modeling Language", "XML"},
 	".gms":                 {"GAMS"},
+	".gmx":                 {"XML"},
 	".gn":                  {"GN"},
 	".gni":                 {"GN"},
 	".gnu":                 {"Gnuplot"},
@@ -368,6 +385,7 @@ var LanguagesByExtension = map[string][]string{
 	".haml":                {"Haml"},
 	".haml.deface":         {"Haml"},
 	".handlebars":          {"Handlebars"},
+	".har":                 {"JSON"},
 	".hats":                {"ATS"},
 	".hb":                  {"Harbour"},
 	".hbs":                 {"Handlebars"},
@@ -387,23 +405,27 @@ var LanguagesByExtension = map[string][]string{
 	".html.hl":             {"HTML"},
 	".http":                {"HTTP"},
 	".hx":                  {"Haxe"},
+	".hxml":                {"HXML"},
 	".hxsl":                {"Haxe"},
 	".hxx":                 {"C++"},
 	".hy":                  {"Hy"},
+	".i3":                  {"Modula-3"},
 	".i7x":                 {"Inform 7"},
+	".ice":                 {"JSON", "Slice"},
 	".iced":                {"CoffeeScript"},
 	".icl":                 {"Clean"},
 	".idc":                 {"C"},
 	".idr":                 {"Idris"},
+	".ig":                  {"Modula-3"},
 	".ihlp":                {"Stata"},
 	".ijs":                 {"J"},
 	".ik":                  {"Ioke"},
 	".ily":                 {"LilyPond"},
 	".iml":                 {"XML"},
-	".inc":                 {"Assembly", "C++", "HTML", "PAWN", "PHP", "POV-Ray SDL", "Pascal", "SQL", "SourcePawn"},
+	".inc":                 {"Assembly", "C++", "HTML", "PHP", "POV-Ray SDL", "Pascal", "Pawn", "SQL", "SourcePawn"},
 	".ini":                 {"INI"},
 	".inl":                 {"C++"},
-	".ino":                 {"Arduino"},
+	".ino":                 {"C++"},
 	".ins":                 {"TeX"},
 	".intr":                {"Dylan"},
 	".io":                  {"Io"},
@@ -422,6 +444,7 @@ var LanguagesByExtension = map[string][]string{
 	".jelly":               {"XML"},
 	".jflex":               {"JFlex"},
 	".jinja":               {"HTML+Django"},
+	".jinja2":              {"HTML+Django"},
 	".jison":               {"Jison"},
 	".jisonlex":            {"Jison Lex"},
 	".jl":                  {"Julia"},
@@ -434,6 +457,7 @@ var LanguagesByExtension = map[string][]string{
 	".json":                {"JSON"},
 	".json-tmlanguage":     {"JSON"},
 	".json5":               {"JSON5"},
+	".jsonl":               {"JSON"},
 	".jsonld":              {"JSONLD"},
 	".jsp":                 {"Java Server Pages"},
 	".jsproj":              {"XML"},
@@ -445,6 +469,7 @@ var LanguagesByExtension = map[string][]string{
 	".kid":                 {"Genshi"},
 	".kit":                 {"Kit"},
 	".kml":                 {"XML"},
+	".kojo":                {"Scala"},
 	".krl":                 {"KRL"},
 	".ksh":                 {"Shell"},
 	".kt":                  {"Kotlin"},
@@ -463,6 +488,7 @@ var LanguagesByExtension = map[string][]string{
 	".ldml":                {"Lasso"},
 	".lds":                 {"Linker Script"},
 	".lean":                {"Lean"},
+	".lektorproject":       {"INI"},
 	".less":                {"Less"},
 	".lex":                 {"Lex"},
 	".lfe":                 {"LFE"},
@@ -487,13 +513,14 @@ var LanguagesByExtension = map[string][]string{
 	".lua":                 {"Lua"},
 	".lvproj":              {"LabVIEW"},
 	".ly":                  {"LilyPond"},
-	".m":                   {"Limbo", "M", "MUF", "Mathematica", "Matlab", "Mercury", "Objective-C"},
+	".m":                   {"Limbo", "M", "MATLAB", "MUF", "Mathematica", "Mercury", "Objective-C"},
+	".m3":                  {"Modula-3"},
 	".m4":                  {"M4", "M4Sugar"},
 	".ma":                  {"Mathematica"},
 	".mak":                 {"Makefile"},
 	".make":                {"Makefile"},
 	".mako":                {"Mako"},
-	".man":                 {"Roff"},
+	".man":                 {"Roff", "Roff Manpage"},
 	".mao":                 {"Mako"},
 	".markdown":            {"Markdown"},
 	".marko":               {"Marko"},
@@ -502,13 +529,15 @@ var LanguagesByExtension = map[string][]string{
 	".mata":                {"Stata"},
 	".matah":               {"Stata"},
 	".mathematica":         {"Mathematica"},
-	".matlab":              {"Matlab"},
+	".matlab":              {"MATLAB"},
 	".mawk":                {"Awk"},
 	".maxhelp":             {"Max"},
 	".maxpat":              {"Max"},
 	".maxproj":             {"Max"},
+	".mbox":                {"EML"},
 	".mcr":                 {"MAXScript"},
 	".md":                  {"GCC Machine Description", "Markdown"},
+	".mdoc":                {"Roff", "Roff Manpage"},
 	".mdown":               {"Markdown"},
 	".mdpolicy":            {"XML"},
 	".mdwn":                {"Markdown"},
@@ -516,8 +545,9 @@ var LanguagesByExtension = map[string][]string{
 	".mediawiki":           {"MediaWiki"},
 	".meta":                {"Unity3D Asset"},
 	".metal":               {"Metal"},
+	".mg":                  {"Modula-3"},
 	".minid":               {"MiniD"},
-	".mir":                 {"Mirah"},
+	".mir":                 {"YAML"},
 	".mirah":               {"Mirah"},
 	".mjml":                {"XML"},
 	".mjs":                 {"JavaScript"},
@@ -562,20 +592,25 @@ var LanguagesByExtension = map[string][]string{
 	".mysql":               {"SQL"},
 	".myt":                 {"Myghty"},
 	".n":                   {"Nemerle", "Roff"},
+	".nanorc":              {"nanorc"},
 	".nasm":                {"Assembly"},
 	".natvis":              {"XML"},
 	".nawk":                {"Awk"},
 	".nb":                  {"Mathematica", "Text"},
 	".nbp":                 {"Mathematica"},
 	".nc":                  {"nesC"},
-	".ncl":                 {"NCL", "Text"},
+	".ncl":                 {"NCL", "Text", "XML"},
 	".ndproj":              {"XML"},
 	".ne":                  {"Nearley"},
 	".nearley":             {"Nearley"},
+	".nf":                  {"Nextflow"},
 	".nginxconf":           {"Nginx"},
 	".ni":                  {"Inform 7"},
 	".nim":                 {"Nim"},
+	".nim.cfg":             {"Nim"},
+	".nimble":              {"Nim"},
 	".nimrod":              {"Nim"},
+	".nims":                {"Nim"},
 	".ninja":               {"Ninja"},
 	".nit":                 {"Nit"},
 	".nix":                 {"Nix"},
@@ -621,6 +656,7 @@ var LanguagesByExtension = map[string][]string{
 	".p6":                  {"Perl 6"},
 	".p6l":                 {"Perl 6"},
 	".p6m":                 {"Perl 6"},
+	".p8":                  {"Lua"},
 	".pac":                 {"JavaScript"},
 	".pan":                 {"Pan"},
 	".parrot":              {"Parrot"},
@@ -633,12 +669,14 @@ var LanguagesByExtension = map[string][]string{
 	".pbi":                 {"PureBasic"},
 	".pbt":                 {"PowerBuilder"},
 	".pck":                 {"PLSQL"},
+	".pcss":                {"PostCSS"},
 	".pd":                  {"Pure Data"},
 	".pd_lua":              {"Lua"},
 	".pde":                 {"Processing"},
 	".pep":                 {"Pep8"},
 	".perl":                {"Perl"},
 	".pfa":                 {"PostScript"},
+	".pgsql":               {"PLpgSQL"},
 	".ph":                  {"Perl"},
 	".php":                 {"Hack", "PHP"},
 	".php3":                {"PHP"},
@@ -665,11 +703,12 @@ var LanguagesByExtension = map[string][]string{
 	".plt":                 {"Gnuplot"},
 	".pluginspec":          {"Ruby", "XML"},
 	".plx":                 {"Perl"},
-	".pm":                  {"Perl", "Perl 6", "XPM"},
+	".pm":                  {"Perl", "Perl 6", "X PixMap"},
 	".pm6":                 {"Perl 6"},
 	".pmod":                {"Pike"},
 	".po":                  {"Gettext Catalog"},
-	".pod":                 {"Perl", "Pod"},
+	".pod":                 {"Pod", "Pod 6"},
+	".pod6":                {"Pod 6"},
 	".podsl":               {"Common Lisp"},
 	".podspec":             {"Ruby"},
 	".pogo":                {"PogoScript"},
@@ -686,7 +725,7 @@ var LanguagesByExtension = map[string][]string{
 	".pro":                 {"IDL", "INI", "Prolog", "QMake"},
 	".proj":                {"XML"},
 	".prolog":              {"Prolog"},
-	".properties":          {"INI"},
+	".properties":          {"INI", "Java Properties"},
 	".props":               {"XML"},
 	".proto":               {"Protocol Buffer"},
 	".prw":                 {"xBase"},
@@ -702,7 +741,7 @@ var LanguagesByExtension = map[string][]string{
 	".pub":                 {"Public Key"},
 	".pug":                 {"Pug"},
 	".purs":                {"PureScript"},
-	".pwn":                 {"PAWN"},
+	".pwn":                 {"Pawn"},
 	".pxd":                 {"Cython"},
 	".pxi":                 {"Cython"},
 	".py":                  {"Python"},
@@ -714,6 +753,7 @@ var LanguagesByExtension = map[string][]string{
 	".pytb":                {"Python traceback"},
 	".pyw":                 {"Python"},
 	".pyx":                 {"Cython"},
+	".q":                   {"HiveQL", "q"},
 	".qbs":                 {"QML"},
 	".qml":                 {"QML"},
 	".r":                   {"R", "Rebol"},
@@ -743,6 +783,7 @@ var LanguagesByExtension = map[string][]string{
 	".red":                 {"Red"},
 	".reds":                {"Red"},
 	".reek":                {"YAML"},
+	".reg":                 {"Windows Registry Entries"},
 	".regex":               {"Regular Expression"},
 	".regexp":              {"Regular Expression"},
 	".rei":                 {"Reason"},
@@ -763,7 +804,7 @@ var LanguagesByExtension = map[string][]string{
 	".rno":                 {"RUNOFF", "Roff"},
 	".robot":               {"RobotFramework"},
 	".roff":                {"Roff"},
-	".ron":                 {"Markdown"},
+	".ronn":                {"Markdown"},
 	".rpy":                 {"Python", "Ren'Py"},
 	".rq":                  {"SPARQL"},
 	".rs":                  {"RenderScript", "Rust"},
@@ -798,6 +839,7 @@ var LanguagesByExtension = map[string][]string{
 	".scrbl":               {"Racket"},
 	".scss":                {"SCSS"},
 	".scxml":               {"XML"},
+	".sed":                 {"sed"},
 	".self":                {"Self"},
 	".sexp":                {"Common Lisp"},
 	".sfd":                 {"Spline Font Database"},
@@ -836,6 +878,7 @@ var LanguagesByExtension = map[string][]string{
 	".srw":                 {"PowerBuilder"},
 	".ss":                  {"Scheme"},
 	".ssjs":                {"JavaScript"},
+	".sss":                 {"SugarSS"},
 	".st":                  {"HTML", "Smalltalk"},
 	".stan":                {"Stan"},
 	".sthlp":               {"Stata"},
@@ -844,21 +887,21 @@ var LanguagesByExtension = map[string][]string{
 	".sttheme":             {"XML"},
 	".sty":                 {"TeX"},
 	".styl":                {"Stylus"},
-	".sublime-build":       {"Sublime Text Config"},
+	".sublime-build":       {"JSON with Comments"},
-	".sublime-commands":    {"Sublime Text Config"},
+	".sublime-commands":    {"JSON with Comments"},
-	".sublime-completions": {"Sublime Text Config"},
+	".sublime-completions": {"JSON with Comments"},
-	".sublime-keymap":      {"Sublime Text Config"},
+	".sublime-keymap":      {"JSON with Comments"},
-	".sublime-macro":       {"Sublime Text Config"},
+	".sublime-macro":       {"JSON with Comments"},
-	".sublime-menu":        {"Sublime Text Config"},
+	".sublime-menu":        {"JSON with Comments"},
-	".sublime-mousemap":    {"Sublime Text Config"},
+	".sublime-mousemap":    {"JSON with Comments"},
-	".sublime-project":     {"Sublime Text Config"},
+	".sublime-project":     {"JSON with Comments"},
-	".sublime-settings":    {"Sublime Text Config"},
+	".sublime-settings":    {"JSON with Comments"},
 	".sublime-snippet":     {"XML"},
 	".sublime-syntax":      {"YAML"},
-	".sublime-theme":       {"Sublime Text Config"},
+	".sublime-theme":       {"JSON with Comments"},
-	".sublime-workspace":   {"Sublime Text Config"},
+	".sublime-workspace":   {"JSON with Comments"},
-	".sublime_metrics":     {"Sublime Text Config"},
+	".sublime_metrics":     {"JSON with Comments"},
-	".sublime_session":     {"Sublime Text Config"},
+	".sublime_session":     {"JSON with Comments"},
 	".sv":                  {"SystemVerilog"},
 	".svg":                 {"SVG"},
 	".svh":                 {"SystemVerilog"},
@@ -877,6 +920,9 @@ var LanguagesByExtension = map[string][]string{
 	".tex":                 {"TeX"},
 	".textile":             {"Textile"},
 	".tf":                  {"HCL"},
+	".tfstate":             {"JSON"},
+	".tfstate.backup":      {"JSON"},
+	".tfvars":              {"HCL"},
 	".thor":                {"Ruby"},
 	".thrift":              {"Thrift"},
 	".thy":                 {"Isabelle"},
@@ -955,11 +1001,14 @@ var LanguagesByExtension = map[string][]string{
 	".vue":                 {"Vue"},
 	".vw":                  {"PLSQL"},
 	".vxml":                {"XML"},
-	".w":                   {"CWeb"},
+	".w":                   {"CWeb", "OpenEdge ABL"},
 	".wast":                {"WebAssembly"},
 	".wat":                 {"WebAssembly"},
 	".watchr":              {"Ruby"},
+	".wdl":                 {"wdl"},
+	".webapp":              {"JSON"},
 	".webidl":              {"WebIDL"},
+	".webmanifest":         {"JSON"},
 	".weechatlog":          {"IRC log"},
 	".wiki":                {"MediaWiki"},
 	".wisp":                {"wisp"},
@@ -968,17 +1017,19 @@ var LanguagesByExtension = map[string][]string{
 	".wlt":                 {"Mathematica"},
 	".wlua":                {"Lua"},
 	".workbook":            {"Markdown"},
+	".workflow":            {"HCL", "XML"},
 	".wsdl":                {"XML"},
 	".wsf":                 {"XML"},
 	".wsgi":                {"Python"},
 	".wxi":                 {"XML"},
 	".wxl":                 {"XML"},
 	".wxs":                 {"XML"},
-	".x":                   {"Logos"},
+	".x":                   {"Linker Script", "Logos", "RPC"},
 	".x10":                 {"X10"},
 	".x3d":                 {"XML"},
 	".xacro":               {"XML"},
 	".xaml":                {"XML"},
+	".xbm":                 {"X BitMap"},
 	".xc":                  {"XC"},
 	".xht":                 {"HTML"},
 	".xhtml":               {"HTML"},
@@ -997,7 +1048,7 @@ var LanguagesByExtension = map[string][]string{
 	".xojo_toolbar":        {"Xojo"},
 	".xojo_window":         {"Xojo"},
 	".xpl":                 {"XProc"},
-	".xpm":                 {"XPM"},
+	".xpm":                 {"X PixMap"},
 	".xproc":               {"XProc"},
 	".xproj":               {"XML"},
 	".xpy":                 {"Python"},
@@ -1024,12 +1075,17 @@ var LanguagesByExtension = map[string][]string{
 	".yaml-tmlanguage":     {"YAML"},
 	".yang":                {"YANG"},
 	".yap":                 {"Prolog"},
+	".yar":                 {"YARA"},
+	".yara":                {"YARA"},
+	".yasnippet":           {"YASnippet"},
 	".yml":                 {"YAML"},
 	".yml.mysql":           {"YAML"},
 	".yrl":                 {"Erlang"},
-	".yy":                  {"Yacc"},
+	".yy":                  {"JSON", "Yacc"},
+	".yyp":                 {"JSON"},
 	".zcml":                {"XML"},
 	".zep":                 {"Zephir"},
+	".zig":                 {"Zig"},
 	".zimpl":               {"Zimpl"},
 	".zmpl":                {"Zimpl"},
 	".zone":                {"DNS Zone"},
@@ -1054,15 +1110,16 @@ var ExtensionsByLanguage = map[string][]string{
 	"Adobe Font Metrics": {".afm"},
 	"Agda":               {".agda"},
 	"Alloy":              {".als"},
+	"AngelScript":        {".as", ".angelscript"},
 	"ApacheConf":         {".apacheconf", ".vhost"},
 	"Apex":               {".cls"},
 	"Apollo Guidance Computer": {".agc"},
 	"AppleScript":              {".applescript", ".scpt"},
 	"Arc":                      {".arc"},
-	"Arduino":                  {".ino"},
 	"AsciiDoc":                 {".asciidoc", ".adoc", ".asc"},
 	"AspectJ":                  {".aj"},
 	"Assembly":                 {".asm", ".a51", ".inc", ".nasm"},
+	"Asymptote":                {".asy"},
 	"Augeas":                   {".aug"},
 	"AutoHotkey":               {".ahk", ".ahkl"},
 	"AutoIt":                   {".au3"},
@@ -1081,8 +1138,8 @@ var ExtensionsByLanguage = map[string][]string{
 	"Brightscript":             {".brs"},
 	"Bro":                      {".bro"},
 	"C":                        {".c", ".cats", ".h", ".idc"},
-	"C#":                       {".cs", ".cake", ".cshtml", ".csx"},
+	"C#":                       {".cs", ".cake", ".csx"},
-	"C++":                      {".cpp", ".c++", ".cc", ".cp", ".cxx", ".h", ".h++", ".hh", ".hpp", ".hxx", ".inc", ".inl", ".ipp", ".re", ".tcc", ".tpp"},
+	"C++":                      {".cpp", ".c++", ".cc", ".cp", ".cxx", ".h", ".h++", ".hh", ".hpp", ".hxx", ".inc", ".inl", ".ino", ".ipp", ".re", ".tcc", ".tpp"},
 	"C-ObjDump":                {".c-objdump"},
 	"C2hs Haskell":             {".chs"},
 	"CLIPS":                    {".clp"},
@@ -1105,10 +1162,12 @@ var ExtensionsByLanguage = map[string][]string{
 	"Click":                    {".click"},
 	"Clojure":                  {".clj", ".boot", ".cl2", ".cljc", ".cljs", ".cljs.hl", ".cljscm", ".cljx", ".hic"},
 	"Closure Templates":        {".soy"},
+	"CoNLL-U":                  {".conllu", ".conll"},
 	"CoffeeScript":             {".coffee", "._coffee", ".cake", ".cjsx", ".iced"},
 	"ColdFusion":               {".cfm", ".cfml"},
 	"ColdFusion CFC":           {".cfc"},
 	"Common Lisp":              {".lisp", ".asd", ".cl", ".l", ".lsp", ".ny", ".podsl", ".sexp"},
+	"Common Workflow Language": {".cwl"},
 	"Component Pascal":         {".cp", ".cps"},
 	"Cool":                     {".cl"},
 	"Coq":                      {".coq", ".v"},
@@ -1139,10 +1198,12 @@ var ExtensionsByLanguage = map[string][]string{
 	"ECL":                  {".ecl", ".eclxml"},
 	"ECLiPSe":              {".ecl"},
 	"EJS":                  {".ejs"},
+	"EML":                  {".eml", ".mbox"},
 	"EQ":                   {".eq"},
 	"Eagle":                {".sch", ".brd"},
 	"Easybuild":            {".eb"},
 	"Ecere Projects":       {".epj"},
+	"Edje Data Collection": {".edc"},
 	"Eiffel":               {".e"},
 	"Elixir":               {".ex", ".exs"},
 	"Elm":                  {".elm"},
@@ -1150,6 +1211,8 @@ var ExtensionsByLanguage = map[string][]string{
 	"EmberScript":          {".em", ".emberscript"},
 	"Erlang":               {".erl", ".app.src", ".es", ".escript", ".hrl", ".xrl", ".yrl"},
 	"F#":                   {".fs", ".fsi", ".fsx"},
+	"F*":                   {".fst"},
+	"FIGlet Font":          {".flf"},
 	"FLUX":                 {".fx", ".flux"},
 	"Factor":               {".factor"},
 	"Fancy":                {".fy", ".fancypack"},
@@ -1177,7 +1240,9 @@ var ExtensionsByLanguage = map[string][]string{
 	"Gerber Image":        {".gbr", ".gbl", ".gbo", ".gbp", ".gbs", ".gko", ".gpb", ".gpt", ".gtl", ".gto", ".gtp", ".gts"},
 	"Gettext Catalog":     {".po", ".pot"},
 	"Gherkin":             {".feature"},
+	"Git Config":          {".gitconfig"},
 	"Glyph":               {".glf"},
+	"Glyph Bitmap Distribution Format": {".bdf"},
 	"Gnuplot":                          {".gp", ".gnu", ".gnuplot", ".plot", ".plt"},
 	"Go":                               {".go"},
 	"Golo":                             {".golo"},
@@ -1190,28 +1255,33 @@ var ExtensionsByLanguage = map[string][]string{
 	"Graphviz (DOT)":                   {".dot", ".gv"},
 	"Groovy":                           {".groovy", ".grt", ".gtpl", ".gvy"},
 	"Groovy Server Pages":              {".gsp"},
-	"HCL":                      {".hcl", ".tf"},
+	"HAProxy":                          {".cfg"},
+	"HCL":                              {".hcl", ".tf", ".tfvars", ".workflow"},
 	"HLSL":                             {".hlsl", ".cginc", ".fx", ".fxh", ".hlsli"},
 	"HTML":                             {".html", ".htm", ".html.hl", ".inc", ".st", ".xht", ".xhtml"},
-	"HTML+Django":              {".jinja", ".mustache", ".njk"},
+	"HTML+Django":                      {".jinja", ".jinja2", ".mustache", ".njk"},
 	"HTML+ECR":                         {".ecr"},
 	"HTML+EEX":                         {".eex"},
 	"HTML+ERB":                         {".erb", ".erb.deface"},
 	"HTML+PHP":                         {".phtml"},
+	"HTML+Razor":                       {".cshtml"},
 	"HTTP":                             {".http"},
+	"HXML":                             {".hxml"},
 	"Hack":                             {".hh", ".php"},
 	"Haml":                             {".haml", ".haml.deface"},
 	"Handlebars":                       {".handlebars", ".hbs"},
 	"Harbour":                          {".hb"},
 	"Haskell":                          {".hs", ".hsc"},
 	"Haxe":                             {".hx", ".hxsl"},
+	"HiveQL":                           {".q"},
 	"Hy":                               {".hy"},
 	"HyPhy":                            {".bf"},
 	"IDL":                              {".pro", ".dlm"},
 	"IGOR Pro":                         {".ipf"},
-	"INI":                      {".ini", ".cfg", ".prefs", ".pro", ".properties"},
+	"INI":                              {".ini", ".cfg", ".lektorproject", ".prefs", ".pro", ".properties"},
 	"IRC log":                          {".irclog", ".weechatlog"},
 	"Idris":                            {".idr", ".lidr"},
+	"Ignore List":                      {".gitignore"},
 	"Inform 7":                         {".ni", ".i7x"},
 	"Inno Setup":                       {".iss"},
 	"Io":                               {".io"},
@@ -1219,13 +1289,15 @@ var ExtensionsByLanguage = map[string][]string{
 	"Isabelle":                         {".thy"},
 	"J":                                {".ijs"},
 	"JFlex":                            {".flex", ".jflex"},
-	"JSON":                     {".json", ".geojson", ".json-tmlanguage", ".topojson"},
+	"JSON":                             {".json", ".avsc", ".geojson", ".gltf", ".har", ".ice", ".json-tmlanguage", ".jsonl", ".tfstate", ".tfstate.backup", ".topojson", ".webapp", ".webmanifest", ".yy", ".yyp"},
+	"JSON with Comments":               {".sublime-build", ".sublime-commands", ".sublime-completions", ".sublime-keymap", ".sublime-macro", ".sublime-menu", ".sublime-mousemap", ".sublime-project", ".sublime-settings", ".sublime-theme", ".sublime-workspace", ".sublime_metrics", ".sublime_session"},
 	"JSON5":                            {".json5"},
 	"JSONLD":                           {".jsonld"},
 	"JSONiq":                           {".jq"},
 	"JSX":                              {".jsx"},
 	"Jasmin":                           {".j"},
 	"Java":                             {".java"},
+	"Java Properties":                  {".properties"},
 	"Java Server Pages":                {".jsp"},
 	"JavaScript":                       {".js", "._js", ".bones", ".es", ".es6", ".frag", ".gs", ".jake", ".jsb", ".jscad", ".jsfl", ".jsm", ".jss", ".mjs", ".njs", ".pac", ".sjs", ".ssjs", ".xsjs", ".xsjslib"},
 	"Jison":                            {".jison"},
@@ -1243,6 +1315,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"LLVM":                             {".ll"},
 	"LOLCODE":                          {".lol"},
 	"LSL":                              {".lsl", ".lslp"},
+	"LTspice Symbol":                   {".asy"},
 	"LabVIEW":                          {".lvproj"},
 	"Lasso":                            {".lasso", ".las", ".lasso8", ".lasso9", ".ldml"},
 	"Latte":                            {".latte"},
@@ -1251,7 +1324,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"Lex":                              {".l", ".lex"},
 	"LilyPond":                         {".ly", ".ily"},
 	"Limbo":                            {".b", ".m"},
-	"Linker Script":            {".ld", ".lds"},
+	"Linker Script":                    {".ld", ".lds", ".x"},
 	"Linux Kernel Module":              {".mod"},
 	"Liquid":                           {".liquid"},
 	"Literate Agda":                    {".lagda"},
@@ -1262,10 +1335,11 @@ var ExtensionsByLanguage = map[string][]string{
 	"Logtalk":                          {".lgt", ".logtalk"},
 	"LookML":                           {".lookml", ".model.lkml", ".view.lkml"},
 	"LoomScript":                       {".ls"},
-	"Lua":                      {".lua", ".fcgi", ".nse", ".pd_lua", ".rbxs", ".wlua"},
+	"Lua":                              {".lua", ".fcgi", ".nse", ".p8", ".pd_lua", ".rbxs", ".wlua"},
 	"M":                                {".mumps", ".m"},
 	"M4":                               {".m4"},
 	"M4Sugar":                          {".m4"},
+	"MATLAB":                           {".matlab", ".m"},
 	"MAXScript":                        {".ms", ".mcr"},
 	"MQL4":                             {".mq4", ".mqh"},
 	"MQL5":                             {".mq5", ".mqh"},
@@ -1273,19 +1347,19 @@ var ExtensionsByLanguage = map[string][]string{
 	"MUF":                              {".muf", ".m"},
 	"Makefile":                         {".mak", ".d", ".make", ".mk", ".mkfile"},
 	"Mako":                             {".mako", ".mao"},
-	"Markdown":                 {".md", ".markdown", ".mdown", ".mdwn", ".mkd", ".mkdn", ".mkdown", ".ron", ".workbook"},
+	"Markdown":                         {".md", ".markdown", ".mdown", ".mdwn", ".mkd", ".mkdn", ".mkdown", ".ronn", ".workbook"},
 	"Marko":                            {".marko"},
 	"Mask":                             {".mask"},
 	"Mathematica":                      {".mathematica", ".cdf", ".m", ".ma", ".mt", ".nb", ".nbp", ".wl", ".wlt"},
-	"Matlab":                   {".matlab", ".m"},
 	"Max":                              {".maxpat", ".maxhelp", ".maxproj", ".mxt", ".pat"},
 	"MediaWiki":                        {".mediawiki", ".wiki"},
 	"Mercury":                          {".m", ".moo"},
 	"Metal":                            {".metal"},
 	"MiniD":                            {".minid"},
-	"Mirah":                    {".druby", ".duby", ".mir", ".mirah"},
+	"Mirah":                            {".druby", ".duby", ".mirah"},
 	"Modelica":                         {".mo"},
 	"Modula-2":                         {".mod"},
+	"Modula-3":                         {".i3", ".ig", ".m3", ".mg"},
 	"Module Management System":         {".mms", ".mmk"},
 	"Monkey":                           {".monkey", ".monkey2"},
 	"Moocode":                          {".moo"},
@@ -1300,8 +1374,9 @@ var ExtensionsByLanguage = map[string][]string{
 	"NetLinx+ERB":                      {".axs.erb", ".axi.erb"},
 	"NetLogo":                          {".nlogo"},
 	"NewLisp":                          {".nl", ".lisp", ".lsp"},
+	"Nextflow":                         {".nf"},
 	"Nginx":                            {".nginxconf", ".vhost"},
-	"Nim":                      {".nim", ".nimrod"},
+	"Nim":                              {".nim", ".nim.cfg", ".nimble", ".nimrod", ".nims"},
 	"Ninja":                            {".ninja"},
 	"Nit":                              {".nit"},
 	"Nix":                              {".nix"},
@@ -1316,7 +1391,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"Opa":                              {".opa"},
 	"Opal":                             {".opal"},
 	"OpenCL":                           {".cl", ".opencl"},
-	"OpenEdge ABL":             {".p", ".cls"},
+	"OpenEdge ABL":                     {".p", ".cls", ".w"},
 	"OpenSCAD":                         {".scad"},
 	"OpenType Feature File":            {".fea"},
 	"Org":                            {".org"},
@@ -1324,10 +1399,9 @@ var ExtensionsByLanguage = map[string][]string{
 	"Oxygene":                        {".oxygene"},
 	"Oz":                             {".oz"},
 	"P4":                             {".p4"},
-	"PAWN":                           {".pwn", ".inc"},
 	"PHP":                            {".php", ".aw", ".ctp", ".fcgi", ".inc", ".php3", ".php4", ".php5", ".phps", ".phpt"},
 	"PLSQL":                          {".pls", ".bdy", ".ddl", ".fnc", ".pck", ".pkb", ".pks", ".plb", ".plsql", ".prc", ".spc", ".sql", ".tpb", ".tps", ".trg", ".vw"},
-	"PLpgSQL":                        {".sql"},
+	"PLpgSQL":                        {".pgsql", ".sql"},
 	"POV-Ray SDL":                    {".pov", ".inc"},
 	"Pan":                            {".pan"},
 	"Papyrus":                        {".psc"},
@@ -1335,8 +1409,9 @@ var ExtensionsByLanguage = map[string][]string{
 	"Parrot Assembly":                {".pasm"},
 	"Parrot Internal Representation": {".pir"},
 	"Pascal":                       {".pas", ".dfm", ".dpr", ".inc", ".lpr", ".pascal", ".pp"},
+	"Pawn":                         {".pwn", ".inc"},
 	"Pep8":                         {".pep"},
-	"Perl":                         {".pl", ".al", ".cgi", ".fcgi", ".perl", ".ph", ".plx", ".pm", ".pod", ".psgi", ".t"},
+	"Perl":                         {".pl", ".al", ".cgi", ".fcgi", ".perl", ".ph", ".plx", ".pm", ".psgi", ".t"},
 	"Perl 6":                       {".6pl", ".6pm", ".nqp", ".p6", ".p6l", ".p6m", ".pl", ".pl6", ".pm", ".pm6", ".t"},
 	"Pic":                          {".pic", ".chem"},
 	"Pickle":                       {".pkl"},
@@ -1344,8 +1419,10 @@ var ExtensionsByLanguage = map[string][]string{
 	"PigLatin":                     {".pig"},
 	"Pike":                         {".pike", ".pmod"},
 	"Pod":                          {".pod"},
+	"Pod 6":                        {".pod", ".pod6"},
 	"PogoScript":                   {".pogo"},
 	"Pony":                         {".pony"},
+	"PostCSS":                      {".pcss"},
 	"PostScript":                   {".ps", ".eps", ".pfa"},
 	"PowerBuilder":                 {".pbt", ".sra", ".sru", ".srw"},
 	"PowerShell":                   {".ps1", ".psd1", ".psm1"},
@@ -1370,6 +1447,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"REXX":                         {".rexx", ".pprx", ".rex"},
 	"RHTML":                        {".rhtml"},
 	"RMarkdown":                    {".rmd"},
+	"RPC":                          {".x"},
 	"RPM Spec":                     {".spec"},
 	"RUNOFF":                       {".rnh", ".rno"},
 	"Racket":                       {".rkt", ".rktd", ".rktl", ".scrbl"},
@@ -1385,7 +1463,8 @@ var ExtensionsByLanguage = map[string][]string{
 	"RenderScript":                 {".rs", ".rsh"},
 	"Ring":                         {".ring"},
 	"RobotFramework":               {".robot"},
-	"Roff":                         {".man", ".1", ".1in", ".1m", ".1x", ".2", ".3", ".3in", ".3m", ".3qt", ".3x", ".4", ".5", ".6", ".7", ".8", ".9", ".l", ".me", ".ms", ".n", ".nr", ".rno", ".roff", ".tmac"},
+	"Roff":                         {".roff", ".1", ".1in", ".1m", ".1x", ".2", ".3", ".3in", ".3m", ".3p", ".3pm", ".3qt", ".3x", ".4", ".5", ".6", ".7", ".8", ".9", ".l", ".man", ".mdoc", ".me", ".ms", ".n", ".nr", ".rno", ".tmac"},
+	"Roff Manpage":                 {".1", ".1in", ".1m", ".1x", ".2", ".3", ".3in", ".3m", ".3p", ".3pm", ".3qt", ".3x", ".4", ".5", ".6", ".7", ".8", ".9", ".man", ".mdoc"},
 	"Rouge":                        {".rg"},
 	"Ruby":                         {".rb", ".builder", ".eye", ".fcgi", ".gemspec", ".god", ".jbuilder", ".mspec", ".pluginspec", ".podspec", ".rabl", ".rake", ".rbuild", ".rbw", ".rbx", ".ru", ".ruby", ".spec", ".thor", ".watchr"},
 	"Rust":                         {".rs", ".rs.in"},
@@ -1402,7 +1481,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"Sage":                         {".sage", ".sagews"},
 	"SaltStack":                    {".sls"},
 	"Sass":                         {".sass"},
-	"Scala":                        {".scala", ".sbt", ".sc"},
+	"Scala":                        {".scala", ".kojo", ".sbt", ".sc"},
 	"Scaml":                        {".scaml"},
 	"Scheme":                       {".scm", ".sch", ".sld", ".sls", ".sps", ".ss"},
 	"Scilab":                       {".sci", ".sce", ".tst"},
@@ -1412,6 +1491,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"ShellSession":                 {".sh-session"},
 	"Shen":                         {".shen"},
 	"Slash":                        {".sl"},
+	"Slice":                        {".ice"},
 	"Slim":                         {".slim"},
 	"Smali":                        {".smali"},
 	"Smalltalk":                    {".st", ".cs"},
@@ -1424,7 +1504,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"Stata":                        {".do", ".ado", ".doh", ".ihlp", ".mata", ".matah", ".sthlp"},
 	"Stylus":                       {".styl"},
 	"SubRip Text":                  {".srt"},
-	"Sublime Text Config":          {".sublime-build", ".sublime-commands", ".sublime-completions", ".sublime-keymap", ".sublime-macro", ".sublime-menu", ".sublime-mousemap", ".sublime-project", ".sublime-settings", ".sublime-theme", ".sublime-workspace", ".sublime_metrics", ".sublime_session"},
+	"SugarSS":                      {".sss"},
 	"SuperCollider":                {".sc", ".scd"},
 	"Swift":                        {".swift"},
 	"SystemVerilog":                {".sv", ".svh", ".vh"},
@@ -1464,11 +1544,13 @@ var ExtensionsByLanguage = map[string][]string{
 	"Web Ontology Language":        {".owl"},
 	"WebAssembly":                  {".wast", ".wat"},
 	"WebIDL":                       {".webidl"},
+	"Windows Registry Entries":     {".reg"},
 	"World of Warcraft Addon Data": {".toc"},
+	"X BitMap":                     {".xbm"},
+	"X PixMap":                     {".xpm", ".pm"},
 	"X10":                          {".x10"},
 	"XC":                           {".xc"},
-	"XML":              {".xml", ".adml", ".admx", ".ant", ".axml", ".builds", ".ccproj", ".ccxml", ".clixml", ".cproject", ".cscfg", ".csdef", ".csl", ".csproj", ".ct", ".depproj", ".dita", ".ditamap", ".ditaval", ".dll.config", ".dotsettings", ".filters", ".fsproj", ".fxml", ".glade", ".gml", ".grxml", ".iml", ".ivy", ".jelly", ".jsproj", ".kml", ".launch", ".mdpolicy", ".mjml", ".mm", ".mod", ".mxml", ".natvis", ".ndproj", ".nproj", ".nuspec", ".odd", ".osm", ".pkgproj", ".plist", ".pluginspec", ".proj", ".props", ".ps1xml", ".psc1", ".pt", ".rdf", ".resx", ".rss", ".sch", ".scxml", ".sfproj", ".shproj", ".srdf", ".storyboard", ".sttheme", ".sublime-snippet", ".targets", ".tmcommand", ".tml", ".tmlanguage", ".tmpreferences", ".tmsnippet", ".tmtheme", ".ts", ".tsx", ".ui", ".urdf", ".ux", ".vbproj", ".vcxproj", ".vsixmanifest", ".vssettings", ".vstemplate", ".vxml", ".wixproj", ".wsdl", ".wsf", ".wxi", ".wxl", ".wxs", ".x3d", ".xacro", ".xaml", ".xib", ".xlf", ".xliff", ".xmi", ".xml.dist", ".xproj", ".xsd", ".xspec", ".xul", ".zcml"},
+	"XML":                          {".xml", ".adml", ".admx", ".ant", ".axml", ".builds", ".ccproj", ".ccxml", ".clixml", ".cproject", ".cscfg", ".csdef", ".csl", ".csproj", ".ct", ".depproj", ".dita", ".ditamap", ".ditaval", ".dll.config", ".dotsettings", ".filters", ".fsproj", ".fxml", ".glade", ".gml", ".gmx", ".grxml", ".iml", ".ivy", ".jelly", ".jsproj", ".kml", ".launch", ".mdpolicy", ".mjml", ".mm", ".mod", ".mxml", ".natvis", ".ncl", ".ndproj", ".nproj", ".nuspec", ".odd", ".osm", ".pkgproj", ".plist", ".pluginspec", ".proj", ".props", ".ps1xml", ".psc1", ".pt", ".rdf", ".resx", ".rss", ".sch", ".scxml", ".sfproj", ".shproj", ".srdf", ".storyboard", ".sttheme", ".sublime-snippet", ".targets", ".tmcommand", ".tml", ".tmlanguage", ".tmpreferences", ".tmsnippet", ".tmtheme", ".ts", ".tsx", ".ui", ".urdf", ".ux", ".vbproj", ".vcxproj", ".vsixmanifest", ".vssettings", ".vstemplate", ".vxml", ".wixproj", ".workflow", ".wsdl", ".wsf", ".wxi", ".wxl", ".wxs", ".x3d", ".xacro", ".xaml", ".xib", ".xlf", ".xliff", ".xmi", ".xml.dist", ".xproj", ".xsd", ".xspec", ".xul", ".zcml"},
-	"XPM":              {".xpm", ".pm"},
 	"XPages":                       {".xsp-config", ".xsp.metadata"},
 	"XProc":                        {".xpl", ".xproc"},
 	"XQuery":                       {".xquery", ".xq", ".xql", ".xqm", ".xqy"},
@@ -1476,19 +1558,26 @@ var ExtensionsByLanguage = map[string][]string{
 	"XSLT":                         {".xslt", ".xsl"},
 	"Xojo":                         {".xojo_code", ".xojo_menu", ".xojo_report", ".xojo_script", ".xojo_toolbar", ".xojo_window"},
 	"Xtend":                        {".xtend"},
-	"YAML":             {".yml", ".reek", ".rviz", ".sublime-syntax", ".syntax", ".yaml", ".yaml-tmlanguage", ".yml.mysql"},
+	"YAML":                         {".yml", ".mir", ".reek", ".rviz", ".sublime-syntax", ".syntax", ".yaml", ".yaml-tmlanguage", ".yml.mysql"},
 	"YANG":                         {".yang"},
+	"YARA":                         {".yar", ".yara"},
+	"YASnippet":                    {".yasnippet"},
 	"Yacc":                         {".y", ".yacc", ".yy"},
 	"Zephir":                       {".zep"},
+	"Zig":                          {".zig"},
 	"Zimpl":                        {".zimpl", ".zmpl", ".zpl"},
 	"desktop":                      {".desktop", ".desktop.in"},
 	"eC":                           {".ec", ".eh"},
 	"edn":                          {".edn"},
 	"fish":                         {".fish"},
 	"mupad":                        {".mu"},
+	"nanorc":                       {".nanorc"},
 	"nesC":                         {".nc"},
 	"ooc":                          {".ooc"},
+	"q":                            {".q"},
 	"reStructuredText":             {".rst", ".rest", ".rest.txt", ".rst.txt"},
+	"sed":                          {".sed"},
+	"wdl":                          {".wdl"},
 	"wisp":                         {".wisp"},
 	"xBase":                        {".prg", ".ch", ".prw"},
 }

data/filename.go

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -8,41 +8,65 @@ var LanguagesByFilename = map[string][]string{
 	".XCompose":          {"XCompose"},
 	".abbrev_defs":       {"Emacs Lisp"},
 	".arcconfig":         {"JSON"},
-	".babelrc":           {"JSON5"},
+	".atomignore":        {"Ignore List"},
+	".babelignore":       {"Ignore List"},
+	".babelrc":           {"JSON with Comments"},
+	".bash_aliases":      {"Shell"},
 	".bash_history":      {"Shell"},
 	".bash_logout":       {"Shell"},
 	".bash_profile":      {"Shell"},
 	".bashrc":            {"Shell"},
+	".bzrignore":         {"Ignore List"},
 	".clang-format":      {"YAML"},
 	".clang-tidy":        {"YAML"},
 	".classpath":         {"XML"},
+	".coffeelintignore":  {"Ignore List"},
 	".cproject":          {"XML"},
 	".cshrc":             {"Shell"},
+	".cvsignore":         {"Ignore List"},
+	".dockerignore":      {"Ignore List"},
 	".editorconfig":      {"INI"},
 	".emacs":             {"Emacs Lisp"},
 	".emacs.desktop":     {"Emacs Lisp"},
+	".eslintignore":      {"Ignore List"},
+	".eslintrc.json":     {"JSON with Comments"},
 	".factor-boot-rc":    {"Factor"},
 	".factor-rc":         {"Factor"},
 	".gclient":           {"Python"},
 	".gemrc":             {"YAML"},
-	".gitconfig":         {"INI"},
+	".gitattributes":     {"Git Attributes"},
+	".gitconfig":         {"Git Config"},
+	".gitignore":         {"Ignore List"},
+	".gitmodules":        {"Git Config"},
 	".gn":                {"GN"},
 	".gnus":              {"Emacs Lisp"},
 	".gvimrc":            {"Vim script"},
 	".htaccess":          {"ApacheConf"},
+	".htmlhintrc":        {"JSON"},
 	".irbrc":             {"Ruby"},
-	".jshintrc":          {"JSON"},
+	".jscsrc":            {"JSON with Comments"},
+	".jshintrc":          {"JSON with Comments"},
+	".jslintrc":          {"JSON with Comments"},
 	".login":             {"Shell"},
+	".nanorc":            {"nanorc"},
+	".nodemonignore":     {"Ignore List"},
+	".npmignore":         {"Ignore List"},
 	".nvimrc":            {"Vim script"},
 	".php":               {"PHP"},
 	".php_cs":            {"PHP"},
 	".php_cs.dist":       {"PHP"},
+	".prettierignore":    {"Ignore List"},
 	".profile":           {"Shell"},
 	".project":           {"XML"},
 	".pryrc":             {"Ruby"},
 	".spacemacs":         {"Emacs Lisp"},
+	".stylelintignore":   {"Ignore List"},
+	".tern-config":       {"JSON"},
+	".tern-project":      {"JSON"},
 	".vimrc":             {"Vim script"},
 	".viper":             {"Emacs Lisp"},
+	".vscodeignore":      {"Ignore List"},
+	".watchmanconfig":    {"JSON"},
 	".zlogin":            {"Shell"},
 	".zlogout":           {"Shell"},
 	".zprofile":          {"Shell"},
@@ -55,6 +79,7 @@ var LanguagesByFilename = map[string][]string{
 	"BSDmakefile":        {"Makefile"},
 	"BUCK":               {"Python"},
 	"BUILD":              {"Python"},
+	"BUILD.bazel":        {"Python"},
 	"Berksfile":          {"Ruby"},
 	"Brewfile":           {"Ruby"},
 	"Buildfile":          {"Ruby"},
@@ -64,6 +89,7 @@ var LanguagesByFilename = map[string][]string{
 	"COPYRIGHT.regex":    {"Text"},
 	"Cakefile":           {"CoffeeScript"},
 	"Capfile":            {"Ruby"},
+	"Cargo.lock":         {"TOML"},
 	"Cask":               {"Emacs Lisp"},
 	"Dangerfile":         {"Ruby"},
 	"Deliverfile":        {"Ruby"},
@@ -75,6 +101,7 @@ var LanguagesByFilename = map[string][]string{
 	"GNUmakefile":        {"Makefile"},
 	"Gemfile":            {"Ruby"},
 	"Gemfile.lock":       {"Ruby"},
+	"Gopkg.lock":         {"TOML"},
 	"Guardfile":          {"Ruby"},
 	"INSTALL":            {"Text"},
 	"INSTALL.mysql":      {"Text"},
@@ -85,6 +112,7 @@ var LanguagesByFilename = map[string][]string{
 	"LICENSE":            {"Text"},
 	"LICENSE.mysql":      {"Text"},
 	"Makefile":           {"Makefile"},
+	"Makefile.PL":        {"Perl"},
 	"Makefile.am":        {"Makefile"},
 	"Makefile.boot":      {"Makefile"},
 	"Makefile.frag":      {"Makefile"},
@@ -107,7 +135,7 @@ var LanguagesByFilename = map[string][]string{
 	"README.mysql":       {"Text"},
 	"ROOT":               {"Isabelle ROOT"},
 	"Rakefile":           {"Ruby"},
-	"Rexfile":            {"Perl 6"},
+	"Rexfile":            {"Perl"},
 	"SConscript":         {"Python"},
 	"SConstruct":         {"Python"},
 	"Settings.StyleCop":  {"XML"},
@@ -127,6 +155,7 @@ var LanguagesByFilename = map[string][]string{
 	"ack":                {"Perl"},
 	"ant.xml":            {"Ant Build System"},
 	"apache2.conf":       {"ApacheConf"},
+	"bash_aliases":       {"Shell"},
 	"bash_logout":        {"Shell"},
 	"bash_profile":       {"Shell"},
 	"bashrc":             {"Shell"},
@@ -136,19 +165,34 @@ var LanguagesByFilename = map[string][]string{
 	"click.me":           {"Text"},
 	"composer.lock":      {"JSON"},
 	"configure.ac":       {"M4Sugar"},
+	"contents.lr":        {"Markdown"},
 	"cpanfile":           {"Perl"},
 	"cshrc":              {"Shell"},
 	"delete.me":          {"Text"},
 	"descrip.mmk":        {"Module Management System"},
 	"descrip.mms":        {"Module Management System"},
+	"encodings.dir":      {"X Font Directory Index"},
 	"expr-dist":          {"R"},
+	"firestore.rules":    {"Cloud Firestore Security Rules"},
+	"fonts.alias":        {"X Font Directory Index"},
+	"fonts.dir":          {"X Font Directory Index"},
+	"fonts.scale":        {"X Font Directory Index"},
 	"fp-lib-table":       {"KiCad Layout"},
+	"gitignore-global":   {"Ignore List"},
+	"gitignore_global":   {"Ignore List"},
+	"glide.lock":         {"YAML"},
+	"go.mod":             {"Text"},
+	"go.sum":             {"Text"},
 	"gradlew":            {"Shell"},
 	"gvimrc":             {"Vim script"},
+	"haproxy.cfg":        {"HAProxy"},
 	"httpd.conf":         {"ApacheConf"},
+	"jsconfig.json":      {"JSON with Comments"},
 	"keep.me":            {"Text"},
 	"ld.script":          {"Linker Script"},
 	"login":              {"Shell"},
+	"m3makefile":         {"Quake"},
+	"m3overrides":        {"Quake"},
 	"makefile":           {"Makefile"},
 	"makefile.sco":       {"Makefile"},
 	"man":                {"Shell"},
@@ -159,7 +203,10 @@ var LanguagesByFilename = map[string][]string{
 	"mkfile":             {"Makefile"},
 	"mmn":                {"Roff"},
 	"mmt":                {"Roff"},
+	"nanorc":             {"nanorc"},
+	"nextflow.config":    {"Nextflow"},
 	"nginx.conf":         {"Nginx"},
+	"nim.cfg":            {"Nim"},
 	"nvimrc":             {"Vim script"},
 	"owh":                {"Tcl"},
 	"packages.config":    {"XML"},
@@ -171,9 +218,9 @@ var LanguagesByFilename = map[string][]string{
 	"rebar.config.lock":  {"Erlang"},
 	"rebar.lock":         {"Erlang"},
 	"riemann.config":     {"Clojure"},
-	"script":             {"C"},
 	"starfield":          {"Tcl"},
 	"test.me":            {"Text"},
+	"tsconfig.json":      {"JSON with Comments"},
 	"vimrc":              {"Vim script"},
 	"wscript":            {"Python"},
 	"xcompose":           {"XCompose"},

data/heuristics.go

@@ -0,0 +1,35 @@
+package data
+
+import "gopkg.in/src-d/enry.v1/data/rule"
+
+// Heuristics implements a rule-based content matching engine.
+
+// Heuristics is a number of sequntially applied rule.Heuristic where a
+// matching one disambiguages language(s) for a single file extension.
+type Heuristics []rule.Heuristic
+
+// Match returns languages identified by the matching rule of the heuristic.
+func (hs Heuristics) Match(data []byte) []string {
+	var matchedLangs []string
+	for _, heuristic := range hs {
+		if heuristic.Match(data) {
+			for _, langOrAlias := range heuristic.Languages() {
+				lang, ok := LanguageByAlias(langOrAlias)
+				if !ok { // should never happen
+					// reaching here means language name/alias in heuristics.yml
+					// is not consistent with languages.yml
+					// but we do not surface any such error at the API
+					continue
+				}
+				matchedLangs = append(matchedLangs, lang)
+			}
+			break
+		}
+	}
+	return matchedLangs
+}
+
+// matchString is a convenience used only in tests.
+func (hs *Heuristics) matchString(data string) []string {
+	return hs.Match([]byte(data))
+}

data/heuristics_test.go

@@ -0,0 +1,61 @@
+package data
+
+import (
+	"regexp"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"gopkg.in/src-d/enry.v1/data/rule"
+)
+
+var testContentHeuristics = map[string]*Heuristics{
+	".md": &Heuristics{ // final pattern for parsed YAML rule
+		rule.Or(
+			rule.MatchingLanguages("Markdown"),
+			regexp.MustCompile(`(^[-A-Za-z0-9=#!\*\[|>])|<\/ | \A\z`),
+		),
+		rule.Or(
+			rule.MatchingLanguages("GCC Machine Description"),
+			regexp.MustCompile(`^(;;|\(define_)`),
+		),
+		rule.Always(
+			rule.MatchingLanguages("Markdown"),
+		),
+	},
+	".ms": &Heuristics{
+		// Order defines precedence: And, Or, Not, Named, Always
+		rule.And(
+			rule.MatchingLanguages("Unix Assembly"),
+			rule.Not(rule.MatchingLanguages(""), regexp.MustCompile(`/\*`)),
+			rule.Or(
+				rule.MatchingLanguages(""),
+				regexp.MustCompile(`^\s*\.(?:include\s|globa?l\s|[A-Za-z][_A-Za-z0-9]*:)`),
+			),
+		),
+		rule.Or(
+			rule.MatchingLanguages("Roff"),
+			regexp.MustCompile(`^[.''][A-Za-z]{2}(\s|$)`),
+		),
+		rule.Always(
+			rule.MatchingLanguages("MAXScript"),
+		),
+	},
+}
+
+func TestContentHeuristic_MatchingAlways(t *testing.T) {
+	lang := testContentHeuristics[".md"].matchString("")
+	assert.Equal(t, []string{"Markdown"}, lang)
+
+	lang = testContentHeuristics[".ms"].matchString("")
+	assert.Equal(t, []string{"MAXScript"}, lang)
+}
+
+func TestContentHeuristic_MatchingAnd(t *testing.T) {
+	lang := testContentHeuristics[".md"].matchString(";;")
+	assert.Equal(t, []string{"GCC Machine Description"}, lang)
+}
+
+func TestContentHeuristic_MatchingOr(t *testing.T) {
+	lang := testContentHeuristics[".ms"].matchString("	.include \"math.s\"")
+	assert.Equal(t, []string{"Unix Assembly"}, lang)
+}

data/interpreter.go

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -8,6 +8,7 @@ var LanguagesByInterpreter = map[string][]string{
 	"apl":         {"APL"},
 	"aplx":        {"APL"},
 	"ash":         {"Shell"},
+	"asy":         {"Asymptote"},
 	"awk":         {"Awk"},
 	"bash":        {"Shell"},
 	"bigloo":      {"Scheme"},
@@ -16,9 +17,11 @@ var LanguagesByInterpreter = map[string][]string{
 	"chicken":     {"Scheme"},
 	"clisp":       {"Common Lisp"},
 	"coffee":      {"CoffeeScript"},
+	"cperl":       {"Perl"},
 	"crystal":     {"Crystal"},
 	"csi":         {"Scheme"},
 	"cvc4":        {"SMT"},
+	"cwl-runner":  {"Common Workflow Language"},
 	"dart":        {"Dart"},
 	"dash":        {"Shell"},
 	"dtrace":      {"DTrace"},
@@ -34,7 +37,9 @@ var LanguagesByInterpreter = map[string][]string{
 	"gnuplot":     {"Gnuplot"},
 	"gosh":        {"Scheme"},
 	"groovy":      {"Groovy"},
+	"gsed":        {"sed"},
 	"guile":       {"Scheme"},
+	"hy":          {"Hy"},
 	"instantfpc":  {"Pascal"},
 	"io":          {"Io"},
 	"ioke":        {"Ioke"},
@@ -50,12 +55,14 @@ var LanguagesByInterpreter = map[string][]string{
 	"make":        {"Makefile"},
 	"mathsat5":    {"SMT"},
 	"mawk":        {"Awk"},
+	"minised":     {"sed"},
 	"mksh":        {"Shell"},
 	"mmi":         {"Mercury"},
 	"moon":        {"MoonScript"},
 	"nawk":        {"Awk"},
 	"newlisp":     {"NewLisp"},
-	"node":        {"JavaScript"},
+	"nextflow":    {"Nextflow"},
+	"node":        {"JavaScript", "TypeScript"},
 	"nush":        {"Nu"},
 	"ocaml":       {"OCaml", "Reason"},
 	"ocamlrun":    {"OCaml"},
@@ -66,11 +73,12 @@ var LanguagesByInterpreter = map[string][]string{
 	"parrot":      {"Parrot Assembly", "Parrot Internal Representation"},
 	"pdksh":       {"Shell"},
 	"perl":        {"Perl", "Pod"},
-	"perl6":       {"Perl 6"},
+	"perl6":       {"Perl 6", "Pod 6"},
 	"php":         {"PHP"},
 	"picolisp":    {"PicoLisp"},
 	"pike":        {"Pike"},
 	"pil":         {"PicoLisp"},
+	"pwsh":        {"PowerShell"},
 	"python":      {"Python"},
 	"python2":     {"Python"},
 	"python3":     {"Python"},
@@ -87,11 +95,14 @@ var LanguagesByInterpreter = map[string][]string{
 	"runhaskell":  {"Haskell"},
 	"sbcl":        {"Common Lisp"},
 	"scala":       {"Scala"},
+	"scheme":      {"Scheme"},
 	"sclang":      {"SuperCollider"},
 	"scsynth":     {"SuperCollider"},
+	"sed":         {"sed"},
 	"sh":          {"Shell"},
 	"smt-rat":     {"SMT"},
 	"smtinterpol": {"SMT"},
+	"ssed":        {"sed"},
 	"stp":         {"SMT"},
 	"swipl":       {"Prolog"},
 	"tcc":         {"C"},

data/mimeType.go

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -9,9 +9,10 @@ var LanguagesMime = map[string]string{
 	"ASN.1":                          "text/x-ttcn-asn",
 	"ASP":                            "application/x-aspx",
 	"Alpine Abuild":                  "text/x-sh",
+	"AngelScript":                    "text/x-c++src",
 	"Ant Build System":               "application/xml",
 	"Apex":                           "text/x-java",
-	"Arduino":                 "text/x-c++src",
+	"Asymptote":                      "text/x-kotlin",
 	"Brainfuck":                      "text/x-brainfuck",
 	"C":                              "text/x-csrc",
 	"C#":                             "text/x-csharp",
@@ -25,8 +26,10 @@ var LanguagesMime = map[string]string{
 	"ChucK":                          "text/x-java",
 	"Clojure":                        "text/x-clojure",
 	"Closure Templates":              "text/x-soy",
+	"Cloud Firestore Security Rules": "text/css",
 	"CoffeeScript":                   "text/x-coffeescript",
 	"Common Lisp":                    "text/x-common-lisp",
+	"Common Workflow Language":       "text/x-yaml",
 	"Component Pascal":               "text/x-pascal",
 	"Crystal":                        "text/x-crystal",
 	"Cuda":                           "text/x-c++src",
@@ -44,6 +47,7 @@ var LanguagesMime = map[string]string{
 	"Eagle":                          "text/xml",
 	"Easybuild":                      "text/x-python",
 	"Ecere Projects":                 "application/json",
+	"Edje Data Collection":           "application/json",
 	"Eiffel":                         "text/x-eiffel",
 	"Elm":                            "text/x-elm",
 	"Emacs Lisp":                     "text/x-common-lisp",
@@ -59,6 +63,8 @@ var LanguagesMime = map[string]string{
 	"Genshi":              "text/xml",
 	"Gentoo Ebuild":       "text/x-sh",
 	"Gentoo Eclass":       "text/x-sh",
+	"Git Attributes":      "text/x-sh",
+	"Git Config":          "text/x-properties",
 	"Glyph":               "text/x-tcl",
 	"Go":                  "text/x-go",
 	"Grammatical Framework": "text/x-haskell",
@@ -71,6 +77,7 @@ var LanguagesMime = map[string]string{
 	"HTML+EEX":                 "text/html",
 	"HTML+ERB":                 "application/x-erb",
 	"HTML+PHP":                 "application/x-httpd-php",
+	"HTML+Razor":               "text/html",
 	"HTTP":                     "message/http",
 	"Hack":                     "application/x-httpd-php",
 	"Haml":                     "text/x-haml",
@@ -79,11 +86,15 @@ var LanguagesMime = map[string]string{
 	"IDL":                      "text/x-idl",
 	"INI":                      "text/x-properties",
 	"IRC log":                  "text/mirc",
+	"Ignore List":              "text/x-sh",
 	"JSON":                     "application/json",
+	"JSON with Comments":       "text/javascript",
 	"JSON5":                    "application/json",
+	"JSONLD":                   "application/json",
 	"JSONiq":                   "application/json",
 	"JSX":                      "text/jsx",
 	"Java":                     "text/x-java",
+	"Java Properties":          "text/x-properties",
 	"Java Server Pages":        "application/x-jsp",
 	"JavaScript":               "text/javascript",
 	"Julia":                    "text/x-julia",
@@ -92,6 +103,7 @@ var LanguagesMime = map[string]string{
 	"Kit":                      "text/html",
 	"Kotlin":                   "text/x-kotlin",
 	"LFE":                      "text/x-common-lisp",
+	"LTspice Symbol":           "text/x-spreadsheet",
 	"LabVIEW":                  "text/xml",
 	"Latte":                    "text/x-smarty",
 	"Less":                     "text/css",
@@ -100,13 +112,13 @@ var LanguagesMime = map[string]string{
 	"LookML":                   "text/x-yaml",
 	"Lua":                      "text/x-lua",
 	"M":                        "text/x-mumps",
+	"MATLAB":                   "text/x-octave",
 	"MTML":                     "text/html",
 	"MUF":                      "text/x-forth",
 	"Makefile":                 "text/x-cmake",
 	"Markdown":                 "text/x-gfm",
 	"Marko":                    "text/html",
 	"Mathematica":              "text/x-mathematica",
-	"Matlab":              "text/x-octave",
 	"Maven POM":                "text/xml",
 	"Max":                      "application/json",
 	"Metal":                    "text/x-c++src",
@@ -146,6 +158,7 @@ var LanguagesMime = map[string]string{
 	"RPM Spec":                 "text/x-rpm-spec",
 	"Reason":                   "text/x-rustsrc",
 	"Roff":                     "text/troff",
+	"Roff Manpage":             "text/troff",
 	"Rouge":                    "text/x-clojure",
 	"Ruby":                     "text/x-ruby",
 	"Rust":                     "text/x-rustsrc",
@@ -168,7 +181,6 @@ var LanguagesMime = map[string]string{
 	"Smarty":                   "text/x-smarty",
 	"Squirrel":                 "text/x-c++src",
 	"Standard ML":              "text/x-ocaml",
-	"Sublime Text Config": "text/javascript",
 	"Swift":                    "text/x-swift",
 	"SystemVerilog":            "text/x-systemverilog",
 	"TOML":                     "text/x-toml",
@@ -190,6 +202,9 @@ var LanguagesMime = map[string]string{
 	"Volt":                     "text/x-d",
 	"WebAssembly":              "text/x-common-lisp",
 	"WebIDL":                   "text/x-webidl",
+	"Windows Registry Entries": "text/x-properties",
+	"X BitMap":                 "text/x-csrc",
+	"X PixMap":                 "text/x-csrc",
 	"XC":                       "text/x-csrc",
 	"XML":                      "text/xml",
 	"XPages":                   "text/xml",

data/rule/rule.go

@@ -0,0 +1,109 @@
+// Package rule contains rule-based heuristic implementations.
+// It is used in the generated code in content.go for disambiguation of languages
+// with colliding extensions, based on regexps from Linguist data.
+package rule
+
+// Heuristic consist of (a number of) rules where each, if matches,
+// identifes content as belonging to a programming language(s).
+type Heuristic interface {
+	Matcher
+	Languages() []string
+}
+
+// Matcher checks if the data matches (number of) pattern.
+// Every heuristic rule below implements this interface.
+// A regexp.Regexp satisfies this interface and can be used instead.
+type Matcher interface {
+	Match(data []byte) bool
+}
+
+// languages struct incapsulate data common to every Matcher: all languages
+// that it identifies.
+type languages struct {
+	langs []string
+}
+
+// Languages returns all languages, identified by this Matcher.
+func (l languages) Languages() []string {
+	return l.langs
+}
+
+// MatchingLanguages is a helper to create new languages.
+func MatchingLanguages(langs ...string) languages {
+	return languages{langs}
+}
+
+// Implements a Heuristic.
+type or struct {
+	languages
+	pattern Matcher
+}
+
+// Or rule matches, if a single matching pattern exists.
+// It recives only one pattern as it relies on compile-time optimization that
+// represtes union with | inside a single regexp.
+func Or(l languages, r Matcher) Heuristic {
+	return or{l, r}
+}
+
+// Match implements rule.Matcher.
+func (r or) Match(data []byte) bool {
+	return r.pattern.Match(data)
+}
+
+// Implements a Heuristic.
+type and struct {
+	languages
+	patterns []Matcher
+}
+
+// And rule matches, if each of the patterns does match.
+func And(l languages, m ...Matcher) Heuristic {
+	return and{l, m}
+}
+
+// Match implements data.Matcher.
+func (r and) Match(data []byte) bool {
+	for _, p := range r.patterns {
+		if !p.Match(data) {
+			return false
+		}
+	}
+	return true
+}
+
+// Implements a Heuristic.
+type not struct {
+	languages
+	Patterns []Matcher
+}
+
+// Not rule matches if none of the patterns match.
+func Not(l languages, r ...Matcher) Heuristic {
+	return not{l, r}
+}
+
+// Match implements data.Matcher.
+func (r not) Match(data []byte) bool {
+	for _, p := range r.Patterns {
+		if p.Match(data) {
+			return false
+		}
+	}
+	return true
+}
+
+// Implements a Heuristic.
+type always struct {
+	languages
+}
+
+// Always rule always matches. Often is used as a default fallback.
+func Always(l languages) Heuristic {
+	return always{l}
+}
+
+// Match implements Matcher.
+func (r always) Match(data []byte) bool {
+	return true
+}

data/rule/rule_test.go

@@ -0,0 +1,39 @@
+package rule
+
+import (
+	"regexp"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+const lang = "ActionScript"
+
+var fixtures = []struct {
+	name     string
+	rule     Heuristic
+	numLangs int
+	matching string
+	noMatch  string
+}{
+	{"Always", Always(MatchingLanguages(lang)), 1, "a", ""},
+	{"Not", Not(MatchingLanguages(lang), regexp.MustCompile(`a`)), 1, "b", "a"},
+	{"And", And(MatchingLanguages(lang), regexp.MustCompile(`a`), regexp.MustCompile(`b`)), 1, "ab", "a"},
+	{"Or", Or(MatchingLanguages(lang), regexp.MustCompile(`a|b`)), 1, "ab", "c"},
+}
+
+func TestRules(t *testing.T) {
+	for _, f := range fixtures {
+		t.Run(f.name, func(t *testing.T) {
+			assert.NotNil(t, f.rule)
+			assert.NotNil(t, f.rule.Languages())
+			assert.Equal(t, f.numLangs, len(f.rule.Languages()))
+			assert.Truef(t, f.rule.Match([]byte(f.matching)),
+				"'%s' is expected to .Match() by rule %s%v", f.matching, f.name, f.rule)
+			if f.noMatch != "" {
+				assert.Falsef(t, f.rule.Match([]byte(f.noMatch)),
+					"'%s' is expected NOT to .Match() by rule %s%v", f.noMatch, f.name, f.rule)
+			}
+		})
+	}
+}

data/type.go

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -21,16 +21,17 @@ var LanguagesType = map[string]int{
 	"Agda":               2,
 	"Alloy":              2,
 	"Alpine Abuild":      2,
+	"AngelScript":        2,
 	"Ant Build System":   1,
 	"ApacheConf":         1,
 	"Apex":               2,
 	"Apollo Guidance Computer":       2,
 	"AppleScript":                    2,
 	"Arc":                            2,
-	"Arduino":                  2,
 	"AsciiDoc":                       4,
 	"AspectJ":                        2,
 	"Assembly":                       2,
+	"Asymptote":                      2,
 	"Augeas":                         2,
 	"AutoHotkey":                     2,
 	"AutoIt":                         2,
@@ -73,10 +74,13 @@ var LanguagesType = map[string]int{
 	"Click":                          2,
 	"Clojure":                        2,
 	"Closure Templates":              3,
+	"Cloud Firestore Security Rules": 1,
+	"CoNLL-U":                        1,
 	"CoffeeScript":                   2,
 	"ColdFusion":                     2,
 	"ColdFusion CFC":                 2,
 	"Common Lisp":                    2,
+	"Common Workflow Language":       2,
 	"Component Pascal":               2,
 	"Cool":                           2,
 	"Coq":                            2,
@@ -99,7 +103,7 @@ var LanguagesType = map[string]int{
 	"Dart":                 2,
 	"DataWeave":            2,
 	"Diff":                 1,
-	"Dockerfile":     1,
+	"Dockerfile":           2,
 	"Dogescript":           2,
 	"Dylan":                2,
 	"E":                    2,
@@ -107,10 +111,12 @@ var LanguagesType = map[string]int{
 	"ECL":                  2,
 	"ECLiPSe":              2,
 	"EJS":                  3,
+	"EML":                  1,
 	"EQ":                   2,
 	"Eagle":                1,
 	"Easybuild":            1,
 	"Ecere Projects":       1,
+	"Edje Data Collection": 1,
 	"Eiffel":               2,
 	"Elixir":               2,
 	"Elm":                  2,
@@ -118,6 +124,8 @@ var LanguagesType = map[string]int{
 	"EmberScript":          2,
 	"Erlang":               2,
 	"F#":                   2,
+	"F*":                   2,
+	"FIGlet Font":          1,
 	"FLUX":                 2,
 	"Factor":               2,
 	"Fancy":                2,
@@ -145,7 +153,10 @@ var LanguagesType = map[string]int{
 	"Gerber Image":        1,
 	"Gettext Catalog":     4,
 	"Gherkin":             2,
+	"Git Attributes":      1,
+	"Git Config":          1,
 	"Glyph":               2,
+	"Glyph Bitmap Distribution Format": 1,
 	"Gnuplot":                          2,
 	"Go":                               2,
 	"Golo":                             2,
@@ -158,6 +169,7 @@ var LanguagesType = map[string]int{
 	"Graphviz (DOT)":                   1,
 	"Groovy":                           2,
 	"Groovy Server Pages":              2,
+	"HAProxy":                          1,
 	"HCL":                              2,
 	"HLSL":                             2,
 	"HTML":                             3,
@@ -166,13 +178,16 @@ var LanguagesType = map[string]int{
 	"HTML+EEX":                         3,
 	"HTML+ERB":                         3,
 	"HTML+PHP":                         3,
+	"HTML+Razor":                       3,
 	"HTTP":                             1,
+	"HXML":                             1,
 	"Hack":                             2,
 	"Haml":                             3,
 	"Handlebars":                       3,
 	"Harbour":                          2,
 	"Haskell":                          2,
 	"Haxe":                             2,
+	"HiveQL":                           2,
 	"Hy":                               2,
 	"HyPhy":                            2,
 	"IDL":                              2,
@@ -180,6 +195,7 @@ var LanguagesType = map[string]int{
 	"INI":                              1,
 	"IRC log":                          1,
 	"Idris":                            2,
+	"Ignore List":                      1,
 	"Inform 7":                         2,
 	"Inno Setup":                       2,
 	"Io":                               2,
@@ -189,12 +205,14 @@ var LanguagesType = map[string]int{
 	"J":                                2,
 	"JFlex":                            2,
 	"JSON":                             1,
+	"JSON with Comments":               1,
 	"JSON5":                            1,
 	"JSONLD":                           1,
 	"JSONiq":                           2,
 	"JSX":                              2,
 	"Jasmin":                           2,
 	"Java":                             2,
+	"Java Properties":                  1,
 	"Java Server Pages":                2,
 	"JavaScript":                       2,
 	"Jison":                            2,
@@ -212,6 +230,7 @@ var LanguagesType = map[string]int{
 	"LLVM":                             2,
 	"LOLCODE":                          2,
 	"LSL":                              2,
+	"LTspice Symbol":                   1,
 	"LabVIEW":                          2,
 	"Lasso":                            2,
 	"Latte":                            3,
@@ -235,6 +254,7 @@ var LanguagesType = map[string]int{
 	"M":                                2,
 	"M4":                               2,
 	"M4Sugar":                          2,
+	"MATLAB":                           2,
 	"MAXScript":                        2,
 	"MQL4":                             2,
 	"MQL5":                             2,
@@ -246,7 +266,6 @@ var LanguagesType = map[string]int{
 	"Marko":                            3,
 	"Mask":                             3,
 	"Mathematica":                      2,
-	"Matlab":                   2,
 	"Maven POM":                        1,
 	"Max":                              2,
 	"MediaWiki":                        4,
@@ -257,6 +276,7 @@ var LanguagesType = map[string]int{
 	"Mirah":                            2,
 	"Modelica":                         2,
 	"Modula-2":                         2,
+	"Modula-3":                         2,
 	"Module Management System":         2,
 	"Monkey":                           2,
 	"Moocode":                          2,
@@ -271,6 +291,7 @@ var LanguagesType = map[string]int{
 	"NetLinx+ERB":                      2,
 	"NetLogo":                          2,
 	"NewLisp":                          2,
+	"Nextflow":                         2,
 	"Nginx":                            1,
 	"Nim":                              2,
 	"Ninja":                            1,
@@ -296,7 +317,6 @@ var LanguagesType = map[string]int{
 	"Oxygene":                        2,
 	"Oz":                             2,
 	"P4":                             2,
-	"PAWN":                           2,
 	"PHP":                            2,
 	"PLSQL":                          2,
 	"PLpgSQL":                        2,
@@ -307,6 +327,7 @@ var LanguagesType = map[string]int{
 	"Parrot Assembly":                2,
 	"Parrot Internal Representation": 2,
 	"Pascal":                       2,
+	"Pawn":                         2,
 	"Pep8":                         2,
 	"Perl":                         2,
 	"Perl 6":                       2,
@@ -316,8 +337,10 @@ var LanguagesType = map[string]int{
 	"PigLatin":                     2,
 	"Pike":                         2,
 	"Pod":                          4,
+	"Pod 6":                        4,
 	"PogoScript":                   2,
 	"Pony":                         2,
+	"PostCSS":                      3,
 	"PostScript":                   3,
 	"PowerBuilder":                 2,
 	"PowerShell":                   2,
@@ -336,6 +359,7 @@ var LanguagesType = map[string]int{
 	"Python traceback":             1,
 	"QML":                          2,
 	"QMake":                        2,
+	"Quake":                        2,
 	"R":                            2,
 	"RAML":                         3,
 	"RDoc":                         4,
@@ -343,6 +367,7 @@ var LanguagesType = map[string]int{
 	"REXX":                         2,
 	"RHTML":                        3,
 	"RMarkdown":                    4,
+	"RPC":                          2,
 	"RPM Spec":                     1,
 	"RUNOFF":                       3,
 	"Racket":                       2,
@@ -359,6 +384,7 @@ var LanguagesType = map[string]int{
 	"Ring":                         2,
 	"RobotFramework":               2,
 	"Roff":                         3,
+	"Roff Manpage":                 3,
 	"Rouge":                        2,
 	"Ruby":                         2,
 	"Rust":                         2,
@@ -385,10 +411,12 @@ var LanguagesType = map[string]int{
 	"ShellSession":                 2,
 	"Shen":                         2,
 	"Slash":                        2,
+	"Slice":                        2,
 	"Slim":                         3,
 	"Smali":                        2,
 	"Smalltalk":                    2,
 	"Smarty":                       2,
+	"Solidity":                     2,
 	"SourcePawn":                   2,
 	"Spline Font Database":         1,
 	"Squirrel":                     2,
@@ -397,7 +425,7 @@ var LanguagesType = map[string]int{
 	"Stata":                        2,
 	"Stylus":                       3,
 	"SubRip Text":                  1,
-	"Sublime Text Config":          1,
+	"SugarSS":                      3,
 	"SuperCollider":                2,
 	"Swift":                        2,
 	"SystemVerilog":                2,
@@ -437,12 +465,15 @@ var LanguagesType = map[string]int{
 	"Web Ontology Language":        1,
 	"WebAssembly":                  2,
 	"WebIDL":                       2,
+	"Windows Registry Entries":     1,
 	"World of Warcraft Addon Data": 1,
+	"X BitMap":                     1,
+	"X Font Directory Index":       1,
+	"X PixMap":                     1,
 	"X10":                          2,
 	"XC":                           2,
 	"XCompose":                     1,
 	"XML":                          1,
-	"XPM":              1,
 	"XPages":                       1,
 	"XProc":                        2,
 	"XQuery":                       2,
@@ -452,17 +483,24 @@ var LanguagesType = map[string]int{
 	"Xtend":                        2,
 	"YAML":                         1,
 	"YANG":                         1,
+	"YARA":                         2,
+	"YASnippet":                    3,
 	"Yacc":                         2,
 	"Zephir":                       2,
+	"Zig":                          2,
 	"Zimpl":                        2,
 	"desktop":                      1,
 	"eC":                           2,
 	"edn":                          1,
 	"fish":                         2,
 	"mupad":                        2,
+	"nanorc":                       1,
 	"nesC":                         2,
 	"ooc":                          2,
+	"q":                            2,
 	"reStructuredText":             4,
+	"sed":                          2,
+	"wdl":                          2,
 	"wisp":                         2,
 	"xBase":                        2,
 }

data/vendor.go

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: 4cd558c37482e8d2c535d8107f2d11b49afbc5b5
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -10,7 +10,6 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`^[Dd]ependencies/`),
 	substring.Regexp(`(^|/)dist/`),
 	substring.Regexp(`^deps/`),
-	substring.Regexp(`^tools/`),
 	substring.Regexp(`(^|/)configure$`),
 	substring.Regexp(`(^|/)config.guess$`),
 	substring.Regexp(`(^|/)config.sub$`),
@@ -32,13 +31,15 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`(^|/)bootstrap([^.]*)\.(js|css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)custom\.bootstrap([^\s]*)(js|css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)font-awesome\.(css|less|scss|styl)$`),
+	substring.Regexp(`(^|/)font-awesome/.*\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)foundation\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)normalize\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)skeleton\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)[Bb]ourbon/.*\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)animate\.(css|less|scss|styl)$`),
-	substring.Regexp(`third[-_]?party/`),
+	substring.Regexp(`(^|/)materialize\.(css|less|scss|styl|js)$`),
-	substring.Regexp(`3rd[-_]?party/`),
+	substring.Regexp(`(^|/)select2/.*\.(css|scss|js)$`),
+	substring.Regexp(`(3rd|[Tt]hird)[-_]?[Pp]arty/`),
 	substring.Regexp(`vendors?/`),
 	substring.Regexp(`extern(al)?/`),
 	substring.Regexp(`(^|/)[Vv]+endor/`),
@@ -53,6 +54,9 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`jquery.fancybox.(js|css)`),
 	substring.Regexp(`fuelux.js`),
 	substring.Regexp(`(^|/)jquery\.fileupload(-\w+)?\.js$`),
+	substring.Regexp(`jquery.dataTables.js`),
+	substring.Regexp(`bootbox.js`),
+	substring.Regexp(`pdf.worker.js`),
 	substring.Regexp(`(^|/)slick\.\w+.js$`),
 	substring.Regexp(`(^|/)Leaflet\.Coordinates-\d+\.\d+\.\d+\.src\.js$`),
 	substring.Regexp(`leaflet.draw-src.js`),
@@ -63,6 +67,7 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`wicket-leaflet.js`),
 	substring.Regexp(`.sublime-project`),
 	substring.Regexp(`.sublime-workspace`),
+	substring.Regexp(`.vscode`),
 	substring.Regexp(`(^|/)prototype(.*)\.js$`),
 	substring.Regexp(`(^|/)effects\.js$`),
 	substring.Regexp(`(^|/)controls\.js$`),
@@ -99,8 +104,7 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`^.osx$`),
 	substring.Regexp(`\.xctemplate/`),
 	substring.Regexp(`\.imageset/`),
-	substring.Regexp(`^Carthage/`),
+	substring.Regexp(`(^|/)Carthage/`),
-	substring.Regexp(`^Pods/`),
 	substring.Regexp(`(^|/)Sparkle/`),
 	substring.Regexp(`Crashlytics.framework/`),
 	substring.Regexp(`Fabric.framework/`),
@@ -113,6 +117,9 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`(^|/)gradlew$`),
 	substring.Regexp(`(^|/)gradlew\.bat$`),
 	substring.Regexp(`(^|/)gradle/wrapper/`),
+	substring.Regexp(`(^|/)mvnw$`),
+	substring.Regexp(`(^|/)mvnw\.cmd$`),
+	substring.Regexp(`(^|/)\.mvn/wrapper/`),
 	substring.Regexp(`-vsdoc\.js$`),
 	substring.Regexp(`\.intellisense\.js$`),
 	substring.Regexp(`(^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$`),

internal/code-generator/assets/alias.go.tmpl

@@ -1,9 +1,34 @@
 package data
 
-// LanguagesByAlias keeps alias for different languages and use the name of the languages as an alias too.
+import "strings"
+
+// LanguageByAliasMap keeps alias for different languages and use the name of the languages as an alias too.
 // All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores.
-var LanguagesByAlias = map[string]string{
+var LanguageByAliasMap = map[string]string{
 	{{range $alias, $language := . -}}
 		"{{ $alias }}":	{{ printf "%q" $language -}},
 	{{end -}}
 }
+
+// LanguageByAlias looks up the language name by it's alias or name.
+// It mirrors the logic of github linguist and is needed e.g for heuristcs.yml
+// that mixes names and aliases in a language field (see XPM example).
+func LanguageByAlias(langOrAlias string) (lang string, ok bool) {
+	k := convertToAliasKey(langOrAlias)
+	lang, ok = LanguageByAliasMap[k]
+	return
+}
+
+
+// convertToAliasKey converts language name to a key in LanguageByAliasMap.
+// Following
+//  - internal.code-generator.generator.convertToAliasKey()
+//  - GetLanguageByAlias()
+// conventions.
+// It is here to avoid dependency on "generate" and "enry" packages.
+func convertToAliasKey(langName string) string {
+	ak := strings.SplitN(langName, `,`, 2)[0]
+	ak = strings.Replace(ak, ` `, `_`, -1)
+	ak = strings.ToLower(ak)
+	return ak
+}

internal/code-generator/assets/content.go.tmpl

@@ -1,33 +1,51 @@
 package data
 
-import "gopkg.in/toqueteos/substring.v1"
+import (
+	"regexp"
 
-type languageMatcher func ([]byte) []string
+	"gopkg.in/src-d/enry.v1/data/rule"
+)
 
-var ContentMatchers = map[string]languageMatcher{
+var ContentHeuristics = map[string]*Heuristics{
-	{{ range $index, $disambiguator := . -}}
+	{{ range $ext, $rules := . -}}
-	{{ printf "%q" $disambiguator.Extension }}: func(i []byte) []string {
+		{{ printf "%q" $ext }}: &Heuristics{
-		{{ range $i, $language := $disambiguator.Languages -}}
+			{{ range $rule := $rules -}}
-
+				{{template "Rule" $rule}}
-		{{- if not (avoidLanguage $language) }}
+			{{ end -}}
-		{{- if gt (len $language.Heuristics) 0 }}
-		{{- if gt $i 0 }} else {{ end -}} 
-		if {{- range $j, $heuristic := $language.Heuristics }} {{ $heuristic.Name }}.Match(string(i)) 
-		{{- if lt $j (len $language.LogicRelations) }} {{index $language.LogicRelations $j}} {{- end -}} {{ end }} {
-			return []string{ {{- printf "%q" $language.Language -}} }
-		}
-
-		{{- end -}}
-		{{- end -}}
-		{{- end}}
-
-		return {{ returnLanguages $disambiguator.Languages | returnStringSlice }}
 		},
 	{{ end -}}
 }
 
-var (
+{{ define "Rule" -}}
-    {{ range $index, $heuristic := getAllHeuristics . -}}
+	{{ if eq .Op "And" -}}
-    {{ $heuristic.Name }} = substring.Regexp(`{{ $heuristic.Regexp }}`)
+		rule.And(
+			{{ template "Languages" .Langs -}}
+			{{ range $rule := .Rules -}}
+				{{template "Rule" $rule}}
 			{{ end -}}
-)
+		),
+	{{- else if eq .Op "Or" -}}
+		rule.Or(
+			{{ template "Languages" .Langs -}}
+			regexp.MustCompile(`{{ .Pattern }}`),
+		),
+	{{- else if eq .Op "Not" -}}
+		rule.Not(
+			{{ template "Languages" .Langs -}}
+			regexp.MustCompile(`{{ .Pattern }}`),
+		),
+	{{- else if eq .Op "Always" -}}
+		rule.Always(
+			{{ template "Languages" .Langs -}}
+		),
+	{{ end -}}
+{{ end -}}
+
+
+{{define "Languages" -}}
+	{{with . -}}
+		rule.MatchingLanguages( {{range .}} {{printf "\"%s\"" .}}, {{end}} ),
+	{{ else -}}
+		rule.MatchingLanguages(""),
+	{{end -}}
+{{end}}

internal/code-generator/generator/aliases.go

@@ -2,10 +2,11 @@ package generator
 
 import (
 	"bytes"
-	"gopkg.in/yaml.v2"
 	"io"
 	"io/ioutil"
 	"strings"
+
+	"gopkg.in/yaml.v2"
 )
 
 // Aliases reads from fileToParse and builds source file from tmplPath. It complies with type File signature.
@@ -21,10 +22,10 @@ func Aliases(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string
 	}
 
 	orderedLangList := getAlphabeticalOrderedKeys(languages)
-	languagesByAlias := buildAliasLanguageMap(languages, orderedLangList)
+	languageByAlias := buildAliasLanguageMap(languages, orderedLangList)
 
 	buf := &bytes.Buffer{}
-	if err := executeAliasesTemplate(buf, languagesByAlias, tmplPath, tmplName, commit); err != nil {
+	if err := executeAliasesTemplate(buf, languageByAlias, tmplPath, tmplName, commit); err != nil {
 		return err
 	}
 
@@ -52,6 +53,6 @@ func convertToAliasKey(s string) (key string) {
 	return
 }
 
-func executeAliasesTemplate(out io.Writer, languagesByAlias map[string]string, aliasesTmplPath, aliasesTmpl, commit string) error {
+func executeAliasesTemplate(out io.Writer, languageByAlias map[string]string, aliasesTmplPath, aliasesTmpl, commit string) error {
-	return executeTemplate(out, aliasesTmpl, aliasesTmplPath, commit, nil, languagesByAlias)
+	return executeTemplate(out, aliasesTmpl, aliasesTmplPath, commit, nil, languageByAlias)
 }

internal/code-generator/generator/documentation.go

@@ -2,13 +2,14 @@ package generator
 
 import (
 	"bytes"
-	"gopkg.in/yaml.v2"
-	"io"
 	"io/ioutil"
+
+	"gopkg.in/yaml.v2"
 )
 
-// Documentation reads from fileToParse and builds source file from tmplPath. It complies with type File signature.
+// Documentation generates regex matchers in Go for documentation files/dirs.
-func Documentation(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error {
+// It is of generator.File type.
+func Documentation(fileToParse, _, outFile, tmplPath, tmplName, commit string) error {
 	data, err := ioutil.ReadFile(fileToParse)
 	if err != nil {
 		return err
@@ -20,13 +21,10 @@ func Documentation(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit
 	}
 
 	buf := &bytes.Buffer{}
-	if err := executeDocumentationTemplate(buf, regexpList, tmplPath, tmplName, commit); err != nil {
+	err = executeTemplate(buf, tmplName, tmplPath, commit, nil, regexpList)
+	if err != nil {
 		return err
 	}
 
-	return formatedWrite(outPath, buf.Bytes())
+	return formatedWrite(outFile, buf.Bytes())
-}
-
-func executeDocumentationTemplate(out io.Writer, regexpList []string, tmplPath, tmplName, commit string) error {
-	return executeTemplate(out, tmplName, tmplPath, commit, nil, regexpList)
 }

internal/code-generator/generator/generator.go

@@ -1,3 +1,5 @@
+// Package generator provides facilities to generate Go code for the
+// package data in enry from YAML files describing supported languages in Linguist.
 package generator
 
 import (
@@ -9,7 +11,10 @@ import (
 	"text/template"
 )
 
-// File is the function's type that generate source file from a file to be parsed, linguist's samples dir and a template.
+// File is a common type for all generator functions.
+// It generates Go source code file based on template in tmplPath,
+// by parsing the data in fileToParse and linguist's samplesDir
+// saving results to an outFile.
 type File func(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error
 
 func formatedWrite(outPath string, source []byte) error {
@@ -28,16 +33,14 @@ func executeTemplate(w io.Writer, name, path, commit string, fmap template.FuncM
 		return commit
 	}
 
-	buf := bytes.NewBuffer(nil)
-
 	const headerTmpl = "header.go.tmpl"
-
 	headerPath := filepath.Join(filepath.Dir(path), headerTmpl)
 
 	h := template.Must(template.New(headerTmpl).Funcs(template.FuncMap{
 		"getCommit": getCommit,
 	}).ParseFiles(headerPath))
 
+	buf := bytes.NewBuffer(nil)
 	if err := h.Execute(buf, data); err != nil {
 		return err
 	}

internal/code-generator/generator/generator_test.go

@@ -1,6 +1,7 @@
 package generator
 
 import (
+	"flag"
 	"fmt"
 	"io/ioutil"
 	"os"
@@ -15,7 +16,7 @@ import (
 const (
 	linguistURL          = "https://github.com/github/linguist.git"
 	linguistClonedEnvVar = "ENRY_TEST_REPO"
-	commit               = "d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68"
+	commit               = "e4560984058b4726010ca4b8f03ed9d0f8f464db"
 	samplesDir           = "samples"
 	languagesFile        = "lib/linguist/languages.yml"
 
@@ -28,7 +29,7 @@ const (
 	extensionTestTmplName = "extension.go.tmpl"
 
 	// Heuristics test
-	heuristicsTestFile  = "lib/linguist/heuristics.rb"
+	heuristicsTestFile  = "lib/linguist/heuristics.yml"
 	contentGold         = testDir + "/content.gold"
 	contentTestTmplPath = assetsDir + "/content.go.tmpl"
 	contentTestTmplName = "content.go.tmpl"
@@ -85,13 +86,27 @@ type GeneratorTestSuite struct {
 	suite.Suite
 	tmpLinguist string
 	cloned      bool
+	testCases   []testCase
 }
 
-func TestGeneratorTestSuite(t *testing.T) {
+type testCase struct {
+	name        string
+	fileToParse string
+	samplesDir  string
+	tmplPath    string
+	tmplName    string
+	commit      string
+	generate    File
+	wantOut     string
+}
+
+var updateGold = flag.Bool("update_gold", false, "Update golden test files")
+
+func Test_GeneratorTestSuite(t *testing.T) {
 	suite.Run(t, new(GeneratorTestSuite))
 }
 
-func (s *GeneratorTestSuite) SetupSuite() {
+func (s *GeneratorTestSuite) maybeCloneLinguist() {
 	var err error
 	s.tmpLinguist = os.Getenv(linguistClonedEnvVar)
 	s.cloned = s.tmpLinguist == ""
@@ -101,7 +116,6 @@ func (s *GeneratorTestSuite) SetupSuite() {
 		cmd := exec.Command("git", "clone", linguistURL, s.tmpLinguist)
 		err = cmd.Run()
 		assert.NoError(s.T(), err)
-	}
 
 		cwd, err := os.Getwd()
 		assert.NoError(s.T(), err)
@@ -109,32 +123,18 @@ func (s *GeneratorTestSuite) SetupSuite() {
 		err = os.Chdir(s.tmpLinguist)
 		assert.NoError(s.T(), err)
 
-	cmd := exec.Command("git", "checkout", commit)
+		cmd = exec.Command("git", "checkout", commit)
 		err = cmd.Run()
 		assert.NoError(s.T(), err)
 
 		err = os.Chdir(cwd)
 		assert.NoError(s.T(), err)
 	}
-
-func (s *GeneratorTestSuite) TearDownSuite() {
-	if s.cloned {
-		err := os.RemoveAll(s.tmpLinguist)
-		assert.NoError(s.T(), err)
-	}
 }
 
-func (s *GeneratorTestSuite) TestGenerationFiles() {
+func (s *GeneratorTestSuite) SetupSuite() {
-	tests := []struct {
+	s.maybeCloneLinguist()
-		name        string
+	s.testCases = []testCase{
-		fileToParse string
-		samplesDir  string
-		tmplPath    string
-		tmplName    string
-		commit      string
-		generate    File
-		wantOut     string
-	}{
 		{
 			name:        "Extensions()",
 			fileToParse: filepath.Join(s.tmpLinguist, languagesFile),
@@ -152,7 +152,7 @@ func (s *GeneratorTestSuite) TestGenerationFiles() {
 			tmplPath:    contentTestTmplPath,
 			tmplName:    contentTestTmplName,
 			commit:      commit,
-			generate:    Heuristics,
+			generate:    GenHeuristics,
 			wantOut:     contentGold,
 		},
 		{
@@ -244,8 +244,35 @@ func (s *GeneratorTestSuite) TestGenerationFiles() {
 			wantOut:     mimeTypeGold,
 		},
 	}
+}
 
-	for _, test := range tests {
+func (s *GeneratorTestSuite) TearDownSuite() {
+	if s.cloned {
+		err := os.RemoveAll(s.tmpLinguist)
+		if err != nil {
+			s.T().Logf("Failed to clean up %s after the test.\n", s.tmpLinguist)
+		}
+	}
+}
+
+// TestUpdateGeneratorTestSuiteGold is a Gold results generation automation.
+// It should only be enabled&run manually on every new Linguist version
+// to update *.gold files.
+func (s *GeneratorTestSuite) TestUpdateGeneratorTestSuiteGold() {
+	if !*updateGold {
+		s.T().Skip()
+	}
+	s.T().Logf("Generating new *.gold test files")
+	for _, test := range s.testCases {
+		dst := test.wantOut
+		s.T().Logf("Generating %s from %s\n", dst, test.fileToParse)
+		err := test.generate(test.fileToParse, test.samplesDir, dst, test.tmplPath, test.tmplName, test.commit)
+		assert.NoError(s.T(), err)
+	}
+}
+
+func (s *GeneratorTestSuite) TestGenerationFiles() {
+	for _, test := range s.testCases {
 		gold, err := ioutil.ReadFile(test.wantOut)
 		assert.NoError(s.T(), err)
 

internal/code-generator/generator/heuristics.go

@@ -1,483 +1,176 @@
 package generator
 
 import (
-	"bufio"
 	"bytes"
 	"fmt"
-	"io"
 	"io/ioutil"
-	"strconv"
+	"log"
 	"strings"
-	"text/template"
 
-	"gopkg.in/src-d/enry.v1/regex"
+	yaml "gopkg.in/yaml.v2"
 )
 
-// Heuristics reads from fileToParse and builds source file from tmplPath. It complies with type File signature.
+const (
-func Heuristics(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error {
+	multilinePrefix = "(?m)"
-	data, err := ioutil.ReadFile(fileToParse)
+	orPipe          = "|"
+)
+
+// GenHeuristics generates language identification heuristics in Go.
+// It is of generator.File type.
+func GenHeuristics(fileToParse, _, outPath, tmplPath, tmplName, commit string) error {
+	heuristicsYaml, err := parseYaml(fileToParse)
 	if err != nil {
 		return err
 	}
 
-	disambiguators, err := getDisambiguators(data)
+	langPatterns, err := loadHeuristics(heuristicsYaml)
 	if err != nil {
 		return err
 	}
 
 	buf := &bytes.Buffer{}
-	if err := executeContentTemplate(buf, disambiguators, tmplPath, tmplName, commit); err != nil {
+	err = executeTemplate(buf, tmplName, tmplPath, commit, nil, langPatterns)
+	if err != nil {
 		return err
 	}
 
 	return formatedWrite(outPath, buf.Bytes())
 }
 
-const (
+// loadHeuristics transforms parsed YAML to map[".ext"]->IR for code generation.
-	unknownLanguage = "OtherLanguage"
+func loadHeuristics(yaml *Heuristics) (map[string][]*LanguagePattern, error) {
-	emptyFile       = "^$"
+	var patterns = make(map[string][]*LanguagePattern)
-)
+	for _, disambiguation := range yaml.Disambiguations {
-
+		var rules []*LanguagePattern
-var (
+		for _, rule := range disambiguation.Rules {
-	disambLine       = regex.MustCompile(`^(\s*)disambiguate`)
+			langPattern := loadRule(yaml.NamedPatterns, rule)
-	definedRegs      = make(map[string]string)
+			if langPattern != nil {
-	illegalCharacter = map[string]string{
+				rules = append(rules, langPattern)
-		"#": "Sharp",
-		"+": "Plus",
-		"-": "Dash",
 			}
-)
+		}
-
+		// unroll to a single map
-type disambiguator struct {
+		for _, ext := range disambiguation.Extensions {
-	Extension string                `json:"extension,omitempty"`
+			if _, ok := patterns[ext]; ok {
-	Languages []*languageHeuristics `json:"languages,omitempty"`
+				return nil, fmt.Errorf("cannt add extension '%s', it already exists for %q", ext, patterns[ext])
+			}
+			patterns[ext] = rules
 		}
 
-func (d *disambiguator) setHeuristicsNames() {
-	for _, lang := range d.Languages {
-		for i, heuristic := range lang.Heuristics {
-			name := buildName(d.Extension, lang.Language, i)
-			heuristic.Name = name
-		}
 	}
+	return patterns, nil
 }
 
-func buildName(extension, language string, id int) string {
+// loadRule transforms single rule from parsed YAML to IR for code generation.
-	extension = strings.TrimPrefix(extension, `.`)
+// For OrPattern case, it always combines multiple patterns into a single one.
-	language = strings.Join(strings.Fields(language), ``)
+func loadRule(namedPatterns map[string]StringArray, rule *Rule) *LanguagePattern {
-	name := strings.Join([]string{extension, language, "Matcher", strconv.Itoa(id)}, `_`)
+	var result *LanguagePattern
-	for k, v := range illegalCharacter {
+	if len(rule.And) != 0 { // AndPattern
-		if strings.Contains(name, k) {
+		var subPatterns []*LanguagePattern
-			name = strings.Replace(name, k, v, -1)
+		for _, r := range rule.And {
+			subp := loadRule(namedPatterns, r)
+			subPatterns = append(subPatterns, subp)
 		}
+		result = &LanguagePattern{"And", rule.Languages, "", subPatterns}
+	} else if len(rule.Pattern) != 0 { // OrPattern
+		conjunction := strings.Join(rule.Pattern, orPipe)
+		pattern := convertToValidRegexp(conjunction)
+		result = &LanguagePattern{"Or", rule.Languages, pattern, nil}
+	} else if rule.NegativePattern != "" { // NotPattern
+		pattern := convertToValidRegexp(rule.NegativePattern)
+		result = &LanguagePattern{"Not", rule.Languages, pattern, nil}
+	} else if rule.NamedPattern != "" { // Named OrPattern
+		conjunction := strings.Join(namedPatterns[rule.NamedPattern], orPipe)
+		pattern := convertToValidRegexp(conjunction)
+		result = &LanguagePattern{"Or", rule.Languages, pattern, nil}
+	} else { // AlwaysPattern
+		result = &LanguagePattern{"Always", rule.Languages, "", nil}
 	}
 
-	return name
+	if isUnsupportedRegexpSyntax(result.Pattern) {
+		log.Printf("skipping rule: language:'%q', rule:'%q'\n", rule.Languages, result.Pattern)
+		return nil
+	}
+	return result
 }
 
-type languageHeuristics struct {
+// LanguagePattern is an IR of parsed Rule suitable for code generations.
-	Language       string       `json:"language,omitempty"`
+// Strings are used as this is to be be consumed by text/template.
-	Heuristics     []*heuristic `json:"heuristics,omitempty"`
+type LanguagePattern struct {
-	LogicRelations []string     `json:"logic_relations,omitempty"`
+	Op      string
+	Langs   []string
+	Pattern string
+	Rules   []*LanguagePattern
 }
 
-func (l *languageHeuristics) clone() (*languageHeuristics, error) {
+type Heuristics struct {
-	language := l.Language
+	Disambiguations []*Disambiguation
-	logicRels := make([]string, len(l.LogicRelations))
+	NamedPatterns   map[string]StringArray `yaml:"named_patterns"`
-	if copy(logicRels, l.LogicRelations) != len(l.LogicRelations) {
-		return nil, fmt.Errorf("error copying logic relations")
 }
 
-	heuristics := make([]*heuristic, 0, len(l.Heuristics))
+type Disambiguation struct {
-	for _, h := range l.Heuristics {
+	Extensions []string `yaml:"extensions,flow"`
-		heuristic := *h
+	Rules      []*Rule  `yaml:"rules"`
-		heuristics = append(heuristics, &heuristic)
 }
 
-	clone := &languageHeuristics{
+type Rule struct {
-		Language:       language,
+	Patterns  `yaml:",inline"`
-		Heuristics:     heuristics,
+	Languages StringArray `yaml:"language"`
-		LogicRelations: logicRels,
+	And       []*Rule
 }
 
-	return clone, nil
+type Patterns struct {
+	Pattern         StringArray `yaml:"pattern,omitempty"`
+	NamedPattern    string      `yaml:"named_pattern,omitempty"`
+	NegativePattern string      `yaml:"negative_pattern,omitempty"`
 }
 
-type heuristic struct {
+// StringArray is workaround for parsing named_pattern,
-	Name   string `json:"name,omitempty"`
+// wich is sometimes arry and sometimes not.
-	Regexp string `json:"regexp,omitempty"`
+// See https://github.com/go-yaml/yaml/issues/100
-}
+type StringArray []string
 
-// A disambiguate block looks like:
+// UnmarshalYAML allowes to parse element always as a []string
-// disambiguate ".mod", ".extension" do |data|
+func (sa *StringArray) UnmarshalYAML(unmarshal func(interface{}) error) error {
-// 	if data.include?('<!ENTITY ') && data.include?('patata')
+	var multi []string
-// 		Language["XML"]
+	if err := unmarshal(&multi); err != nil {
-// 	elsif /^\s*MODULE [\w\.]+;/i.match(data) || /^\s*END [\w\.]+;/i.match(data) || data.empty?
+		var single string
-// 		Language["Modula-2"]
+		if err := unmarshal(&single); err != nil {
-//	elsif (/^\s*import (scala|java)\./.match(data) || /^\s*val\s+\w+\s*=/.match(data) || /^\s*class\b/.match(data))
+			return err
-//              Language["Scala"]
-//      elsif (data.include?("gap> "))
-//		Language["GAP"]
-// 	else
-// 		[Language["Linux Kernel Module"], Language["AMPL"]]
-// 	end
-// end
-func getDisambiguators(heuristics []byte) ([]*disambiguator, error) {
-	seenExtensions := map[string]bool{}
-	buf := bufio.NewScanner(bytes.NewReader(heuristics))
-	disambiguators := make([]*disambiguator, 0, 50)
-	for buf.Scan() {
-		line := buf.Text()
-		if disambLine.MatchString(line) {
-			d, err := parseDisambiguators(line, buf, seenExtensions)
-			if err != nil {
-				return nil, err
 		}
-
+		*sa = []string{single}
-			disambiguators = append(disambiguators, d...)
-		}
-
-		lookForRegexpVariables(line)
-	}
-
-	if err := buf.Err(); err != nil {
-		return nil, err
-	}
-
-	return disambiguators, nil
-}
-
-func lookForRegexpVariables(line string) {
-	if strings.Contains(line, "ObjectiveCRegex = ") {
-		line = strings.TrimSpace(line)
-		reg := strings.TrimPrefix(line, "ObjectiveCRegex = ")
-		definedRegs["ObjectiveCRegex"] = reg
-	}
-
-	if strings.Contains(line, "fortran_rx = ") {
-		line = strings.TrimSpace(line)
-		reg := strings.TrimPrefix(line, "fortran_rx = ")
-		definedRegs["fortran_rx"] = reg
-	}
-}
-
-func parseDisambiguators(line string, buf *bufio.Scanner, seenExtensions map[string]bool) ([]*disambiguator, error) {
-	disambList := make([]*disambiguator, 0, 2)
-	splitted := strings.Fields(line)
-
-	for _, v := range splitted {
-		if strings.HasPrefix(v, `"`) {
-			extension := strings.Trim(v, `",`)
-			if _, ok := seenExtensions[extension]; !ok {
-				d := &disambiguator{Extension: extension}
-				disambList = append(disambList, d)
-				seenExtensions[extension] = true
-			}
-		}
-	}
-
-	langsHeuristics, err := getLanguagesHeuristics(buf)
-	if err != nil {
-		return nil, err
-	}
-
-	for i, disamb := range disambList {
-		lh := langsHeuristics
-		if i != 0 {
-			lh = cloneLanguagesHeuristics(langsHeuristics)
-		}
-
-		disamb.Languages = lh
-		disamb.setHeuristicsNames()
-	}
-
-	return disambList, nil
-}
-
-func cloneLanguagesHeuristics(list []*languageHeuristics) []*languageHeuristics {
-	cloneList := make([]*languageHeuristics, 0, len(list))
-	for _, langHeu := range list {
-		clone, _ := langHeu.clone()
-		cloneList = append(cloneList, clone)
-	}
-
-	return cloneList
-}
-
-func getLanguagesHeuristics(buf *bufio.Scanner) ([]*languageHeuristics, error) {
-	langsList := make([][]string, 0, 2)
-	heuristicsList := make([][]*heuristic, 0, 1)
-	logicRelsList := make([][]string, 0, 1)
-
-	lastWasMatch := false
-	for buf.Scan() {
-		line := buf.Text()
-		if strings.TrimSpace(line) == "end" {
-			break
-		}
-
-		if hasRegExp(line) {
-			line := cleanRegExpLine(line)
-
-			logicRels := getLogicRelations(line)
-			heuristics := getHeuristics(line)
-			if lastWasMatch {
-				i := len(heuristicsList) - 1
-				heuristicsList[i] = append(heuristicsList[i], heuristics...)
-				i = len(logicRelsList) - 1
-				logicRelsList[i] = append(logicRelsList[i], logicRels...)
 	} else {
-				heuristicsList = append(heuristicsList, heuristics)
+		*sa = multi
-				logicRelsList = append(logicRelsList, logicRels)
+	}
+	return nil
 }
 
-			lastWasMatch = true
+func parseYaml(file string) (*Heuristics, error) {
-		}
+	data, err := ioutil.ReadFile(file)
-
+	if err != nil {
-		if strings.Contains(line, "Language") {
-			langs := getLanguages(line)
-			langsList = append(langsList, langs)
-			lastWasMatch = false
-		}
-
-	}
-
-	if err := buf.Err(); err != nil {
 		return nil, err
 	}
 
-	langsHeuristics := buildLanguagesHeuristics(langsList, heuristicsList, logicRelsList)
+	h := &Heuristics{}
-	return langsHeuristics, nil
+	if err := yaml.Unmarshal(data, &h); err != nil {
+		return nil, err
 	}
 
-func hasRegExp(line string) bool {
+	return h, nil
-	return strings.Contains(line, ".match") || strings.Contains(line, ".include?") || strings.Contains(line, ".empty?")
 }
 
-func cleanRegExpLine(line string) string {
+// isUnsupportedRegexpSyntax filters regexp syntax that is not supported by RE2.
-	if strings.Contains(line, "if ") {
+// In particular, we stumbled up on usage of next cases:
-		line = line[strings.Index(line, `if `)+3:]
+// - named & numbered capturing group/after text matching
+// - backreference
+// For referece on supported syntax see https://github.com/google/re2/wiki/Syntax
+func isUnsupportedRegexpSyntax(reg string) bool {
+	return strings.Contains(reg, `(?<`) || strings.Contains(reg, `\1`) ||
+		// See https://github.com/github/linguist/pull/4243#discussion_r246105067
+		(strings.HasPrefix(reg, multilinePrefix+`/`) && strings.HasSuffix(reg, `/`))
 }
 
-	line = strings.TrimSpace(line)
+// convertToValidRegexp converts Ruby regexp syntaxt to RE2 equivalent.
-	line = strings.TrimPrefix(line, `(`)
+// Does not work with Ruby regexp literals.
-	if strings.Contains(line, "))") {
+func convertToValidRegexp(rubyRegexp string) string {
-		line = strings.TrimSuffix(line, `)`)
+	return multilinePrefix + rubyRegexp
-	}
-
-	return line
-}
-
-func getLogicRelations(line string) []string {
-	rels := make([]string, 0)
-	splitted := strings.Split(line, "||")
-	for i, v := range splitted {
-		if strings.Contains(v, "&&") {
-			rels = append(rels, "&&")
-		}
-
-		if i < len(splitted)-1 {
-			rels = append(rels, "||")
-		}
-	}
-
-	if len(rels) == 0 {
-		rels = nil
-	}
-
-	return rels
-}
-
-func getHeuristics(line string) []*heuristic {
-	splitted := splitByLogicOps(line)
-	heuristics := make([]*heuristic, 0, len(splitted))
-	for _, v := range splitted {
-		v = strings.TrimSpace(v)
-		var reg string
-
-		if strings.Contains(v, ".match") {
-			reg = v[:strings.Index(v, ".match")]
-			reg = replaceRegexpVariables(reg)
-		}
-
-		if strings.Contains(v, ".include?") {
-			reg = includeToRegExp(v)
-		}
-
-		if strings.Contains(v, ".empty?") {
-			reg = emptyFile
-		}
-
-		if reg != "" {
-			reg = convertToValidRegexp(reg)
-			heuristics = append(heuristics, &heuristic{Regexp: reg})
-		}
-	}
-
-	return heuristics
-}
-
-func splitByLogicOps(line string) []string {
-	splitted := make([]string, 0, 1)
-	splitOr := strings.Split(line, "||")
-	for _, v := range splitOr {
-		splitAnd := strings.Split(v, "&&")
-		splitted = append(splitted, splitAnd...)
-	}
-
-	return splitted
-}
-
-func replaceRegexpVariables(reg string) string {
-	repl := reg
-	if v, ok := definedRegs[reg]; ok {
-		repl = v
-	}
-
-	return repl
-}
-
-func convertToValidRegexp(reg string) string {
-	// example: `/^(\s*)(<Project|<Import|<Property|<?xml|xmlns)/i``
-	// Ruby modifier "m" matches multiple lines, recognizing newlines as normal characters, Go use flag "s" for that.
-	const (
-		caseSensitive = "i"
-		matchEOL      = "s"
-
-		rubyCaseSensitive = "i"
-		rubyMultiLine     = "m"
-	)
-
-	if reg == emptyFile {
-		return reg
-	}
-
-	reg = strings.TrimPrefix(reg, `/`)
-	flags := "(?m"
-	lastSlash := strings.LastIndex(reg, `/`)
-	if lastSlash == -1 {
-		return flags + ")" + reg
-	}
-
-	specialChars := reg[lastSlash:]
-	reg = reg[:lastSlash]
-	if lastSlash == len(reg)-1 {
-		return flags + ")" + reg
-	}
-
-	if strings.Contains(specialChars, rubyCaseSensitive) {
-		flags = flags + caseSensitive
-	}
-
-	if strings.Contains(specialChars, rubyMultiLine) {
-		flags = flags + matchEOL
-	}
-
-	return flags + ")" + reg
-}
-
-func includeToRegExp(include string) string {
-	content := include[strings.Index(include, `(`)+1 : strings.Index(include, `)`)]
-	content = strings.Trim(content, `"'`)
-	return regex.QuoteMeta(content)
-}
-
-func getLanguages(line string) []string {
-	languages := make([]string, 0)
-	splitted := strings.Split(line, `,`)
-	for _, lang := range splitted {
-		lang = trimLanguage(lang)
-		languages = append(languages, lang)
-	}
-
-	return languages
-}
-
-func trimLanguage(enclosedLang string) string {
-	lang := strings.TrimSpace(enclosedLang)
-	lang = lang[strings.Index(lang, `"`)+1:]
-	lang = lang[:strings.Index(lang, `"`)]
-	return lang
-}
-
-func buildLanguagesHeuristics(langsList [][]string, heuristicsList [][]*heuristic, logicRelsList [][]string) []*languageHeuristics {
-	langsHeuristics := make([]*languageHeuristics, 0, len(langsList))
-	for i, langSlice := range langsList {
-		var heuristics []*heuristic
-		if i < len(heuristicsList) {
-			heuristics = heuristicsList[i]
-		}
-
-		var rels []string
-		if i < len(logicRelsList) {
-			rels = logicRelsList[i]
-		}
-
-		for _, lang := range langSlice {
-			lh := &languageHeuristics{
-				Language:       lang,
-				Heuristics:     heuristics,
-				LogicRelations: rels,
-			}
-
-			langsHeuristics = append(langsHeuristics, lh)
-		}
-	}
-
-	return langsHeuristics
-}
-
-func executeContentTemplate(out io.Writer, disambiguators []*disambiguator, tmplPath, tmplName, commit string) error {
-	fmap := template.FuncMap{
-		"getAllHeuristics": getAllHeuristics,
-		"returnStringSlice": func(slice []string) string {
-			if len(slice) == 0 {
-				return "nil"
-			}
-
-			return `[]string{` + strings.Join(slice, `, `) + `}`
-		},
-		"returnLanguages": returnLanguages,
-		"avoidLanguage":   avoidLanguage,
-	}
-	return executeTemplate(out, tmplName, tmplPath, commit, fmap, disambiguators)
-}
-
-func getAllHeuristics(disambiguators []*disambiguator) []*heuristic {
-	heuristics := make([]*heuristic, 0)
-	for _, disamb := range disambiguators {
-		for _, lang := range disamb.Languages {
-			if !avoidLanguage(lang) {
-				heuristics = append(heuristics, lang.Heuristics...)
-			}
-		}
-	}
-
-	return heuristics
-}
-
-func avoidLanguage(lang *languageHeuristics) bool {
-	// necessary to avoid corner cases
-	for _, heuristic := range lang.Heuristics {
-		if containsInvalidRegexp(heuristic.Regexp) {
-			return true
-		}
-	}
-
-	return false
-}
-
-func containsInvalidRegexp(reg string) bool {
-	return strings.Contains(reg, `(?<`) || strings.Contains(reg, `\1`)
-}
-
-func returnLanguages(langsHeuristics []*languageHeuristics) []string {
-	langs := make([]string, 0)
-	for _, langHeu := range langsHeuristics {
-		if len(langHeu.Heuristics) == 0 {
-			langs = append(langs, `"`+langHeu.Language+`"`)
-		}
-	}
-
-	return langs
 }

internal/code-generator/generator/heuristics_test.go

@@ -0,0 +1,123 @@
+package generator
+
+import (
+	"bytes"
+	"fmt"
+	"go/format"
+	"testing"
+	"text/template"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestYAMLParsing(t *testing.T) {
+	heuristics, err := parseYaml("test_files/heuristics.yml")
+
+	require.NoError(t, err)
+	assert.NotNil(t, heuristics)
+
+	// extensions
+	require.NotNil(t, heuristics.Disambiguations)
+	assert.Equal(t, 4, len(heuristics.Disambiguations))
+	assert.Equal(t, 2, len(heuristics.Disambiguations[0].Extensions))
+
+	rules := heuristics.Disambiguations[0].Rules
+	assert.Equal(t, 2, len(rules))
+	require.Equal(t, "Objective-C", rules[0].Languages[0])
+	assert.Equal(t, 1, len(rules[0].Pattern))
+
+	rules = heuristics.Disambiguations[1].Rules
+	assert.Equal(t, 3, len(rules))
+	require.Equal(t, "Forth", rules[0].Languages[0])
+	require.Equal(t, 2, len(rules[0].Pattern))
+
+	rules = heuristics.Disambiguations[2].Rules
+	assert.Equal(t, 3, len(rules))
+	require.Equal(t, "Unix Assembly", rules[1].Languages[0])
+	require.NotNil(t, rules[1].And)
+	assert.Equal(t, 2, len(rules[1].And))
+	require.NotNil(t, rules[1].And[0].NegativePattern)
+	assert.Equal(t, "np", rules[1].And[0].NegativePattern)
+
+	rules = heuristics.Disambiguations[3].Rules
+	assert.Equal(t, 1, len(rules))
+	assert.Equal(t, "Linux Kernel Module", rules[0].Languages[0])
+	assert.Equal(t, "AMPL", rules[0].Languages[1])
+
+	// named_patterns
+	require.NotNil(t, heuristics.NamedPatterns)
+	assert.Equal(t, 2, len(heuristics.NamedPatterns))
+	assert.Equal(t, 1, len(heuristics.NamedPatterns["fortran"]))
+	assert.Equal(t, 2, len(heuristics.NamedPatterns["cpp"]))
+}
+
+func TestSingleRuleLoading(t *testing.T) {
+	namedPatterns := map[string]StringArray{"cpp": []string{"cpp_ptrn1", "cpp_ptrn2"}}
+	rules := []*Rule{
+		&Rule{Languages: []string{"a"}, Patterns: Patterns{NamedPattern: "cpp"}},
+		&Rule{Languages: []string{"b"}, And: []*Rule{}},
+	}
+
+	// named_pattern case
+	langPattern := loadRule(namedPatterns, rules[0])
+	require.Equal(t, "a", langPattern.Langs[0])
+	assert.NotEmpty(t, langPattern.Pattern)
+
+	// and case
+	langPattern = loadRule(namedPatterns, rules[1])
+	require.Equal(t, "b", langPattern.Langs[0])
+}
+
+func TestLoadingAllHeuristics(t *testing.T) {
+	parsedYaml, err := parseYaml("test_files/heuristics.yml")
+	require.NoError(t, err)
+
+	hs, err := loadHeuristics(parsedYaml)
+
+	// grep -Eo "extensions:\ (.*)" internal/code-generator/generator/test_files/heuristics.yml
+	assert.Equal(t, 5, len(hs))
+}
+
+func TestLoadingHeuristicsForSameExt(t *testing.T) {
+	parsedYaml := &Heuristics{
+		Disambiguations: []*Disambiguation{
+			&Disambiguation{
+				Extensions: []string{".a", ".b"},
+				Rules:      []*Rule{&Rule{Languages: []string{"A"}}},
+			},
+			&Disambiguation{
+				Extensions: []string{".b"},
+				Rules:      []*Rule{&Rule{Languages: []string{"B"}}},
+			},
+		},
+	}
+
+	_, err := loadHeuristics(parsedYaml)
+	require.Error(t, err)
+}
+
+func TestTemplateMatcherVars(t *testing.T) {
+	parsed, err := parseYaml("test_files/heuristics.yml")
+	require.NoError(t, err)
+
+	heuristics, err := loadHeuristics(parsed)
+	require.NoError(t, err)
+
+	// render a tmpl
+	const contentTmpl = "../assets/content.go.tmpl"
+	tmpl, err := template.ParseFiles(contentTmpl)
+	require.NoError(t, err)
+
+	buf := bytes.NewBuffer(nil)
+	err = tmpl.Execute(buf, heuristics)
+	require.NoError(t, err, fmt.Sprintf("%+v", tmpl))
+	require.NotEmpty(t, buf)
+
+	// TODO(bzz) add more advanced test using go/ast package, to verify the
+	// strucutre of generated code:
+	//  - check key literal exists in map for each extension:
+
+	src, err := format.Source(buf.Bytes())
+	require.NoError(t, err, "\n%s\n", string(src))
+}

internal/code-generator/generator/mimeType.go

@@ -2,11 +2,14 @@ package generator
 
 import (
 	"bytes"
-	"gopkg.in/yaml.v2"
 	"io"
 	"io/ioutil"
+
+	"gopkg.in/yaml.v2"
 )
 
+// MimeType generates a map in Go with language name -> MIME string.
+// It is of generator.File type.
 func MimeType(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error {
 	data, err := ioutil.ReadFile(fileToParse)
 	if err != nil {

internal/code-generator/generator/test_files/alias.gold

@@ -1,16 +1,19 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
-// LanguagesByAlias keeps alias for different languages and use the name of the languages as an alias too.
+import "strings"
+
+// LanguageByAliasMap keeps alias for different languages and use the name of the languages as an alias too.
 // All the keys (alias or not) are written in lower case and the whitespaces has been replaced by underscores.
-var LanguagesByAlias = map[string]string{
+var LanguageByAliasMap = map[string]string{
 	"1c_enterprise":  "1C Enterprise",
 	"abap":           "ABAP",
 	"abl":            "OpenEdge ABL",
 	"abnf":           "ABNF",
 	"abuild":         "Alpine Abuild",
+	"acfm":           "Adobe Font Metrics",
 	"aconf":          "ApacheConf",
 	"actionscript":   "ActionScript",
 	"actionscript3":  "ActionScript",
@@ -18,6 +21,9 @@ var LanguagesByAlias = map[string]string{
 	"ada":            "Ada",
 	"ada2005":        "Ada",
 	"ada95":          "Ada",
+	"adobe_composite_font_metrics": "Adobe Font Metrics",
+	"adobe_font_metrics":           "Adobe Font Metrics",
+	"adobe_multiple_font_metrics":  "Adobe Font Metrics",
 	"advpl":            "xBase",
 	"afdko":            "OpenType Feature File",
 	"agda":             "Agda",
@@ -26,7 +32,9 @@ var LanguagesByAlias = map[string]string{
 	"ahk":              "AutoHotkey",
 	"alloy":            "Alloy",
 	"alpine_abuild":    "Alpine Abuild",
+	"amfm":             "Adobe Font Metrics",
 	"ampl":             "AMPL",
+	"angelscript":      "AngelScript",
 	"ant_build_system": "Ant Build System",
 	"antlr":            "ANTLR",
 	"apache":           "ApacheConf",
@@ -38,16 +46,17 @@ var LanguagesByAlias = map[string]string{
 	"apollo_guidance_computer":       "Apollo Guidance Computer",
 	"applescript":                    "AppleScript",
 	"arc":                            "Arc",
-	"arduino":                  "Arduino",
 	"arexx":                          "REXX",
 	"as3":                            "ActionScript",
 	"asciidoc":                       "AsciiDoc",
+	"asm":                            "Assembly",
 	"asn.1":                          "ASN.1",
 	"asp":                            "ASP",
 	"aspectj":                        "AspectJ",
 	"aspx":                           "ASP",
 	"aspx-vb":                        "ASP",
 	"assembly":                       "Assembly",
+	"asymptote":                      "Asymptote",
 	"ats":                            "ATS",
 	"ats2":                           "ATS",
 	"au3":                            "AutoIt",
@@ -59,6 +68,7 @@ var LanguagesByAlias = map[string]string{
 	"autoitscript":                   "AutoIt",
 	"awk":                            "Awk",
 	"b3d":                            "BlitzBasic",
+	"ballerina":                      "Ballerina",
 	"bash":                           "Shell",
 	"bash_session":                   "ShellSession",
 	"bat":                            "Batchfile",
@@ -107,6 +117,7 @@ var LanguagesByAlias = map[string]string{
 	"clips":                          "CLIPS",
 	"clojure":                        "Clojure",
 	"closure_templates":              "Closure Templates",
+	"cloud_firestore_security_rules": "Cloud Firestore Security Rules",
 	"cmake":                    "CMake",
 	"cobol":                    "COBOL",
 	"coffee":                   "CoffeeScript",
@@ -117,10 +128,15 @@ var LanguagesByAlias = map[string]string{
 	"coldfusion_html":          "ColdFusion",
 	"collada":                  "COLLADA",
 	"common_lisp":              "Common Lisp",
+	"common_workflow_language": "Common Workflow Language",
 	"component_pascal":         "Component Pascal",
+	"conll":                    "CoNLL-U",
+	"conll-u":                  "CoNLL-U",
+	"conll-x":                  "CoNLL-U",
 	"console":                  "ShellSession",
 	"cool":                     "Cool",
 	"coq":                      "Coq",
+	"cperl":                    "Perl",
 	"cpp":                      "C++",
 	"cpp-objdump":              "Cpp-ObjDump",
 	"creole":                   "Creole",
@@ -138,12 +154,14 @@ var LanguagesByAlias = map[string]string{
 	"cucumber":                 "Gherkin",
 	"cuda":                     "Cuda",
 	"cweb":                     "CWeb",
+	"cwl":                      "Common Workflow Language",
 	"cycript":                  "Cycript",
 	"cython":                   "Cython",
 	"d":                        "D",
 	"d-objdump":                "D-ObjDump",
 	"darcs_patch":              "Darcs Patch",
 	"dart":                     "Dart",
+	"dataweave":                "DataWeave",
 	"dcl":                      "DIGITAL Command Language",
 	"delphi":                   "Component Pascal",
 	"desktop":                  "desktop",
@@ -169,7 +187,9 @@ var LanguagesByAlias = map[string]string{
 	"ecl":                      "ECL",
 	"eclipse":                  "ECLiPSe",
 	"ecr":                      "HTML+ECR",
+	"edje_data_collection": "Edje Data Collection",
 	"edn":                 "edn",
+	"eeschema_schematic":  "KiCad Schematic",
 	"eex":                 "HTML+EEX",
 	"eiffel":              "Eiffel",
 	"ejs":                 "EJS",
@@ -179,13 +199,17 @@ var LanguagesByAlias = map[string]string{
 	"emacs":               "Emacs Lisp",
 	"emacs_lisp":          "Emacs Lisp",
 	"emberscript":         "EmberScript",
+	"eml":                 "EML",
 	"eq":                  "EQ",
 	"erb":                 "HTML+ERB",
 	"erlang":              "Erlang",
 	"f#":                  "F#",
+	"f*":                  "F*",
 	"factor":              "Factor",
 	"fancy":               "Fancy",
 	"fantom":              "Fantom",
+	"figfont":             "FIGlet Font",
+	"figlet_font":         "FIGlet Font",
 	"filebench_wml":       "Filebench WML",
 	"filterscript":        "Filterscript",
 	"fish":                "fish",
@@ -198,6 +222,7 @@ var LanguagesByAlias = map[string]string{
 	"freemarker":          "FreeMarker",
 	"frege":               "Frege",
 	"fsharp":              "F#",
+	"fstar":               "F*",
 	"ftl":                 "FreeMarker",
 	"fundamental":         "Text",
 	"g-code":              "G-code",
@@ -211,11 +236,20 @@ var LanguagesByAlias = map[string]string{
 	"genshi":          "Genshi",
 	"gentoo_ebuild":   "Gentoo Ebuild",
 	"gentoo_eclass":   "Gentoo Eclass",
+	"gerber_image":    "Gerber Image",
 	"gettext_catalog": "Gettext Catalog",
 	"gf":              "Grammatical Framework",
 	"gherkin":         "Gherkin",
+	"git-ignore":      "Ignore List",
+	"git_attributes":  "Git Attributes",
+	"git_config":      "Git Config",
+	"gitattributes":   "Git Attributes",
+	"gitconfig":       "Git Config",
+	"gitignore":       "Ignore List",
+	"gitmodules":      "Git Config",
 	"glsl":            "GLSL",
 	"glyph":           "Glyph",
+	"glyph_bitmap_distribution_format": "Glyph Bitmap Distribution Format",
 	"gn":                      "GN",
 	"gnuplot":                 "Gnuplot",
 	"go":                      "Go",
@@ -228,17 +262,20 @@ var LanguagesByAlias = map[string]string{
 	"graph_modeling_language": "Graph Modeling Language",
 	"graphql":                 "GraphQL",
 	"graphviz_(dot)":          "Graphviz (DOT)",
+	"groff":                   "Roff",
 	"groovy":                  "Groovy",
 	"groovy_server_pages":     "Groovy Server Pages",
 	"gsp":                      "Groovy Server Pages",
 	"hack":                     "Hack",
 	"haml":                     "Haml",
 	"handlebars":               "Handlebars",
+	"haproxy":                  "HAProxy",
 	"harbour":                  "Harbour",
 	"haskell":                  "Haskell",
 	"haxe":                     "Haxe",
 	"hbs":                      "Handlebars",
 	"hcl":                      "HCL",
+	"hiveql":                   "HiveQL",
 	"hlsl":                     "HLSL",
 	"html":                     "HTML",
 	"html+django":              "HTML+Django",
@@ -248,16 +285,20 @@ var LanguagesByAlias = map[string]string{
 	"html+erb":                 "HTML+ERB",
 	"html+jinja":               "HTML+Django",
 	"html+php":                 "HTML+PHP",
+	"html+razor":               "HTML+Razor",
 	"html+ruby":                "RHTML",
 	"htmlbars":                 "Handlebars",
 	"htmldjango":               "HTML+Django",
 	"http":                     "HTTP",
+	"hxml":                     "HXML",
 	"hy":                       "Hy",
 	"hylang":                   "Hy",
 	"hyphy":                    "HyPhy",
 	"i7":                       "Inform 7",
 	"idl":                      "IDL",
 	"idris":                    "Idris",
+	"ignore":                   "Ignore List",
+	"ignore_list":              "Ignore List",
 	"igor":                     "IGOR Pro",
 	"igor_pro":                 "IGOR Pro",
 	"igorpro":                  "IGOR Pro",
@@ -277,6 +318,7 @@ var LanguagesByAlias = map[string]string{
 	"j":                        "J",
 	"jasmin":                   "Jasmin",
 	"java":                     "Java",
+	"java_properties":          "Java Properties",
 	"java_server_page":         "Groovy Server Pages",
 	"java_server_pages":        "Java Server Pages",
 	"javascript":               "JavaScript",
@@ -288,13 +330,17 @@ var LanguagesByAlias = map[string]string{
 	"js":                       "JavaScript",
 	"json":                     "JSON",
 	"json5":                    "JSON5",
+	"json_with_comments":       "JSON with Comments",
+	"jsonc":                    "JSON with Comments",
 	"jsoniq":                   "JSONiq",
 	"jsonld":                   "JSONLD",
 	"jsp":                      "Java Server Pages",
 	"jsx":                      "JSX",
 	"julia":                    "Julia",
 	"jupyter_notebook":         "Jupyter Notebook",
-	"kicad":                    "KiCad",
+	"kicad_layout":             "KiCad Layout",
+	"kicad_legacy_layout":      "KiCad Legacy Layout",
+	"kicad_schematic":          "KiCad Schematic",
 	"kit":                      "Kit",
 	"kotlin":                   "Kotlin",
 	"krl":                      "KRL",
@@ -329,6 +375,7 @@ var LanguagesByAlias = map[string]string{
 	"loomscript":               "LoomScript",
 	"ls":                       "LiveScript",
 	"lsl":                      "LSL",
+	"ltspice_symbol":           "LTspice Symbol",
 	"lua":                      "Lua",
 	"m":                        "M",
 	"m4":                       "M4",
@@ -337,17 +384,22 @@ var LanguagesByAlias = map[string]string{
 	"make":                     "Makefile",
 	"makefile":                 "Makefile",
 	"mako":                     "Mako",
+	"man":                      "Roff",
+	"man-page":                 "Roff",
+	"man_page":                 "Roff",
+	"manpage":                  "Roff",
 	"markdown":                 "Markdown",
 	"marko":                    "Marko",
 	"markojs":                  "Marko",
 	"mask":                     "Mask",
 	"mathematica":              "Mathematica",
-	"matlab":                   "Matlab",
+	"matlab":                   "MATLAB",
 	"maven_pom":                "Maven POM",
 	"max":                      "Max",
 	"max/msp":                  "Max",
 	"maxmsp":                   "Max",
 	"maxscript":                "MAXScript",
+	"mdoc":                     "Roff",
 	"mediawiki":                "MediaWiki",
 	"mercury":                  "Mercury",
 	"meson":                    "Meson",
@@ -358,6 +410,7 @@ var LanguagesByAlias = map[string]string{
 	"mma":                      "Mathematica",
 	"modelica":                 "Modelica",
 	"modula-2":                 "Modula-2",
+	"modula-3":                 "Modula-3",
 	"module_management_system": "Module Management System",
 	"monkey":                   "Monkey",
 	"moocode":                  "Moocode",
@@ -369,14 +422,17 @@ var LanguagesByAlias = map[string]string{
 	"mumps":                    "M",
 	"mupad":                    "mupad",
 	"myghty":                   "Myghty",
+	"nanorc":                   "nanorc",
 	"nasm":                     "Assembly",
 	"ncl":                      "NCL",
+	"nearley":                  "Nearley",
 	"nemerle":                  "Nemerle",
 	"nesc":                     "nesC",
 	"netlinx":                  "NetLinx",
 	"netlinx+erb":              "NetLinx+ERB",
 	"netlogo":                  "NetLogo",
 	"newlisp":                  "NewLisp",
+	"nextflow":                 "Nextflow",
 	"nginx":                    "Nginx",
 	"nginx_configuration_file": "Nginx",
 	"nim":                   "Nim",
@@ -409,8 +465,9 @@ var LanguagesByAlias = map[string]string{
 	"objectpascal":          "Component Pascal",
 	"objj":                  "Objective-J",
 	"ocaml":                 "OCaml",
-	"octave":                "Matlab",
+	"octave":                "MATLAB",
 	"omgrofl":               "Omgrofl",
+	"oncrpc":                "RPC",
 	"ooc":                   "ooc",
 	"opa":                   "Opa",
 	"opal":                  "Opal",
@@ -435,9 +492,11 @@ var LanguagesByAlias = map[string]string{
 	"parrot_internal_representation": "Parrot Internal Representation",
 	"pascal":               "Pascal",
 	"pasm":                 "Parrot Assembly",
-	"pawn":                  "PAWN",
+	"pawn":                 "Pawn",
+	"pcbnew":               "KiCad Layout",
 	"pep8":                 "Pep8",
 	"perl":                 "Perl",
+	"perl6":                "Perl 6",
 	"perl_6":               "Perl 6",
 	"php":                  "PHP",
 	"pic":                  "Pic",
@@ -449,9 +508,11 @@ var LanguagesByAlias = map[string]string{
 	"plpgsql":              "PLpgSQL",
 	"plsql":                "PLSQL",
 	"pod":                  "Pod",
+	"pod_6":                "Pod 6",
 	"pogoscript":           "PogoScript",
 	"pony":                 "Pony",
 	"posh":                 "PowerShell",
+	"postcss":              "PostCSS",
 	"postscr":              "PostScript",
 	"postscript":           "PostScript",
 	"pot":                  "Gettext Catalog",
@@ -473,13 +534,17 @@ var LanguagesByAlias = map[string]string{
 	"pure_data":            "Pure Data",
 	"purebasic":            "PureBasic",
 	"purescript":           "PureScript",
+	"pwsh":                 "PowerShell",
 	"pycon":                "Python console",
 	"pyrex":                "Cython",
 	"python":               "Python",
+	"python3":              "Python",
 	"python_console":       "Python console",
 	"python_traceback":     "Python traceback",
+	"q":                    "q",
 	"qmake":                "QMake",
 	"qml":                  "QML",
+	"quake":                "Quake",
 	"r":                    "R",
 	"racket":               "Racket",
 	"ragel":                "Ragel",
@@ -490,6 +555,7 @@ var LanguagesByAlias = map[string]string{
 	"rascal":               "Rascal",
 	"raw":                  "Raw token data",
 	"raw_token_data":       "Raw token data",
+	"razor":                "HTML+Razor",
 	"rb":                   "Ruby",
 	"rbx":                  "Ruby",
 	"rdoc":                 "RDoc",
@@ -512,8 +578,12 @@ var LanguagesByAlias = map[string]string{
 	"rmarkdown":            "RMarkdown",
 	"robotframework":       "RobotFramework",
 	"roff":                 "Roff",
+	"roff_manpage":         "Roff Manpage",
 	"rouge":                "Rouge",
+	"rpc":                  "RPC",
+	"rpcgen":               "RPC",
 	"rpm_spec":             "RPM Spec",
+	"rs-274x":              "Gerber Image",
 	"rscript":              "R",
 	"rss":                  "XML",
 	"rst":                  "reStructuredText",
@@ -532,6 +602,7 @@ var LanguagesByAlias = map[string]string{
 	"scheme":               "Scheme",
 	"scilab":               "Scilab",
 	"scss":                 "SCSS",
+	"sed":                  "sed",
 	"self":                 "Self",
 	"sh":                   "Shell",
 	"shaderlab":            "ShaderLab",
@@ -540,14 +611,18 @@ var LanguagesByAlias = map[string]string{
 	"shellsession":         "ShellSession",
 	"shen":                 "Shen",
 	"slash":                "Slash",
+	"slice":                "Slice",
 	"slim":                 "Slim",
 	"smali":                "Smali",
 	"smalltalk":            "Smalltalk",
 	"smarty":               "Smarty",
 	"sml":                  "Standard ML",
 	"smt":                  "SMT",
+	"snippet":              "YASnippet",
+	"solidity":             "Solidity",
 	"sourcemod":            "SourcePawn",
 	"sourcepawn":           "SourcePawn",
+	"soy":                  "Closure Templates",
 	"sparql":               "SPARQL",
 	"specfile":             "RPM Spec",
 	"spline_font_database": "Spline Font Database",
@@ -563,8 +638,8 @@ var LanguagesByAlias = map[string]string{
 	"stata":                "Stata",
 	"ston":                 "STON",
 	"stylus":               "Stylus",
-	"sublime_text_config":   "Sublime Text Config",
 	"subrip_text":          "SubRip Text",
+	"sugarss":              "SugarSS",
 	"supercollider":        "SuperCollider",
 	"svg":                  "SVG",
 	"swift":                "Swift",
@@ -573,6 +648,7 @@ var LanguagesByAlias = map[string]string{
 	"tcsh":                 "Tcsh",
 	"tea":                  "Tea",
 	"terra":                "Terra",
+	"terraform":            "HCL",
 	"tex":                  "TeX",
 	"text":                 "Text",
 	"textile":              "Textile",
@@ -581,6 +657,7 @@ var LanguagesByAlias = map[string]string{
 	"tl":                   "Type Language",
 	"tla":                  "TLA",
 	"toml":                 "TOML",
+	"troff":                "Roff",
 	"ts":                   "TypeScript",
 	"turing":               "Turing",
 	"turtle":               "Turtle",
@@ -613,23 +690,31 @@ var LanguagesByAlias = map[string]string{
 	"wast":                 "WebAssembly",
 	"wavefront_material":   "Wavefront Material",
 	"wavefront_object":     "Wavefront Object",
+	"wdl":                  "wdl",
 	"web_ontology_language":    "Web Ontology Language",
 	"webassembly":              "WebAssembly",
 	"webidl":                   "WebIDL",
 	"winbatch":                 "Batchfile",
+	"windows_registry_entries": "Windows Registry Entries",
 	"wisp": "wisp",
 	"world_of_warcraft_addon_data": "World of Warcraft Addon Data",
 	"wsdl":                   "XML",
 	"x10":                    "X10",
+	"x_bitmap":               "X BitMap",
+	"x_font_directory_index": "X Font Directory Index",
+	"x_pixmap":               "X PixMap",
 	"xbase":                  "xBase",
+	"xbm":                    "X BitMap",
 	"xc":                     "XC",
 	"xcompose":               "XCompose",
+	"xdr":                    "RPC",
 	"xhtml":                  "HTML",
 	"xml":                    "XML",
 	"xml+genshi":             "Genshi",
 	"xml+kid":                "Genshi",
 	"xojo":                   "Xojo",
 	"xpages":                 "XPages",
+	"xpm":                    "X PixMap",
 	"xproc":                  "XProc",
 	"xquery":                 "XQuery",
 	"xs":                     "XS",
@@ -641,8 +726,34 @@ var LanguagesByAlias = map[string]string{
 	"yacc":                   "Yacc",
 	"yaml":                   "YAML",
 	"yang":                   "YANG",
+	"yara":                   "YARA",
+	"yas":                    "YASnippet",
+	"yasnippet":              "YASnippet",
 	"yml":                    "YAML",
 	"zephir":                 "Zephir",
+	"zig":                    "Zig",
 	"zimpl":                  "Zimpl",
 	"zsh":                    "Shell",
 }
+
+// LanguageByAlias looks up the language name by it's alias or name.
+// It mirrors the logic of github linguist and is needed e.g for heuristcs.yml
+// that mixes names and aliases in a language field (see XPM example).
+func LanguageByAlias(langOrAlias string) (lang string, ok bool) {
+	k := convertToAliasKey(langOrAlias)
+	lang, ok = LanguageByAliasMap[k]
+	return
+}
+
+// convertToAliasKey converts language name to a key in LanguageByAliasMap.
+// Following
+//  - internal.code-generator.generator.convertToAliasKey()
+//  - GetLanguageByAlias()
+// conventions.
+// It is here to avoid dependency on "generate" and "enry" packages.
+func convertToAliasKey(langName string) string {
+	ak := strings.SplitN(langName, `,`, 2)[0]
+	ak = strings.Replace(ak, ` `, `_`, -1)
+	ak = strings.ToLower(ak)
+	return ak
+}

internal/code-generator/generator/test_files/commit.gold

@@ -1,7 +1,7 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
 // linguist's commit from which files were generated.
-var LinguistCommit = "d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68"
+var LinguistCommit = "e4560984058b4726010ca4b8f03ed9d0f8f464db"

internal/code-generator/generator/test_files/documentation.gold

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -8,10 +8,12 @@ import "gopkg.in/toqueteos/substring.v1"
 var DocumentationMatchers = substring.Or(
 	substring.Regexp(`^[Dd]ocs?/`),
 	substring.Regexp(`(^|/)[Dd]ocumentation/`),
+	substring.Regexp(`(^|/)[Gg]roovydoc/`),
 	substring.Regexp(`(^|/)[Jj]avadoc/`),
 	substring.Regexp(`^[Mm]an/`),
 	substring.Regexp(`^[Ee]xamples/`),
 	substring.Regexp(`^[Dd]emos?/`),
+	substring.Regexp(`(^|/)inst/doc/`),
 	substring.Regexp(`(^|/)CHANGE(S|LOG)?(\.|$)`),
 	substring.Regexp(`(^|/)CONTRIBUTING(\.|$)`),
 	substring.Regexp(`(^|/)COPYING(\.|$)`),

internal/code-generator/generator/test_files/extension.gold

@@ -1,32 +1,34 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
 var LanguagesByExtension = map[string][]string{
-	".1":                   {"Roff"},
+	".1":                   {"Roff", "Roff Manpage"},
-	".1in":                 {"Roff"},
+	".1in":                 {"Roff", "Roff Manpage"},
-	".1m":                  {"Roff"},
+	".1m":                  {"Roff", "Roff Manpage"},
-	".1x":                  {"Roff"},
+	".1x":                  {"Roff", "Roff Manpage"},
-	".2":                   {"Roff"},
+	".2":                   {"Roff", "Roff Manpage"},
-	".3":                   {"Roff"},
+	".3":                   {"Roff", "Roff Manpage"},
-	".3in":                 {"Roff"},
+	".3in":                 {"Roff", "Roff Manpage"},
-	".3m":                  {"Roff"},
+	".3m":                  {"Roff", "Roff Manpage"},
-	".3qt":                 {"Roff"},
+	".3p":                  {"Roff", "Roff Manpage"},
-	".3x":                  {"Roff"},
+	".3pm":                 {"Roff", "Roff Manpage"},
-	".4":                   {"Roff"},
+	".3qt":                 {"Roff", "Roff Manpage"},
+	".3x":                  {"Roff", "Roff Manpage"},
+	".4":                   {"Roff", "Roff Manpage"},
 	".4th":                 {"Forth"},
-	".5":                   {"Roff"},
+	".5":                   {"Roff", "Roff Manpage"},
-	".6":                   {"Roff"},
+	".6":                   {"Roff", "Roff Manpage"},
 	".6pl":                 {"Perl 6"},
 	".6pm":                 {"Perl 6"},
-	".7":                   {"Roff"},
+	".7":                   {"Roff", "Roff Manpage"},
-	".8":                   {"Roff"},
+	".8":                   {"Roff", "Roff Manpage"},
 	".8xk":                 {"TI Program"},
 	".8xk.txt":             {"TI Program"},
 	".8xp":                 {"TI Program"},
 	".8xp.txt":             {"TI Program"},
-	".9":                   {"Roff"},
+	".9":                   {"Roff", "Roff Manpage"},
 	"._coffee":             {"CoffeeScript"},
 	"._js":                 {"JavaScript"},
 	"._ls":                 {"LiveScript"},
@@ -41,6 +43,7 @@ var LanguagesByExtension = map[string][]string{
 	".adoc":                {"AsciiDoc"},
 	".adp":                 {"Tcl"},
 	".ads":                 {"Ada"},
+	".afm":                 {"Adobe Font Metrics"},
 	".agc":                 {"Apollo Guidance Computer"},
 	".agda":                {"Agda"},
 	".ahk":                 {"AutoHotkey"},
@@ -49,6 +52,7 @@ var LanguagesByExtension = map[string][]string{
 	".al":                  {"Perl"},
 	".als":                 {"Alloy"},
 	".ampl":                {"AMPL"},
+	".angelscript":         {"AngelScript"},
 	".anim":                {"Unity3D Asset"},
 	".ant":                 {"XML"},
 	".apacheconf":          {"ApacheConf"},
@@ -58,7 +62,7 @@ var LanguagesByExtension = map[string][]string{
 	".applescript":         {"AppleScript"},
 	".arc":                 {"Arc"},
 	".arpa":                {"DNS Zone"},
-	".as":                  {"ActionScript"},
+	".as":                  {"ActionScript", "AngelScript"},
 	".asax":                {"ASP"},
 	".asc":                 {"AGS Script", "AsciiDoc", "Public Key"},
 	".asciidoc":            {"AsciiDoc"},
@@ -73,10 +77,12 @@ var LanguagesByExtension = map[string][]string{
 	".asp":                 {"ASP"},
 	".aspx":                {"ASP"},
 	".asset":               {"Unity3D Asset"},
+	".asy":                 {"Asymptote", "LTspice Symbol"},
 	".au3":                 {"AutoIt"},
 	".aug":                 {"Augeas"},
 	".auk":                 {"Awk"},
 	".aux":                 {"TeX"},
+	".avsc":                {"JSON"},
 	".aw":                  {"PHP"},
 	".awk":                 {"Awk"},
 	".axd":                 {"ASP"},
@@ -86,12 +92,14 @@ var LanguagesByExtension = map[string][]string{
 	".axs":                 {"NetLinx"},
 	".axs.erb":             {"NetLinx+ERB"},
 	".b":                   {"Brainfuck", "Limbo"},
+	".bal":                 {"Ballerina"},
 	".bas":                 {"Visual Basic"},
 	".bash":                {"Shell"},
 	".bat":                 {"Batchfile"},
 	".bats":                {"Shell"},
 	".bb":                  {"BitBake", "BlitzBasic"},
 	".bbx":                 {"TeX"},
+	".bdf":                 {"Glyph Bitmap Distribution Format"},
 	".bdy":                 {"PLSQL"},
 	".befunge":             {"Befunge"},
 	".bf":                  {"Brainfuck", "HyPhy"},
@@ -103,7 +111,7 @@ var LanguagesByExtension = map[string][]string{
 	".bones":               {"JavaScript"},
 	".boo":                 {"Boo"},
 	".boot":                {"Clojure"},
-	".brd":                 {"Eagle", "KiCad"},
+	".brd":                 {"Eagle", "KiCad Legacy Layout"},
 	".bro":                 {"Bro"},
 	".brs":                 {"Brightscript"},
 	".bsl":                 {"1C Enterprise"},
@@ -123,11 +131,12 @@ var LanguagesByExtension = map[string][]string{
 	".cbx":                 {"TeX"},
 	".cc":                  {"C++"},
 	".ccp":                 {"COBOL"},
+	".ccproj":              {"XML"},
 	".ccxml":               {"XML"},
 	".cdf":                 {"Mathematica"},
 	".ceylon":              {"Ceylon"},
 	".cfc":                 {"ColdFusion CFC"},
-	".cfg":                 {"INI"},
+	".cfg":                 {"HAProxy", "INI"},
 	".cfm":                 {"ColdFusion"},
 	".cfml":                {"ColdFusion"},
 	".cgi":                 {"Perl", "Python", "Shell"},
@@ -160,6 +169,8 @@ var LanguagesByExtension = map[string][]string{
 	".coffee":              {"CoffeeScript"},
 	".com":                 {"DIGITAL Command Language"},
 	".command":             {"Shell"},
+	".conll":               {"CoNLL-U"},
+	".conllu":              {"CoNLL-U"},
 	".coq":                 {"Coq"},
 	".cp":                  {"C++", "Component Pascal"},
 	".cpp":                 {"C++"},
@@ -172,9 +183,11 @@ var LanguagesByExtension = map[string][]string{
 	".cr":                  {"Crystal"},
 	".creole":              {"Creole"},
 	".cs":                  {"C#", "Smalltalk"},
+	".cscfg":               {"XML"},
 	".csd":                 {"Csound Document"},
+	".csdef":               {"XML"},
 	".csh":                 {"Tcsh"},
-	".cshtml":              {"C#"},
+	".cshtml":              {"HTML+Razor"},
 	".csl":                 {"XML"},
 	".cson":                {"CSON"},
 	".csproj":              {"XML"},
@@ -186,6 +199,7 @@ var LanguagesByExtension = map[string][]string{
 	".cu":                  {"Cuda"},
 	".cuh":                 {"Cuda"},
 	".cw":                  {"Redcode"},
+	".cwl":                 {"Common Workflow Language"},
 	".cxx":                 {"C++"},
 	".cxx-objdump":         {"Cpp-ObjDump"},
 	".cy":                  {"Cycript"},
@@ -199,6 +213,7 @@ var LanguagesByExtension = map[string][]string{
 	".dcl":                 {"Clean"},
 	".ddl":                 {"PLSQL", "SQL"},
 	".decls":               {"BlitzBasic"},
+	".depproj":             {"XML"},
 	".desktop":             {"desktop"},
 	".desktop.in":          {"desktop"},
 	".dfm":                 {"Pascal"},
@@ -221,6 +236,7 @@ var LanguagesByExtension = map[string][]string{
 	".druby":               {"Mirah"},
 	".dtx":                 {"TeX"},
 	".duby":                {"Mirah"},
+	".dwl":                 {"DataWeave"},
 	".dyalog":              {"APL"},
 	".dyl":                 {"Dylan"},
 	".dylan":               {"Dylan"},
@@ -234,6 +250,7 @@ var LanguagesByExtension = map[string][]string{
 	".eclass":              {"Gentoo Eclass"},
 	".eclxml":              {"ECL"},
 	".ecr":                 {"HTML+ECR"},
+	".edc":                 {"Edje Data Collection"},
 	".edn":                 {"edn"},
 	".eex":                 {"HTML+EEX"},
 	".eh":                  {"eC"},
@@ -246,6 +263,7 @@ var LanguagesByExtension = map[string][]string{
 	".emacs":               {"Emacs Lisp"},
 	".emacs.desktop":       {"Emacs Lisp"},
 	".emberscript":         {"EmberScript"},
+	".eml":                 {"EML"},
 	".epj":                 {"Ecere Projects"},
 	".eps":                 {"PostScript"},
 	".eq":                  {"EQ"},
@@ -273,6 +291,7 @@ var LanguagesByExtension = map[string][]string{
 	".filters":             {"XML"},
 	".fish":                {"fish"},
 	".flex":                {"JFlex"},
+	".flf":                 {"FIGlet Font"},
 	".flux":                {"FLUX"},
 	".fnc":                 {"PLSQL"},
 	".for":                 {"Formatted", "Forth", "Fortran"},
@@ -290,6 +309,7 @@ var LanguagesByExtension = map[string][]string{
 	".fshader":             {"GLSL"},
 	".fsi":                 {"F#"},
 	".fsproj":              {"XML"},
+	".fst":                 {"F*"},
 	".fsx":                 {"F#"},
 	".fth":                 {"Forth"},
 	".ftl":                 {"FreeMarker"},
@@ -302,6 +322,11 @@ var LanguagesByExtension = map[string][]string{
 	".g4":                  {"ANTLR"},
 	".gap":                 {"GAP"},
 	".gawk":                {"Awk"},
+	".gbl":                 {"Gerber Image"},
+	".gbo":                 {"Gerber Image"},
+	".gbp":                 {"Gerber Image"},
+	".gbr":                 {"Gerber Image"},
+	".gbs":                 {"Gerber Image"},
 	".gco":                 {"G-code"},
 	".gcode":               {"G-code"},
 	".gd":                  {"GAP", "GDScript"},
@@ -313,12 +338,17 @@ var LanguagesByExtension = map[string][]string{
 	".geom":                {"GLSL"},
 	".gf":                  {"Grammatical Framework"},
 	".gi":                  {"GAP"},
+	".gitconfig":           {"Git Config"},
+	".gitignore":           {"Ignore List"},
+	".gko":                 {"Gerber Image"},
 	".glade":               {"XML"},
 	".glf":                 {"Glyph"},
 	".glsl":                {"GLSL"},
 	".glslv":               {"GLSL"},
+	".gltf":                {"JSON"},
 	".gml":                 {"Game Maker Language", "Graph Modeling Language", "XML"},
 	".gms":                 {"GAMS"},
+	".gmx":                 {"XML"},
 	".gn":                  {"GN"},
 	".gni":                 {"GN"},
 	".gnu":                 {"Gnuplot"},
@@ -327,6 +357,9 @@ var LanguagesByExtension = map[string][]string{
 	".god":                 {"Ruby"},
 	".golo":                {"Golo"},
 	".gp":                  {"Gnuplot"},
+	".gpb":                 {"Gerber Image"},
+	".gpt":                 {"Gerber Image"},
+	".gql":                 {"GraphQL"},
 	".grace":               {"Grace"},
 	".gradle":              {"Gradle"},
 	".graphql":             {"GraphQL"},
@@ -338,7 +371,11 @@ var LanguagesByExtension = map[string][]string{
 	".gsp":                 {"Groovy Server Pages"},
 	".gst":                 {"Gosu"},
 	".gsx":                 {"Gosu"},
+	".gtl":                 {"Gerber Image"},
+	".gto":                 {"Gerber Image"},
+	".gtp":                 {"Gerber Image"},
 	".gtpl":                {"Groovy"},
+	".gts":                 {"Gerber Image"},
 	".gv":                  {"Graphviz (DOT)"},
 	".gvy":                 {"Groovy"},
 	".gyp":                 {"Python"},
@@ -348,6 +385,7 @@ var LanguagesByExtension = map[string][]string{
 	".haml":                {"Haml"},
 	".haml.deface":         {"Haml"},
 	".handlebars":          {"Handlebars"},
+	".har":                 {"JSON"},
 	".hats":                {"ATS"},
 	".hb":                  {"Harbour"},
 	".hbs":                 {"Handlebars"},
@@ -367,23 +405,27 @@ var LanguagesByExtension = map[string][]string{
 	".html.hl":             {"HTML"},
 	".http":                {"HTTP"},
 	".hx":                  {"Haxe"},
+	".hxml":                {"HXML"},
 	".hxsl":                {"Haxe"},
 	".hxx":                 {"C++"},
 	".hy":                  {"Hy"},
+	".i3":                  {"Modula-3"},
 	".i7x":                 {"Inform 7"},
+	".ice":                 {"JSON", "Slice"},
 	".iced":                {"CoffeeScript"},
 	".icl":                 {"Clean"},
 	".idc":                 {"C"},
 	".idr":                 {"Idris"},
+	".ig":                  {"Modula-3"},
 	".ihlp":                {"Stata"},
 	".ijs":                 {"J"},
 	".ik":                  {"Ioke"},
 	".ily":                 {"LilyPond"},
 	".iml":                 {"XML"},
-	".inc":                 {"Assembly", "C++", "HTML", "PAWN", "PHP", "POV-Ray SDL", "Pascal", "SQL", "SourcePawn"},
+	".inc":                 {"Assembly", "C++", "HTML", "PHP", "POV-Ray SDL", "Pascal", "Pawn", "SQL", "SourcePawn"},
 	".ini":                 {"INI"},
 	".inl":                 {"C++"},
-	".ino":                 {"Arduino"},
+	".ino":                 {"C++"},
 	".ins":                 {"TeX"},
 	".intr":                {"Dylan"},
 	".io":                  {"Io"},
@@ -402,6 +444,7 @@ var LanguagesByExtension = map[string][]string{
 	".jelly":               {"XML"},
 	".jflex":               {"JFlex"},
 	".jinja":               {"HTML+Django"},
+	".jinja2":              {"HTML+Django"},
 	".jison":               {"Jison"},
 	".jisonlex":            {"Jison Lex"},
 	".jl":                  {"Julia"},
@@ -414,15 +457,19 @@ var LanguagesByExtension = map[string][]string{
 	".json":                {"JSON"},
 	".json-tmlanguage":     {"JSON"},
 	".json5":               {"JSON5"},
+	".jsonl":               {"JSON"},
 	".jsonld":              {"JSONLD"},
 	".jsp":                 {"Java Server Pages"},
 	".jsproj":              {"XML"},
 	".jss":                 {"JavaScript"},
 	".jsx":                 {"JSX"},
-	".kicad_pcb":           {"KiCad"},
+	".kicad_mod":           {"KiCad Layout"},
+	".kicad_pcb":           {"KiCad Layout"},
+	".kicad_wks":           {"KiCad Layout"},
 	".kid":                 {"Genshi"},
 	".kit":                 {"Kit"},
 	".kml":                 {"XML"},
+	".kojo":                {"Scala"},
 	".krl":                 {"KRL"},
 	".ksh":                 {"Shell"},
 	".kt":                  {"Kotlin"},
@@ -441,6 +488,7 @@ var LanguagesByExtension = map[string][]string{
 	".ldml":                {"Lasso"},
 	".lds":                 {"Linker Script"},
 	".lean":                {"Lean"},
+	".lektorproject":       {"INI"},
 	".less":                {"Less"},
 	".lex":                 {"Lex"},
 	".lfe":                 {"LFE"},
@@ -465,13 +513,14 @@ var LanguagesByExtension = map[string][]string{
 	".lua":                 {"Lua"},
 	".lvproj":              {"LabVIEW"},
 	".ly":                  {"LilyPond"},
-	".m":                   {"Limbo", "M", "MUF", "Mathematica", "Matlab", "Mercury", "Objective-C"},
+	".m":                   {"Limbo", "M", "MATLAB", "MUF", "Mathematica", "Mercury", "Objective-C"},
+	".m3":                  {"Modula-3"},
 	".m4":                  {"M4", "M4Sugar"},
 	".ma":                  {"Mathematica"},
 	".mak":                 {"Makefile"},
 	".make":                {"Makefile"},
 	".mako":                {"Mako"},
-	".man":                 {"Roff"},
+	".man":                 {"Roff", "Roff Manpage"},
 	".mao":                 {"Mako"},
 	".markdown":            {"Markdown"},
 	".marko":               {"Marko"},
@@ -480,13 +529,15 @@ var LanguagesByExtension = map[string][]string{
 	".mata":                {"Stata"},
 	".matah":               {"Stata"},
 	".mathematica":         {"Mathematica"},
-	".matlab":              {"Matlab"},
+	".matlab":              {"MATLAB"},
 	".mawk":                {"Awk"},
 	".maxhelp":             {"Max"},
 	".maxpat":              {"Max"},
 	".maxproj":             {"Max"},
+	".mbox":                {"EML"},
 	".mcr":                 {"MAXScript"},
 	".md":                  {"GCC Machine Description", "Markdown"},
+	".mdoc":                {"Roff", "Roff Manpage"},
 	".mdown":               {"Markdown"},
 	".mdpolicy":            {"XML"},
 	".mdwn":                {"Markdown"},
@@ -494,10 +545,12 @@ var LanguagesByExtension = map[string][]string{
 	".mediawiki":           {"MediaWiki"},
 	".meta":                {"Unity3D Asset"},
 	".metal":               {"Metal"},
+	".mg":                  {"Modula-3"},
 	".minid":               {"MiniD"},
-	".mir":                 {"Mirah"},
+	".mir":                 {"YAML"},
 	".mirah":               {"Mirah"},
 	".mjml":                {"XML"},
+	".mjs":                 {"JavaScript"},
 	".mk":                  {"Makefile"},
 	".mkd":                 {"Markdown"},
 	".mkdn":                {"Markdown"},
@@ -518,6 +571,7 @@ var LanguagesByExtension = map[string][]string{
 	".mod":                 {"AMPL", "Linux Kernel Module", "Modula-2", "XML"},
 	".model.lkml":          {"LookML"},
 	".monkey":              {"Monkey"},
+	".monkey2":             {"Monkey"},
 	".moo":                 {"Mercury", "Moocode"},
 	".moon":                {"MoonScript"},
 	".mq4":                 {"MQL4"},
@@ -538,16 +592,25 @@ var LanguagesByExtension = map[string][]string{
 	".mysql":               {"SQL"},
 	".myt":                 {"Myghty"},
 	".n":                   {"Nemerle", "Roff"},
+	".nanorc":              {"nanorc"},
 	".nasm":                {"Assembly"},
+	".natvis":              {"XML"},
 	".nawk":                {"Awk"},
 	".nb":                  {"Mathematica", "Text"},
 	".nbp":                 {"Mathematica"},
 	".nc":                  {"nesC"},
-	".ncl":                 {"NCL", "Text"},
+	".ncl":                 {"NCL", "Text", "XML"},
+	".ndproj":              {"XML"},
+	".ne":                  {"Nearley"},
+	".nearley":             {"Nearley"},
+	".nf":                  {"Nextflow"},
 	".nginxconf":           {"Nginx"},
 	".ni":                  {"Inform 7"},
 	".nim":                 {"Nim"},
+	".nim.cfg":             {"Nim"},
+	".nimble":              {"Nim"},
 	".nimrod":              {"Nim"},
+	".nims":                {"Nim"},
 	".ninja":               {"Ninja"},
 	".nit":                 {"Nit"},
 	".nix":                 {"Nix"},
@@ -593,6 +656,7 @@ var LanguagesByExtension = map[string][]string{
 	".p6":                  {"Perl 6"},
 	".p6l":                 {"Perl 6"},
 	".p6m":                 {"Perl 6"},
+	".p8":                  {"Lua"},
 	".pac":                 {"JavaScript"},
 	".pan":                 {"Pan"},
 	".parrot":              {"Parrot"},
@@ -605,11 +669,14 @@ var LanguagesByExtension = map[string][]string{
 	".pbi":                 {"PureBasic"},
 	".pbt":                 {"PowerBuilder"},
 	".pck":                 {"PLSQL"},
+	".pcss":                {"PostCSS"},
 	".pd":                  {"Pure Data"},
 	".pd_lua":              {"Lua"},
 	".pde":                 {"Processing"},
 	".pep":                 {"Pep8"},
 	".perl":                {"Perl"},
+	".pfa":                 {"PostScript"},
+	".pgsql":               {"PLpgSQL"},
 	".ph":                  {"Perl"},
 	".php":                 {"Hack", "PHP"},
 	".php3":                {"PHP"},
@@ -636,11 +703,12 @@ var LanguagesByExtension = map[string][]string{
 	".plt":                 {"Gnuplot"},
 	".pluginspec":          {"Ruby", "XML"},
 	".plx":                 {"Perl"},
-	".pm":                  {"Perl", "Perl 6"},
+	".pm":                  {"Perl", "Perl 6", "X PixMap"},
 	".pm6":                 {"Perl 6"},
 	".pmod":                {"Pike"},
 	".po":                  {"Gettext Catalog"},
-	".pod":                 {"Perl", "Pod"},
+	".pod":                 {"Pod", "Pod 6"},
+	".pod6":                {"Pod 6"},
 	".podsl":               {"Common Lisp"},
 	".podspec":             {"Ruby"},
 	".pogo":                {"PogoScript"},
@@ -655,8 +723,9 @@ var LanguagesByExtension = map[string][]string{
 	".prg":                 {"xBase"},
 	".pri":                 {"QMake"},
 	".pro":                 {"IDL", "INI", "Prolog", "QMake"},
+	".proj":                {"XML"},
 	".prolog":              {"Prolog"},
-	".properties":          {"INI"},
+	".properties":          {"INI", "Java Properties"},
 	".props":               {"XML"},
 	".proto":               {"Protocol Buffer"},
 	".prw":                 {"xBase"},
@@ -672,7 +741,7 @@ var LanguagesByExtension = map[string][]string{
 	".pub":                 {"Public Key"},
 	".pug":                 {"Pug"},
 	".purs":                {"PureScript"},
-	".pwn":                 {"PAWN"},
+	".pwn":                 {"Pawn"},
 	".pxd":                 {"Cython"},
 	".pxi":                 {"Cython"},
 	".py":                  {"Python"},
@@ -684,6 +753,7 @@ var LanguagesByExtension = map[string][]string{
 	".pytb":                {"Python traceback"},
 	".pyw":                 {"Python"},
 	".pyx":                 {"Cython"},
+	".q":                   {"HiveQL", "q"},
 	".qbs":                 {"QML"},
 	".qml":                 {"QML"},
 	".r":                   {"R", "Rebol"},
@@ -713,6 +783,7 @@ var LanguagesByExtension = map[string][]string{
 	".red":                 {"Red"},
 	".reds":                {"Red"},
 	".reek":                {"YAML"},
+	".reg":                 {"Windows Registry Entries"},
 	".regex":               {"Regular Expression"},
 	".regexp":              {"Regular Expression"},
 	".rei":                 {"Reason"},
@@ -733,7 +804,7 @@ var LanguagesByExtension = map[string][]string{
 	".rno":                 {"RUNOFF", "Roff"},
 	".robot":               {"RobotFramework"},
 	".roff":                {"Roff"},
-	".ron":                 {"Markdown"},
+	".ronn":                {"Markdown"},
 	".rpy":                 {"Python", "Ren'Py"},
 	".rq":                  {"SPARQL"},
 	".rs":                  {"RenderScript", "Rust"},
@@ -760,7 +831,7 @@ var LanguagesByExtension = map[string][]string{
 	".scaml":               {"Scaml"},
 	".scd":                 {"SuperCollider"},
 	".sce":                 {"Scilab"},
-	".sch":                 {"Eagle", "KiCad", "XML"},
+	".sch":                 {"Eagle", "KiCad Schematic", "Scheme", "XML"},
 	".sci":                 {"Scilab"},
 	".scm":                 {"Scheme"},
 	".sco":                 {"Csound Score"},
@@ -768,6 +839,7 @@ var LanguagesByExtension = map[string][]string{
 	".scrbl":               {"Racket"},
 	".scss":                {"SCSS"},
 	".scxml":               {"XML"},
+	".sed":                 {"sed"},
 	".self":                {"Self"},
 	".sexp":                {"Common Lisp"},
 	".sfd":                 {"Spline Font Database"},
@@ -777,6 +849,7 @@ var LanguagesByExtension = map[string][]string{
 	".sh.in":               {"Shell"},
 	".shader":              {"GLSL", "ShaderLab"},
 	".shen":                {"Shen"},
+	".shproj":              {"XML"},
 	".sig":                 {"Standard ML"},
 	".sj":                  {"Objective-J"},
 	".sjs":                 {"JavaScript"},
@@ -805,6 +878,7 @@ var LanguagesByExtension = map[string][]string{
 	".srw":                 {"PowerBuilder"},
 	".ss":                  {"Scheme"},
 	".ssjs":                {"JavaScript"},
+	".sss":                 {"SugarSS"},
 	".st":                  {"HTML", "Smalltalk"},
 	".stan":                {"Stan"},
 	".sthlp":               {"Stata"},
@@ -813,21 +887,21 @@ var LanguagesByExtension = map[string][]string{
 	".sttheme":             {"XML"},
 	".sty":                 {"TeX"},
 	".styl":                {"Stylus"},
-	".sublime-build":       {"Sublime Text Config"},
+	".sublime-build":       {"JSON with Comments"},
-	".sublime-commands":    {"Sublime Text Config"},
+	".sublime-commands":    {"JSON with Comments"},
-	".sublime-completions": {"Sublime Text Config"},
+	".sublime-completions": {"JSON with Comments"},
-	".sublime-keymap":      {"Sublime Text Config"},
+	".sublime-keymap":      {"JSON with Comments"},
-	".sublime-macro":       {"Sublime Text Config"},
+	".sublime-macro":       {"JSON with Comments"},
-	".sublime-menu":        {"Sublime Text Config"},
+	".sublime-menu":        {"JSON with Comments"},
-	".sublime-mousemap":    {"Sublime Text Config"},
+	".sublime-mousemap":    {"JSON with Comments"},
-	".sublime-project":     {"Sublime Text Config"},
+	".sublime-project":     {"JSON with Comments"},
-	".sublime-settings":    {"Sublime Text Config"},
+	".sublime-settings":    {"JSON with Comments"},
 	".sublime-snippet":     {"XML"},
 	".sublime-syntax":      {"YAML"},
-	".sublime-theme":       {"Sublime Text Config"},
+	".sublime-theme":       {"JSON with Comments"},
-	".sublime-workspace":   {"Sublime Text Config"},
+	".sublime-workspace":   {"JSON with Comments"},
-	".sublime_metrics":     {"Sublime Text Config"},
+	".sublime_metrics":     {"JSON with Comments"},
-	".sublime_session":     {"Sublime Text Config"},
+	".sublime_session":     {"JSON with Comments"},
 	".sv":                  {"SystemVerilog"},
 	".svg":                 {"SVG"},
 	".svh":                 {"SystemVerilog"},
@@ -846,6 +920,9 @@ var LanguagesByExtension = map[string][]string{
 	".tex":                 {"TeX"},
 	".textile":             {"Textile"},
 	".tf":                  {"HCL"},
+	".tfstate":             {"JSON"},
+	".tfstate.backup":      {"JSON"},
+	".tfvars":              {"HCL"},
 	".thor":                {"Ruby"},
 	".thrift":              {"Thrift"},
 	".thy":                 {"Isabelle"},
@@ -924,11 +1001,14 @@ var LanguagesByExtension = map[string][]string{
 	".vue":                 {"Vue"},
 	".vw":                  {"PLSQL"},
 	".vxml":                {"XML"},
-	".w":                   {"CWeb"},
+	".w":                   {"CWeb", "OpenEdge ABL"},
 	".wast":                {"WebAssembly"},
 	".wat":                 {"WebAssembly"},
 	".watchr":              {"Ruby"},
+	".wdl":                 {"wdl"},
+	".webapp":              {"JSON"},
 	".webidl":              {"WebIDL"},
+	".webmanifest":         {"JSON"},
 	".weechatlog":          {"IRC log"},
 	".wiki":                {"MediaWiki"},
 	".wisp":                {"wisp"},
@@ -937,17 +1017,19 @@ var LanguagesByExtension = map[string][]string{
 	".wlt":                 {"Mathematica"},
 	".wlua":                {"Lua"},
 	".workbook":            {"Markdown"},
+	".workflow":            {"HCL", "XML"},
 	".wsdl":                {"XML"},
 	".wsf":                 {"XML"},
 	".wsgi":                {"Python"},
 	".wxi":                 {"XML"},
 	".wxl":                 {"XML"},
 	".wxs":                 {"XML"},
-	".x":                   {"Logos"},
+	".x":                   {"Linker Script", "Logos", "RPC"},
 	".x10":                 {"X10"},
 	".x3d":                 {"XML"},
 	".xacro":               {"XML"},
 	".xaml":                {"XML"},
+	".xbm":                 {"X BitMap"},
 	".xc":                  {"XC"},
 	".xht":                 {"HTML"},
 	".xhtml":               {"HTML"},
@@ -966,6 +1048,7 @@ var LanguagesByExtension = map[string][]string{
 	".xojo_toolbar":        {"Xojo"},
 	".xojo_window":         {"Xojo"},
 	".xpl":                 {"XProc"},
+	".xpm":                 {"X PixMap"},
 	".xproc":               {"XProc"},
 	".xproj":               {"XML"},
 	".xpy":                 {"Python"},
@@ -983,6 +1066,7 @@ var LanguagesByExtension = map[string][]string{
 	".xslt":                {"XSLT"},
 	".xsp-config":          {"XPages"},
 	".xsp.metadata":        {"XPages"},
+	".xspec":               {"XML"},
 	".xtend":               {"Xtend"},
 	".xul":                 {"XML"},
 	".y":                   {"Yacc"},
@@ -991,12 +1075,17 @@ var LanguagesByExtension = map[string][]string{
 	".yaml-tmlanguage":     {"YAML"},
 	".yang":                {"YANG"},
 	".yap":                 {"Prolog"},
+	".yar":                 {"YARA"},
+	".yara":                {"YARA"},
+	".yasnippet":           {"YASnippet"},
 	".yml":                 {"YAML"},
 	".yml.mysql":           {"YAML"},
 	".yrl":                 {"Erlang"},
-	".yy":                  {"Yacc"},
+	".yy":                  {"JSON", "Yacc"},
+	".yyp":                 {"JSON"},
 	".zcml":                {"XML"},
 	".zep":                 {"Zephir"},
+	".zig":                 {"Zig"},
 	".zimpl":               {"Zimpl"},
 	".zmpl":                {"Zimpl"},
 	".zone":                {"DNS Zone"},
@@ -1018,21 +1107,24 @@ var ExtensionsByLanguage = map[string][]string{
 	"ATS":                {".dats", ".hats", ".sats"},
 	"ActionScript":       {".as"},
 	"Ada":                {".adb", ".ada", ".ads"},
+	"Adobe Font Metrics": {".afm"},
 	"Agda":               {".agda"},
 	"Alloy":              {".als"},
+	"AngelScript":        {".as", ".angelscript"},
 	"ApacheConf":         {".apacheconf", ".vhost"},
 	"Apex":               {".cls"},
 	"Apollo Guidance Computer": {".agc"},
 	"AppleScript":              {".applescript", ".scpt"},
 	"Arc":                      {".arc"},
-	"Arduino":                  {".ino"},
 	"AsciiDoc":                 {".asciidoc", ".adoc", ".asc"},
 	"AspectJ":                  {".aj"},
 	"Assembly":                 {".asm", ".a51", ".inc", ".nasm"},
+	"Asymptote":                {".asy"},
 	"Augeas":                   {".aug"},
 	"AutoHotkey":               {".ahk", ".ahkl"},
 	"AutoIt":                   {".au3"},
 	"Awk":                      {".awk", ".auk", ".gawk", ".mawk", ".nawk"},
+	"Ballerina":                {".bal"},
 	"Batchfile":                {".bat", ".cmd"},
 	"Befunge":                  {".befunge"},
 	"Bison":                    {".bison"},
@@ -1046,8 +1138,8 @@ var ExtensionsByLanguage = map[string][]string{
 	"Brightscript":             {".brs"},
 	"Bro":                      {".bro"},
 	"C":                        {".c", ".cats", ".h", ".idc"},
-	"C#":                       {".cs", ".cake", ".cshtml", ".csx"},
+	"C#":                       {".cs", ".cake", ".csx"},
-	"C++":                      {".cpp", ".c++", ".cc", ".cp", ".cxx", ".h", ".h++", ".hh", ".hpp", ".hxx", ".inc", ".inl", ".ipp", ".re", ".tcc", ".tpp"},
+	"C++":                      {".cpp", ".c++", ".cc", ".cp", ".cxx", ".h", ".h++", ".hh", ".hpp", ".hxx", ".inc", ".inl", ".ino", ".ipp", ".re", ".tcc", ".tpp"},
 	"C-ObjDump":                {".c-objdump"},
 	"C2hs Haskell":             {".chs"},
 	"CLIPS":                    {".clp"},
@@ -1070,10 +1162,12 @@ var ExtensionsByLanguage = map[string][]string{
 	"Click":                    {".click"},
 	"Clojure":                  {".clj", ".boot", ".cl2", ".cljc", ".cljs", ".cljs.hl", ".cljscm", ".cljx", ".hic"},
 	"Closure Templates":        {".soy"},
+	"CoNLL-U":                  {".conllu", ".conll"},
 	"CoffeeScript":             {".coffee", "._coffee", ".cake", ".cjsx", ".iced"},
 	"ColdFusion":               {".cfm", ".cfml"},
 	"ColdFusion CFC":           {".cfc"},
 	"Common Lisp":              {".lisp", ".asd", ".cl", ".l", ".lsp", ".ny", ".podsl", ".sexp"},
+	"Common Workflow Language": {".cwl"},
 	"Component Pascal":         {".cp", ".cps"},
 	"Cool":                     {".cl"},
 	"Coq":                      {".coq", ".v"},
@@ -1094,6 +1188,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"DTrace":               {".d"},
 	"Darcs Patch":          {".darcspatch", ".dpatch"},
 	"Dart":                 {".dart"},
+	"DataWeave":            {".dwl"},
 	"Diff":                 {".diff", ".patch"},
 	"Dockerfile":           {".dockerfile"},
 	"Dogescript":           {".djs"},
@@ -1103,10 +1198,12 @@ var ExtensionsByLanguage = map[string][]string{
 	"ECL":                  {".ecl", ".eclxml"},
 	"ECLiPSe":              {".ecl"},
 	"EJS":                  {".ejs"},
+	"EML":                  {".eml", ".mbox"},
 	"EQ":                   {".eq"},
 	"Eagle":                {".sch", ".brd"},
 	"Easybuild":            {".eb"},
 	"Ecere Projects":       {".epj"},
+	"Edje Data Collection": {".edc"},
 	"Eiffel":               {".e"},
 	"Elixir":               {".ex", ".exs"},
 	"Elm":                  {".elm"},
@@ -1114,6 +1211,8 @@ var ExtensionsByLanguage = map[string][]string{
 	"EmberScript":          {".em", ".emberscript"},
 	"Erlang":               {".erl", ".app.src", ".es", ".escript", ".hrl", ".xrl", ".yrl"},
 	"F#":                   {".fs", ".fsi", ".fsx"},
+	"F*":                   {".fst"},
+	"FIGlet Font":          {".flf"},
 	"FLUX":                 {".fx", ".flux"},
 	"Factor":               {".factor"},
 	"Fancy":                {".fy", ".fancypack"},
@@ -1138,9 +1237,12 @@ var ExtensionsByLanguage = map[string][]string{
 	"Genshi":              {".kid"},
 	"Gentoo Ebuild":       {".ebuild"},
 	"Gentoo Eclass":       {".eclass"},
+	"Gerber Image":        {".gbr", ".gbl", ".gbo", ".gbp", ".gbs", ".gko", ".gpb", ".gpt", ".gtl", ".gto", ".gtp", ".gts"},
 	"Gettext Catalog":     {".po", ".pot"},
 	"Gherkin":             {".feature"},
+	"Git Config":          {".gitconfig"},
 	"Glyph":               {".glf"},
+	"Glyph Bitmap Distribution Format": {".bdf"},
 	"Gnuplot":                          {".gp", ".gnu", ".gnuplot", ".plot", ".plt"},
 	"Go":                               {".go"},
 	"Golo":                             {".golo"},
@@ -1149,32 +1251,37 @@ var ExtensionsByLanguage = map[string][]string{
 	"Gradle":                           {".gradle"},
 	"Grammatical Framework":            {".gf"},
 	"Graph Modeling Language":          {".gml"},
-	"GraphQL":                 {".graphql"},
+	"GraphQL":                          {".graphql", ".gql"},
 	"Graphviz (DOT)":                   {".dot", ".gv"},
 	"Groovy":                           {".groovy", ".grt", ".gtpl", ".gvy"},
 	"Groovy Server Pages":              {".gsp"},
-	"HCL":                      {".hcl", ".tf"},
+	"HAProxy":                          {".cfg"},
+	"HCL":                              {".hcl", ".tf", ".tfvars", ".workflow"},
 	"HLSL":                             {".hlsl", ".cginc", ".fx", ".fxh", ".hlsli"},
 	"HTML":                             {".html", ".htm", ".html.hl", ".inc", ".st", ".xht", ".xhtml"},
-	"HTML+Django":              {".jinja", ".mustache", ".njk"},
+	"HTML+Django":                      {".jinja", ".jinja2", ".mustache", ".njk"},
 	"HTML+ECR":                         {".ecr"},
 	"HTML+EEX":                         {".eex"},
 	"HTML+ERB":                         {".erb", ".erb.deface"},
 	"HTML+PHP":                         {".phtml"},
+	"HTML+Razor":                       {".cshtml"},
 	"HTTP":                             {".http"},
+	"HXML":                             {".hxml"},
 	"Hack":                             {".hh", ".php"},
 	"Haml":                             {".haml", ".haml.deface"},
 	"Handlebars":                       {".handlebars", ".hbs"},
 	"Harbour":                          {".hb"},
 	"Haskell":                          {".hs", ".hsc"},
 	"Haxe":                             {".hx", ".hxsl"},
+	"HiveQL":                           {".q"},
 	"Hy":                               {".hy"},
 	"HyPhy":                            {".bf"},
 	"IDL":                              {".pro", ".dlm"},
 	"IGOR Pro":                         {".ipf"},
-	"INI":                      {".ini", ".cfg", ".prefs", ".pro", ".properties"},
+	"INI":                              {".ini", ".cfg", ".lektorproject", ".prefs", ".pro", ".properties"},
 	"IRC log":                          {".irclog", ".weechatlog"},
 	"Idris":                            {".idr", ".lidr"},
+	"Ignore List":                      {".gitignore"},
 	"Inform 7":                         {".ni", ".i7x"},
 	"Inno Setup":                       {".iss"},
 	"Io":                               {".io"},
@@ -1182,28 +1289,33 @@ var ExtensionsByLanguage = map[string][]string{
 	"Isabelle":                         {".thy"},
 	"J":                                {".ijs"},
 	"JFlex":                            {".flex", ".jflex"},
-	"JSON":                     {".json", ".geojson", ".json-tmlanguage", ".topojson"},
+	"JSON":                             {".json", ".avsc", ".geojson", ".gltf", ".har", ".ice", ".json-tmlanguage", ".jsonl", ".tfstate", ".tfstate.backup", ".topojson", ".webapp", ".webmanifest", ".yy", ".yyp"},
+	"JSON with Comments":               {".sublime-build", ".sublime-commands", ".sublime-completions", ".sublime-keymap", ".sublime-macro", ".sublime-menu", ".sublime-mousemap", ".sublime-project", ".sublime-settings", ".sublime-theme", ".sublime-workspace", ".sublime_metrics", ".sublime_session"},
 	"JSON5":                            {".json5"},
 	"JSONLD":                           {".jsonld"},
 	"JSONiq":                           {".jq"},
 	"JSX":                              {".jsx"},
 	"Jasmin":                           {".j"},
 	"Java":                             {".java"},
+	"Java Properties":                  {".properties"},
 	"Java Server Pages":                {".jsp"},
-	"JavaScript":               {".js", "._js", ".bones", ".es", ".es6", ".frag", ".gs", ".jake", ".jsb", ".jscad", ".jsfl", ".jsm", ".jss", ".njs", ".pac", ".sjs", ".ssjs", ".xsjs", ".xsjslib"},
+	"JavaScript":                       {".js", "._js", ".bones", ".es", ".es6", ".frag", ".gs", ".jake", ".jsb", ".jscad", ".jsfl", ".jsm", ".jss", ".mjs", ".njs", ".pac", ".sjs", ".ssjs", ".xsjs", ".xsjslib"},
 	"Jison":                            {".jison"},
 	"Jison Lex":                        {".jisonlex"},
 	"Jolie":                            {".ol", ".iol"},
 	"Julia":                            {".jl"},
 	"Jupyter Notebook":                 {".ipynb"},
 	"KRL":                              {".krl"},
-	"KiCad":                    {".sch", ".brd", ".kicad_pcb"},
+	"KiCad Layout":                     {".kicad_pcb", ".kicad_mod", ".kicad_wks"},
+	"KiCad Legacy Layout":              {".brd"},
+	"KiCad Schematic":                  {".sch"},
 	"Kit":                              {".kit"},
 	"Kotlin":                           {".kt", ".ktm", ".kts"},
 	"LFE":                              {".lfe"},
 	"LLVM":                             {".ll"},
 	"LOLCODE":                          {".lol"},
 	"LSL":                              {".lsl", ".lslp"},
+	"LTspice Symbol":                   {".asy"},
 	"LabVIEW":                          {".lvproj"},
 	"Lasso":                            {".lasso", ".las", ".lasso8", ".lasso9", ".ldml"},
 	"Latte":                            {".latte"},
@@ -1212,7 +1324,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"Lex":                              {".l", ".lex"},
 	"LilyPond":                         {".ly", ".ily"},
 	"Limbo":                            {".b", ".m"},
-	"Linker Script":            {".ld", ".lds"},
+	"Linker Script":                    {".ld", ".lds", ".x"},
 	"Linux Kernel Module":              {".mod"},
 	"Liquid":                           {".liquid"},
 	"Literate Agda":                    {".lagda"},
@@ -1223,10 +1335,11 @@ var ExtensionsByLanguage = map[string][]string{
 	"Logtalk":                          {".lgt", ".logtalk"},
 	"LookML":                           {".lookml", ".model.lkml", ".view.lkml"},
 	"LoomScript":                       {".ls"},
-	"Lua":                      {".lua", ".fcgi", ".nse", ".pd_lua", ".rbxs", ".wlua"},
+	"Lua":                              {".lua", ".fcgi", ".nse", ".p8", ".pd_lua", ".rbxs", ".wlua"},
 	"M":                                {".mumps", ".m"},
 	"M4":                               {".m4"},
 	"M4Sugar":                          {".m4"},
+	"MATLAB":                           {".matlab", ".m"},
 	"MAXScript":                        {".ms", ".mcr"},
 	"MQL4":                             {".mq4", ".mqh"},
 	"MQL5":                             {".mq5", ".mqh"},
@@ -1234,34 +1347,36 @@ var ExtensionsByLanguage = map[string][]string{
 	"MUF":                              {".muf", ".m"},
 	"Makefile":                         {".mak", ".d", ".make", ".mk", ".mkfile"},
 	"Mako":                             {".mako", ".mao"},
-	"Markdown":                 {".md", ".markdown", ".mdown", ".mdwn", ".mkd", ".mkdn", ".mkdown", ".ron", ".workbook"},
+	"Markdown":                         {".md", ".markdown", ".mdown", ".mdwn", ".mkd", ".mkdn", ".mkdown", ".ronn", ".workbook"},
 	"Marko":                            {".marko"},
 	"Mask":                             {".mask"},
 	"Mathematica":                      {".mathematica", ".cdf", ".m", ".ma", ".mt", ".nb", ".nbp", ".wl", ".wlt"},
-	"Matlab":                   {".matlab", ".m"},
 	"Max":                              {".maxpat", ".maxhelp", ".maxproj", ".mxt", ".pat"},
 	"MediaWiki":                        {".mediawiki", ".wiki"},
 	"Mercury":                          {".m", ".moo"},
 	"Metal":                            {".metal"},
 	"MiniD":                            {".minid"},
-	"Mirah":                    {".druby", ".duby", ".mir", ".mirah"},
+	"Mirah":                            {".druby", ".duby", ".mirah"},
 	"Modelica":                         {".mo"},
 	"Modula-2":                         {".mod"},
+	"Modula-3":                         {".i3", ".ig", ".m3", ".mg"},
 	"Module Management System":         {".mms", ".mmk"},
-	"Monkey":                   {".monkey"},
+	"Monkey":                           {".monkey", ".monkey2"},
 	"Moocode":                          {".moo"},
 	"MoonScript":                       {".moon"},
 	"Myghty":                           {".myt"},
 	"NCL":                              {".ncl"},
 	"NL":                               {".nl"},
 	"NSIS":                             {".nsi", ".nsh"},
+	"Nearley":                          {".ne", ".nearley"},
 	"Nemerle":                          {".n"},
 	"NetLinx":                          {".axs", ".axi"},
 	"NetLinx+ERB":                      {".axs.erb", ".axi.erb"},
 	"NetLogo":                          {".nlogo"},
 	"NewLisp":                          {".nl", ".lisp", ".lsp"},
+	"Nextflow":                         {".nf"},
 	"Nginx":                            {".nginxconf", ".vhost"},
-	"Nim":                      {".nim", ".nimrod"},
+	"Nim":                              {".nim", ".nim.cfg", ".nimble", ".nimrod", ".nims"},
 	"Ninja":                            {".ninja"},
 	"Nit":                              {".nit"},
 	"Nix":                              {".nix"},
@@ -1276,7 +1391,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"Opa":                              {".opa"},
 	"Opal":                             {".opal"},
 	"OpenCL":                           {".cl", ".opencl"},
-	"OpenEdge ABL":             {".p", ".cls"},
+	"OpenEdge ABL":                     {".p", ".cls", ".w"},
 	"OpenSCAD":                         {".scad"},
 	"OpenType Feature File":            {".fea"},
 	"Org":                            {".org"},
@@ -1284,10 +1399,9 @@ var ExtensionsByLanguage = map[string][]string{
 	"Oxygene":                        {".oxygene"},
 	"Oz":                             {".oz"},
 	"P4":                             {".p4"},
-	"PAWN":                           {".pwn", ".inc"},
 	"PHP":                            {".php", ".aw", ".ctp", ".fcgi", ".inc", ".php3", ".php4", ".php5", ".phps", ".phpt"},
 	"PLSQL":                          {".pls", ".bdy", ".ddl", ".fnc", ".pck", ".pkb", ".pks", ".plb", ".plsql", ".prc", ".spc", ".sql", ".tpb", ".tps", ".trg", ".vw"},
-	"PLpgSQL":                        {".sql"},
+	"PLpgSQL":                        {".pgsql", ".sql"},
 	"POV-Ray SDL":                    {".pov", ".inc"},
 	"Pan":                            {".pan"},
 	"Papyrus":                        {".psc"},
@@ -1295,8 +1409,9 @@ var ExtensionsByLanguage = map[string][]string{
 	"Parrot Assembly":                {".pasm"},
 	"Parrot Internal Representation": {".pir"},
 	"Pascal":                       {".pas", ".dfm", ".dpr", ".inc", ".lpr", ".pascal", ".pp"},
+	"Pawn":                         {".pwn", ".inc"},
 	"Pep8":                         {".pep"},
-	"Perl":                         {".pl", ".al", ".cgi", ".fcgi", ".perl", ".ph", ".plx", ".pm", ".pod", ".psgi", ".t"},
+	"Perl":                         {".pl", ".al", ".cgi", ".fcgi", ".perl", ".ph", ".plx", ".pm", ".psgi", ".t"},
 	"Perl 6":                       {".6pl", ".6pm", ".nqp", ".p6", ".p6l", ".p6m", ".pl", ".pl6", ".pm", ".pm6", ".t"},
 	"Pic":                          {".pic", ".chem"},
 	"Pickle":                       {".pkl"},
@@ -1304,9 +1419,11 @@ var ExtensionsByLanguage = map[string][]string{
 	"PigLatin":                     {".pig"},
 	"Pike":                         {".pike", ".pmod"},
 	"Pod":                          {".pod"},
+	"Pod 6":                        {".pod", ".pod6"},
 	"PogoScript":                   {".pogo"},
 	"Pony":                         {".pony"},
-	"PostScript":                   {".ps", ".eps"},
+	"PostCSS":                      {".pcss"},
+	"PostScript":                   {".ps", ".eps", ".pfa"},
 	"PowerBuilder":                 {".pbt", ".sra", ".sru", ".srw"},
 	"PowerShell":                   {".ps1", ".psd1", ".psm1"},
 	"Processing":                   {".pde"},
@@ -1330,6 +1447,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"REXX":                         {".rexx", ".pprx", ".rex"},
 	"RHTML":                        {".rhtml"},
 	"RMarkdown":                    {".rmd"},
+	"RPC":                          {".x"},
 	"RPM Spec":                     {".spec"},
 	"RUNOFF":                       {".rnh", ".rno"},
 	"Racket":                       {".rkt", ".rktd", ".rktl", ".scrbl"},
@@ -1345,7 +1463,8 @@ var ExtensionsByLanguage = map[string][]string{
 	"RenderScript":                 {".rs", ".rsh"},
 	"Ring":                         {".ring"},
 	"RobotFramework":               {".robot"},
-	"Roff":                         {".man", ".1", ".1in", ".1m", ".1x", ".2", ".3", ".3in", ".3m", ".3qt", ".3x", ".4", ".5", ".6", ".7", ".8", ".9", ".l", ".me", ".ms", ".n", ".nr", ".rno", ".roff", ".tmac"},
+	"Roff":                         {".roff", ".1", ".1in", ".1m", ".1x", ".2", ".3", ".3in", ".3m", ".3p", ".3pm", ".3qt", ".3x", ".4", ".5", ".6", ".7", ".8", ".9", ".l", ".man", ".mdoc", ".me", ".ms", ".n", ".nr", ".rno", ".tmac"},
+	"Roff Manpage":                 {".1", ".1in", ".1m", ".1x", ".2", ".3", ".3in", ".3m", ".3p", ".3pm", ".3qt", ".3x", ".4", ".5", ".6", ".7", ".8", ".9", ".man", ".mdoc"},
 	"Rouge":                        {".rg"},
 	"Ruby":                         {".rb", ".builder", ".eye", ".fcgi", ".gemspec", ".god", ".jbuilder", ".mspec", ".pluginspec", ".podspec", ".rabl", ".rake", ".rbuild", ".rbw", ".rbx", ".ru", ".ruby", ".spec", ".thor", ".watchr"},
 	"Rust":                         {".rs", ".rs.in"},
@@ -1362,9 +1481,9 @@ var ExtensionsByLanguage = map[string][]string{
 	"Sage":                         {".sage", ".sagews"},
 	"SaltStack":                    {".sls"},
 	"Sass":                         {".sass"},
-	"Scala":                        {".scala", ".sbt", ".sc"},
+	"Scala":                        {".scala", ".kojo", ".sbt", ".sc"},
 	"Scaml":                        {".scaml"},
-	"Scheme":                       {".scm", ".sld", ".sls", ".sps", ".ss"},
+	"Scheme":                       {".scm", ".sch", ".sld", ".sls", ".sps", ".ss"},
 	"Scilab":                       {".sci", ".sce", ".tst"},
 	"Self":                         {".self"},
 	"ShaderLab":                    {".shader"},
@@ -1372,6 +1491,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"ShellSession":                 {".sh-session"},
 	"Shen":                         {".shen"},
 	"Slash":                        {".sl"},
+	"Slice":                        {".ice"},
 	"Slim":                         {".slim"},
 	"Smali":                        {".smali"},
 	"Smalltalk":                    {".st", ".cs"},
@@ -1384,7 +1504,7 @@ var ExtensionsByLanguage = map[string][]string{
 	"Stata":                        {".do", ".ado", ".doh", ".ihlp", ".mata", ".matah", ".sthlp"},
 	"Stylus":                       {".styl"},
 	"SubRip Text":                  {".srt"},
-	"Sublime Text Config":          {".sublime-build", ".sublime-commands", ".sublime-completions", ".sublime-keymap", ".sublime-macro", ".sublime-menu", ".sublime-mousemap", ".sublime-project", ".sublime-settings", ".sublime-theme", ".sublime-workspace", ".sublime_metrics", ".sublime_session"},
+	"SugarSS":                      {".sss"},
 	"SuperCollider":                {".sc", ".scd"},
 	"Swift":                        {".swift"},
 	"SystemVerilog":                {".sv", ".svh", ".vh"},
@@ -1424,10 +1544,13 @@ var ExtensionsByLanguage = map[string][]string{
 	"Web Ontology Language":        {".owl"},
 	"WebAssembly":                  {".wast", ".wat"},
 	"WebIDL":                       {".webidl"},
+	"Windows Registry Entries":     {".reg"},
 	"World of Warcraft Addon Data": {".toc"},
+	"X BitMap":                     {".xbm"},
+	"X PixMap":                     {".xpm", ".pm"},
 	"X10":                          {".x10"},
 	"XC":                           {".xc"},
-	"XML":              {".xml", ".adml", ".admx", ".ant", ".axml", ".builds", ".ccxml", ".clixml", ".cproject", ".csl", ".csproj", ".ct", ".dita", ".ditamap", ".ditaval", ".dll.config", ".dotsettings", ".filters", ".fsproj", ".fxml", ".glade", ".gml", ".grxml", ".iml", ".ivy", ".jelly", ".jsproj", ".kml", ".launch", ".mdpolicy", ".mjml", ".mm", ".mod", ".mxml", ".nproj", ".nuspec", ".odd", ".osm", ".pkgproj", ".plist", ".pluginspec", ".props", ".ps1xml", ".psc1", ".pt", ".rdf", ".resx", ".rss", ".sch", ".scxml", ".sfproj", ".srdf", ".storyboard", ".sttheme", ".sublime-snippet", ".targets", ".tmcommand", ".tml", ".tmlanguage", ".tmpreferences", ".tmsnippet", ".tmtheme", ".ts", ".tsx", ".ui", ".urdf", ".ux", ".vbproj", ".vcxproj", ".vsixmanifest", ".vssettings", ".vstemplate", ".vxml", ".wixproj", ".wsdl", ".wsf", ".wxi", ".wxl", ".wxs", ".x3d", ".xacro", ".xaml", ".xib", ".xlf", ".xliff", ".xmi", ".xml.dist", ".xproj", ".xsd", ".xul", ".zcml"},
+	"XML":                          {".xml", ".adml", ".admx", ".ant", ".axml", ".builds", ".ccproj", ".ccxml", ".clixml", ".cproject", ".cscfg", ".csdef", ".csl", ".csproj", ".ct", ".depproj", ".dita", ".ditamap", ".ditaval", ".dll.config", ".dotsettings", ".filters", ".fsproj", ".fxml", ".glade", ".gml", ".gmx", ".grxml", ".iml", ".ivy", ".jelly", ".jsproj", ".kml", ".launch", ".mdpolicy", ".mjml", ".mm", ".mod", ".mxml", ".natvis", ".ncl", ".ndproj", ".nproj", ".nuspec", ".odd", ".osm", ".pkgproj", ".plist", ".pluginspec", ".proj", ".props", ".ps1xml", ".psc1", ".pt", ".rdf", ".resx", ".rss", ".sch", ".scxml", ".sfproj", ".shproj", ".srdf", ".storyboard", ".sttheme", ".sublime-snippet", ".targets", ".tmcommand", ".tml", ".tmlanguage", ".tmpreferences", ".tmsnippet", ".tmtheme", ".ts", ".tsx", ".ui", ".urdf", ".ux", ".vbproj", ".vcxproj", ".vsixmanifest", ".vssettings", ".vstemplate", ".vxml", ".wixproj", ".workflow", ".wsdl", ".wsf", ".wxi", ".wxl", ".wxs", ".x3d", ".xacro", ".xaml", ".xib", ".xlf", ".xliff", ".xmi", ".xml.dist", ".xproj", ".xsd", ".xspec", ".xul", ".zcml"},
 	"XPages":                       {".xsp-config", ".xsp.metadata"},
 	"XProc":                        {".xpl", ".xproc"},
 	"XQuery":                       {".xquery", ".xq", ".xql", ".xqm", ".xqy"},
@@ -1435,19 +1558,26 @@ var ExtensionsByLanguage = map[string][]string{
 	"XSLT":                         {".xslt", ".xsl"},
 	"Xojo":                         {".xojo_code", ".xojo_menu", ".xojo_report", ".xojo_script", ".xojo_toolbar", ".xojo_window"},
 	"Xtend":                        {".xtend"},
-	"YAML":             {".yml", ".reek", ".rviz", ".sublime-syntax", ".syntax", ".yaml", ".yaml-tmlanguage", ".yml.mysql"},
+	"YAML":                         {".yml", ".mir", ".reek", ".rviz", ".sublime-syntax", ".syntax", ".yaml", ".yaml-tmlanguage", ".yml.mysql"},
 	"YANG":                         {".yang"},
+	"YARA":                         {".yar", ".yara"},
+	"YASnippet":                    {".yasnippet"},
 	"Yacc":                         {".y", ".yacc", ".yy"},
 	"Zephir":                       {".zep"},
+	"Zig":                          {".zig"},
 	"Zimpl":                        {".zimpl", ".zmpl", ".zpl"},
 	"desktop":                      {".desktop", ".desktop.in"},
 	"eC":                           {".ec", ".eh"},
 	"edn":                          {".edn"},
 	"fish":                         {".fish"},
 	"mupad":                        {".mu"},
+	"nanorc":                       {".nanorc"},
 	"nesC":                         {".nc"},
 	"ooc":                          {".ooc"},
+	"q":                            {".q"},
 	"reStructuredText":             {".rst", ".rest", ".rest.txt", ".rst.txt"},
+	"sed":                          {".sed"},
+	"wdl":                          {".wdl"},
 	"wisp":                         {".wisp"},
 	"xBase":                        {".prg", ".ch", ".prw"},
 }

internal/code-generator/generator/test_files/filename.gold

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -8,40 +8,65 @@ var LanguagesByFilename = map[string][]string{
 	".XCompose":          {"XCompose"},
 	".abbrev_defs":       {"Emacs Lisp"},
 	".arcconfig":         {"JSON"},
-	".babelrc":           {"JSON5"},
+	".atomignore":        {"Ignore List"},
+	".babelignore":       {"Ignore List"},
+	".babelrc":           {"JSON with Comments"},
+	".bash_aliases":      {"Shell"},
 	".bash_history":      {"Shell"},
 	".bash_logout":       {"Shell"},
 	".bash_profile":      {"Shell"},
 	".bashrc":            {"Shell"},
+	".bzrignore":         {"Ignore List"},
 	".clang-format":      {"YAML"},
+	".clang-tidy":        {"YAML"},
 	".classpath":         {"XML"},
+	".coffeelintignore":  {"Ignore List"},
 	".cproject":          {"XML"},
 	".cshrc":             {"Shell"},
+	".cvsignore":         {"Ignore List"},
+	".dockerignore":      {"Ignore List"},
 	".editorconfig":      {"INI"},
 	".emacs":             {"Emacs Lisp"},
 	".emacs.desktop":     {"Emacs Lisp"},
+	".eslintignore":      {"Ignore List"},
+	".eslintrc.json":     {"JSON with Comments"},
 	".factor-boot-rc":    {"Factor"},
 	".factor-rc":         {"Factor"},
 	".gclient":           {"Python"},
 	".gemrc":             {"YAML"},
-	".gitconfig":         {"INI"},
+	".gitattributes":     {"Git Attributes"},
+	".gitconfig":         {"Git Config"},
+	".gitignore":         {"Ignore List"},
+	".gitmodules":        {"Git Config"},
 	".gn":                {"GN"},
 	".gnus":              {"Emacs Lisp"},
 	".gvimrc":            {"Vim script"},
 	".htaccess":          {"ApacheConf"},
+	".htmlhintrc":        {"JSON"},
 	".irbrc":             {"Ruby"},
-	".jshintrc":          {"JSON"},
+	".jscsrc":            {"JSON with Comments"},
+	".jshintrc":          {"JSON with Comments"},
+	".jslintrc":          {"JSON with Comments"},
 	".login":             {"Shell"},
+	".nanorc":            {"nanorc"},
+	".nodemonignore":     {"Ignore List"},
+	".npmignore":         {"Ignore List"},
 	".nvimrc":            {"Vim script"},
 	".php":               {"PHP"},
 	".php_cs":            {"PHP"},
 	".php_cs.dist":       {"PHP"},
+	".prettierignore":    {"Ignore List"},
 	".profile":           {"Shell"},
 	".project":           {"XML"},
 	".pryrc":             {"Ruby"},
 	".spacemacs":         {"Emacs Lisp"},
+	".stylelintignore":   {"Ignore List"},
+	".tern-config":       {"JSON"},
+	".tern-project":      {"JSON"},
 	".vimrc":             {"Vim script"},
 	".viper":             {"Emacs Lisp"},
+	".vscodeignore":      {"Ignore List"},
+	".watchmanconfig":    {"JSON"},
 	".zlogin":            {"Shell"},
 	".zlogout":           {"Shell"},
 	".zprofile":          {"Shell"},
@@ -54,6 +79,7 @@ var LanguagesByFilename = map[string][]string{
 	"BSDmakefile":        {"Makefile"},
 	"BUCK":               {"Python"},
 	"BUILD":              {"Python"},
+	"BUILD.bazel":        {"Python"},
 	"Berksfile":          {"Ruby"},
 	"Brewfile":           {"Ruby"},
 	"Buildfile":          {"Ruby"},
@@ -63,6 +89,7 @@ var LanguagesByFilename = map[string][]string{
 	"COPYRIGHT.regex":    {"Text"},
 	"Cakefile":           {"CoffeeScript"},
 	"Capfile":            {"Ruby"},
+	"Cargo.lock":         {"TOML"},
 	"Cask":               {"Emacs Lisp"},
 	"Dangerfile":         {"Ruby"},
 	"Deliverfile":        {"Ruby"},
@@ -74,6 +101,7 @@ var LanguagesByFilename = map[string][]string{
 	"GNUmakefile":        {"Makefile"},
 	"Gemfile":            {"Ruby"},
 	"Gemfile.lock":       {"Ruby"},
+	"Gopkg.lock":         {"TOML"},
 	"Guardfile":          {"Ruby"},
 	"INSTALL":            {"Text"},
 	"INSTALL.mysql":      {"Text"},
@@ -84,6 +112,7 @@ var LanguagesByFilename = map[string][]string{
 	"LICENSE":            {"Text"},
 	"LICENSE.mysql":      {"Text"},
 	"Makefile":           {"Makefile"},
+	"Makefile.PL":        {"Perl"},
 	"Makefile.am":        {"Makefile"},
 	"Makefile.boot":      {"Makefile"},
 	"Makefile.frag":      {"Makefile"},
@@ -106,7 +135,7 @@ var LanguagesByFilename = map[string][]string{
 	"README.mysql":       {"Text"},
 	"ROOT":               {"Isabelle ROOT"},
 	"Rakefile":           {"Ruby"},
-	"Rexfile":            {"Perl 6"},
+	"Rexfile":            {"Perl"},
 	"SConscript":         {"Python"},
 	"SConstruct":         {"Python"},
 	"Settings.StyleCop":  {"XML"},
@@ -126,25 +155,44 @@ var LanguagesByFilename = map[string][]string{
 	"ack":                {"Perl"},
 	"ant.xml":            {"Ant Build System"},
 	"apache2.conf":       {"ApacheConf"},
+	"bash_aliases":       {"Shell"},
 	"bash_logout":        {"Shell"},
 	"bash_profile":       {"Shell"},
 	"bashrc":             {"Shell"},
 	"build.xml":          {"Ant Build System"},
 	"buildfile":          {"Ruby"},
+	"buildozer.spec":     {"INI"},
 	"click.me":           {"Text"},
 	"composer.lock":      {"JSON"},
 	"configure.ac":       {"M4Sugar"},
+	"contents.lr":        {"Markdown"},
+	"cpanfile":           {"Perl"},
 	"cshrc":              {"Shell"},
 	"delete.me":          {"Text"},
 	"descrip.mmk":        {"Module Management System"},
 	"descrip.mms":        {"Module Management System"},
+	"encodings.dir":      {"X Font Directory Index"},
 	"expr-dist":          {"R"},
+	"firestore.rules":    {"Cloud Firestore Security Rules"},
+	"fonts.alias":        {"X Font Directory Index"},
+	"fonts.dir":          {"X Font Directory Index"},
+	"fonts.scale":        {"X Font Directory Index"},
+	"fp-lib-table":       {"KiCad Layout"},
+	"gitignore-global":   {"Ignore List"},
+	"gitignore_global":   {"Ignore List"},
+	"glide.lock":         {"YAML"},
+	"go.mod":             {"Text"},
+	"go.sum":             {"Text"},
 	"gradlew":            {"Shell"},
 	"gvimrc":             {"Vim script"},
+	"haproxy.cfg":        {"HAProxy"},
 	"httpd.conf":         {"ApacheConf"},
+	"jsconfig.json":      {"JSON with Comments"},
 	"keep.me":            {"Text"},
 	"ld.script":          {"Linker Script"},
 	"login":              {"Shell"},
+	"m3makefile":         {"Quake"},
+	"m3overrides":        {"Quake"},
 	"makefile":           {"Makefile"},
 	"makefile.sco":       {"Makefile"},
 	"man":                {"Shell"},
@@ -155,7 +203,10 @@ var LanguagesByFilename = map[string][]string{
 	"mkfile":             {"Makefile"},
 	"mmn":                {"Roff"},
 	"mmt":                {"Roff"},
+	"nanorc":             {"nanorc"},
+	"nextflow.config":    {"Nextflow"},
 	"nginx.conf":         {"Nginx"},
+	"nim.cfg":            {"Nim"},
 	"nvimrc":             {"Vim script"},
 	"owh":                {"Tcl"},
 	"packages.config":    {"XML"},
@@ -167,9 +218,9 @@ var LanguagesByFilename = map[string][]string{
 	"rebar.config.lock":  {"Erlang"},
 	"rebar.lock":         {"Erlang"},
 	"riemann.config":     {"Clojure"},
-	"script":             {"C"},
 	"starfield":          {"Tcl"},
 	"test.me":            {"Text"},
+	"tsconfig.json":      {"JSON with Comments"},
 	"vimrc":              {"Vim script"},
 	"wscript":            {"Python"},
 	"xcompose":           {"XCompose"},

internal/code-generator/generator/test_files/heuristics.yml

@@ -0,0 +1,40 @@
+# Tests care about number and order of heuristics in this fixture
+disambiguations:
+
+- extensions: ['.h', '.hh']
+  rules:
+  - language: Objective-C
+    pattern: 'objc'
+  - language: C++
+    named_pattern: cpp
+
+- extensions: ['.f']
+  rules:
+  - language: Forth
+    pattern:  #as in .md
+      - 'f'
+      - 'f1'
+  - language: Filebench WML
+    pattern: 'f2'
+  - language: Fortran
+    named_pattern: fortran
+
+- extensions: ['.ms']
+  rules:
+  - language: Roff
+    pattern: 'rp'
+  - language: Unix Assembly
+    and:
+      - negative_pattern: 'np'
+      - pattern: 'p'
+  - language: MAXScript
+
+- extensions: ['.mod']
+  rules:
+  - language: [Linux Kernel Module, AMPL]
+
+named_patterns:
+  cpp:
+  - 'regex1'
+  - 'regex2'
+  fortran: 'regex3'

internal/code-generator/generator/test_files/interpreter.gold

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -7,6 +7,8 @@ var LanguagesByInterpreter = map[string][]string{
 	"Rscript":     {"R"},
 	"apl":         {"APL"},
 	"aplx":        {"APL"},
+	"ash":         {"Shell"},
+	"asy":         {"Asymptote"},
 	"awk":         {"Awk"},
 	"bash":        {"Shell"},
 	"bigloo":      {"Scheme"},
@@ -15,10 +17,13 @@ var LanguagesByInterpreter = map[string][]string{
 	"chicken":     {"Scheme"},
 	"clisp":       {"Common Lisp"},
 	"coffee":      {"CoffeeScript"},
+	"cperl":       {"Perl"},
 	"crystal":     {"Crystal"},
 	"csi":         {"Scheme"},
 	"cvc4":        {"SMT"},
+	"cwl-runner":  {"Common Workflow Language"},
 	"dart":        {"Dart"},
+	"dash":        {"Shell"},
 	"dtrace":      {"DTrace"},
 	"dyalog":      {"APL"},
 	"ecl":         {"Common Lisp"},
@@ -26,11 +31,15 @@ var LanguagesByInterpreter = map[string][]string{
 	"escript":     {"Erlang"},
 	"fish":        {"fish"},
 	"gawk":        {"Awk"},
+	"gerbv":       {"Gerber Image"},
+	"gerbview":    {"Gerber Image"},
 	"gn":          {"GN"},
 	"gnuplot":     {"Gnuplot"},
 	"gosh":        {"Scheme"},
 	"groovy":      {"Groovy"},
+	"gsed":        {"sed"},
 	"guile":       {"Scheme"},
+	"hy":          {"Hy"},
 	"instantfpc":  {"Pascal"},
 	"io":          {"Io"},
 	"ioke":        {"Ioke"},
@@ -38,6 +47,7 @@ var LanguagesByInterpreter = map[string][]string{
 	"jolie":       {"Jolie"},
 	"jruby":       {"Ruby"},
 	"julia":       {"Julia"},
+	"ksh":         {"Shell"},
 	"lisp":        {"Common Lisp"},
 	"lsl":         {"LSL"},
 	"lua":         {"Lua", "Terra"},
@@ -45,11 +55,14 @@ var LanguagesByInterpreter = map[string][]string{
 	"make":        {"Makefile"},
 	"mathsat5":    {"SMT"},
 	"mawk":        {"Awk"},
+	"minised":     {"sed"},
+	"mksh":        {"Shell"},
 	"mmi":         {"Mercury"},
 	"moon":        {"MoonScript"},
 	"nawk":        {"Awk"},
 	"newlisp":     {"NewLisp"},
-	"node":        {"JavaScript"},
+	"nextflow":    {"Nextflow"},
+	"node":        {"JavaScript", "TypeScript"},
 	"nush":        {"Nu"},
 	"ocaml":       {"OCaml", "Reason"},
 	"ocamlrun":    {"OCaml"},
@@ -58,12 +71,14 @@ var LanguagesByInterpreter = map[string][]string{
 	"opensmt":     {"SMT"},
 	"osascript":   {"AppleScript"},
 	"parrot":      {"Parrot Assembly", "Parrot Internal Representation"},
-	"perl":        {"Perl"},
+	"pdksh":       {"Shell"},
-	"perl6":       {"Perl 6"},
+	"perl":        {"Perl", "Pod"},
+	"perl6":       {"Perl 6", "Pod 6"},
 	"php":         {"PHP"},
 	"picolisp":    {"PicoLisp"},
 	"pike":        {"Pike"},
 	"pil":         {"PicoLisp"},
+	"pwsh":        {"PowerShell"},
 	"python":      {"Python"},
 	"python2":     {"Python"},
 	"python3":     {"Python"},
@@ -80,11 +95,14 @@ var LanguagesByInterpreter = map[string][]string{
 	"runhaskell":  {"Haskell"},
 	"sbcl":        {"Common Lisp"},
 	"scala":       {"Scala"},
+	"scheme":      {"Scheme"},
 	"sclang":      {"SuperCollider"},
 	"scsynth":     {"SuperCollider"},
+	"sed":         {"sed"},
 	"sh":          {"Shell"},
 	"smt-rat":     {"SMT"},
 	"smtinterpol": {"SMT"},
+	"ssed":        {"sed"},
 	"stp":         {"SMT"},
 	"swipl":       {"Prolog"},
 	"tcc":         {"C"},

internal/code-generator/generator/test_files/mimeType.gold

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -9,9 +9,10 @@ var LanguagesMime = map[string]string{
 	"ASN.1":                          "text/x-ttcn-asn",
 	"ASP":                            "application/x-aspx",
 	"Alpine Abuild":                  "text/x-sh",
+	"AngelScript":                    "text/x-c++src",
 	"Ant Build System":               "application/xml",
 	"Apex":                           "text/x-java",
-	"Arduino":                 "text/x-c++src",
+	"Asymptote":                      "text/x-kotlin",
 	"Brainfuck":                      "text/x-brainfuck",
 	"C":                              "text/x-csrc",
 	"C#":                             "text/x-csharp",
@@ -25,8 +26,10 @@ var LanguagesMime = map[string]string{
 	"ChucK":                          "text/x-java",
 	"Clojure":                        "text/x-clojure",
 	"Closure Templates":              "text/x-soy",
+	"Cloud Firestore Security Rules": "text/css",
 	"CoffeeScript":                   "text/x-coffeescript",
 	"Common Lisp":                    "text/x-common-lisp",
+	"Common Workflow Language":       "text/x-yaml",
 	"Component Pascal":               "text/x-pascal",
 	"Crystal":                        "text/x-crystal",
 	"Cuda":                           "text/x-c++src",
@@ -44,6 +47,7 @@ var LanguagesMime = map[string]string{
 	"Eagle":                          "text/xml",
 	"Easybuild":                      "text/x-python",
 	"Ecere Projects":                 "application/json",
+	"Edje Data Collection":           "application/json",
 	"Eiffel":                         "text/x-eiffel",
 	"Elm":                            "text/x-elm",
 	"Emacs Lisp":                     "text/x-common-lisp",
@@ -59,6 +63,8 @@ var LanguagesMime = map[string]string{
 	"Genshi":              "text/xml",
 	"Gentoo Ebuild":       "text/x-sh",
 	"Gentoo Eclass":       "text/x-sh",
+	"Git Attributes":      "text/x-sh",
+	"Git Config":          "text/x-properties",
 	"Glyph":               "text/x-tcl",
 	"Go":                  "text/x-go",
 	"Grammatical Framework": "text/x-haskell",
@@ -71,6 +77,7 @@ var LanguagesMime = map[string]string{
 	"HTML+EEX":                 "text/html",
 	"HTML+ERB":                 "application/x-erb",
 	"HTML+PHP":                 "application/x-httpd-php",
+	"HTML+Razor":               "text/html",
 	"HTTP":                     "message/http",
 	"Hack":                     "application/x-httpd-php",
 	"Haml":                     "text/x-haml",
@@ -79,18 +86,24 @@ var LanguagesMime = map[string]string{
 	"IDL":                      "text/x-idl",
 	"INI":                      "text/x-properties",
 	"IRC log":                  "text/mirc",
+	"Ignore List":              "text/x-sh",
 	"JSON":                     "application/json",
+	"JSON with Comments":       "text/javascript",
 	"JSON5":                    "application/json",
+	"JSONLD":                   "application/json",
 	"JSONiq":                   "application/json",
 	"JSX":                      "text/jsx",
 	"Java":                     "text/x-java",
+	"Java Properties":          "text/x-properties",
 	"Java Server Pages":        "application/x-jsp",
 	"JavaScript":               "text/javascript",
 	"Julia":                    "text/x-julia",
 	"Jupyter Notebook":         "application/json",
+	"KiCad Layout":             "text/x-common-lisp",
 	"Kit":                      "text/html",
 	"Kotlin":                   "text/x-kotlin",
 	"LFE":                      "text/x-common-lisp",
+	"LTspice Symbol":           "text/x-spreadsheet",
 	"LabVIEW":                  "text/xml",
 	"Latte":                    "text/x-smarty",
 	"Less":                     "text/css",
@@ -99,13 +112,13 @@ var LanguagesMime = map[string]string{
 	"LookML":                   "text/x-yaml",
 	"Lua":                      "text/x-lua",
 	"M":                        "text/x-mumps",
+	"MATLAB":                   "text/x-octave",
 	"MTML":                     "text/html",
 	"MUF":                      "text/x-forth",
 	"Makefile":                 "text/x-cmake",
 	"Markdown":                 "text/x-gfm",
 	"Marko":                    "text/html",
 	"Mathematica":              "text/x-mathematica",
-	"Matlab":              "text/x-octave",
 	"Maven POM":                "text/xml",
 	"Max":                      "application/json",
 	"Metal":                    "text/x-c++src",
@@ -145,6 +158,7 @@ var LanguagesMime = map[string]string{
 	"RPM Spec":                 "text/x-rpm-spec",
 	"Reason":                   "text/x-rustsrc",
 	"Roff":                     "text/troff",
+	"Roff Manpage":             "text/troff",
 	"Rouge":                    "text/x-clojure",
 	"Ruby":                     "text/x-ruby",
 	"Rust":                     "text/x-rustsrc",
@@ -167,7 +181,6 @@ var LanguagesMime = map[string]string{
 	"Smarty":                   "text/x-smarty",
 	"Squirrel":                 "text/x-c++src",
 	"Standard ML":              "text/x-ocaml",
-	"Sublime Text Config": "text/javascript",
 	"Swift":                    "text/x-swift",
 	"SystemVerilog":            "text/x-systemverilog",
 	"TOML":                     "text/x-toml",
@@ -189,6 +202,9 @@ var LanguagesMime = map[string]string{
 	"Volt":                     "text/x-d",
 	"WebAssembly":              "text/x-common-lisp",
 	"WebIDL":                   "text/x-webidl",
+	"Windows Registry Entries": "text/x-properties",
+	"X BitMap":                 "text/x-csrc",
+	"X PixMap":                 "text/x-csrc",
 	"XC":                       "text/x-csrc",
 	"XML":                      "text/xml",
 	"XPages":                   "text/xml",

internal/code-generator/generator/test_files/type.gold

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -17,23 +17,26 @@ var LanguagesType = map[string]int{
 	"ATS":                2,
 	"ActionScript":       2,
 	"Ada":                2,
+	"Adobe Font Metrics": 1,
 	"Agda":               2,
 	"Alloy":              2,
 	"Alpine Abuild":      2,
+	"AngelScript":        2,
 	"Ant Build System":   1,
-	"ApacheConf":       3,
+	"ApacheConf":         1,
 	"Apex":               2,
 	"Apollo Guidance Computer":       2,
 	"AppleScript":                    2,
 	"Arc":                            2,
-	"Arduino":                  2,
 	"AsciiDoc":                       4,
 	"AspectJ":                        2,
 	"Assembly":                       2,
+	"Asymptote":                      2,
 	"Augeas":                         2,
 	"AutoHotkey":                     2,
 	"AutoIt":                         2,
 	"Awk":                            2,
+	"Ballerina":                      2,
 	"Batchfile":                      2,
 	"Befunge":                        2,
 	"Bison":                          2,
@@ -71,10 +74,13 @@ var LanguagesType = map[string]int{
 	"Click":                          2,
 	"Clojure":                        2,
 	"Closure Templates":              3,
+	"Cloud Firestore Security Rules": 1,
+	"CoNLL-U":                        1,
 	"CoffeeScript":                   2,
 	"ColdFusion":                     2,
 	"ColdFusion CFC":                 2,
 	"Common Lisp":                    2,
+	"Common Workflow Language":       2,
 	"Component Pascal":               2,
 	"Cool":                           2,
 	"Coq":                            2,
@@ -95,8 +101,9 @@ var LanguagesType = map[string]int{
 	"DTrace":               2,
 	"Darcs Patch":          1,
 	"Dart":                 2,
+	"DataWeave":            2,
 	"Diff":                 1,
-	"Dockerfile":     1,
+	"Dockerfile":           2,
 	"Dogescript":           2,
 	"Dylan":                2,
 	"E":                    2,
@@ -104,10 +111,12 @@ var LanguagesType = map[string]int{
 	"ECL":                  2,
 	"ECLiPSe":              2,
 	"EJS":                  3,
+	"EML":                  1,
 	"EQ":                   2,
-	"Eagle":          3,
+	"Eagle":                1,
 	"Easybuild":            1,
 	"Ecere Projects":       1,
+	"Edje Data Collection": 1,
 	"Eiffel":               2,
 	"Elixir":               2,
 	"Elm":                  2,
@@ -115,6 +124,8 @@ var LanguagesType = map[string]int{
 	"EmberScript":          2,
 	"Erlang":               2,
 	"F#":                   2,
+	"F*":                   2,
+	"FIGlet Font":          1,
 	"FLUX":                 2,
 	"Factor":               2,
 	"Fancy":                2,
@@ -139,9 +150,13 @@ var LanguagesType = map[string]int{
 	"Genshi":              2,
 	"Gentoo Ebuild":       2,
 	"Gentoo Eclass":       2,
+	"Gerber Image":        1,
 	"Gettext Catalog":     4,
 	"Gherkin":             2,
+	"Git Attributes":      1,
+	"Git Config":          1,
 	"Glyph":               2,
+	"Glyph Bitmap Distribution Format": 1,
 	"Gnuplot":                          2,
 	"Go":                               2,
 	"Golo":                             2,
@@ -154,6 +169,7 @@ var LanguagesType = map[string]int{
 	"Graphviz (DOT)":                   1,
 	"Groovy":                           2,
 	"Groovy Server Pages":              2,
+	"HAProxy":                          1,
 	"HCL":                              2,
 	"HLSL":                             2,
 	"HTML":                             3,
@@ -162,13 +178,16 @@ var LanguagesType = map[string]int{
 	"HTML+EEX":                         3,
 	"HTML+ERB":                         3,
 	"HTML+PHP":                         3,
+	"HTML+Razor":                       3,
 	"HTTP":                             1,
+	"HXML":                             1,
 	"Hack":                             2,
 	"Haml":                             3,
 	"Handlebars":                       3,
 	"Harbour":                          2,
 	"Haskell":                          2,
 	"Haxe":                             2,
+	"HiveQL":                           2,
 	"Hy":                               2,
 	"HyPhy":                            2,
 	"IDL":                              2,
@@ -176,6 +195,7 @@ var LanguagesType = map[string]int{
 	"INI":                              1,
 	"IRC log":                          1,
 	"Idris":                            2,
+	"Ignore List":                      1,
 	"Inform 7":                         2,
 	"Inno Setup":                       2,
 	"Io":                               2,
@@ -185,12 +205,14 @@ var LanguagesType = map[string]int{
 	"J":                                2,
 	"JFlex":                            2,
 	"JSON":                             1,
+	"JSON with Comments":               1,
 	"JSON5":                            1,
 	"JSONLD":                           1,
 	"JSONiq":                           2,
 	"JSX":                              2,
 	"Jasmin":                           2,
 	"Java":                             2,
+	"Java Properties":                  1,
 	"Java Server Pages":                2,
 	"JavaScript":                       2,
 	"Jison":                            2,
@@ -199,13 +221,16 @@ var LanguagesType = map[string]int{
 	"Julia":                            2,
 	"Jupyter Notebook":                 3,
 	"KRL":                              2,
-	"KiCad":                    2,
+	"KiCad Layout":                     1,
+	"KiCad Legacy Layout":              1,
+	"KiCad Schematic":                  1,
 	"Kit":                              3,
 	"Kotlin":                           2,
 	"LFE":                              2,
 	"LLVM":                             2,
 	"LOLCODE":                          2,
 	"LSL":                              2,
+	"LTspice Symbol":                   1,
 	"LabVIEW":                          2,
 	"Lasso":                            2,
 	"Latte":                            3,
@@ -229,6 +254,7 @@ var LanguagesType = map[string]int{
 	"M":                                2,
 	"M4":                               2,
 	"M4Sugar":                          2,
+	"MATLAB":                           2,
 	"MAXScript":                        2,
 	"MQL4":                             2,
 	"MQL5":                             2,
@@ -240,7 +266,6 @@ var LanguagesType = map[string]int{
 	"Marko":                            3,
 	"Mask":                             3,
 	"Mathematica":                      2,
-	"Matlab":                   2,
 	"Maven POM":                        1,
 	"Max":                              2,
 	"MediaWiki":                        4,
@@ -251,6 +276,7 @@ var LanguagesType = map[string]int{
 	"Mirah":                            2,
 	"Modelica":                         2,
 	"Modula-2":                         2,
+	"Modula-3":                         2,
 	"Module Management System":         2,
 	"Monkey":                           2,
 	"Moocode":                          2,
@@ -259,12 +285,14 @@ var LanguagesType = map[string]int{
 	"NCL":                              2,
 	"NL":                               1,
 	"NSIS":                             2,
+	"Nearley":                          2,
 	"Nemerle":                          2,
 	"NetLinx":                          2,
 	"NetLinx+ERB":                      2,
 	"NetLogo":                          2,
 	"NewLisp":                          2,
-	"Nginx":                    3,
+	"Nextflow":                         2,
+	"Nginx":                            1,
 	"Nim":                              2,
 	"Ninja":                            1,
 	"Nit":                              2,
@@ -289,7 +317,6 @@ var LanguagesType = map[string]int{
 	"Oxygene":                        2,
 	"Oz":                             2,
 	"P4":                             2,
-	"PAWN":                           2,
 	"PHP":                            2,
 	"PLSQL":                          2,
 	"PLpgSQL":                        2,
@@ -300,6 +327,7 @@ var LanguagesType = map[string]int{
 	"Parrot Assembly":                2,
 	"Parrot Internal Representation": 2,
 	"Pascal":                       2,
+	"Pawn":                         2,
 	"Pep8":                         2,
 	"Perl":                         2,
 	"Perl 6":                       2,
@@ -309,19 +337,21 @@ var LanguagesType = map[string]int{
 	"PigLatin":                     2,
 	"Pike":                         2,
 	"Pod":                          4,
+	"Pod 6":                        4,
 	"PogoScript":                   2,
 	"Pony":                         2,
+	"PostCSS":                      3,
 	"PostScript":                   3,
 	"PowerBuilder":                 2,
 	"PowerShell":                   2,
 	"Processing":                   2,
 	"Prolog":                       2,
 	"Propeller Spin":               2,
-	"Protocol Buffer":              3,
+	"Protocol Buffer":              1,
 	"Public Key":                   1,
 	"Pug":                          3,
 	"Puppet":                       2,
-	"Pure Data":                    2,
+	"Pure Data":                    1,
 	"PureBasic":                    2,
 	"PureScript":                   2,
 	"Python":                       2,
@@ -329,6 +359,7 @@ var LanguagesType = map[string]int{
 	"Python traceback":             1,
 	"QML":                          2,
 	"QMake":                        2,
+	"Quake":                        2,
 	"R":                            2,
 	"RAML":                         3,
 	"RDoc":                         4,
@@ -336,6 +367,7 @@ var LanguagesType = map[string]int{
 	"REXX":                         2,
 	"RHTML":                        3,
 	"RMarkdown":                    4,
+	"RPC":                          2,
 	"RPM Spec":                     1,
 	"RUNOFF":                       3,
 	"Racket":                       2,
@@ -352,6 +384,7 @@ var LanguagesType = map[string]int{
 	"Ring":                         2,
 	"RobotFramework":               2,
 	"Roff":                         3,
+	"Roff Manpage":                 3,
 	"Rouge":                        2,
 	"Ruby":                         2,
 	"Rust":                         2,
@@ -378,10 +411,12 @@ var LanguagesType = map[string]int{
 	"ShellSession":                 2,
 	"Shen":                         2,
 	"Slash":                        2,
+	"Slice":                        2,
 	"Slim":                         3,
 	"Smali":                        2,
 	"Smalltalk":                    2,
 	"Smarty":                       2,
+	"Solidity":                     2,
 	"SourcePawn":                   2,
 	"Spline Font Database":         1,
 	"Squirrel":                     2,
@@ -390,7 +425,7 @@ var LanguagesType = map[string]int{
 	"Stata":                        2,
 	"Stylus":                       3,
 	"SubRip Text":                  1,
-	"Sublime Text Config":          1,
+	"SugarSS":                      3,
 	"SuperCollider":                2,
 	"Swift":                        2,
 	"SystemVerilog":                2,
@@ -427,15 +462,19 @@ var LanguagesType = map[string]int{
 	"Vue":                          3,
 	"Wavefront Material":           1,
 	"Wavefront Object":             1,
-	"Web Ontology Language":        3,
+	"Web Ontology Language":        1,
 	"WebAssembly":                  2,
 	"WebIDL":                       2,
+	"Windows Registry Entries":     1,
 	"World of Warcraft Addon Data": 1,
+	"X BitMap":                     1,
+	"X Font Directory Index":       1,
+	"X PixMap":                     1,
 	"X10":                          2,
 	"XC":                           2,
 	"XCompose":                     1,
 	"XML":                          1,
-	"XPages":           2,
+	"XPages":                       1,
 	"XProc":                        2,
 	"XQuery":                       2,
 	"XS":                           2,
@@ -444,17 +483,24 @@ var LanguagesType = map[string]int{
 	"Xtend":                        2,
 	"YAML":                         1,
 	"YANG":                         1,
+	"YARA":                         2,
+	"YASnippet":                    3,
 	"Yacc":                         2,
 	"Zephir":                       2,
+	"Zig":                          2,
 	"Zimpl":                        2,
 	"desktop":                      1,
 	"eC":                           2,
 	"edn":                          1,
 	"fish":                         2,
 	"mupad":                        2,
+	"nanorc":                       1,
 	"nesC":                         2,
 	"ooc":                          2,
+	"q":                            2,
 	"reStructuredText":             4,
+	"sed":                          2,
+	"wdl":                          2,
 	"wisp":                         2,
 	"xBase":                        2,
 }

internal/code-generator/generator/test_files/vendor.gold

@@ -1,5 +1,5 @@
 // Code generated by gopkg.in/src-d/enry.v1/internal/code-generator DO NOT EDIT.
-// Extracted from github/linguist commit: d5c8db3fb91963c4b2762ca2ea2ff7cfac109f68
+// Extracted from github/linguist commit: e4560984058b4726010ca4b8f03ed9d0f8f464db
 
 package data
 
@@ -10,7 +10,6 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`^[Dd]ependencies/`),
 	substring.Regexp(`(^|/)dist/`),
 	substring.Regexp(`^deps/`),
-	substring.Regexp(`^tools/`),
 	substring.Regexp(`(^|/)configure$`),
 	substring.Regexp(`(^|/)config.guess$`),
 	substring.Regexp(`(^|/)config.sub$`),
@@ -32,13 +31,15 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`(^|/)bootstrap([^.]*)\.(js|css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)custom\.bootstrap([^\s]*)(js|css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)font-awesome\.(css|less|scss|styl)$`),
+	substring.Regexp(`(^|/)font-awesome/.*\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)foundation\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)normalize\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)skeleton\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)[Bb]ourbon/.*\.(css|less|scss|styl)$`),
 	substring.Regexp(`(^|/)animate\.(css|less|scss|styl)$`),
-	substring.Regexp(`third[-_]?party/`),
+	substring.Regexp(`(^|/)materialize\.(css|less|scss|styl|js)$`),
-	substring.Regexp(`3rd[-_]?party/`),
+	substring.Regexp(`(^|/)select2/.*\.(css|scss|js)$`),
+	substring.Regexp(`(3rd|[Tt]hird)[-_]?[Pp]arty/`),
 	substring.Regexp(`vendors?/`),
 	substring.Regexp(`extern(al)?/`),
 	substring.Regexp(`(^|/)[Vv]+endor/`),
@@ -53,6 +54,9 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`jquery.fancybox.(js|css)`),
 	substring.Regexp(`fuelux.js`),
 	substring.Regexp(`(^|/)jquery\.fileupload(-\w+)?\.js$`),
+	substring.Regexp(`jquery.dataTables.js`),
+	substring.Regexp(`bootbox.js`),
+	substring.Regexp(`pdf.worker.js`),
 	substring.Regexp(`(^|/)slick\.\w+.js$`),
 	substring.Regexp(`(^|/)Leaflet\.Coordinates-\d+\.\d+\.\d+\.src\.js$`),
 	substring.Regexp(`leaflet.draw-src.js`),
@@ -63,6 +67,7 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`wicket-leaflet.js`),
 	substring.Regexp(`.sublime-project`),
 	substring.Regexp(`.sublime-workspace`),
+	substring.Regexp(`.vscode`),
 	substring.Regexp(`(^|/)prototype(.*)\.js$`),
 	substring.Regexp(`(^|/)effects\.js$`),
 	substring.Regexp(`(^|/)controls\.js$`),
@@ -87,6 +92,7 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`(^|/)angular([^.]*)\.js$`),
 	substring.Regexp(`(^|\/)d3(\.v\d+)?([^.]*)\.js$`),
 	substring.Regexp(`(^|/)react(-[^.]*)?\.js$`),
+	substring.Regexp(`(^|/)flow-typed/.*\.js$`),
 	substring.Regexp(`(^|/)modernizr\-\d\.\d+(\.\d+)?\.js$`),
 	substring.Regexp(`(^|/)modernizr\.custom\.\d+\.js$`),
 	substring.Regexp(`(^|/)knockout-(\d+\.){3}(debug\.)?js$`),
@@ -98,8 +104,7 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`^.osx$`),
 	substring.Regexp(`\.xctemplate/`),
 	substring.Regexp(`\.imageset/`),
-	substring.Regexp(`^Carthage/`),
+	substring.Regexp(`(^|/)Carthage/`),
-	substring.Regexp(`^Pods/`),
 	substring.Regexp(`(^|/)Sparkle/`),
 	substring.Regexp(`Crashlytics.framework/`),
 	substring.Regexp(`Fabric.framework/`),
@@ -112,6 +117,9 @@ var VendorMatchers = substring.Or(
 	substring.Regexp(`(^|/)gradlew$`),
 	substring.Regexp(`(^|/)gradlew\.bat$`),
 	substring.Regexp(`(^|/)gradle/wrapper/`),
+	substring.Regexp(`(^|/)mvnw$`),
+	substring.Regexp(`(^|/)mvnw\.cmd$`),
+	substring.Regexp(`(^|/)\.mvn/wrapper/`),
 	substring.Regexp(`-vsdoc\.js$`),
 	substring.Regexp(`\.intellisense\.js$`),
 	substring.Regexp(`(^|/)jquery([^.]*)\.validate(\.unobtrusive)?\.js$`),

internal/code-generator/generator/vendor.go

@@ -2,12 +2,14 @@ package generator
 
 import (
 	"bytes"
-	"gopkg.in/yaml.v2"
 	"io"
 	"io/ioutil"
+
+	"gopkg.in/yaml.v2"
 )
 
-// Vendor reads from fileToParse and builds source file from tmplPath. It complies with type File signature.
+// Vendor generates regex matchers in Go for vendoring files/dirs.
+// It is of generator.File type.
 func Vendor(fileToParse, samplesDir, outPath, tmplPath, tmplName, commit string) error {
 	data, err := ioutil.ReadFile(fileToParse)
 	if err != nil {

internal/code-generator/main.go

@@ -20,7 +20,7 @@ const (
 	extensionsTmpl     = "extension.go.tmpl"
 
 	// content.go generation
-	heuristicsRuby  = ".linguist/lib/linguist/heuristics.rb"
+	heuristicsYAML  = ".linguist/lib/linguist/heuristics.yml"
 	contentFile     = "data/content.go"
 	contentTmplPath = "internal/code-generator/assets/content.go.tmpl"
 	contentTmpl     = "content.go.tmpl"
@@ -93,7 +93,7 @@ func main() {
 
 	fileList := []*generatorFiles{
 		{generator.Extensions, languagesYAML, "", extensionsFile, extensionsTmplPath, extensionsTmpl, commit},
-		{generator.Heuristics, heuristicsRuby, "", contentFile, contentTmplPath, contentTmpl, commit},
+		{generator.GenHeuristics, heuristicsYAML, "", contentFile, contentTmplPath, contentTmpl, commit},
 		{generator.Vendor, vendorYAML, "", vendorFile, vendorTmplPath, vendorTmpl, commit},
 		{generator.Documentation, documentationYAML, "", documentationFile, documentationTmplPath, documentationTmpl, commit},
 		{generator.Types, languagesYAML, "", typeFile, typeTmplPath, typeTmpl, commit},