enry

File programming language detector and toolbox to ignore binary or vendored files. enry, started as a port to Go of the original linguist Ruby library, that has an improved 2x performance.

Installation

The recommended way to install simple-linguist

go get gopkg.in/src-d/enry.v1/...

To build enry's CLI you must run

make build-cli

it generates a binary in the project's root directory called enry. You can move this binary to anywhere in your PATH.

Examples

lang, safe := enry.GetLanguageByExtension("foo.go")
fmt.Println(lang)
// result: Go

lang, safe := enry.GetLanguageByContent("foo.m", "<matlab-code>")
fmt.Println(lang)
// result: Matlab

lang, safe := enry.GetLanguageByContent("bar.m", "<objective-c-code>")
fmt.Println(lang)
// result: Objective-C

// all strategies together
lang := enry.GetLanguage("foo.cpp", "<cpp-code>")

Note the returned boolean value "safe" is set either to true, if there is only one possible language detected or, to false otherwise.

To get a list of possible languages for a given file, you can use the plural version of the detecting functions.

langs := enry.GetLanguages("foo.h",  "<cpp-code>")
// result: []string{"C++", "C"}

langs := enry.GetLanguagesByExtension("foo.asc", "<content>", nil)
// result: []string{"AGS Script", "AsciiDoc", "Public Key"}

langs := enry.GetLanguagesByFilename("Gemfile", "<content>", []string{})
// result: []string{"Ruby"}

CLI

You can use enry as a command,

$ enry --help
enry, A simple (and faster) implementation of github/linguist
usage: enry <path>
              enry <path> [--json] [--breakdown]
              enry [--json] [--breakdown]

and it will return an output similar to linguist's output,

$ enry
11.11%    Gnuplot
22.22%    Ruby
55.56%    Shell
11.11%    Go

but not only the output, also its flags are the same as linguist's ones,

$ enry --breakdown
11.11%    Gnuplot
22.22%    Ruby
55.56%    Shell
11.11%    Go

Gnuplot
plot-histogram.gp

Ruby
linguist-samples.rb
linguist-total.rb

Shell
parse.sh
plot-histogram.sh
run-benchmark.sh
run-slow-benchmark.sh
run.sh

Go
parser/main.go

even the JSON flag,

$ enry --json
{"Gnuplot":["plot-histogram.gp"],"Go":["parser/main.go"],"Ruby":["linguist-samples.rb","linguist-total.rb"],"Shell":["parse.sh","plot-histogram.sh","run-benchmark.sh","run-slow-benchmark.sh","run.sh"]}

Note that even if enry's CLI is compatible with linguist's, its main point is that, contrary to linguist, enry doesn't need a git repository to work!

Development

enry re-uses parts of original linguist to generate internal data structures. In order to update to latest upstream and generate the necessary code you must run:

go generate

We update enry due to changes in linguist's master branch related to the following files:

• languages.yml
• heuristics.rb
• vendor.yml
• documentation.yml

For the moment we don't have any procedure established to detect changes in the linguist project automatically and regenerate the code. So we are updating the generated code as needed, without any specific criteria.

If you want update enry because of changes in linguist, you can run the go generate command and do a pull request that only contains the changes in generated files (those files in the subdirectory data).

To run the tests

make test

Divergences from linguist

Using linguist/samples as a set against run tests the following issues were found:

• with hello.ms we can't detect the language (Unix Assembly) because we don't have a matcher in contentMatchers (content.go) for Unix Assembly. Linguist uses this regexp in its code,

  elsif /(?<!\S)\.(include|globa?l)\s/.match(data) || /(?<!\/\*)(\A|\n)\s*\.[A-Za-z][_A-Za-z0-9]*:/.match(data.gsub(/"([^\\"]|\\.)*"|'([^\\']|\\.)*'|\\\s*(?:--.*)?\n/, ""))

  which we can't port.

• all files for SQL language fall to the classifier because we don't parse this disambiguator expresion for *.sql files right. This expression doesn't comply with the pattern for the rest of heuristics.rb file.

Benchmarks

Enry's language detection has been compared with Linguist's language detection. In order to do that, linguist's project directory linguist/samples was used as a set of files to run benchmarks against.

Following results were obtained:

The histogram represents the number of files for which spent time in language detection was in the range of the time interval indicated in x axis.

So reviewing the comparison enry/linguist, you can see the most of the files were detected in less time than linguist does.

We detected some few cases enry turns slower than linguist. This is due to Golang's regexp engine being slower than Ruby's, which uses oniguruma library, written in C.

You can find scripts and additional information (as software and hardware used, and benchmarks' results per sample file) in benchmarks directory.

If you want to reproduce the same experiment you can run:

benchmarks/run.sh

from the root's project directory and It runs benchmarks for enry and linguist, parse the output, create csv files and create a histogram (you must have installed gnuplot in your system to get the histogram). It can take to much time, so to run local benchmarks to take a quick look you can run either:

make benchmarks

to get time averages for main detection function and strategies for the whole samples set or:

make benchmarks-samples

if you want see measures by sample file

Why Enry?

In the movie My Fair Lady, Professor Henry Higgins is one of the main characters. Henry is a linguist and at the very beginning of the movie enjoys guessing the nationality of people based on their accent.

Enry Iggins is how Eliza Doolittle, pronounces the name of the Professor during the first half of the movie.

License

MIT, see LICENSE