Golang · Tokenizers

Tokenizers

• go-ego/gse on GitHub, opens in a new tab

  2.8Kstars

  Go efficient text segmentation; support english, chinese, japanese and other.

  Go·Apache License 2.0·active·2 months ago

• yanyiwu/gojieba on GitHub, opens in a new tab

  2.6Kstars

  This is a Go implementation of jieba which a Chinese word splitting algorithm.

  Go·MIT License·active·4 weeks ago

• neurosnap/sentences on GitHub, opens in a new tab

  473stars

  Sentence tokenizer: converts text into a list of sentences.

  Go·MIT License·slowing·2 years ago

• blevesearch/segment on GitHub, opens in a new tab

  89stars

  Go library for performing Unicode Text Segmentation as described in Unicode Standard Annex #29

  Go·Apache License 2.0·active·3 weeks ago

• pebbe/textcat on GitHub, opens in a new tab

  73stars

  Go package for n-gram based text categorization, with support for utf-8 and raw text.

  Go·BSD 2-Clause "Simplified" License·slowing·2 years ago

• awsong/MMSEGO on GitHub, opens in a new tab

  62stars

  This is a GO implementation of MMSEG which a Chinese word splitting algorithm.

  Go·Other·dormant·14 years ago

• dchest/stemmer on GitHub, opens in a new tab

  56stars

  Stemmer packages for Go programming language. Includes English and German stemmers.

  Go·BSD 2-Clause "Simplified" License·dormant·10 years ago

• xujiajun/gotokenizer on GitHub, opens in a new tab

  21stars

  A tokenizer based on the dictionary and Bigram language models for Golang. (Now only support chinese segmentation)

  Go·Apache License 2.0·dormant·7 years ago

• osamingo/shamoji on GitHub, opens in a new tab

  13stars

  The shamoji is word filtering package written in Go.

  Go·MIT License·slowing·last year