pythainlp.tokenize

The pythainlp.tokenize module contains a comprehensive set of functions and classes for tokenizing Thai text into various units, such as sentences, words, subwords, and more. This module is a fundamental component of the PyThaiNLP library, providing tools for natural language processing in the Thai language.

Modules

pythainlp.tokenize.sent_tokenize(text: str | list[str], engine: str = 'crfcut', keep_whitespace: bool = True) → list[str] | list[list[str]][source]

Tokenize text into sentences.

The text can be a string or a list of words.

Parameters:

text (Union[str, list[str]]) – text, or list of words, to be tokenized
engine (str) –
name of the sentence tokenizer engine. Options:
- crfcut - (default) split by a CRF trained on the TED dataset
- thaisum - sentence segmentor from Nakhun Chumpolsathien, 2020
- tltk - split by TLTK
- wtp - split by wtpsplit. Supports many model sizes: wtp uses the mini model (default), wtp-tiny uses wtp-bert-tiny, wtp-mini uses wtp-bert-mini, wtp-base uses wtp-canine-s-1l, wtp-large uses wtp-canine-s-12l
- whitespace+newline - split by whitespace and newline
- whitespace - split by whitespace, using the regular expression r" +"
keep_whitespace (bool) – True to keep whitespace; otherwise omit whitespace

Returns:

list of sentences

Return type:

Union[list[str], list[list[str]]]

Example:

Split the text based on whitespace:

>>> from pythainlp.tokenize import sent_tokenize
>>> sentence_1 = "ฉันไปประชุมเมื่อวันที่ 11 มีนาคม"
>>> sent_tokenize(sentence_1, engine="whitespace")
['ฉันไปประชุมเมื่อวันที่', '11', 'มีนาคม']

Split the text based on whitespace and newline:

>>> sent_tokenize(sentence_1, engine="whitespace+newline")
['ฉันไปประชุมเมื่อวันที่', '11', 'มีนาคม']

Split the text using CRF trained on TED dataset:

>>> sent_tokenize(sentence_1, engine="crfcut")
['ฉันไปประชุมเมื่อวันที่ 11 มีนาคม']

Splits Thai text into sentences. This function identifies sentence boundaries, which is essential for text segmentation and analysis.

pythainlp.tokenize.paragraph_tokenize(text: str, engine: str = 'wtp-mini', paragraph_threshold: float = 0.5, style: str = 'newline') → list[list[str]][source]

Tokenize text into paragraphs.

Parameters:

text (str) – text to be tokenized
engine (str) –
name of the paragraph tokenizer engine. Options:
- wtp - split by wtpsplit. Supports many model sizes: wtp uses the mini model (default), wtp-tiny uses wtp-bert-tiny, wtp-mini uses wtp-bert-mini, wtp-base uses wtp-canine-s-1l, wtp-large uses wtp-canine-s-12l
paragraph_threshold (float) – threshold for paragraph boundaries
style (str) – paragraph style passed to the engine, "newline" (default) or "opus100"

Returns:

list of paragraphs, each a list of sentences

Return type:

list[list[str]]

Raises:

ValueError – if the engine is unknown

Example:

Split the text based on wtp:

>>> from pythainlp.tokenize import paragraph_tokenize
>>> sent = (
...     "(1) บทความนี้ผู้เขียนสังเคราะห์ขึ้นมาจากผลงานวิจัยที่เคยทำมาในอดีต"
...     + "  มิได้ทำการศึกษาค้นคว้าใหม่อย่างกว้างขวางแต่อย่างใด"
...     + " จึงใคร่ขออภัยในความบกพร่องทั้งปวงมา ณ ที่นี้"
... )
>>> paragraph_tokenize(sent)
[['(1) '], ['บทความนี้ผู้เขียนสังเคราะห์ขึ้นมาจากผลงานวิจัยที่เคยทำมาในอดีต  ', 'มิได้ทำการศึกษาค้นคว้าใหม่อย่างกว้างขวางแต่อย่างใด ', 'จึงใคร่ขออภัยในความบกพร่องทั้งปวงมา ', 'ณ ที่นี้']]

Segments text into paragraphs, which can be valuable for document-level analysis or summarization.

pythainlp.tokenize.subword_tokenize(text: str, engine: str = 'tcc', keep_whitespace: bool = True) → list[str][source]

Tokenize text into subwords, units smaller than syllables.

The function tokenizes text into inseparable units of contiguous Thai characters, namely Thai Character Clusters (TCCs). See https://www.researchgate.net/publication/2853284_Character_Cluster_Based_Thai_Information_Retrieval

TCCs are units based on Thai spelling features. A TCC cannot be separated any further, such as ‘ก็’, ‘จะ’, ‘ไม่’, and ‘ฝา’. If the units are separated, they cannot be spelled out. The function applies TCC rules to tokenize the text into the smallest units.

For example, the function tokenizes the word ‘ขนมชั้น’ into ‘ข’, ‘น’, ‘ม’, and ‘ชั้น’.

Parameters:

text (str) – text to be tokenized
engine (str) –
name of the subword tokenizer engine. Options:
- dict - newmm word tokenizer with a syllable dictionary
- etcc - Enhanced Thai Character Cluster (Inrut et al. 2001)
- han_solo - CRF syllable tokenizer for Thai that can work in the Thai social media domain. See PyThaiNLP/Han-solo
- phayathai - tokenizer from the PhayaThaiBERT model
- ssg - CRF syllable tokenizer for Thai. See ponrawee/ssg
- tcc (default) - Thai Character Cluster (Theeramunkong et al. 2000)
- tcc_p - Thai Character Cluster with improved rules used in newmm
- tltk - syllable tokenizer from tltk. See tltk
- wangchanberta - SentencePiece from the WangChanBERTa model
keep_whitespace (bool) – True to keep whitespace; otherwise omit whitespace

Returns:

list of subwords

Return type:

list[str]

Example:

Tokenize text into subwords based on tcc:

>>> from pythainlp.tokenize import subword_tokenize
>>> text_1 = "ยุคเริ่มแรกของ ราชวงศ์หมิง"
>>> text_2 = "ความแปลกแยกและพัฒนาการ"
>>> subword_tokenize(text_1, engine="tcc")
['ยุ', 'ค', 'เริ่ม', 'แร', 'ก', 'ข', 'อ', 'ง', ' ', 'รา', 'ช', 'วงศ์', 'ห', 'มิ', 'ง']
>>> subword_tokenize(text_2, engine="tcc")
['ค', 'วา', 'ม', 'แป', 'ล', 'ก', 'แย', 'ก', 'และ', 'พั', 'ฒ', 'นา', 'กา', 'ร']

Tokenize text into subwords based on etcc:

>>> subword_tokenize(text_1, engine="etcc")
['ยุ', 'ค', 'เริ่', 'ม', 'แร', 'ก', 'ข', 'อ', 'ง', ' ', 'รา', 'ช', 'ว', 'งศ์', 'ห', 'มิง']
>>> subword_tokenize(text_2, engine="etcc")
['ค', 'วา', 'ม', 'แป', 'ล', 'ก', 'แย', 'ก', 'และ', 'พัฒ', 'นา', 'กา', 'ร']

Tokenize text into subwords based on wangchanberta:

>>> subword_tokenize(text_1, engine="wangchanberta")
['▁', 'ยุค', 'เริ่มแรก', 'ของ', '▁', 'ราชวงศ์', 'หมิง']
>>> subword_tokenize(text_2, engine="wangchanberta")
['▁ความ', 'แปลก', 'แยก', 'และ', 'พัฒนาการ']

Tokenizes text into subwords, which can be helpful for various NLP tasks, including subword embeddings.

pythainlp.tokenize.syllable_tokenize(text: str, engine: str = 'han_solo', keep_whitespace: bool = True) → list[str][source]

Tokenize text into syllables.

The function tokenizes text into inseparable units of Thai syllables.

Parameters:

text (str) – text to be tokenized
engine (str) –
name of the syllable tokenizer engine. Options:
- dict - newmm word tokenizer with a syllable dictionary
- han_solo - (default) CRF syllable tokenizer for Thai that can work in the Thai social media domain. See PyThaiNLP/Han-solo
- ssg - CRF syllable tokenizer for Thai. See ponrawee/ssg
- tltk - syllable tokenizer from tltk. See tltk
keep_whitespace (bool) – True to keep whitespace; otherwise omit whitespace

Returns:

list of syllables

Return type:

list[str]

Raises:

ValueError – if the engine is unknown

Example:

>>> from pythainlp.tokenize import syllable_tokenize
>>> syllable_tokenize("สวัสดีครับ", engine="dict")
['สวัส', 'ดี', 'ครับ']
>>> syllable_tokenize("ประเทศไทย", engine="dict")
['ประ', 'เทศ', 'ไทย']

Divides text into syllables, allowing you to work with individual Thai language phonetic units.

pythainlp.tokenize.word_tokenize(text: str, custom_dict: Trie | None = None, engine: str = 'newmm', keep_whitespace: bool = True, join_broken_num: bool = True) → list[str][source]

Tokenize text into words.

Parameters:

text (str) – text to be tokenized
engine (str) –
name of the word tokenizer engine. Options:
- attacut - wrapper for AttaCut, learning-based approach, as an ONNX model via LEKCut, learning-based approach
- budoux - wrapper for budoux
- deepcut - wrapper for DeepCut, learning-based approach
- icu - wrapper for a word tokenizer in PyICU, from International Components for Unicode (ICU), dictionary-based approach
- longest - dictionary-based approach, longest matching
- mm (alias multi_cut) - “multi-cut”, dictionary-based approach, maximum matching
- nercut - named entity tagger-based approach, combining words that are parts of the same named entity
- newmm (default, alias onecut) - “new multi-cut”, dictionary-based approach, maximum matching, constrained by Thai Character Cluster (TCC) boundaries with improved TCC rules
- newmm-safe - newmm with a mechanism to avoid long processing time for text with continuously ambiguous breaking points
- nlpo3 - wrapper for a word tokenizer in nlpO3, adaptation of newmm in Rust (2.5x faster)
- oskut - wrapper for OSKut, Out-of-domain StacKed cut for Word Segmentation, as an ONNX model via LEKCut
- sefr_cut - wrapper for SEFR CUT, Stacked Ensemble Filter and Refine for Word Segmentation
- tltk - wrapper for TLTK, maximum collocation approach
custom_dict (pythainlp.util.Trie) – dictionary trie (some engines do not support this)
keep_whitespace (bool) – True to keep whitespace, a common marker for end of phrase in Thai; otherwise omit whitespace
join_broken_num (bool) – True to rejoin formatted numbers that a tokenizer could wrongly split (e.g., times and IP addresses); otherwise leave them as split

Returns:

list of words

Return type:

list[str]

Note:

The custom_dict parameter works only with the longest, mm, newmm, and newmm-safe engines.
The built-in tokenizers (longest, mm, newmm, and newmm-safe) are thread-safe.
The wrappers of external tokenizers are designed to be thread-safe, but thread-safety depends on the external tokenizer.
WARNING: When using custom_dict in a multi-threaded environment, do NOT modify the trie (with its add or remove methods) while tokenization is in progress. The trie is not thread-safe for concurrent modification. Create the dictionary before starting threads, and only read from it during tokenization.

Example:

Tokenize text with different tokenizers:

>>> from pythainlp.tokenize import word_tokenize
>>> text = "โอเคบ่พวกเรารักภาษาบ้านเกิด"
>>> word_tokenize(text, engine="newmm")
['โอเค', 'บ่', 'พวกเรา', 'รัก', 'ภาษา', 'บ้านเกิด']
>>> word_tokenize(text, engine="attacut")
['โอเค', 'บ่', 'พวกเรา', 'รัก', 'ภาษา', 'บ้านเกิด']

Tokenize text with whitespace omitted:

>>> text = "วรรณกรรม ภาพวาด และการแสดงงิ้ว "
>>> word_tokenize(text, engine="newmm")
['วรรณกรรม', ' ', 'ภาพวาด', ' ', 'และ', 'การแสดง', 'งิ้ว', ' ']
>>> word_tokenize(text, engine="newmm", keep_whitespace=False)
['วรรณกรรม', 'ภาพวาด', 'และ', 'การแสดง', 'งิ้ว']

Join broken formatted numeric (e.g. time, decimals, IP addresses):

>>> text = "เงิน1,234บาท19:32น 127.0.0.1"
>>> word_tokenize(
...     text, engine="attacut", join_broken_num=False
... )
['เงิน', '1', ',', '234', 'บาท', '19', ':', '32น', ' ', '127', '.', '0', '.', '0', '.', '1']
>>> word_tokenize(
...     text, engine="attacut", join_broken_num=True
... )
['เงิน', '1,234', 'บาท', '19:32น', ' ', '127.0.0.1']

Tokenize with default and custom dictionaries:

>>> from pythainlp.corpus.common import thai_words
>>> from pythainlp.tokenize import dict_trie
>>> text = "ชินโซ อาเบะ เกิด 21 กันยายน"
>>> word_tokenize(text, engine="newmm")
['ชิน', 'โซ', ' ', 'อา', 'เบะ', ' ', 'เกิด', ' ', '21', ' ', 'กันยายน']
>>> custom_dict_japanese_name = set(thai_words())
>>> custom_dict_japanese_name.add("ชินโซ")
>>> custom_dict_japanese_name.add("อาเบะ")
>>> trie = dict_trie(
...     dict_source=custom_dict_japanese_name
... )
>>> word_tokenize(
...     text, engine="newmm", custom_dict=trie
... )
['ชินโซ', ' ', 'อาเบะ', ' ', 'เกิด', ' ', '21', ' ', 'กันยายน']

Splits text into words. This function is a fundamental tool for Thai language text analysis.

pythainlp.tokenize.word_detokenize(segments: list[list[str]] | list[str], output: str = 'str') → list[list[str]] | str[source]

Detokenize lists of words into text.

Join the words in each sentence into text.

Parameters:

segments (Union[list[list[str]], list[str]]) – list of words, or list of sentences each with a list of words
output (str) – output type, "str" or "list"

Returns:

Thai text

Return type:

Union[list[list[str]], str]

Example:

>>> from pythainlp.tokenize import word_detokenize
>>> word_detokenize(["เรา", "เล่น"])
'เราเล่น'

Reverses the tokenization process, reconstructing text from tokenized units. Useful for text generation tasks.

class pythainlp.tokenize.Tokenizer(custom_dict: Trie | Iterable[str] | str | None = None, engine: str = 'newmm', keep_whitespace: bool = True, join_broken_num: bool = True)[source]

Bundle a custom dictionary and a word tokenizer engine in one object.

This class allows users to pre-define a custom dictionary along with an engine. It wraps both pythainlp.tokenize.word_tokenize() and pythainlp.util.dict_trie().

Example:

Tokenizer object instantiated with pythainlp.util.Trie:

>>> from pythainlp.tokenize import Tokenizer
>>> from pythainlp.corpus.common import thai_words
>>> from pythainlp.util import dict_trie
>>> custom_words_list = set(thai_words())
>>> custom_words_list.add("อะเฟเซีย")
>>> custom_words_list.add("Aphasia")
>>> trie = dict_trie(dict_source=custom_words_list)
>>> text = "อะเฟเซีย (Aphasia*) เป็นอาการผิดปกติของการพูด"
>>> _tokenizer = Tokenizer(
...     custom_dict=trie, engine="newmm"
... )
>>> _tokenizer.word_tokenize(text)
['อะเฟเซีย', ' ', '(', 'Aphasia', ')', ' ', 'เป็น', 'อาการ', 'ผิดปกติ', 'ของ', 'การ', 'พูด']

Tokenizer object instantiated with a list of words:

>>> text = "อะเฟเซีย (Aphasia) เป็นอาการผิดปกติของการพูด"
>>> _tokenizer = Tokenizer(
...     custom_dict=list(thai_words()), engine="newmm"
... )
>>> _tokenizer.word_tokenize(text)
['อะ', 'เฟเซีย', ' ', '(', 'Aphasia', ')', ' ', 'เป็น', 'อาการ', 'ผิดปกติ', 'ของ', 'การ', 'พูด']

Tokenizer object instantiated with a file path containing a list of words separated with newline and explicitly setting a new tokenizer after initiation:

>>> PATH_TO_CUSTOM_DICTIONARY = (
...     "./custom_dictionary.txt"
... )
>>> with open(
...     PATH_TO_CUSTOM_DICTIONARY, "w", encoding="utf-8"
... ) as f:
...     f.write("อะเฟเซีย\nAphasia\nผิด\nปกติ")
>>> text = "อะเฟเซีย (Aphasia) เป็นอาการผิดปกติของการพูด"
>>> _tokenizer = Tokenizer(
...     custom_dict=PATH_TO_CUSTOM_DICTIONARY, engine="attacut"
... )
>>> _tokenizer.word_tokenize(text)
['อะเฟเซีย', ' ', '(', 'Aphasia', ')', ' ', 'เป็น', 'อาการ', 'ผิด', 'ปกติ', 'ของ', 'การ', 'พูด']
>>> _tokenizer.set_tokenize_engine(engine="newmm")
>>> _tokenizer.word_tokenize(text)
['อะเฟเซีย', ' ', '(', 'Aphasia', ')', ' ', 'เป็นอาการ', 'ผิด', 'ปกติ', 'ของการพูด']

The Tokenizer class is a versatile tool for customizing tokenization processes and managing tokenization models. It provides various methods and attributes to fine-tune tokenization according to your specific needs.

__init__(custom_dict: Trie | Iterable[str] | str | None = None, engine: str = 'newmm', keep_whitespace: bool = True, join_broken_num: bool = True) → None[source]

Initialize the tokenizer object.

Parameters:

custom_dict (Union[pythainlp.util.Trie, Iterable[str], str, None]) – file path, list of words to create a trie from, or pythainlp.util.Trie object
engine (str) – name of the word tokenizer engine (newmm, mm, longest, or deepcut)
keep_whitespace (bool) – True to keep whitespace, a common marker for end of phrase in Thai; otherwise omit whitespace
join_broken_num (bool) – True to rejoin formatted numbers that a tokenizer could wrongly split; otherwise leave them as split

Raises:

NotImplementedError – if the engine is not supported

word_tokenize(text: str) → list[str][source]

Tokenize text into words.

Parameters:

text (str) – text to be tokenized

Returns:

list of words

Return type:

list[str]

Example:

>>> from pythainlp.tokenize import Tokenizer
>>> tokenizer = Tokenizer()
>>> tokenizer.word_tokenize("สวัสดีครับ")
['สวัสดี', 'ครับ']

set_tokenize_engine(engine: str) → None[source]

Set the word tokenizer engine.

Parameters:

engine (str) – name of the word tokenizer engine (newmm, mm, longest, or deepcut)

Example:

>>> from pythainlp.tokenize import Tokenizer
>>> tokenizer = Tokenizer()
>>> tokenizer.set_tokenize_engine("newmm")
>>> tokenizer.word_tokenize("สวัสดีครับ")
['สวัสดี', 'ครับ']

class pythainlp.tokenize.display_cell_tokenize(text: str)[source]

Tokenize Thai text into display cells.

The function does not split tone marks from their base characters.

Parameters:: text (str) – text to be tokenized
Returns:: list of display cells
Return type:: list[str]
Example:

Tokenize Thai text into display cells:

>>> from pythainlp.tokenize import display_cell_tokenize
>>> text = "แม่น้ำอยู่ที่ไหน"
>>> display_cell_tokenize(text)
['แ', 'ม่', 'น้ํ', 'า', 'อ', 'ยู่', 'ที่', 'ไ', 'ห', 'น']

Tokenization Engines

This module offers multiple tokenization engines designed for different levels of text analysis.

Sentence level

crfcut

Segment Thai text into sentences with CRFCut.

CRFCut uses a conditional random field (CRF). The default model is trained on the TED dataset.

Performance:

ORCHID - space-correct accuracy 87% vs 95% state-of-the-art (Zhou et al, 2016; https://www.aclweb.org/anthology/C16-1031.pdf)
TED dataset - space-correct accuracy 82%

See the development notebooks at https://github.com/vistec-AI/ted_crawler; the model does not use part-of-speech (POS) features, because the available POS tagging is unreliable.

A tokenizer that operates at the sentence level using Conditional Random Fields (CRF). It is suitable for segmenting text into sentences accurately.

pythainlp.tokenize.crfcut.segment(text: str) → list[str][source]

Tokenize text into sentences with a CRF model.

Parameters:: text (str) – text to be tokenized
Returns:: list of sentences
Return type:: list[str]

thaisumcut

Tokenize Thai text into sentences with the ThaiSum sentence segmentor.

This is an implementation of the sentence segmentor by Nakhun Chumpolsathien, 2020. The original code is from https://github.com/nakhunchumpolsathien/ThaiSum

Cite:

@mastersthesis{chumpolsathien_2020,
    title={Using Knowledge Distillation from Keyword Extraction
        to Improve the Informativeness of Neural Cross-lingual
        Summarization},
    author={Chumpolsathien, Nakhun},
    year={2020},
    school={Beijing Institute of Technology}
}

A sentence tokenizer based on a maximum entropy model. It’s a great choice for sentence boundary detection in Thai text.

pythainlp.tokenize.thaisumcut.list_to_string(tokens: list[str]) → str[source]

Join a list of strings and collapse the whitespace.

Parameters:: tokens (list[str]) – list of strings
Returns:: joined string
Return type:: str

pythainlp.tokenize.thaisumcut.middle_cut(sentences: list[str]) → list[str][source]

Split long sentences at the space nearest to the middle.

Parameters:: sentences (list[str]) – list of sentences
Returns:: list of sentences
Return type:: list[str]

class pythainlp.tokenize.thaisumcut.ThaiSentenceSegmentor[source]

Tokenize Thai text into sentences.

split_into_sentences(text: str, isMiddleCut: bool = False) → list[str][source]

Split text into sentences.

Parameters:

text (str) – text to be tokenized
isMiddleCut (bool) – also cut long sentences at the middle

Returns:

list of sentences

Return type:

list[str]

Word level

attacut

deepcut

multi_cut

Tokenize Thai text into words with multi-cut, a maximum matching approach.

The original code is by Korakot Chaovavanich.

See Also:

Facebook post: https://www.facebook.com/groups/408004796247683/permalink/431283740586455/
GitHub Gist: https://gist.github.com/korakot/fe26c65dc9eed467f4497f784a805716

An ensemble tokenizer that combines multiple tokenization strategies for improved word segmentation.

class pythainlp.tokenize.multi_cut.LatticeString(value: str, multi: list[str] | None = None, in_dict: bool = True)[source]

Keep the possible tokenizations of a string.

__init__(value: str, multi: list[str] | None = None, in_dict: bool = True) → None[source]

Initialize the lattice string.

Parameters:

value (str) – string value
multi (Optional[list[str]]) – possible tokenizations of the value
in_dict (bool) – whether the value is in the dictionary

unique: bool

multi: list[str]

in_dict: bool

pythainlp.tokenize.multi_cut.mmcut(text: str) → list[str][source]

Tokenize text into words with the minimum-cut selection.

Parameters:: text (str) – text to be tokenized
Returns:: list of words
Return type:: list[str]

pythainlp.tokenize.multi_cut.segment(text: str, custom_dict: Trie | None = None) → list[str][source]

Tokenize text into words with dictionary-based maximum matching.

Parameters:

text (str) – text to be tokenized
custom_dict (pythainlp.util.Trie) – dictionary trie (default: a trie of pythainlp.corpus.thai_words())

Returns:

list of words

Return type:

list[str]

pythainlp.tokenize.multi_cut.find_all_segment(text: str, custom_dict: Trie | None = None) → list[str][source]

Get all possible tokenizations of text.

Parameters:

text (str) – text to be tokenized
custom_dict (pythainlp.util.Trie) – dictionary trie (default: a trie of pythainlp.corpus.thai_words())

Returns:

list of tokenizations, each with words separated by |

Return type:

list[str]

nlpo3

Wrap nlpo3, a Rust implementation of the Thai word tokenizer.

A word tokenizer based on the NLPO3 model. It offers advanced word boundary detection and is suitable for various NLP tasks.

pythainlp.tokenize.nlpo3.load_dict(file_path: str, dict_name: str) → bool[source]

Load a dictionary file into an in-memory dictionary collection.

The loaded dictionary is accessible through the assigned dict_name. This function does not override an existing dictionary name.

Parameters:

file_path (str) – path to a dictionary file
dict_name (str) – unique dictionary name, used for reference

Returns:

True if the dictionary loaded successfully, False otherwise

Return type:

bool

Raises:

ImportError – if nlpo3 is not installed

See Also:

https://github.com/PyThaiNLP/nlpo3

pythainlp.tokenize.nlpo3.segment(text: str, custom_dict: str = '_73bcj049dzbu9t49b4va170k', safe_mode: bool = False, parallel_mode: bool = False) → list[str][source]

Tokenize text into words with nlpO3.

This function is the Python binding for nlpO3, a Rust implementation of the newmm engine.

Parameters:

text (str) – text to be tokenized
custom_dict (str) – dictionary name, as assigned with load_dict() (default: the dictionary from pythainlp/corpus/words_th.txt)
safe_mode (bool) – True to reduce the chance of long processing time for long text with many ambiguous breaking points (default: False)
parallel_mode (bool) – True to use multithread mode (default: False)

Returns:

list of words

Return type:

list[str]

Raises:

ImportError – if nlpo3 is not installed

See Also:

https://github.com/PyThaiNLP/nlpo3

longest

Tokenize Thai text into words with dictionary-based longest matching.

The implementation is based on code from Patorn Utenpattanun.

See Also:

GitHub repository: https://github.com/patorn/thaitokenizer/blob/master/thaitokenizer/tokenizer.py

A tokenizer that identifies word boundaries by selecting the longest possible words in a text.

class pythainlp.tokenize.longest.LongestMatchTokenizer(trie: Trie)[source]

Tokenize text into words with dictionary-based longest matching.

__init__(trie: Trie) → None[source]

Initialize the tokenizer.

Parameters:: trie (pythainlp.util.Trie) – dictionary trie

tokenize(text: str) → list[str][source]

Tokenize text into words.

Parameters:: text (str) – text to be tokenized
Returns:: list of words
Return type:: list[str]

pythainlp.tokenize.longest.segment(text: str, custom_dict: Trie | None = None) → list[str][source]

Tokenize text into words with dictionary-based longest matching.

This function is thread-safe. It uses a lock to protect access to the internal tokenizer cache.

Parameters:

text (str) – text to be tokenized
custom_dict (pythainlp.util.Trie) – dictionary trie (default: the default word trie)

Returns:

list of words

Return type:

list[str]

pyicu

nercut

Tokenize Thai text into words with nercut 0.2.

The tokenizer uses a named entity tagger and combines words that are parts of the same named entity.

The code is by Wannaphong Phatthiyaphaibun.

A tokenizer optimized for Named Entity Recognition (NER) tasks, ensuring accurate tokenization for entity recognition.

pythainlp.tokenize.nercut.segment(text: str, taglist: Iterable[str] = ['ORGANIZATION', 'PERSON', 'PHONE', 'EMAIL', 'DATE', 'TIME'], tagger: NER = <pythainlp.tag.named_entity.NER object>) → list[str][source]

Tokenize text into words, combining words of the same named entity.

Parameters:

text (str) – text to be tokenized
taglist (Iterable[str]) – named entity tags to combine
tagger (pythainlp.tag.named_entity.NER) – named entity tagger

Returns:

list of words

Return type:

list[str]

sefr_cut

oskut

newmm (Default)

Tokenize Thai text into words with dictionary-based maximal matching.

The tokenizer constrains the matching by Thai Character Cluster (TCC) boundaries with improved rules.

The code is based on notebooks created by Korakot Chaovavanich, with a heuristic graph size limit added to avoid exponential waiting time.

See Also:

Colab notebook, version 1: https://colab.research.google.com/notebook#fileId=1V1Z657_5eSWPo8rLfVRwA0A5E4vkg7SI
Colab notebook, version 2: https://colab.research.google.com/drive/14Ibg-ngZXj15RKwjNwoZlOT32fQBOrBx#scrollTo=MYZ7NzAR7Dmw

The default word tokenization engine that provides a balance between accuracy and efficiency for most use cases.

pythainlp.tokenize.newmm.segment(text: str, custom_dict: Trie | None = None, safe_mode: bool = False) → list[str][source]

Tokenize text into words with maximal matching and TCC boundaries.

This is a dictionary-based tokenizer that uses a maximal matching algorithm, constrained by Thai Character Cluster (TCC) boundaries. A custom dictionary can be supplied.

For very long text (hundreds of kilobytes or more), consider using safe_mode=True to enable chunk-based processing and reduce memory use.

Parameters:

text (str) – text to be tokenized
custom_dict (pythainlp.util.Trie) – dictionary trie (default: the default word trie)
safe_mode (bool) – True to use chunk-based processing, which reduces memory use and processing time for long text with many ambiguous breaking points (default: False)

Returns:

list of words

Return type:

list[str]

Subword level

tcc

Tokenize text into Thai Character Clusters (TCCs).

The implementation follows the TCC rules proposed by Theeramunkong et al. 2000: https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.59.2548

Credits:

TCC: Jakkrit TeCho
Grammar: Wittawat Jitkrittum (https://github.com/wittawatj/jtcc/blob/master/TCC.g)
Python code: Korakot Chaovavanich

Tokenizes text into Thai Character Clusters (TCCs), a subword level representation.

pythainlp.tokenize.tcc.tcc(text: str) → Iterator[str][source]

Generate Thai Character Clusters (TCCs) from text.

Parameters:: text (str) – text to be tokenized
Returns:: iterator of character clusters
Return type:: collections.abc.Iterator[str]

pythainlp.tokenize.tcc.tcc_pos(text: str) → set[int][source]

Get the ending positions of Thai Character Clusters (TCCs) in text.

Parameters:: text (str) – text to be tokenized
Returns:: set of ending positions of character clusters
Return type:: set[int]

pythainlp.tokenize.tcc.segment(text: str) → list[str][source]

Tokenize text into Thai Character Clusters (TCCs).

Parameters:: text (str) – text to be tokenized
Returns:: list of character clusters
Return type:: list[str]

tcc+

Tokenize text into Thai Character Clusters (TCCs) with improved rules.

The implementation follows the TCC rules proposed by Theeramunkong et al. 2000 (https://doi.org/10.1145/355214.355225), and the improved rules that newmm uses.

Credits:

TCC: Jakkrit TeCho
Grammar: Wittawat Jitkrittum (https://github.com/wittawatj/jtcc/blob/master/TCC.g)
Python code: Korakot Chaovavanich

A subword tokenizer that includes additional rules for more precise subword segmentation.

pythainlp.tokenize.tcc_p.tcc(text: str) → Iterator[str][source]

Generate Thai Character Clusters (TCCs) from text.

Parameters:: text (str) – text to be tokenized
Returns:: iterator of character clusters
Return type:: collections.abc.Iterator[str]

pythainlp.tokenize.tcc_p.tcc_pos(text: str) → set[int][source]

Get the ending positions of Thai Character Clusters (TCCs) in text.

Parameters:: text (str) – text to be tokenized
Returns:: set of ending positions of character clusters
Return type:: set[int]

pythainlp.tokenize.tcc_p.tcc_pos_array(text: str) → bytearray[source]

Get the valid Thai Character Cluster (TCC) boundaries as a bytearray.

The bytearray has length len(text) + 1. Index i is 1 if position i is a valid TCC boundary, and 0 otherwise. Array-index lookup is faster and uses less memory than set membership for large text.

Parameters:: text (str) – text to be tokenized
Returns:: bytearray of valid TCC boundary flags, indexed by position
Return type:: bytearray

pythainlp.tokenize.tcc_p.segment(text: str) → list[str][source]

Tokenize text into Thai Character Clusters (TCCs).

Parameters:: text (str) – text to be tokenized
Returns:: list of character clusters
Return type:: list[str]

etcc

Tokenize text into Enhanced Thai Character Clusters (ETCCs).

This Python implementation is by Wannaphong Phatthiyaphaibun. It relies on a dictionary of ETCCs created from etcc.txt in pythainlp/corpus.

Notebook: https://colab.research.google.com/drive/1UTQgxxMRxOr9Jp1B1jcq1frBNvorhtBQ

See Also:

Jeeragone Inrut, Patiroop Yuanghirun, Sarayut Paludkong, Supot Nitsuwat, and Para Limmaneepraserth. “Thai word segmentation using combination of forward and backward longest matching techniques.” In International Symposium on Communications and Information Technology (ISCIT), pp. 37-40. 2001.

Enhanced Thai Character Clusters (eTCC) tokenizer for subword-level analysis.

pythainlp.tokenize.etcc.segment(text: str) → list[str][source]

Tokenize text into ETCCs.

An Enhanced Thai Character Cluster (ETCC) is a kind of subword unit. Inrut, Jeeragone, Patiroop Yuanghirun, Sarayut Paludkong, Supot Nitsuwat, and Para Limmaneepraserth presented the concept in “Thai word segmentation using combination of forward and backward longest matching techniques.” In International Symposium on Communications and Information Technology (ISCIT), pp. 37-40. 2001.

Parameters:: text (str) – text to be tokenized
Returns:: list of character clusters
Return type:: list[str]

han_solo

🪿 Han-solo: Thai syllable tokenizer.

GitHub: https://github.com/PyThaiNLP/Han-solo

A subword tokenizer specialized for Han characters and mixed scripts, suitable for various text processing scenarios.

class pythainlp.tokenize.han_solo.Featurizer(N: int = 2, sequence_size: int = 1, delimiter: str | None = None)[source]

Extract character n-gram features for Han-solo.

__init__(N: int = 2, sequence_size: int = 1, delimiter: str | None = None) → None[source]

Initialize the featurizer.

Parameters:

N (int) – size of the character n-gram
sequence_size (int) – size of the character sequence
delimiter (Optional[str]) – delimiter character in the text

N: int

delimiter: str | None

radius: int

pad(sentence: str, padder: str = '#') → str[source]

Pad a sentence with a padding character on both sides.

Parameters:

sentence (str) – sentence to pad
padder (str) – padding character

Returns:

padded sentence

Return type:

str

featurize(sentence: str, padding: bool = True, indiv_char: bool = True, return_type: str = 'list') → dict[str, list[Any]][source]

Extract features and labels from a sentence.

Parameters:

sentence (str) – sentence to featurize
padding (bool) – pad the sentence first
indiv_char (bool) – include features of individual characters
return_type (str) – type of the features (list or dict)

Returns:

features (X) and labels (Y)

Return type:

dict[str, list[Any]]

pythainlp.tokenize.han_solo.segment(text: str) → list[str][source]

Tokenize text into words with Han-solo.

Parameters:: text (str) – text to be tokenized
Returns:: list of words
Return type:: list[str]