pythainlp.lm.wangchanberta

Note

pythainlp.wangchanberta has moved to pythainlp.lm.wangchanberta. Importing from pythainlp.wangchanberta still works but emits a DeprecationWarning and will be removed in 6.0.

The pythainlp.lm.wangchanberta module is built upon the WangchanBERTa base model, specifically the wangchanberta-base-att-spm-uncased model, as detailed in the paper by Lowphansirikul et al. [1].

This base model is utilized for various natural language processing tasks in the Thai language, including named entity recognition, part-of-speech tagging, and subword tokenization.

If you intend to fine-tune the model or explore its capabilities further, please refer to the thai2transformers repository.

Speed Benchmark

Function	Named Entity Recognition	Part of Speech
PyThaiNLP basic function	89.7 ms	312 ms
pythainlp.wangchanberta (CPU)	9.64 s	9.65 s
pythainlp.wangchanberta (GPU)	8.02 s	8 s

For a comprehensive performance benchmark, the following notebooks are available:

• `PyThaiNLP basic function and pythainlp.wangchanberta CPU at Google Colab`_

• `pythainlp.wangchanberta GPU`_

Modules

class pythainlp.lm.wangchanberta.NamedEntityRecognition(model: str = 'pythainlp/thainer-corpus-v2-base-model', revision: str | None = None)

Recognize Thai named entities with a WangchanBERTa model.

The NamedEntityRecognition class is a fundamental component for identifying named entities in Thai text. It allows you to extract entities such as names, locations, and organizations from text data.

__init__(model: str = 'pythainlp/thainer-corpus-v2-base-model', revision: str | None = None) → None

Initialize a named entity recognizer.

Use a model pretrained with WangchanBERTa.

Parameters:

• model (str) – pretrained model name

• revision (Optional[str]) – git revision ID (branch, tag, or commit hash). Pin to a full commit hash for secure downloads.

tokenizer: PreTrainedTokenizerBase

model: PreTrainedModel

get_ner(text: str, pos: bool = False, tag: bool = False) → list[tuple[str, str]] | str

Tag named entities in Thai text.

Parameters:

• text (str) – Thai text to be tagged

• pos (bool) – whether to request part-of-speech output; the model does not support it, so a warning is raised

• tag (bool) – whether to return HTML-like tags instead of tuples

Returns:

tagged words, or an HTML-like tagged string if tag is True

Return type:

Union[list[tuple[str, str]], str]

class pythainlp.lm.wangchanberta.ThaiNameTagger(dataset_name: str = 'thainer', grouped_entities: bool = True)

Tag named entities with the WangchanBERTa pipeline.

The ThaiNameTagger class is designed for tagging Thai names within text. This is essential for tasks such as entity recognition, information extraction, and text classification.

json_ner: list[dict[str, str]]

output: str

sent_ner: list[tuple[str, str]]

__init__(dataset_name: str = 'thainer', grouped_entities: bool = True) → None

Initialize a named entity tagger in IOB format.

Use WangchanBERTa from the VISTEC-depa AI Research Institute of Thailand.

Parameters:

• dataset_name (str) – dataset the model is fine-tuned on * thainer - ThaiNER dataset (default)

• grouped_entities (bool) – whether to group word pieces of the same entity

dataset_name: str

grouped_entities: bool

classify_tokens: TokenClassificationPipeline

get_ner(text: str, pos: bool = False, tag: bool = False) → list[tuple[str, str]] | str

Tag named entities in Thai text.

Use WangchanBERTa from the VISTEC-depa AI Research Institute of Thailand.

Parameters:

• text (str) – Thai text to be tagged

• pos (bool) – whether to request part-of-speech output; the model does not support it, so a warning is raised

• tag (bool) – whether to return HTML-like tags instead of tuples

Returns:

tagged words, or an HTML-like tagged string if tag is True

Return type:

Union[list[tuple[str, str]], str]

pythainlp.lm.wangchanberta.segment(text: str) → list[str]

Tokenize text into subwords with the WangchanBERTa tokenizer.

Parameters:

text (str) – text to be tokenized

Returns:

list of subwords

Return type:

list[str]

The segment function is a subword tokenization tool that breaks down text into subword units, offering a foundation for further text processing and analysis.

References

[1]

Lowphansirikul L, Polpanumas C, Jantrakulchai N, Nutanong S. WangchanBERTa: Pretraining transformer-based Thai Language Models. ArXiv:2101.09635 [Internet]. 2021 Jan 23 [cited 2021 Feb 27].