pythainlp.wangchanberta

The pythainlp.wangchanberta module is built upon the WangchanBERTa base model, specifically the wangchanberta-base-att-spm-uncased model, as detailed in the paper by Lowphansirikul et al. [1].

This base model is utilized for various natural language processing tasks in the Thai language, including named entity recognition, part-of-speech tagging, and subword tokenization.

If you intend to fine-tune the model or explore its capabilities further, please refer to the thai2transformers repository.

Speed Benchmark

Function	Named Entity Recognition	Part of Speech
PyThaiNLP basic function	89.7 ms	312 ms
pythainlp.wangchanberta (CPU)	9.64 s	9.65 s
pythainlp.wangchanberta (GPU)	8.02 s	8 s

For a comprehensive performance benchmark, the following notebooks are available:

• `PyThaiNLP basic function and pythainlp.wangchanberta CPU at Google Colab`_

• `pythainlp.wangchanberta GPU`_

Modules

class pythainlp.wangchanberta.NamedEntityRecognition(model: str = 'pythainlp/thainer-corpus-v2-base-model', revision: str | None = None)

Recognize named entities in Thai text using WangchanBERTa.

The NamedEntityRecognition class is a fundamental component for identifying named entities in Thai text. It allows you to extract entities such as names, locations, and organizations from text data.

__init__(model: str = 'pythainlp/thainer-corpus-v2-base-model', revision: str | None = None) → None

Initialize a named entity tagger in IOB format.

Powered by WangchanBERTa from the VISTEC-depa AI Research Institute of Thailand.

Parameters:

• model (str) – name of a model pretrained from WangchanBERTa

• revision (Optional[str]) – git revision id (branch, tag, or commit hash). Pin to a full commit hash for secure downloads.

tokenizer: PreTrainedTokenizerBase

model: PreTrainedModel

get_ner(text: str, pos: bool = False, tag: bool = False) → list[tuple[str, str]] | str

Tag named entities in text in IOB format.

Powered by WangchanBERTa from the VISTEC-depa AI Research Institute of Thailand.

Parameters:

• text (str) – Thai text to be tagged

• pos (bool) – output part-of-speech tags. This model does not support them, so a warning is raised.

• tag (bool) – return HTML-like tags in a string instead of a list of tuples

Returns:

list of tuples (word group, named entity tag), or a string with HTML-like tags if tag is True

Return type:

Union[list[tuple[str, str]], str]

class pythainlp.wangchanberta.ThaiNameTagger(dataset_name: str = 'thainer', grouped_entities: bool = True)

Tag named entities in Thai text using WangchanBERTa.

The ThaiNameTagger class is designed for tagging Thai names within text. This is essential for tasks such as entity recognition, information extraction, and text classification.

json_ner: list[dict[str, str]]

output: str

sent_ner: list[tuple[str, str]]

__init__(dataset_name: str = 'thainer', grouped_entities: bool = True) → None

Initialize a named entity tagger in IOB format.

Powered by WangchanBERTa from the VISTEC-depa AI Research Institute of Thailand.

Parameters:

• dataset_name (str) –

  dataset the model is fine-tuned on

  • thainer - ThaiNER dataset (default)

• grouped_entities (bool) – whether to group word pieces of the same entity

dataset_name: str

grouped_entities: bool

classify_tokens: TokenClassificationPipeline

get_ner(text: str, pos: bool = False, tag: bool = False) → list[tuple[str, str]] | str

Tag named entities in text in IOB format.

Powered by WangchanBERTa from the VISTEC-depa AI Research Institute of Thailand.

Parameters:

• text (str) – Thai text to be tagged

• pos (bool) – output part-of-speech tags. This model does not support them, so a warning is raised.

• tag (bool) – return HTML-like tags in a string instead of a list of tuples

Returns:

list of tuples (word group, named entity tag), or a string with HTML-like tags if tag is True

Return type:

Union[list[tuple[str, str]], str]

pythainlp.wangchanberta.segment(text: str) → list[str]

Tokenize text into subwords with the WangchanBERTa tokenizer.

The tokenizer is a SentencePiece model.

Parameters:

text (str) – text to be tokenized

Returns:

list of subwords

Return type:

list[str]

The segment function is a subword tokenization tool that breaks down text into subword units, offering a foundation for further text processing and analysis.

References

[1]

Lowphansirikul L, Polpanumas C, Jantrakulchai N, Nutanong S. WangchanBERTa: Pretraining transformer-based Thai Language Models. ArXiv:2101.09635 [Internet]. 2021 Jan 23 [cited 2021 Feb 27].