pythainlp.lm.phayathaibert

Note

pythainlp.phayathaibert has moved to pythainlp.lm.phayathaibert. Importing from pythainlp.phayathaibert still works but emits a DeprecationWarning and will be removed in 6.0.

The pythainlp.lm.phayathaibert module is built upon the phayathaibert base model.

Modules

class pythainlp.lm.phayathaibert.ThaiTextProcessor

Preprocess Thai text for PhayaThaiBERT.

__init__() → None

Initialize the text processor.

replace_url(text: str) → str

Replace URLs in text with the URL token.

See https://stackoverflow.com/a/6041965

Parameters:

text (str) – text to replace url

Returns:

text where urls are replaced

Return type:

str

Example:

>>> replace_url("go to https://github.com")
'go to <url>'

rm_brackets(text: str) → str

Remove empty brackets and artifacts within brackets from text.

Parameters:

text (str) – text to remove useless brackets

Returns:

text where all useless brackets are removed

Return type:

str

Example:

>>> rm_brackets("hey() whats[;] up{*&} man(hey)")
'hey whats up man(hey)'

replace_newlines(text: str) → str

Replace newlines in text with spaces.

param str text:

text to replace all newlines with spaces

return:

text where all newlines are replaced with spaces

rtype:

str

Example:

>>> rm_useless_spaces("hey whats

up”)

hey whats up

rm_useless_spaces(text: str) → str

Collapse repeated spaces in text (code from fastai).

Parameters:

text (str) – text to replace useless spaces

Returns:

text where all spaces are reduced to one

Return type:

str

Example:

>>> rm_useless_spaces("oh         no")
oh no

replace_spaces(text: str, space_token: str = '<_>') → str

Replace spaces in text with a space token.

Parameters:

text (str) – text to replace spaces

Returns:

text where all spaces replaced with _

Return type:

str

Example:

>>> replace_spaces("oh no")
oh_no

replace_rep_after(text: str) → str

Remove character repetitions in text.

Parameters:

text (str) – input text to replace character repetition

Returns:

text with repetitive tokens removed.

Return type:

str

Example:

>>> text = "กาาาาาาา"
>>> replace_rep_after(text)
'กา'

replace_wrep_post(toks: list[str]) → list[str]

Remove repeated words after tokenization.

The replace_wrep function of fastai does not work well with Thai.

Parameters:

toks (list[str]) – list of tokens

Returns:

list of tokens where repetitive words are removed.

Return type:

list[str]

Example:

>>> toks = ["กา", "น้ำ", "น้ำ", "น้ำ", "น้ำ"]
>>> replace_wrep_post(toks)
['กา', 'น้ำ']

remove_space(toks: list[str]) → list[str]

Remove spaces from a list of words for bag-of-words models.

Parameters:

toks (list[str]) – list of tokens

Returns:

List of tokens where space tokens (” “) are filtered out

Return type:

list[str]

Example:

>>> toks = ["ฉัน", "เดิน", " ", "กลับ", "บ้าน"]
>>> remove_space(toks)
['ฉัน', 'เดิน', 'กลับ', 'บ้าน']

preprocess(text: str, pre_rules: Optional[list[Callable[..., str]]]=None, tok_func: Callable[..., list[str]]=<function word_tokenize>) → str

Apply preprocessing rules, tokenize text, and join the words.

class pythainlp.lm.phayathaibert.ThaiTextAugmenter

Generate and augment text with PhayaThaiBERT.

__init__() → None

Initialize the text augmenter.

generate(sample_text: str, word_rank: int, max_length: int = 3, sample: bool = False) → str

Generate text from PhayaThaiBERT.

augment(text: str, num_augs: int = 3, sample: bool = False) → list[str]

Text augmentation from PhayaThaiBERT.

Parameters:

• text (str) – Thai text

• num_augs (int) – an amount of augmentation text needed as an output

• sample (bool) – whether to sample the text as an output or not, true if more word diversity is needed

Returns:

list of text augment

Return type:

list[str]

Example:

>>> from pythainlp.augment.lm import ThaiTextAugmenter

>>> aug = ThaiTextAugmenter()
>>> aug.augment("ช้างมีทั้งหมด 50 ตัว บน", num_args=5)

[‘ช้างมีทั้งหมด 50 ตัว บนโลกใบนี้ครับ.’,

‘ช้างมีทั้งหมด 50 ตัว บนพื้นดินครับ…’, ‘ช้างมีทั้งหมด 50 ตัว บนท้องฟ้าครับ…’, ‘ช้างมีทั้งหมด 50 ตัว บนดวงจันทร์.‼’, ‘ช้างมีทั้งหมด 50 ตัว บนเขาค่ะ😁’]

class pythainlp.lm.phayathaibert.PartOfSpeechTagger(model: str = 'lunarlist/pos_thai_phayathai', revision: str | None = None)

Tag Thai text with part-of-speech labels using PhayaThaiBERT.

__init__(model: str = 'lunarlist/pos_thai_phayathai', revision: str | None = None) → None

Initialize the part-of-speech tagger.

get_tag(sentence: str, strategy: str = 'simple') → list[list[tuple[str, str]]]

Tag sentences with part-of-speech (POS) labels.

Parameters:

sentence (str) – a list of lists of tokenized words

Returns:

a list of lists of tuples (word, POS tag)

Return type:

list[list[tuple[str, str]]]

Example:

Labels POS for given sentence:

>>> from pythainlp.lm.phayathaibert.core import (
...     PartOfSpeechTagger,
... )

>>> tagger = PartOfSpeechTagger()
>>> tagger.get_tag("แมวทำอะไรตอนห้าโมงเช้า")
[[('แมว', 'NOUN'), ('ทําอะไร', 'VERB'), ('ตอนห้าโมงเช้า', 'NOUN')]]

class pythainlp.lm.phayathaibert.NamedEntityTagger(model: str = 'Pavarissy/phayathaibert-thainer', revision: str | None = None)

Tag named entities with PhayaThaiBERT.

__init__(model: str = 'Pavarissy/phayathaibert-thainer', revision: str | None = None) → None

Initialize the named entity tagger.

get_ner(text: str, tag: bool = False, pos: bool = False, strategy: str = 'simple') → list[tuple[str, str]] | list[tuple[str, str, str]] | str

Tag named entities in text.

Parameters:

• text (str) – text in Thai to be tagged

• pos (bool) – output with part-of-speech tags. (PhayaThaiBERT is supported in PartOfSpeechTagger)

Returns:

a list of tuples associated with tokenized words, NER tags, POS tags (if the parameter pos is specified as True), and output HTML-like tags (if the parameter tag is specified as True). Otherwise, return a list of tuples associated with tokenized words and NER tags

Return type:

Union[list[tuple[str, str]], list[tuple[str, str, str]], str]

Example:

>>> from pythainlp.lm.phayathaibert.core import NamedEntityTagger
>>>
>>> tagger = NamedEntityTagger()
>>> tagger.get_ner("ทดสอบนายปวริศ เรืองจุติโพธิ์พานจากประเทศไทย")
[('นายปวริศ เรืองจุติโพธิ์พานจากประเทศไทย', 'PERSON'),
('จาก', 'LOCATION'),
('ประเทศไทย', 'LOCATION')]
>>> ner.tag("ทดสอบนายปวริศ เรืองจุติโพธิ์พานจากประเทศไทย", tag=True)
'ทดสอบ<PERSON>นายปวริศ เรืองจุติโพธิ์พาน</PERSON>                <LOCATION>จาก</LOCATION><LOCATION>ประเทศไทย</LOCATION>'

pythainlp.lm.phayathaibert.segment(sentence: str) → list[str]

Tokenize text into subwords with the PhayaThaiBERT tokenizer.

The tokenizer is a SentencePiece model with vocabulary expansion.

Parameters:

sentence (str) – text to be tokenized

Returns:

list of subwords

Return type:

list[str]