Test suites and execution
To run a test suite, run:
python -m unittest tests.<test_suite_name>
This command will run a default set of test suites:
The default test suite includes all test suites listed in tests/__init__.py file.
Currently, it includes tests.core and tests.compact.
To optimize CI/CD resource utilization and manage dependency overhead,
tests are categorized into four tiers based on their resource
requirements and complexity: “core”, “compact”, “extra”, and “noauto”.
Adding a test case to a test suite
To add a test case to a test suite,
add it to tests_packages list in __init__.py
inside that test suite’s directory.
Test matrix for CI
The following table outlines the automated test coverage across
supported Python versions and operating systems:
| Python |
Ubuntu |
Windows |
macOS |
| 3.14 (Latest) |
O+C |
O |
O |
| 3.13 |
O+C+X |
O+C |
O+C |
| 3.12 |
O |
|
|
| 3.11 |
O |
|
|
| 3.10 (Earliest) |
O+C |
O+C |
O+C |
The CI/CD test workflow is at
https://github.com/PyThaiNLP/pythainlp/blob/dev/.github/workflows/unittest.yml.
Core tests (test_*.py)
- Run
python -m unittest tests.core
- Focus on core functionalities.
- Do not rely on external dependencies beyond the standard library.
- Do not perform network access or corpus downloads.
- Tested on all supported operating systems and all active Python versions.
- Test case class suffix:
TestCase
Compact tests (testc_*.py)
- Run
python -m unittest tests.compact
- Need dependencies from
pip install "pythainlp[compact]"
- Test a limited set of functionalities that rely on a stable
and small set of dependencies.
- These dependencies are
PyYAML, nlpo3, numpy,
and pyicu.
- Includes corpus download/remove tests (may require network access).
- Tested on:
- All OSes: earliest and second-latest supported Python versions
- Ubuntu: additionally tested on the latest version
- Test case class suffix:
TestCaseC
- Run
python -m unittest tests.extra
- Need dependencies from
pip install "pythainlp[compact,extra]"
- Test more functionalities that rely on larger set of dependencies
or one that require more time or computation.
- Only tested on Ubuntu using the second-latest Python version.
- Test case class suffix:
TestCaseX
Noauto tests (testn_*.py)
The noauto (no-automated) test suite contains tests for functionalities
that require heavy dependencies which are not feasible to run in automated
CI/CD pipelines. These tests are organized into specialized suites based
on their dependency requirements.
Why separate noauto test suites?
Different ML/AI frameworks often have conflicting version requirements for
their dependencies. For example:
- PyTorch and TensorFlow may require different versions of numpy or protobuf
- Large frameworks take significant time to install (~1-3 GB each)
- Some packages require Cython compilation or system libraries
By separating tests by dependency group, we can:
- Test each framework independently without conflicts
- Optimize CI/CD resources by running only relevant test groups
- Make it easier for developers to test specific functionality
Noauto test suites
Umbrella suite: tests.noauto
- Run
python -m unittest tests.noauto
- Includes all modular noauto test suites
- Use this for comprehensive testing when all dependencies are available
- Test case class suffix:
TestCaseN
Modular suites by dependency
PyTorch-based: tests.noauto_torch
- Run
python -m unittest tests.noauto_torch
- Need dependencies from
pip install "pythainlp[noauto-torch]"
- Tests requiring PyTorch and its ecosystem:
- torch, transformers (PyTorch backend)
- thai-nner, wtpsplit, tltk
- Tests: spell correction (wanchanberta), NER/POS tagging (transformers-based),
subword tokenization (phayathai, wangchanberta),
sentence tokenization (wtp)
- Dependencies: ~2-3 GB
- Test case class suffix:
TestCaseN
TensorFlow-based: tests.noauto_tensorflow
- Run
python -m unittest tests.noauto_tensorflow
- Need dependencies from
pip install "pythainlp[noauto-tensorflow]"
- Tests requiring TensorFlow:
- Dependencies: ~1-2 GB
- Note: May conflict with PyTorch dependencies
- Test case class suffix:
TestCaseN
ONNX Runtime-based: tests.noauto_onnx
- Run
python -m unittest tests.noauto_onnx
- Need dependencies from
pip install "pythainlp[noauto-onnx]"
- Tests requiring ONNX Runtime:
- attacut, deepcut, oskut, sefr_cut tokenizers (via LEKCut / ONNX)
- phayathaibert POS tagger (downloads a ~530 MB model on first run)
- Dependencies: ~200-500 MB
- Test case class suffix:
TestCaseN
Cython-compiled: tests.noauto_cython
- Run
python -m unittest tests.noauto_cython
- Need dependencies from
pip install "pythainlp[noauto-cython]"
- Tests requiring Cython-compiled packages:
- Requires: Cython, C compiler, system libraries (hunspell)
- Platform-specific build requirements
- Test case class suffix:
TestCaseN
Network-dependent: tests.noauto_network
- Run
python -m unittest tests.noauto_network
- Need dependencies from
pip install "pythainlp[noauto-network]"
- Tests requiring network access:
- Hugging Face Hub model downloads
- External API calls
- Requires: Internet connection, may involve large downloads
- Test case class suffix:
TestCaseN
Test coverage
Expected coverage: at least 80% overall, and at least 95% for new and
changed code.
CI runs diff-cover on pull requests, in the job that runs the core,
compact, and extra tests:
- Modules listed in
tests/diff-cover-noauto.txt need dependencies (such as
torch) that CI does not install. CI reports their coverage but does not
gate on it. Run the noauto suites locally and meet the same 95%.
- New and changed code in all other modules must reach 95%.
- Add a module to the list when it needs a heavy dependency that CI cannot
install. Keep the list sorted, one path per line.
To run the same check locally (needs pip install -e ".[compact,extra,dev]"):
make diff-cover DIFF_BASE=upstream/main
DIFF_BASE is the branch to compare against (default: origin/main).
DIFF_COVER_TESTS sets the test suites to run
(default: tests.core tests.compact tests.extra).
Robustness tests (test_robustness.py)
A comprehensive test suite within core tests that tests edge cases important
for real-world usage:
- Empty strings and various whitespace handling (spaces, tabs, Unicode spaces)
- Special characters from encoding issues, BOM, terminal copy/paste
- Truncated/malformed Unicode and surrogate pairs
- Emoji and modern Unicode sequences (ZWJ, modifiers, flags)
- Control and hidden/invisible characters (zero-width, control chars)
- Thai-specific edge cases with combining characters and mixed scripts
- Multi-engine robustness testing across all core tokenization engines
- Very long strings that can cause performance issues (issue #893)
Corpus test (corpus/)
A separate test suite that verifies the integrity, format, parseability,
and catalog functionality of corpus in PyThaiNLP.
These tests are separate from regular unit tests because they test actual
file loading and parsing (not mocked), require network access, and
can be resource intensive.
For detailed information about corpus test, see:
tests/corpus/README.md
The corpus test is triggered automatically via GitHub Actions
when changes are made to pythainlp/corpus/** or tests/corpus/**.
Run corpus test:
python -m unittest tests.corpus