amazon-science/NLPActiveTesting

amazon-science/NLPActiveTesting

Description: Repository containing the results of the paper: Select, Label, Evaluate: Active Testing in NLP, accepted at EMNLP 2026 Main Conference.

Language: Python

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 1

Created: 2026-08-28T21:42:36Z

Pushed: 2026-08-29T08:34:27Z

Default branch: main

Fork: no

Archived: no

README:

Select, Label, Evaluate: Active Testing in NLP

Additional plots of the paper can be found at this link!

If you want to launch an experiment:

Install the required libraries:

pip install -r requirements.txt

You can simply use the bash files with the pre-defined experiments.

For experiments on real data:

bash runs/total/run_{dataset_name}.sh

• If you want to run a specific experiment:

python3 main.py --predictor PREDICTOR_NAME --dataset_name DATASET_NAME --model_name EMBEDDER_NAME --budget B --methods METHOD_NAME --strategy STRATEGY

Example:

python3 main.py --num_seeds 10 --dataset_name agnews --budget 1000 --methods Agreement

• If you want to run a multilingual experiment:

python3 multilingual.py --strategy STRATEGY --model_name EMBEDDER_NAME --methods METHOD_NAME --budget B --prior PRIOR_VALUE

Example:

python3 multilingual.py --predictor nova_pro --model_name bert --prior 0.6 --num_seeds 10 --dataset_name multilingual --budget 1000 --methods Agreement

• If you want to run an experiment on NER/POS tagging:

python3 pos_ner.py --strategy STRATEGY --model_name EMBEDDER_NAME --methods METHOD_NAME --budget B --predictor PREDICTOR_NAME --dataset_name DATASET_NAME

Example:

python3 pos_ner.py --model_name distilbert --predictor nuner --num_seeds 10 --dataset_name uniner --budget 1000 --methods Agreement

• If you want to run an experiment on summatization:

python3 summarization.py --strategy STRATEGY --model_name EMBEDDER_NAME --methods METHOD_NAME --budget B --predictor PREDICTOR_NAME --dataset_name DATASET_NAME

Example:

python3 summarization.py --predictor claude --model_name bert --num_seeds 10 --dataset_name cnn --budget 1000 --methods Agreement

Due to space limits on Github, in order to download the LLM annotations for summarization, you have to go to the autoeval folder here.

If you want to see some plots:

python3 utils.py --task unbiased --base_folder PREDICTOR/results_{EMBEDDER_NAME}

With:

• B: Value of budget (int);
• PRIOR_VALUE value of the prior on english (float);
• EMBEDDER_NAME: qwen,stella,bert,distilbert;
• METHOD_NAME: Random,Coverage,Surrogate,Uncertainty;
• STRATEGY: SVM or RF for Surrogate, max_dist for Coverage, gaussian_prior or mutual_information for Uncertainty.
• PREDICTOR_NAME: claude,nova_pro
• DATASET_NAME: list of all the available datasets can be found in data.py;

Citation

If you use this code in your research or project, please cite us:

@article{purificato2026select,
title={Select, Label, Evaluate: Active Testing in NLP},
author={Purificato, Antonio and Bucarelli, Maria Sofia and Bacciu, Andrea and Mantrach, Amin and Silvestri, Fabrizio},
journal={arXiv preprint arXiv:2603.21840},
year={2026}
}

For doubts or errors feel free to ping purificato@diag.uniroma1.it!

Acknowledgments

The implementation of competitor methods draws from the Active Testing Repository and the paper Active Testing: Sample-Efficient Model Evaluation. We gratefully acknowledge the authors for making their code available.

Security

See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. This code is being released solely for academic and scientific reproducibility purposes, in support of the methods and findings described in the associated publication. Pull requests are not being accepted in order to maintain the code exactly as it was used in the paper.

License

This library is licensed under the CC-BY-NC-4.0 License.