Quickstart

Tau-Eval is designed for flexibility. With just a few lines of code, you can set up and run evaluations.

1. Define Your Anonymization Model

Create a custom anonymization model by extending the Anonymizer interface:

from tau_eval.models import Anonymizer

class TestModel(Anonymizer):
    def __init__(self):
        self.name = "Test Model"

    def anonymize(self, text: str) -> str:
        # Implement anonymization logic
        return text

    def anonymize_batch(self, texts: list[str]) -> list[str]:
        # Batch processing
        return texts

Or use prebuilt models from tau_eval.models.

2. Configure Evaluation Metrics

Use built-in metrics from tau_eval.metrics or define your own following this signature:

Callable[[str | list[str], str | list[str]], dict]

This allows complete control over what and how you evaluate.

3. Instantiate Tasks

Tasks can be created using prebuilt options in tau_eval.tasks, or customized using CustomTask. Tau-Eval also supports tasksource for dataset integration.

from tau_eval.tasks import DeIdentification
from tasknet import AutoTask

anli = AutoTask("anli/a1")
deid = DeIdentification(dataset="ai4privacy/pii-masking-400k")

4. Configure and Run Your Experiment

Define an experiment configuration:

from tau_eval.config import ExperimentConfig

config = ExperimentConfig(
    exp_name="test-experiment",
    classifier_name="answerdotai/ModernBERT-base",
    train_task_models=True,
    train_with_generations=False,
)

Run the experiment:

from tau_eval.experiment import Experiment

Experiment(
    models=[TestModel(), ...],
    metrics=["bertscore", "rouge"],
    tasks=[anli, deid],
    config=config
).run()

5. Visualize Results

Tau-Eval includes built-in visualization tools to compare model anonymization strategies and evaluation results. You can find them with tau_eval.visualization.