tau_eval package

Subpackages

• tau_eval.models package
• tau_eval.tasks package

Submodules

• tau_eval.experiment module
• tau_eval.logger module
• tau_eval.utils module
• tau_eval.visualization module

Module contents

Top-level package for tau-eval.

class tau_eval.Experiment(models: list[Anonymizer], metrics: list[str | Callable], tasks: list[Task | CustomTask], config: ExperimentConfig = ExperimentConfig(exp_name='experiment', classifier_name='answerdotai/ModernBERT-base', train_task_models=False, train_with_generations=False, device='cuda', classifier_args={}))

Bases: object

classmethod from_json(filepath: str)

Loads experiment results from a JSON file.

run(output_dir='results.json')

summary(output_dir=None, to_rich=False)

class tau_eval.ExperimentConfig(exp_name: str = 'experiment', classifier_name: str = 'answerdotai/ModernBERT-base', train_task_models: bool = False, train_with_generations: bool = False, device: str | None = 'cuda', classifier_args: dict = <factory>)

Bases: object

Evaluation experiment config

classifier_args: dict

classifier_name: str = 'answerdotai/ModernBERT-base'

device: str | None = 'cuda'

exp_name: str = 'experiment'

train_task_models: bool = False

train_with_generations: bool = False