tau_eval.experiment module

class tau_eval.experiment.Experiment(models: list[Anonymizer], metrics: list[str | Callable], tasks: list[Task | CustomTask], config: ExperimentConfig = ExperimentConfig(exp_name='experiment', classifier_name='answerdotai/ModernBERT-base', train_task_models=False, train_with_generations=False, device='cuda', classifier_args={}))

Bases: object

classmethod from_json(filepath: str)

Loads experiment results from a JSON file.

run(output_dir='results.json')

summary(output_dir=None, to_rich=False)

class tau_eval.experiment.ExperimentConfig(exp_name: str = 'experiment', classifier_name: str = 'answerdotai/ModernBERT-base', train_task_models: bool = False, train_with_generations: bool = False, device: str | None = 'cuda', classifier_args: dict = <factory>)

Bases: object

Evaluation experiment config

classifier_args: dict

classifier_name: str = 'answerdotai/ModernBERT-base'

device: str | None = 'cuda'

exp_name: str = 'experiment'

train_task_models: bool = False

train_with_generations: bool = False

tau_eval.experiment.rich_display_dataframe(df, title='Dataframe') → None

Display dataframe as table using rich library. :param df: dataframe to display :type df: pd.DataFrame :param title: title of the table. Defaults to “Dataframe”. :type title: str, optional

Raises:

NotRenderableError – if dataframe cannot be rendered

Returns:

rich table

Return type:

rich.table.Table