Attempt

class genderbench.probing.attempt.Attempt(prompt: Prompt, repetition_id: int)

Attempt represents the entire lifecycle of a single call to a generator:

The object is initialized with a Prompt object representing the input.

Generator is used to generate a text answer in Probe.generate.

Probe’s evaluator is used to evaluate this answer and create an evaluation.

Parameters:

prompt (Prompt) – Input to a generator.
repetition_id (int) – Repetition index used when a single Prompt has multiple Attempts. See Probe.num_repetitions.

answer

Text answer generated based on prompt.

Type:: str

evaluation

Evaluation of the answer.

Type:: Any

evaluation_undetected

Did an Evaluator mark the evaluation of this Attempt as undetected?

Type:: Optional[bool]

uuid

UUID identifier.

Type:: uuid.UUID

evaluate(evaluator: Evaluator) → Any

Send the Attempt object to evaluator and store the result in evaluation.

Parameters:: evaluator (Evaluator)
Returns:: evaluator’s output.
Return type:: Any

classmethod from_json_dict(json_dict: dict, prompt: Prompt) → Attempt

Create a new Attempt object from a JSON-serializable dictionary representation.

Parameters:

json_dict (dict) – JSON-serializable dictionary. Generated by to_json_dict.
prompt (Prompt) – Prompt object used in the attempt.

Returns:

Restored Attempt object.

Return type:

Attempt

to_json_dict() → dict

Prepare a JSON-serializable dictionary representation. Used for logging.

Returns:: JSON-serializable dictionary.
Return type:: dict