ProbeItem
- class genderbench.probing.probe_item.ProbeItem(prompts: list[Prompt], num_repetitions: int, metadata: dict[str, Any] | None = None)
ProbeItem is a single test item for Probe. Its main role is to group related Attempts into a single evaluated entity when metrics are calculated. The sampling process performed during confidence interval calculations is also done on a ProbeItem level. For this reason, it is recommended to group related Prompts into a single ProbeItem to avoid overconfident inverval ranges.
ProbeItem object handles the creation of multiple Attempts with identical Prompts when num_repetitions is used in Probe.
Note
Apart from repetitions, ProbeItem can be used to group very similar prompts that we do not want to necessarily treat as separate entities. For example, multiple orderings of options for a multiple-choice question can be grouped within one ProbeItem.
- Parameters:
prompts (list[Prompt]) – Prompts that will be used to create new Attempts.
num_repetitions (int) – How many Attempts will be created for each Prompt.
metadata (Optional[dict[str, Any]], optional) – Metadata related to ProbeItem that can be used during metric calculation process.
- uuid
UUID identifier.
- Type:
uuid.UUID
- classmethod from_json_dict(json_dict: dict) ProbeItem
Create a new ProbeItem object from a JSON-serializable dictionary representation.
- Parameters:
json_dict (dict) – JSON-serializable dictionary. Generated by
to_json_dict.- Returns:
Restored ProbeItem object.
- Return type:
- to_json_dict() dict
Prepare a JSON-serializable dictionary representation. Used for logging.
- Returns:
JSON-serializable dictionary.
- Return type:
dict