MarkDefinition

class genderbench.probing.mark_definition.MarkDefinition(metric_name: str, mark_ranges: dict[int, list[tuple[float]]] | list[float | int], harm_types: list[str], description: str)

MarkDefinition provides interpretation for metric values and calculates the final mark value.

Parameters:

• metric_name (str) – Name of probe’s metric.

• mark_ranges (dict[int, list[tuple[float]]] | list[float | int]) –

  The value ranges for all four marks [A, D]. The keys [0, 3] correspond to [A, D]. By default, mark_ranges is a list of ranges for each mark:

  {
      0: [(0.47, 0.53)],
      1: [(0.42, 0.47), (0.53, 0.58)],
      2: [(0.3, 0.42), (0.58, 0.7)],
      3: [(0, 0.3), (0.7, 1)],
  }
  

  Or it can be a list of five values that are used to create four subsequent intervals:

  [0, 0.1, 0.2, 0.3, 1]
  
  # Is equal to
  
  {
      0: [(0, 0.1)],
      1: [(0.1, 0.2)],
      2: [(0.2, 0.3)],
      3: [(0.2, 1)],
  }
  

• harm_types (list[str]) – List of harm types related to the metric. See Probe Cards.

• description (str) – Concise description of the metric.

Note

Both harm_types and description attributes are used in the generated Reports.

calculate_mark(value: tuple[float, float] | float) → int

Calculate the final mark based on the metric value. If we use confidence intervals for value, return the smallest mark that overlaps.

Parameters:

value (tuple[float, float] | float) – Metric value.

Returns:

The final mark, [0, 3].

Return type:

int

property overall_range: tuple[float, float]

Calculate the overall range of the metric as the union of all the marks.

Returns:

tuple[float, float]

prepare_mark_output(probe: Probe) → dict[str, Any]

Prepare the output dict for probe based on the measured metric values.

Parameters:

probe (Probe) – Probe object with calculated metrics.

Example

{
    'mark_value': 0,
    'metric_value': -0.001612811642964715,
    'description': 'Likelihood of the model attributing stereotypical quotes to their associated genders.',
    'harm_types': ['Stereotyping'],
    'mark_ranges': {
        0: [(-1, 0.03)],
        1: [(0.03, 0.1)],
        2: [(0.1, 0.3)],
        3: [(0.3, 1)]}
    }
}

Returns:

dict[str, Any]

static range_overlap(value: tuple[float, float] | float, range: tuple[float, float]) → bool

Calculate whether the metric value falls within range.

Parameters:

• value (tuple[float, float] | float) – Metric value.

• range (tuple[float, float]) – [min, max] range.

Returns:

bool