repro.common.util#
- repro.common.util.aggregate_metrics_by_group(groups, metrics_list)#
Averages metrics based on groups. This is useful for when a metric only supports single references and you want to generalize it to a multi-reference metric by averaging over the references.
- Parameters
groups (List[int]) – The group which corresponds to each item in metrics_list. The smallest group value must be 0 and must increase by 1. That is, if n is max(groups), then groups 0, 1, …, n-1 must also exist.
metrics_list (List[MetricsType]) – The metrics to aggregate via averaging.
- Returns
A list of length max(groups) + 1 with the average metric values for each group.
- Return type
List[MetricsType]
- repro.common.util.aggregate_parallel_metrics(outputs_list)#
Aggregates the result of parallel metrics computations performed with a
ParallelModel
. It assumes that the metric returns a tuple consisting of the macro results of typeMetricsType
and a list of micro results, one per input, of typeMetricsType
.This function creates a single list of all of the micro results across all of the parallel results and averages them to get a new macro result. If the metric’s macro score is not an average over the input-level results, this function will not correctly compute the metric’s macro score.
- Parameters
outputs_list (List[Tuple[MetricsType, List[MetricsType]]]) – The outputs from the
ParallelModel
computation of a metric. Each item in the list contains the macro and micro results on the batch given to the parallel process.- Returns
MetricsType – The metric’s score averaged over all of the inputs
List[MetricsType] – The per-input scores, in the same order as they were passed as input.
Examples
Here is an example of how this could be used in combination with ROUGE to evaluate texts in parallel:
from repro.common.util import aggregate_parallel_metrics from repro.models import ParallelModel from repro.models.lin2004 import ROUGE # The inputs that will be scored. Here we only have one # input for demonstration purposes inputs = [{"candidate": "Candidate text", "references": ["list of references"]}] # First we show the serial computation serial_model = ROUGE() serial_macro, serial_micro = serial_model.predict_batch(inputs) # The above is equivalent to the following parallel computation parallel_model = ParallelModel(ROUGE, num_models=10) outputs_list = parallel_model.predict_batch(inputs) parallel_macro, parallel_micro = aggregate_parallel_metrics(outputs_list)
serial_macro
andparallel_macro
dicts will have the same result, as well asserial_micro
andparallel_micro
.
- repro.common.util.average_dicts(dicts)#
- repro.common.util.check_for_single_texts(texts_list)#
- repro.common.util.flatten(text, separator=None)#
Flattens the text item to a string. If the input is a string, that same string is returned. Otherwise, the text is joined together with the separator.
- Parameters
text (Union[str, List[str]]) – The text to flatten
separator (str, default=None) – The separator to join the list with. If None, the separator will be ” “
- Returns
The flattened text
- Return type
str
- repro.common.util.flatten_nested_dict(nested_dict)#
- repro.common.util.get_default_dict(d, default)#
- repro.common.util.group_by_references(candidates, references_list)#
Groups the candidates by identical references in references_list. This is enables evaluation metrics which need to do a lot of preprocessing of the references to reducing the amount of duplicate effort required.
- Parameters
candidates (List[TextType]) – The candidate texts
references_list (List[List[TextType]]) – The reference texts.
- Returns
List[List[TextType]] – The grouped candidate texts in which the candidates at index i all share the same references in the grouped references
List[List[TextType]] – The grouped reference texts
List[Tuple[int, int]] – A mapping from the input candidates to the (i, j) position in the grouped candidates that it corresponds to.
- repro.common.util.insert_empty_values(inputs, empty_indices, empty_value)#
- repro.common.util.is_empty_text(text)#
- repro.common.util.remove_empty_inputs(inputs, *contexts)#
- repro.common.util.unflatten_dict(d)#
- repro.common.util.ungroup_values(values, mapping)#
Ungroups the values in values_list based on the group mapping. mapping[i] specifies the (j, k) position in values_list that the value for i corresponds to. The result of the function is the ungrouped values for each item in mapping.
- Parameters
values (Indexable[Indexable[T]]) – The grouped values
mapping (List[Tuple[int, int]]) – The mapping which specifies where in the grouped values each item in the original order is in.
- Returns
The ungrouped values.
- Return type
List[T]