repro.common.util#

repro.common.util.aggregate_metrics_by_group(groups, metrics_list)#

Averages metrics based on groups. This is useful for when a metric only supports single references and you want to generalize it to a multi-reference metric by averaging over the references.

Parameters

  • groups (List[int]) – The group which corresponds to each item in metrics_list. The smallest group value must be 0 and must increase by 1. That is, if n is max(groups), then groups 0, 1, …, n-1 must also exist.

  • metrics_list (List[MetricsType]) – The metrics to aggregate via averaging.

Returns

A list of length max(groups) + 1 with the average metric values for each group.

Return type

List[MetricsType]

repro.common.util.aggregate_parallel_metrics(outputs_list)#

Aggregates the result of parallel metrics computations performed with a ParallelModel. It assumes that the metric returns a tuple consisting of the macro results of type MetricsType and a list of micro results, one per input, of type MetricsType.

This function creates a single list of all of the micro results across all of the parallel results and averages them to get a new macro result. If the metric’s macro score is not an average over the input-level results, this function will not correctly compute the metric’s macro score.

Parameters

outputs_list (List[Tuple[MetricsType, List[MetricsType]]]) – The outputs from the ParallelModel computation of a metric. Each item in the list contains the macro and micro results on the batch given to the parallel process.

Returns

  • MetricsType – The metric’s score averaged over all of the inputs

  • List[MetricsType] – The per-input scores, in the same order as they were passed as input.

Examples

Here is an example of how this could be used in combination with ROUGE to evaluate texts in parallel:

from repro.common.util import aggregate_parallel_metrics
from repro.models import ParallelModel
from repro.models.lin2004 import ROUGE

# The inputs that will be scored. Here we only have one
# input for demonstration purposes
inputs = [{"candidate": "Candidate text", "references": ["list of references"]}]

# First we show the serial computation
serial_model = ROUGE()
serial_macro, serial_micro = serial_model.predict_batch(inputs)

# The above is equivalent to the following parallel computation
parallel_model = ParallelModel(ROUGE, num_models=10)
outputs_list = parallel_model.predict_batch(inputs)
parallel_macro, parallel_micro = aggregate_parallel_metrics(outputs_list)

serial_macro and parallel_macro dicts will have the same result, as well as serial_micro and parallel_micro.

repro.common.util.average_dicts(dicts)#
repro.common.util.check_for_single_texts(texts_list)#
repro.common.util.flatten(text, separator=None)#

Flattens the text item to a string. If the input is a string, that same string is returned. Otherwise, the text is joined together with the separator.

Parameters

  • text (Union[str, List[str]]) – The text to flatten

  • separator (str, default=None) – The separator to join the list with. If None, the separator will be ” “

Returns

The flattened text

Return type

str

repro.common.util.flatten_nested_dict(nested_dict)#
repro.common.util.get_default_dict(d, default)#
repro.common.util.group_by_references(candidates, references_list)#

Groups the candidates by identical references in references_list. This is enables evaluation metrics which need to do a lot of preprocessing of the references to reducing the amount of duplicate effort required.

Parameters

  • candidates (List[TextType]) – The candidate texts

  • references_list (List[List[TextType]]) – The reference texts.

Returns

  • List[List[TextType]] – The grouped candidate texts in which the candidates at index i all share the same references in the grouped references

  • List[List[TextType]] – The grouped reference texts

  • List[Tuple[int, int]] – A mapping from the input candidates to the (i, j) position in the grouped candidates that it corresponds to.

repro.common.util.insert_empty_values(inputs, empty_indices, empty_value)#
repro.common.util.is_empty_text(text)#
repro.common.util.remove_empty_inputs(inputs, *contexts)#
repro.common.util.unflatten_dict(d)#
repro.common.util.ungroup_values(values, mapping)#

Ungroups the values in values_list based on the group mapping. mapping[i] specifies the (j, k) position in values_list that the value for i corresponds to. The result of the function is the ungrouped values for each item in mapping.

Parameters

  • values (Indexable[Indexable[T]]) – The grouped values

  • mapping (List[Tuple[int, int]]) – The mapping which specifies where in the grouped values each item in the original order is in.

Returns

The ungrouped values.

Return type

List[T]