Scialom et al. (2019)#
Publication#
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
Repositories#
https://github.com/ThomasScialom/summa-qa
Available Models#
This implementation wraps the SummaQA
metric.
SummaQA
Description: A reference-free QA-based metric
Name:
scialom2019-summaqa
Usage:
from repro.models.scialom2019 import SummaQA model = SummaQA() inputs = [ {"candidate": "The candidate", "references": ["The reference"], "sources": ["The source"]}, ... ] macro, micro = model.predict_batch(inputs)
macro
is the averaged SummaQA scores over the inputs, andmicro
is the individual scores per input.
Implementation Notes#
The metric only supports 1 source document, so the length of
sources
must be 1.The metric does not support using the GPU
Docker Information#
Image name:
scialom2019
Build command:
repro setup scialom2019 [--silent]
Requires network: No
Testing#
repro setup scialom2019
pytest models/scialom2019/tests
Status#
[x] Regression unit tests pass
[x] Correctness unit tests pass
See here. We have reproduced the examples from the original code’s Github repo.[ ] Model runs on full test dataset
Not tested[ ] Predictions approximately replicate results reported in the paper
Not tested[ ] Predictions exactly replicate results reported in the paper
Not tested