Vasilyev et al. (2020)#
Publication#
Fill in the BLANC: Human-free quality estimation of document summaries
Repositories#
https://github.com/PrimerAI/blanc
Available Models#
BLANCHelp
Description: A reference-free summarization evaluation metric
Name:
vasilyev2020-blanc-help
Usage:
from repro.models.vasilyev2020 import BLANCHelp model = BLANCHelp() inputs = [ {"sources": ["The input document"], "candidate": "The summary"} ] macro, micro = model.predict_batch(inputs)
The output
micro
is the input-level score.macro
is the score averaged over all of the inputs.
BLANCTune
Description: A reference-free summarization evaluation metric
Name:
vasilyev2020-blanc-tune
Usage:
from repro.models.vasilyev2020 import BLANCTune model = BLANCTune() inputs = [ {"sources": ["The input document"], "candidate": "The summary"} ] macro, micro = model.predict_batch(inputs)
The output
micro
is the input-level score.macro
is the score averaged over all of the inputs.
Implementation Notes#
The unit tests here do not replicate the example outputs from the original repository.
However, there were also differences between the original Readme and this Colab notebook which reruns the provided examples.
Further, the scores (at least for BLANCTune
) are different whether you use CPU or GPU.
Docker Information#
Image name:
danieldeutsch/vasilyev2020
Build command: Provide documentation on how to build the image
repro setup vasilyev2020
Requires network: Yes, it still requires connecting to the network even when warmup queries are run.
Testing#
repro setup vasilyev2020
pytest models/vasilyev2020/tests
Status#
[x] Regression unit tests pass
See here.[ ] Correctness unit tests pass
See explanation in “Implementation Notes”[ ] Model runs on full test dataset
Not tested[ ] Predictions approximately replicate results reported in the paper
Not tested[ ] Predictions exactly replicate results reported in the paper
Not tested