Thompson & Post (2020)#
Publication#
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
Repositories#
https://github.com/thompsonb/prism
Available Models#
Prism
Description: A machine translation metric based on paraphrasing
Name:
thompson2020-prism
Usage:
from repro.models.thompson2020 import Prism model = Prism() inputs = [ {"candidate": "The candidate", "references": ["The reference"]} ] macro, micro = model.predict_batch(inputs)
macro
is the average Prism score across inputs, andmicro
is the score per input.
Prism-src
Description: A reference-free machine translation metric based on paraphrasing
Name:
thompson2020-prism-src
Usage:
from repro.models.thompson2020 import PrismSrc model = PrismSrc() inputs = [ {"candidate": "The candidate", "sources": ["The source"]} ] macro, micro = model.predict_batch(inputs)
macro
is the average Prism-src score across inputs, andmicro
is the score per input.
Prism as an MT model
Description: Uses Prism as a machine translation model
Name:
thompson2020-prism
Usage:
from repro.models.thompson2020 import Prism model = Prism() inputs = [ {"source": "The source text."} ] outputs = model.translate_batch("fr", inputs)
"fr"
is the language code for the target language andoutputs
is the list of translations.
Implementation Notes#
The metric requires all inputs to have sources xor references, not both or neither.
The metric only supports single references and/or sources, so the length of
references
andsources
must be 1.
Docker Information#
Image name:
danieldeutsch/thompson2020:1.2
Build command:
repro setup thompson2020 [--silent]
Requires network: No
Testing#
repro setup thompson2020
pytest models/thompson2020/tests
Status#
[x] Regression unit tests pass
[x] Correctness unit tests pass
See here. We replicate the output provided in their Github repository.[ ] Model runs on full test dataset
Not tested[ ] Predictions approximately replicate results reported in the paper
Not tested[ ] Predictions exactly replicate results reported in the paper
Not tested
Changelog#
v1.3#
Separated Prism and PrismSrc. The output metric name for PrismSrc is now “prism-src”
v1.2#
Added a
batch_size
argument to the translation functions
v1.1#
Added using Prism as an MT model