Machine Translation

MT Quality Estimation (TMS)

MT Quality Estimation (MTQE) is an AI-powered feature that provides segment-level quality estimations for machine translation (MT) suggestions. It is similar to the quality estimations for translation memory (TM) matches and non-translatables (NT).

The instant MT quality scores help guide post-editing and can be used to enhance the default (pre-translation) analysis and to assess MT engine quality.

MTQE is only available through the Phrase Translate Add-on.

Using MTQE

MTQE is only available in projects with Phrase Translate as the selected MT engine type.

MTQE does not support all language combinations. See Supported Language Pairs.

Analyses

With MTQE enabled, Default Analysis includes MT scores alongside the TM and NT scores. For example, a 75% MT score falls into the 75%-84% match range, and a 99% MT score into the 95%-99% match range. This can be turned off in Analysis options.

Pre-translation

In addition to the instant, segment-level, quality matches in the CAT editor, MTQE is used in pre-translation. This can be turned off in pre-translate options.

Quality Scores

Scoring categories:

  • 100% -Excellent MT match, probably no post-editing required

  • 99% - Near-perfect MT output, possibly minor post-editing required for mostly typographical errors

  • 75% - Good MT match, but likely to require some post-editing

  • No score - When there is no score, it is very likely that the MT output is of low quality. In general, it is recommended that this output not be post-edited but used for reference only.

MTQE scores appear at the segment level together with other translation resources (TM, NT, TB). Match origin is presented in a tooltip and at the bottom of the CAT panel in the metadata section.

 

Evaluating MTQE Results

Once MTQE has been enabled and employed as part of the Machine Translation process, MTQE scores for content and engine can be measured for accuracy. Direct comparison between segment-level MTQE scores and post-editing analysis is not available, but the following options provide ways to quantitatively and qualitatively evaluate MTQE scores.

Evaluating with Post-editing Analysis

Post-editing analysis indicates editing effort; how much text the Linguist or Proofreader had to edit. For post-editing analysis in projects with Machine Translation and MTQE, results are calculated as the difference between the machine translation suggestions and the final text after post-editing is finished.

In order to evaluate the results of the post-editing analysis, run a Default Analysis before the first step of the workflow to see how MT matches are categorized into MTQE bands.

When post-editing is complete, run a Post-editing Analysis with the Analyze MT option.

If the machine translation or non-translatable suggestion was accepted without any editing, the results will indicate 100%.

If the machine translation has been changed, the match rate is lower and the more the segment is changed, the lower the score will be. This is the same score-counting algorithm as the one used to calculate the score of translation memory fuzzy matches.

If the default analysis indicates a high number of quality MT matches (75% or above), the post-editing analysis reflects the correspondingly minimal to moderate amount of editing to the MT suggestions.

Evaluating the Segment Changes

To evaluate the substance of the changes made during post-editing, create a workflow that generates a report showing the changes on a segment-level.

To create this workflow, follow these steps:

  1. Create a project with two Workflow steps (e.g. pre-translation and post-editing).

  2. In the first Workflow step, pre-translate the job with only MT. This provides a snapshot of the matches to be used.

  3. In the second Workflow step, let the linguist post-edit normally.

  4. Once the workflow is completed, run the post-editing analysis to see the edit distance between the two steps (the number of changes).

  5. Select the relevant jobs, then go to Tools and select Export Workflow Changes.

    The different versions of the segments are presented.

 

MTQE Supported Language Pairs

Codes are based on ISO 639-1.

Source Target
ar en
ca es
cs bg
cs de
cs en
cs es
cs fr
cs hu
cs it
cs pl
cs ro
cs ru
cs sk
cs sl
cs sv
cs uk
cy en
da de
da en
da es
da fi
da fr
da nb
da nl
da pl
da sv
de cs
de da
de en
de es
de fi
de fr
de it
de lv
de nb
de nl
de pl
de ro
de ru
de sk
de sl
de sr
de sv
el en
en af
en ar
en as
en az
en bg
en bn
en bs
en ca
en cs
en cy
en da
en de
en el
en es
en et
en fa
en fi
en fil
en fr
en ga
en gu
en he
en hi
en hr
en hu
en hy
en id
en is
en it
en ja
en kk
en kn
en ko
en lt
en lv
en mi
en ml
en mr
en ms
en mt
en nb
en ne
en nl
en or
en pa
en pl
en pt
en ro
en ru
en sk
en sl
en sq
en sr
en sv
en ta
en te
en th
en tl
en tr
en uk
en ur
en vi
en zh-hans
en zh-hant
es cs
es de
es en
es fr
es it
es nl
es pt
es ru
et en
et fi
et lt
et lv
et ru
fi de
fi en
fi et
fi ru
fi sv
fr ar
fr cs
fr de
fr en
fr es
fr id
fr it
fr nl
fr pt
fr ru
hr de
hr en
id en
it cs
it de
it en
it es
it fr
it pl
it pt
it ru
ja en
ja es
ja fr
ja id
ja ko
ja ru
ja zh-hans
ja zh-hant
ko en
ko ja
lt en
lt et
lt lv
lt pl
lt ru
lv en
lv et
lv ru
nb da
nb de
nb en
nb fi
nb nn
nb sv
nl de
nl en
nl fr
pl cs
pl de
pl en
pl it
pl lt
pl nl
pl ro
pl ru
pl sk
pt en
pt es
pt fr
pt it
ro en
ru cs
ru el
ru en
ru es
ru et
ru fr
ru ky
ru lt
ru lv
ru pt
sk cs
sk de
sk en
sk hu
sl en
sr en
sv ar
sv da
sv de
sv en
sv es
sv et
sv fa
sv fi
sv fr
sv is
sv it
sv nb
sv nl
sv pl
sv ru
th en
tr ar
tr en
uk en
vi en
zh-hans en
zh-hans ja
zh-hant en
 
 
Was this article helpful?

Sorry about that! In what way was it not helpful?

The article didn’t address my problem.
I couldn’t understand the article.
The feature doesn’t do what I need.
Other reason.

Note that feedback is provided anonymously so we aren't able to reply to questions.
If you'd like to ask a question, submit a request to our Support team.
Thank you for your feedback.