General remarks on TM system behavior

Display

For numbers, TM systems have to be divided into two groups. Some recognize and highlight numbers in the source text during editing, while others do not, see table 3.2. When numbers are recognized, they can be placed into the target segment by means of a shortcut. In the TM system documentation, they are referred to as placeables for this reason. Failing to display numbers as placeables does not imply that they are always treated as plain text, see 3.2.1.2 for more information.

Numbers Shortcut highlighted?

Across yes yes Déjà Vu no - Heartsome no - memoQ no - MultiTrans no - SDL Trados yes yes Transit no - Wordfast yes yes

Table 3.2: Highlighting of numbers Table 3.3 shows how recognized numbers are displayed.

Display Across

SDL Trados Wordfast

Table 3.3: Display of recognized numbers

Across: recognized numbers are marked with a blue overline. The current element can be transferred with the shortcut Ctrl+Alt+0.

SDL Trados: recognized numbers are marked with a blue underline. The current element can be transferred with the shortcut Ctrl+Alt+_.

Wordfast: the current recognized number is marked with a red square. The current element can be transferred with the shortcut Ctrl+Alt+_.

Recognition assessment

For TM systems that highlight numbers in their editor (Across, SDL Trados and Wordfast), assessing what has been recognized is relatively straightforward. For all other TM systems, it is necessary to adopt a dierent strategy. This strategy consists of using the quality check function for numbers that is available for almost all TM systems, see table 3.4. This function reveals which numbers are recognized, and in nearly all cases, also indicates whether recognition covers numbers as units or splits them.

Quality check tool Across Quality Management Déjà Vu Terminology Check Heartsome (planned)

memoQ QA

MultiTrans QA add-on

SDL Trados QA checker (with TagEditor) Transit Check formatting codes Wordfast Quality Check

Table 3.4: Quality check tools

Déjà Vu: Déjà Vu does not provide any specic transfer function for numbers (unlike for tags, with the shortcut F8). The system supports users with the functions AutoPropa- gate and AutoAssemble, which copy numbers into the target segment, see 2.4.3.2 for more information. However, these functions are not suitable in order to ascertain whether Déjà Vu recognizes some numbers as a unit or splits them into several digit sequences. Therefore, a dierent strategy was adopted.

With Déjà Vu it is possible to verify numbers (Check Numerals). If the numbers in the source and in the target text dier, the digits will be displayed in red and the current segment will be marked with a red exclamation mark. During the tests, numbers were deliberately changed and successively veried. By changing one digit, it is possible to check whether the number as a unit is red or only some of its digits. An example is provided by segment 4 in 3.2.3.2, where 8:00 is recognized as two numbers separated by the colon.

memoQ: memoQ does not provide any standard transfer function for numbers, but a quality assurance function for verifying numbers was used for test purposes. The QA check in memoQ indicates only if there is a general number dierence between the source and target segment. A general error description is provided, e.g. Non-standard number format in the target side or Numbers in source and target segment do not match, however it is not always easy to ascertain the exact error because the diverging digits are not marked (as e.g. in Déjà Vu). Fortunately, recognition of decimal points and thousand separators as part of the number particularly relevant for these tests can be checked: if the decimal point or the thousand separator is modied, the QA checker recognizes the modication.

MultiTrans: MultiTrans does not provide any standard transfer function for numbers. An add-on, the QA Agent, can be used to verify whether numbers in the source and target text correspond. Because of copyright regulations, it was not possible to obtain the necessary license. Therefore, no recognition assessment could be made and MultiTrans is not included in these tests.

Transit: Transit does not provide any standard transfer function for numbers, but it is possible to verify them with the option Options > Check Formatting Codes > Settings > Numbers. If the numbers in the source and in the target text dier, a warning is displayed quoting the altered number.

Others: for Across and Wordfast, the recognition results in the editor were cross- checked with the quality control function. No divergences were ascertained, i.e., digits that were not recognized were not checked either. In the case of SDL Trados, since MS Word was used as the editor, no number check was available.

Scoring system

For a better comparison of the performance of TM systems, a scoring system was intro- duced. The reasons why usual precision and recall were not suitable have already been explained in 2.3.3.1. As regards precision, false positives are virtually absent in the test suites. Exceptions occur only for Wordfast, which applies extended alphanumeric recogni- tion. These examples will be discussed, but do not in any way oer a basis broad enough for a sound numeric conclusion. As regards recall, a more elaborate metric provides a more accurate account of the performance of the TM systems, see 3.3.2.

A score was calculated for each instance of recognition according to the following rules: ˆ 3 points were assigned when all digits were recognized.

ˆ If digits were separated by non-standard separators (such as a solidus e.g. in 1/4), they were treated as independent sequences.

ˆ 0.5 points were assigned for each correctly recognized thousand or decimal separator, only if preceded or followed by digits (e.g. in 0.70).

ˆ Bonus of 0.5 points was assigned when ranges consisting of two or more independent sequences were recognized (e.g. in 0.20-0.40). Partial recognitions were excluded. No extra bonus was applied to the non-alphanumeric character (usually a hyphen- minus sign) used to dene the range.

ˆ Bonus of 0.25 points was assigned for each correctly recognized additional character (e.g. a mathematical operator such as the plus sign in +10%). When a sequence of the same character type was recognized (e.g. alphabetic characters), the bonus was applied only once.

ˆ Penalty of 0.5 points was applied in the case of over-recognition (e.g. the word Seiten, meaning pages, in A4-Seiten).

ˆ Minor misrecognitions (hyphen-minus sign interpreted as a minus character when used in a range) were ignored (no bonus, no penalty).

A baseline3 _{was calculated: it assumes complete recognition of all digits and of all separa-}

tors (excluding non-standard separators such as a solidus, etc.).

In order to prevent bonuses compensating for poor digit recognition which is deemed the most serious problem low bonus values were set. Standard separators were included in the minimum match value because, in particular for measures and to some extent times, correct recognition of these elements is a prerequisite for any automatic adaptation.

Here are some score examples (the recognized characters are printed in gray, as in the tables):

1.234,5= 3 (all digits recognized) + 0.5 (one separator (comma) correctly recognized). The baseline is 3+0.5*2=4.

30 - 250 V = 3 + 3 (all digits recognized, 2 sequences) + 0.25 (V recognized) + 0.5 (range recognized). The baseline is 3+3=6.

In order to make the score more transparent, each table indicates how the score was calculated and which elements were considered. The elements are referred to with the following abbreviations:

ˆ D: digit(s) ˆ S: separator ˆ R: range

3_{The notion of baseline, see (Resnik and Lin, 2010, 279-280), is very close to that of golden score, see}

ˆ A: additional character(s) ˆ O: over-recognition