Assembling the regular expression - Structure of test descriptions

2.5 Structure of test descriptions

3.4.7 Assembling the regular expression

The regular expressions for numbers, including the improvements, is presented below. Several decisions and trade-os had to be made. In addition, the regular expression was constructed on the basis of the test data. Despite all eorts to keep it as general as possible, it is likely that it will need some adaptations when applied to dierent test data.

1 m/ 2 ( 3 ( ? : 4 ( ? : 5 \p{Sm}\p{Zs }? 6 | 7 \p{Pd } [ , . ] ? 8 ) ? 9 10 \p{N}+ 11 ( ? : 12 \p{P} [ \ p{N}\p{Pd}]*?\ p{N}+ 13 | 14 \p{P}\p{Pd}+(?!\p{L}) 15 | 16 \p{Zs }\p{N}+ 17 ) * 18 )+ 19 20 ( ? : 21 [ \ p{L}\p{M}]+\b 22 | 23 \p{Zs } [ \ p{Lu}\p{M}]+\b 24 | 25 \p{Zs }? [ \ p{Sc}%] 26 | 27 \p{Zs }? ° [CF] ? 28 ) ? 29 ) 30 /gx

Chapter 4 Dates

4.1 Introduction

Because of their peculiarities, dates were not considered as a subset of numbers in chapter 3. There are two types of dates:

Numeric dates, e.g. 02-02-2010.

Alphanumeric dates, e.g. 2nd _{February 2010.}

Alphanumeric dates can be recognized by TM systems only if linguistic knowledge is available. Ideally, this knowledge should be available for each supported language (and most TM systems support virtually any language).

4.2 Tests

The tests are aimed at checking the following issues: Are numeric dates recognized as one single unit? Are alphanumeric dates recognized as one single unit?

As in chapter 3, the automatic conversion of dates from the source language format (e.g. 26.01.2011 in German) into the target language format (e.g. 01-26-2011 in US English) oered by several TM systems is not tested.1 _{Recognition assessment does not require that}

target text be entered, see 2.3.2.1.

In the examples given in this chapter, dates always occur as plain text. If dates have been inserted in a document as elds (e.g. in MS Word 2003 with Insert > Field > Date), they are processed in a dierent way. See chapter 10 for further information.

1_{(Trujillo, 1999, 68) states that dates can be automatically reformatted and the names of the month}

and day reliably translated. Whilst it is often true, it implies some basic linguistic knowledge for all supported languages.

4.2.1 TM system settings

4.2.1.1 Versions

TM system Version Across Standalone Personal Edition 4.00 Déjà Vu X Professional 7.5.303 Heartsome Translation Studio Ultimate 7.0.6 2008-09-12S memoQ Corporate 3.2.17 MultiTrans 4.3.0.84 SDL Trados 2007 Freelance 8.2.0.835 STAR Transit XV Professional 3.1 SP22 631

Wordfast 5.53

Table 4.1: TM systems used in the tests 4.2.1.2 Customizability

The customizability options mainly determine whether date-specic patterns can be modied and whether automatic conversion is performed.

Across: under Tools > System settings > General > Language settings >Locale settings > Language sets > Standard language set > Languages > [specific language] > Format > Date long and Date short, language-depen- dent date formats are specied. They are editable and new ones can be added. In addition, under Standard language set > Format, language-independent date formats are specied. They are editable as well, and can be supplemented. However, only numeric dates are taken into account.

Across oers automatic conversion for dates if the option Use autochanges under Tools > System Settings > General > crossTank is activated (default setting) and a Rich TM is used, see 2.4.3.2 for more information.

Déjà Vu, Transit, Wordfast: no specic setting for the recognition of dates is available. Numeric dates are treated as numbers, see 3.2.1.2.

Heartsome: the tested version does not provide any date recognition. Neither is a quality assurance component available. Consequently, Heartsome will not be included in these tests. These functions are planned for future versions.

memoQ: no embedded function for recognizing or converting dates is available. For this purpose, the function Auto-translatables described in 3.2.1.2 could have been used, but entails writing regular expressions.2 _{As dened in 2.4.3.2, any customizing that goes}

beyond activating/deactivating functions is not considered.

2_{Auto-translatables rules specify how a portion of the source text is converted to its equivalent in}

MultiTrans: as long as dates are numeric only, the same remarks as in 3.2.1.2 apply. MultiTrans does not support number recognition and a test by means of the QA Add-on was not possible due to license restrictions, see 3.2.2. No specic options are available for other types of dates.

SDL Trados: under File > Setup > Substitutions > Dates and Times, it is possible to dene [...] dates [...] as variables rather than normal words in Translator's Workbench. [...] Translator's Workbench recognises designated variables and treats them as non-translatable or placeable elements during translation, SDL (2007). This option is activated by default. However, it is not possible to see or modify the patterns used.

In addition, under Options > Translation Memory Options > Substitution Localisation > Dates and Times, it is possible to dene if, and how, Translator's Workbench should automatically adapt the format of variable elements to suit the target language [...], SDL (2007).

4.2.2 General remarks on TM system behavior

In terms of display and recognition assessment, dates are handled as numbers, see 3.2.2. A scoring system was used for dates too, and is described in the following section.

Scoring system

A score was calculated for each instance of recognition according to the following rules: 3 points were assigned when the date as a whole (including separators) was recog-

nized. Mere digit recognition was not given any point if it excluded separators (e.g. the full stops in 21.1.2007) or months written out in full (e.g. Juli 2008).

Bonus of 0.5 points was assigned when time intervals consisting of two dates (e.g. 1.4.07 - 31.3.08) were recognized.

A baseline assuming complete recognition of all dates (excluding the bonuses for interval recognition) was calculated. The scores achieved by the TM systems and the baseline are compared in 4.3.

In order to make the score more transparent, each table indicates how the score was calculated and which elements were considered. The elements are referred to with the following abbreviations:

D: digit(s) I: interval

Recognition assessment

4.2.3 Test suite

The examples contain minor formal errors, e.g. missing spaces in 3 and 17, taken literally from the source corpus. They have been retained in order to ascertain whether recognition incorporates fault tolerance. The sequences that should be recognized are printed in gray. The following examples were used:

Text Baseline Calculation 1 Juni undJuli 2008 3 D

2 vom 1. bis30. Juni 2007 3 D

3 1.April 2007 3 D 4 ab1. Mai 2008 3 D 5 ab06. Mai 2007 3 D 6 ab13. Mai 07 3 D 7 ab1. Dez. 2006 3 D 8 Freitag,21.09.2007 3 D 9 ab06.03.2008 3 D 10 Dauer 20.-22.9.2007 3 D 11 Dauer29.6-1.7.07 6 (3+3) D+D 12 26.05bis30.06.07 6 (3+3) D+D 13 Zeit 9.-11. Juni 2006 3 D 14 vom 11.-14. Juli 3 D 15 bis 20./21. September 2007 3 D 16 1 März-31. Juli 2007 6 (3+3) D+D 17 Von Juli bisSept.05 3 D 18 am1.5.2008 3 D 19 am1.7.08 3 D 20 am21.1.2007 3 D 21 Stand12.2004 3 D 22 Daten1.4.07-31.3.08 6 (3+3) D+D 23 am1. Januar 3 D 24 für den30. April, 2. und3. Mai 6 (3+3) D+D 25 vom21.4. bis 25.4. 6 (3+3) D+D 93 Total

Table 4.2: Dates: test examples Results

Only complete date recognition is highlighted: simple digits are not highlighted because this recognition has already been dealt with in chapter 3.

Across: see table 4.3. Only examples 8 and 9 as well as 30.06.07 in example 12 are recognized as a single sequence.

Across can recognize numeric dates only if their format conforms to those formats dened under Tools > System settings > General > Language settings >Lo- cale settings > Language sets > Standard language set > Languages > [Deutsch] > Format > Date long or Date short. The default formats for Ger- man are DD.MM.YY and DD.MM.YYYY. If dates have another format, they are not

recognized, unless these formats are added manually, see 4.2.1.2. Alphanumeric dates and intervals are not recognized.

Déjà Vu: see table 4.4. The standard segmentation is problematic because many dates are segmented if the full stop is followed by a space and an uppercase letter or a digit. Adding two exceptions (.^w^# and .^w^A) under Tools > Options > Delimiters >

Exceptions prevents this error from occurring and was used only for this test set. Déjà Vu does not support the recognition of intervals and alphanumeric dates. On the other hand, numeric dates are recognized, e.g. 21.09.2007 in example 20. This recognition can be checked as follows: if one digit is changed (e.g. to 22.09.2007) and Check Numerals is activated, the whole date is marked in red.

memoQ: see table 4.5. Numeric dates are recognized only if they have the pattern

[0−9]+\.[0−9]+, which is not date-specic. Dates such as 21.09.2007 are recognized only

in part. This recognition can be checked as follows: if the rst point (between 22 and 09) is changed and the QA check is run, an error message is displayed. If the second point (between 09 and 2007) and the QA check is run, no error message is displayed. Alphanumeric dates and intervals are not recognized.

SDL Trados: see table 4.6. Alphanumeric dates are generally recognized. However, if the month is abbreviated, there is no recognition.

Patterns of numeric dates are recognized: DD.MM.YYYY, DD.MM.YY as well as DD.MM, which is, however, not specic for dates. Intervals with a start and an end date are not recognized.

Transit: see table 4.7. The recognition of intervals and alphanumeric dates is not supported. On the other hand, numeric dates are recognized. This recognition can be checked as follows: if one of its digits is modied (e.g. 22.09.2007 instead of 21.09.2007) and a number check is carried out, the error message Number "[...]" not found contains the whole date.

Wordfast: see table 4.8. The recognition of alphanumeric dates is not supported. On the other hand, numeric dates are recognized.

If intervals of time are dened by two dates separated by a hyphen-minus sign (e.g. examples 10 and 11), they are recognized as a single unit, but only if there is no space. Otherwise, two separated dates are recognized (e.g. example 22).