The “Translation Automation User Society” (TAUS) is a think tank which
promotes the use of machine translation and technology within the translation
industry. It organises events and offers services such as data sharing and
language technology training. A recent article on the TAUS blog focused
on the problem of quality evaluation in automated translation. It proposes a
model called “dynamic quality evaluation”. This model has also been discussed onthe LinkedIn group “Translation Automation”, and Rahzeb Choudhury of Leeds
University kindly sent me a link to a longer report in PDF format, the DynamicQuality Framework Report.
Looking at these materials, the underlying logic looks to me rather
suspect, like a circular argument. It is worth considering the reasons for
this.
The TAUS
demographics
The Dynamic
Quality Evaluation Framework report is based on a study conducted with a number
of major multinational organisations (“reviewers”) which have a high volume of
text which needs translation. Most of these organisations are large businesses
with high volume technical products such as Dell, Google, Microsoft, Phillips
and Siemens. The organisations also include the EU, which has a high volume of
translations between the national languages in the European Community.
In other
words, the work of TAUS, at least in this particular instance, is based on a
very limited sample, i.e. major international organisations with an extremely
high volume of multilingual text requirements, most of which service a limited
range of subject areas. There is no consideration given to highly complex and
confidential legal texts which will be read in different jurisdictions, no
mention of complicated architectural texts, of urban planning, high-powered
business management documents and much more. Given this highly selective
demographic situation, it is not surprising that TAUS claims broad agreement on
certain priorities in its reports and other documents. I would suggest,
however, that the translation industry is much broader than the demographic
group represented by TAUS.
The part and the whole
This limited demographic
sample would not in itself be a problem if TAUS freely admitted that the study
deliberately focuses on a certain scenario and certain types of translation
work. But the actual usage in the report exacerbates the problem and is often
misleading. For example, there are frequent references to “the translation
industry”, although the actual descriptions and conclusions actually apply to clients (and perhaps selected suppliers) in the translation technology industry working
on high volume automated translation in specified subject domains.
If the work
of TAUS claimed to be impartial academic research, it would take a far more self-critical
approach to its own sampling procedures and would openly point out the
limitations of its material. Instead, it acts like a political pressure group,
presenting its results in the way that most suits its own agenda. In some of
the TAUS material that I have read, I have wondered whether this confusion is deliberate,
or whether it reflects a genuine inability to perceive that there are different
perspectives on the issues.
Dynamic quality evaluation – a definition of
convenience?
The report on
“dynamic quality evaluation” uses this very problem as its starting point. It
states, for example, “Quality evaluation (QE) in the translation industry is
problematic”. The blog post claims “The industry needs common measurable
definitions”. Both of these statements pose more questions than they answer.
Which sector(s) of the translation industry is TAUS referring to? What quality
is referred to, who wants to evaluate this quality, for what purpose and in
what kinds of text? What measurements could be used to define something as
flowing and variable as language? To what extent would industrial-scale
evaluation and defined measurements miss the essential characteristics of the
material they are used on?
Instead of
dealing with these fundamental issues, TAUS posits a quality evaluation system
with three main elements, which it calls utility, time and sentiment. We are
told that utility refers to the functionality of the content, speed refers to
how quickly the translation is needed and sentiment denotes the effect of the
resulting text on the brand image. You may notice that the actual quality of a
text is not one of the three elements. So where does it come in? As far as I
can gather, it seems to be relegated to a sub-category of “Utility” and to be
marginally touched on in the category “Sentiment”. At the stroke of the categoriser's computer keyboard, the quality of the text itself is relegated to a mere sub-category.
The pinnacle
of the “dynamic quality” logic is reached in the blog post. At the conference
which is reported on the blog, there were apparently some participants who did
not agree with the majority opinion – they advocated absolute rather than relative
quality, and they felt that universal measurable standards did not do justice
to the phenomenon of translation. Then comes the classic conclusion: most
participants at the conference felt that “unless we maintain the simplicity of
the model we get lost in endless details and personal requirements, and we end
up … having no generalizable reference …”
Get yourself
a cup of coffee and sit down and consider this sentence for a few moments. I
would paraphrase it like this: some people argue that the world of language and
translation is complicated, but we can’t handle a complex world because we
could then not create the simple and measurable system that we want.
We must have simplicity, so let there be simplicity. Simplicity rules,
simply because we want it to rule.
This is rather
like the semantic principles expressed by Humpty Dumpty in Lewis Carroll's novel “Alice in Wonderland”: “When I use a word, it
means just what I choose it to mean – neither more nor less.” It would be a
wonderfully simple way to use language: I say what I want, and it means what I
want. The only problem is the puzzled expression on the faces of my listeners.
The toxic disclaimer
The final
section of the blog is where TAUS dances on the borderline of Imperialism. In
the title of this section, and three times in the paragraphs, it mentions the
possibility of applying for the “dynamic quality” system to be certified as a
standard. Each time, the possibility is retracted, at least partially, rather like the song of the Mock Turtle in Carroll's novel: “Will you, won't you, will you, won't you, will you join the dance?” In a TAUS context, this translates as “we
would not be so sure that we would want to apply for official standardisation”
and “Whether we go for standard certification is a decision we can take
together when we get to this crossroads”.
Together?
Dear TAUS, does this mean that you will gather all of the translators in the
world and involve us in deciding whether to apply for certification of a
standard? I think not. Your agenda seems to be domination of the translation
industry rather than cooperation with real life translators. You do not look kindly on people like me who have differing opinions,
far less do you take us seriously. For you, we are unwelcome “quality gatekeepers” who
are “blinkered by prior assumptions”. Ho hum, I suppose Humpty would be proud
of these sweeping allegations.
Unintended consequences
The
occupation of Gaul by the Roman Empire gave rise to the insurrection by Asterix
and Obelix in the wonderful French comics and films. Many other literary
parallels come to mind, such as Luke Skywalker and the Empire, Thursday Next and Goliath Corporation, etc. If you continue to play Humpty with the values which
translators hold dear, please do not be surprised when you meet opposition.
Every group which aspires to global domination must expect resistance. The rhetoric
adopted by TAUS and others will bring forth a myriad Luke Skywalkers,
and your glorious automated future will be lit up by the flash of lightsabres all over the globe.
Previous related posts on this blog
