Wednesday 8 May 2013

Humpty Dumpty and the TAUS quality concept



The “Translation Automation User Society” (TAUS) is a think tank which promotes the use of machine translation and technology within the translation industry. It organises events and offers services such as data sharing and language technology training. A recent article on the TAUS blog focused on the problem of quality evaluation in automated translation. It proposes a model called “dynamic quality evaluation”. This model has also been discussed onthe LinkedIn group “Translation Automation”, and Rahzeb Choudhury of Leeds University kindly sent me a link to a longer report in PDF format, the DynamicQuality Framework Report.
Looking at these materials, the underlying logic looks to me rather suspect, like a circular argument. It is worth considering the reasons for this.
The TAUS demographics
The Dynamic Quality Evaluation Framework report is based on a study conducted with a number of major multinational organisations (“reviewers”) which have a high volume of text which needs translation. Most of these organisations are large businesses with high volume technical products such as Dell, Google, Microsoft, Phillips and Siemens. The organisations also include the EU, which has a high volume of translations between the national languages in the European Community.
In other words, the work of TAUS, at least in this particular instance, is based on a very limited sample, i.e. major international organisations with an extremely high volume of multilingual text requirements, most of which service a limited range of subject areas. There is no consideration given to highly complex and confidential legal texts which will be read in different jurisdictions, no mention of complicated architectural texts, of urban planning, high-powered business management documents and much more. Given this highly selective demographic situation, it is not surprising that TAUS claims broad agreement on certain priorities in its reports and other documents. I would suggest, however, that the translation industry is much broader than the demographic group represented by TAUS.
The part and the whole
This limited demographic sample would not in itself be a problem if TAUS freely admitted that the study deliberately focuses on a certain scenario and certain types of translation work. But the actual usage in the report exacerbates the problem and is often misleading. For example, there are frequent references to “the translation industry”, although the actual descriptions and conclusions actually apply to clients (and perhaps selected suppliers) in the translation technology industry working on high volume automated translation in specified subject domains.
If the work of TAUS claimed to be impartial academic research, it would take a far more self-critical approach to its own sampling procedures and would openly point out the limitations of its material. Instead, it acts like a political pressure group, presenting its results in the way that most suits its own agenda. In some of the TAUS material that I have read, I have wondered whether this confusion is deliberate, or whether it reflects a genuine inability to perceive that there are different perspectives on the issues.
Dynamic quality evaluation – a definition of convenience?
The report on “dynamic quality evaluation” uses this very problem as its starting point. It states, for example, “Quality evaluation (QE) in the translation industry is problematic”. The blog post claims “The industry needs common measurable definitions”. Both of these statements pose more questions than they answer. Which sector(s) of the translation industry is TAUS referring to? What quality is referred to, who wants to evaluate this quality, for what purpose and in what kinds of text? What measurements could be used to define something as flowing and variable as language? To what extent would industrial-scale evaluation and defined measurements miss the essential characteristics of the material they are used on?
Instead of dealing with these fundamental issues, TAUS posits a quality evaluation system with three main elements, which it calls utility, time and sentiment. We are told that utility refers to the functionality of the content, speed refers to how quickly the translation is needed and sentiment denotes the effect of the resulting text on the brand image. You may notice that the actual quality of a text is not one of the three elements. So where does it come in? As far as I can gather, it seems to be relegated to a sub-category of “Utility” and to be marginally touched on in the category “Sentiment”. At the stroke of the categoriser's computer keyboard, the quality of the text itself is relegated to a mere sub-category.
The pinnacle of the “dynamic quality” logic is reached in the blog post. At the conference which is reported on the blog, there were apparently some participants who did not agree with the majority opinion – they advocated absolute rather than relative quality, and they felt that universal measurable standards did not do justice to the phenomenon of translation. Then comes the classic conclusion: most participants at the conference felt that “unless we maintain the simplicity of the model we get lost in endless details and personal requirements, and we end up … having no generalizable reference …”
Get yourself a cup of coffee and sit down and consider this sentence for a few moments. I would paraphrase it like this: some people argue that the world of language and translation is complicated, but we can’t handle a complex world because we could then not create the simple and measurable system that we want. We must have simplicity, so let there be simplicity. Simplicity rules, simply because we want it to rule.
This is rather like the semantic principles expressed by Humpty Dumpty in Lewis Carroll's novel “Alice in Wonderland”: “When I use a word, it means just what I choose it to mean – neither more nor less.” It would be a wonderfully simple way to use language: I say what I want, and it means what I want. The only problem is the puzzled expression on the faces of my listeners.
The toxic disclaimer
The final section of the blog is where TAUS dances on the borderline of Imperialism. In the title of this section, and three times in the paragraphs, it mentions the possibility of applying for the “dynamic quality” system to be certified as a standard. Each time, the possibility is retracted, at least partially, rather like the song of the Mock Turtle in Carroll's novel: “Will you, won't you, will you, won't you, will you join the dance?” In a TAUS context, this translates as “we would not be so sure that we would want to apply for official standardisation” and “Whether we go for standard certification is a decision we can take together when we get to this crossroads”.
Together? Dear TAUS, does this mean that you will gather all of the translators in the world and involve us in deciding whether to apply for certification of a standard? I think not. Your agenda seems to be domination of the translation industry rather than cooperation with real life translators. You do not look kindly on people like me who have differing opinions, far less do you take us seriously. For you, we are unwelcome “quality gatekeepers” who are “blinkered by prior assumptions”. Ho hum, I suppose Humpty would be proud of these sweeping allegations.
Unintended consequences
The occupation of Gaul by the Roman Empire gave rise to the insurrection by Asterix and Obelix in the wonderful French comics and films. Many other literary parallels come to mind, such as Luke Skywalker and the Empire, Thursday Next and Goliath Corporation, etc. If you continue to play Humpty with the values which translators hold dear, please do not be surprised when you meet opposition. Every group which aspires to global domination must expect resistance. The rhetoric adopted by TAUS and others will bring forth a myriad Luke Skywalkers, and your glorious automated future will be lit up by the flash of lightsabres all over the globe.
Previous related posts on this blog

9 comments:

  1. Good post! When TAUS first started on translation quality, I was wondering why, as their interest was primarily in promoting MT. Now their agenda is pretty clear. MT faces a threshold too high for it, which cannot be crossed in the short run. That threshold is absolute quality. To push for MT, the effort now is focused on lowering the threshold, and making quality relative. ("Dynamic" is a fancy word only.) Now that massaging quality is justified, MT can proceed without feeling bad about itself. As you said, it is OK with some clients in some subject areas. But to make thsi relative quality model into industry standards is purely a joke.

    ReplyDelete
  2. Hi Victor,

    Here are two of the results of the DQF project, which were published following public consultation -

    http://www.translationautomation.com/adequacy-fluency-guidelines/adequacy-fluency-guidelines

    http://www.translationautomation.com/error-typology/error-typology-guidelines

    The report you mention was published in November 2011, as a TAUS Labs report, documenting progress at that time.

    Here is a playlist of various people speaking about the DQF, warts and all -

    http://www.youtube.com/playlist?feature=edit_ok&list=PLVjXYOjST-AoQ9wCR2Fi17PiHKi9TyZYj

    It is an ongoing project. We welcome your scrutiny.

    You are right that most people regularly involved in the project are from companies. There are small companies involved also, it's not just the behemoths. People involved in the project include ex translators and translator trainers.

    Here is a link to upcoming consultations where everyone is welcome to respond -

    http://www.translationautomation.com/consultations/consultations

    If your schedule allows, I am happy to give you an overview of all DQF resources.

    Rahzeb

    ReplyDelete
  3. Hi Razheb,
    Thanks for the extra links, and especially for your admission of the demographic limitations of the TAUS stakeholders.
    I am certainly always interested in more information to keep myself fully informed, but I do not have the resources to respond on every topic or to contribute to your consultations.
    I am sure that we will continue to differ in our perspective - partly because we focus on different segments of the translation industry and partly because I still find it difficult to accept that a group which is basically made up of non-translators should endeavour to tell translators what to do and what not to do, and even dream of standardising its findings.
    So I will continue to monitor your work and material, although it is not really relevant to the type of expert translation work I do. However, ifI see any signs of distorted definitions or hostile takeovers, I may be tempted to get out my lightsabre again.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Hi Victor,

    There is a consultation open now on translators and quality:

    http://translationautomation.com/consultations/consultations#quality-and-the-translator

    Perhaps you could throw a few stones at Humpty to see if he breaks?

    Best,

    Rahzeb

    ReplyDelete
  6. Hi Rahzeb,
    Thanks for the link and the invitation. I'll try to put a few thoughts together when I have a minute.
    I note that Humpty still seems to feel he's talking about all translations everywhere rather than just a certain type of job scenario in the mass translation and post-editing market. I would struggle to fit my usual translation jobs (whether for direct clients or for agencies) into Humpty's categories.

    ReplyDelete
  7. I submitted a comment to the consultation, as requested by Rahzeb, and it has had at least some effect (probably in combination with comments by other colleagues) in the new version of the document which is now available at http://translationautomation.com/quality-and-the-translator/quality-and-the-translator-guidelines:
    - The first heading in the document is now less confrontative ("Helping Translators to Deliver the Expected Quality" instead of "Ensuring Translators Deliver the Expected Quality").
    - There is now a domain limitation of sorts ("the localization segment"). Not as explicit as I would have liked, but it is at least a first step towards modifying the scope covered by TAUS.
    I also made two other comments, which were not taken into account in the revision:
    - I suggested that there are already translation buyer guidelines issued by a number of translation associations and similar bodies, and that it would help for TAUS to explain what contribution another set of guidelines can make. Rahzeb replied that the TAUS members (i.e. large translation buying companies) selected the topic and scope of the document, not TAUS. I find it disappointing (but not surprising) that nobody within TAUS is able to open the eyes of the corporate TAUS members to what is actually going on in the translator world.
    - I suggested that the document confused translation and post-editing and did not make a sufficient distinction between them. Rahzeb replied that the distinction is blurred anyway because of the way TM and MT are often combined in translation work. I would have thought that this would be an extra reason to present careful definitions and distinctions, but it seems that my idea of clarity is not shared by TAUS.

    ReplyDelete
  8. Great post. There is a lot to be said for group of people translations, especially in areas where companies do not deem it necessary to localize for other markets (such as fan subs of American anime etc.) or there simply isn't a company behind a product (open source software anyone?). But if you are running a business and trying to make the most out of the opportunities presented by globalization, using professional spanish to english translation services.is the only way to go.
    Luckily, I can't see machines taking over the jobs of human translators in the near future, as they have done with so many other professions.

    ReplyDelete
  9. You're right to conserve your energies with regard to TAUS. They inhabit a certain lower layer of muck in the bulk market bog, and I've never seen an action or presentation from a member of that organization that has any real relevance to the sort of work we do. But like so many who huckster on behalf of the myth of machine superiority, they are, time and again, far too broad in their claims, which reveals a certain dishonesty in their entire agenda.

    ReplyDelete