Quality

Translation quality is a difficult and often subjective topic. Traditionally, agencies have often aspired to provide "perfect" quality, and you'll often hear translation services claim they have the best, or unrivaled quality. But ultimately, quality should be determined by what customers need. At Gengo, we've been working for several years on turning this often subjective topic into an objective, rational discussion. We do this by having a clearly defined set of quality standards and by using numeric scoring

and a variety of metrics for quality, rather than relying on personal preference. It's impossible to be "perfect," so instead we aspire to be reliable, consistent and upfront with customers about what they can expect.

The first data set for quality that we’re releasing is our customer feedback—the ratings and comments that customers leave after they receive a translation. After receiving their translation, we ask customers to rate our service in three areas:

Translation Quality, Translation Speed and Comments (how well the translator responded to any communication). Customers can also leave a comment for us and a comment for the translator. We find that around 40% of our Retail customers leave feedback for us. Feedback can also be given through our API, although this is less frequent.

Translation quality is a difficult and often subjective topic. Traditionally, agencies have often aspired to provide "perfect" quality, and you'll often hear translation services claim they have the best, or unrivaled quality. But ultimately, quality should be determined by what customers need. At Gengo, we've been working for several years on turning this often subjective topic into an objective, rational discussion. We do this by having a clearly defined set of quality standards and by using numeric scoring and a variety of metrics for quality, rather than relying on personal preference. It's impossible to be "perfect," so instead we aspire to be reliable, consistent and upfront with customers about what they can expect.

The first data set for quality that we’re releasing is our customer feedback—the ratings and comments that customers leave after they receive a translation. After receiving their translation, we ask customers to rate our service in three areas: Translation Quality, Translation Speed and Comments (how well the translator responded to any communication). Customers can also leave a comment for us and a comment for the translator. We find that around 40% of our Retail customers leave feedback for us. Feedback can also be given through our API, although this is less frequent.

Weeks Months

Feedback Ratings

This chart shows the average scores received across all language pairs over time. You'll see that feedback for Speed and Quality cluster pretty closely together. This happens for two reasons: we work to improve our service across all fronts, and customers who are not able to evaluate quality will often leave a high

Quality rating when they have a good overall experience.

Because customers are often overly generous with their ratings, we generally use Quality feedback as a negative indicator rather than a positive one. This means that we don't take action

when a customer leaves a 5/5 rating, but we'll ask a Language Specialist to look more closely at a translation if it scores 2/5 or 1/5, because this is more likely to be significant. Over time, we aim to reduce the number of translations that have low Feedback ratings, especially if a specific language pair receives a large number of low ratings.

This chart shows the average scores received across all language pairs over time. You'll see that feedback for Speed, Quality and Comments cluster pretty closely together. This happens for two reasons: we work to improve our service across all fronts, and customers who are not able to evaluate quality will often leave a high Quality and Comments rating when they have a good overall experience.

Because customers are often overly generous with their ratings, we generally use Quality feedback as a negative indicator rather than a positive one. This means that we don't take action when a customer leaves a 5/5 rating, but we'll ask a Senior Translator to look more closely at a translation if it scores 2/5 or 1/5, because this is more likely to be significant. Over time, we aim to reduce the number of translations that have low Feedback ratings, especially if a specific language pair receives a large number of low ratings.

Weeks Months

Rejection & Revision Rate

This chart shows our Rejection and Revision Rate as a percentage of overall units translated. Lower numbers are better.

Customers have five days to review their translation after they receive it. This allows time to share the content and get feedback from peers

or users. The vast majority of customers approve their translation within this time. However, customers who are unhappy have two options. They can request revisions, allowing the original translator 24 hours to make corrections to the translation based on the customer's feedback, or they can reject the translation, choosing either to

receive a full refund or have a different translator complete the job. We track the incidence of both actions, as a high Revision or Rejection Rate is an indicator of low perceived quality.

This chart shows our Rejection and Revision Rate as a percentage of overall units translated. Lower numbers are better.

Customers have five days to review their translation after they receive it. This allows time to share the content and get feedback from peers or users. The vast majority of customers approve their translation within this time. However, customers who are unhappy have two options. They can request revisions, allowing the original translator 24 hours to make corrections to the translation based on the customer's feedback, or they can reject the translation, choosing either to receive a full refund or have a different translator complete the job. We track the incidence of both actions, as a high Revision or Rejection Rate is an indicator of low perceived quality.

Feedback Comments

Feedback Comments contain a variety of information, often not related to quality. However, in accordance with our Open Data promise, we include all Feedback Comments below, removing only those that contain sensitive or personally-identifiable information. Where a Quality Score of 3 or greater was received, the comment is marked in green, otherwise it is marked in red.

Feedback Comments contain a variety of information, often not related to quality. However, in accordance with our Open Data promise, we include all Feedback Comments below, removing only those that contain sensitive or personally-identifiable information. Where a Quality Score of 3 or greater was received, the comment is marked in green, otherwise it is marked in red.

Weeks Months

Translation Quality Score

While customer feedback is important, it’s not enough. And the subjective, un-scalable methods that traditional agencies use for checking quality do not work on the scale of the crowd. So in 2012 we developed an internal tool called GoCheck that allows our Senior Translators to objectively rate our translations.

Gengo is used primarily for “everyday content” translations: Emails, blog posts, social media, product descriptions and user reviews. These are items that traditional agencies can’t handle, because they are set up for expensive and

complex work like legal documents, patents and high end marketing materials. GoCheck is set up primarily to work with everyday content translations.

In the GoCheck UI, reviewers indicate the type, severity and frequency of errors, and our algorithm calculates a score out of 10. For example, an omission of a key word in a translation will significantly affect a score, but a minor punctuation error may not. Hundreds of checks happen each week either as random samples or prompted by low reviews, rejections

or other actions, giving us a good idea of quality for each translator and each language pair. Our target is for 95% of Standard Level translations at Gengo to achieve a 7 or higher.

Scores allow us to see over time if we are improving quality, through better tools, better testing, better training or other means. Exposing the scores gives you an idea of how we are doing each month. You will not find this transparency anywhere else.

While customer feedback is important, it’s not enough. And the subjective, un-scalable methods that traditional agencies use for checking quality do not work on the scale of the crowd. So in 2012 we developed an internal tool called GoCheck that allows our Senior Translators to objectively rate our translations.

Gengo is used primarily for “everyday content” translations: Emails, blog posts, social media, product descriptions and user reviews. These are items that traditional agencies can’t handle, because they are set up for expensive and complex work like legal documents, patents and high end marketing materials. GoCheck is set up primarily to work with everyday content translations.

In the GoCheck UI, reviewers indicate the type, severity and frequency of errors, and our algorithm calculates a score out of 10. For example, an omission of a key word in a translation will significantly affect a score, but a minor punctuation error may not. Hundreds of checks happen each week either as random samples or prompted by low reviews, rejections or other actions, giving us a good idea of quality for each translator and each language pair. Our target is for 95% of Standard Level translations at Gengo to achieve a 7 or higher.

Scores allow us to see over time if we are improving quality, through better tools, better testing, better training or other means. Exposing the scores gives you an idea of how we are doing each month. You will not find this transparency anywhere else.

Real Translations

This is the most daring and progressive section of Open Data: You can now see Gengo’s translations publicly, for every job where we have permission to publish. You can see exactly what we translated, unfiltered, unedited and complete. Any variations in quality are visible and public.

No one else in our industry has ever published

this kind of information. You can imagine why. It’s scary. It completely changes the conversation. It puts the power in the hands of you, the customer. You might find that you are paying 4x the price for inferior translations.
So dive in and browse. Next to each translation you can see the customer rating and our internal GoCheck rating if present, so you can see how

others have judged the quality. You can also see the turnaround time, to get an idea of how long something like this might take. In most language pairs, there is enough volume to give you a good sense of Gengo’s quality. We think you’ll like it.

This is the most daring and progressive section of Open Data: You can now see Gengo’s translations publicly, for every job where we have permission to publish. You can see exactly what we translated, unfiltered, unedited and complete. Any variations in quality are visible and public.

No one else in our industry has ever published this kind of information. You can imagine why. It’s scary. It completely changes the conversation. It puts the power in the hands of you, the customer. You might find that you are paying 4x the price for inferior translations.

So dive in and browse. Next to each translation you can see the customer rating and our internal GoCheck rating if present, so you can see how others have judged the quality. You can also see the turnaround time, to get an idea of how long something like this might take. In most language pairs, there is enough volume to give you a good sense of Gengo’s quality. We think you’ll like it.

Add Filter

Add Filter by ...

  • by Language Pair
  • by Content Type
  • by Level
  • by Date
          Level
          Type
          Date