AI you can trust

Bias creeps into training data two ways: lack of diversity and representation in the training examples; and bias of human labelers.

It’s not unusual for a dataset to be, for example, heavily based on males in their 30s. That’s bias in the representation of the dataset. We mitigate this by balancing our datasets across demographics (gender, race, etc.).

Then there’s the risk of bias in human labelers. Before data can be used for AI training purposes, it must be “labeled” or annotated to show the model how it should be interpreting such data.

The problem is that people labeling data can have their own biases. We assess this by having multiple people annotate the same text and using a diverse group of experts to label our data. We measure how often the annotators agree with each other using Cohen’s Kappa statistic. High agreement (in the 75-80% range) indicates better quality data and assures individual bias is not introduced.

In cases where annotators disagree, we have a separate annotator tie-break before it’s used for our model training. If the agreement is not at least 75%, we retrain the annotators and re-annotate the data.

We use classification—a machine learning method—to determine the context of a phrase and whether it should be “flagged” according to what you’re writing (employee feedback, a performance review, a job post, etc.).

For example, is “bubbly” talking about a person or a soda in the sentence “She brings a bubbly presence to work”?

We test bias in our classification models on a common ML metric called an F-score. This is a statistic that measures how well the model finds the correct answers and how many correct answers it misses. If we test our model over the same data with different names (“Sally is a bubbly person” and “Bob is a bubbly person”), we should see consistent F-scores. If the F-scores are not consistent, there is bias.

When we find bias, we identify the source of bias, mediate appropriately, and retest. Some things we consider:

Is the data balanced? Do we need more representative data?
Do we need to choose a different model? Or do we need to re-train the model with different parameters?
Is there bias in the data we use to measure the performance of the model?
Consider incorporating “in-processing fairness techniques” that influence how the model learns

Textio’s generative AI features take input text or answers to prompts and can write, rewrite, or expand the message. We test to make sure that the output doesn’t contain demographic biases based on gender, race, or age.

One way we measure bias in these features is to vary names in the input text and test if the generated content is different. For example, we’d test “Rewrite: Sally is a bubbly person” and “Rewrite: Bob is a bubbly person” and compare the results.

To determine whether the differences are meaningful across demographic groups, we collect different generative AI outputs for each variation (for example, male vs. female names) at a large scale. We then run a paired t-test to compare the distribution of words across each of these groups. If there is a significant difference in the language used in one group over another (defined by the p-value, where p < .05), we can confidently say the output of the generative AI model is biased. If so, we would then:

Do a qualitative analysis of the bias to identify the themes and characteristics of the differences
Iterate on the prompt strategy and add hard-coded rules (if necessary) to correct the behaviors of the AI
Remeasure

For our generative AI models, we also mitigate bias by masking proper names from the input. This neutralizes any potential gender- or race-based biases the model might produce because of someone’s name.

The AI platform purpose-built to prevent bias

Seriously secure

Your data is private

Results are bias-free

Security is best-in-class

How we build mindful AI at Textio

A diverse team

Designing for a specific purpose

Learn more:

Building with bias in mind

Research

Give superhero managers superhuman support