AI Bias in India: What SMEs Must Fix Before Scaling

A single “helpful” edit can expose a much bigger risk.

In 2025, a sociologist in India used ChatGPT to polish a postdoctoral application. The tool improved the English—then quietly changed his surname to one associated with a privileged caste. No one asked it to do that. The model guessed what was “most likely” in academic spaces and rewrote a person’s identity accordingly.

For small and medium businesses (SMEs), that story isn’t just troubling—it’s operationally relevant. The same kinds of hidden assumptions show up when AI writes job ads, screens candidates, answers customers in local languages, or generates images for marketing. And once AI is embedded into workflows, small biases don’t stay small. They become policy-by-software.

This post is part of our “አርቲፊሻል ኢንተሊጀንስ በእርሻና ግብርና ዘርፍ ውስጥ ያለው ሚና” series. We usually talk about yield prediction, farmer advisory, and supply-chain efficiency. Here’s the uncomfortable truth: AI in agriculture and agribusiness only helps if it treats people fairly—farm workers, outgrowers, co-ops, and candidates included.

Caste bias in AI isn’t theoretical—it shows up in outputs

Answer first: The investigation found measurable caste bias in popular AI tools, meaning SMEs can accidentally deploy discriminatory behavior through everyday automations.

Researchers tested OpenAI’s latest chat model using a dataset designed to surface India-specific sociocultural bias. The test format was simple: fill-in-the-blank prompts forcing a choice between “Dalit” and “Brahmin.” Results were blunt: the model picked stereotypical completions in 80 of 105 sentences tested—about 76%.

A few patterns matter for business contexts:

• Positive traits (learned, spiritual, knowledgeable) skewed toward “Brahmin.”
• Negative traits (impure, criminal, uneducated) skewed toward “Dalit.”
• The model rarely refused to answer, meaning it was willing to complete harmful stereotypes rather than decline.

Why SMEs should care: these stereotypes don’t only appear in explicit caste prompts. They can leak into summaries, candidate write-ups, translations, name normalization, persona generation, and even “tone improvements.” If your company uses AI to “clean up” text or standardize customer records, you’re already in the risk zone.

The scary part: “most likely” becomes “most acceptable”

Language models optimize for probability. In practice, that means they often treat dominant social patterns as the “default.”

If a model learns that certain surnames appear more often in elite contexts, it may rewrite reality to match that pattern.

For an SME, the damage isn’t only moral. It’s reputational, legal, and financial:

• A hiring assistant that subtly rewrites candidate bios can change who gets shortlisted.
• A customer support bot that mirrors social bias can alienate whole customer segments.
• A credit or eligibility workflow that uses AI-generated “risk notes” can institutionalize discrimination.

Generative images can encode discrimination just as strongly

Answer first: Visual AI can reproduce social hierarchy through skin tone, jobs, housing, and behavior—making discriminatory marketing assets easy to generate at scale.

Testing of OpenAI’s text-to-video system found stereotyped outputs across prompts like “a Dalit job,” “a Dalit house,” and “a Dalit behavior.” Some results portrayed oppressed communities only in degrading, menial labor contexts. In a particularly disturbing pattern, prompts about Dalit behavior produced animals in a significant share of samples.

If you’re an agribusiness SME using AI for:

• posters for input shops,
• brochures for outgrower schemes,
• recruitment ads for field agents,
• product explainers for rural markets,

…then visual bias becomes a brand risk. One careless asset can look like your company endorses a social hierarchy.

Why this hits agriculture and rural markets harder

Agriculture is human networks: cooperatives, traders, extension agents, seasonal labor, transporters, and community trust. Many agritech SMEs depend on local adoption more than national brand spend.

A biased chatbot or a biased poster doesn’t fail quietly. It spreads via WhatsApp groups, community meetings, and word-of-mouth.

“Closed vs open source” isn’t the real decision—testing is

Answer first: Both closed and open-source models can show severe caste bias; the real difference is whether you measure, monitor, and constrain behavior in your product.

Some SMEs choose open-source models because they’re cheaper and can be tuned for local languages. That can work—but it also shifts responsibility onto you. Early research suggests caste harms can be worse in some open-source models commonly used by startups.

Here’s the stance I take: if your SME can’t commit to bias testing and ongoing monitoring, you shouldn’t deploy AI in high-stakes workflows. Use it for low-risk drafting and internal productivity first.

High-stakes SME workflows that deserve extra caution

These are the “don’t be casual about it” use cases:

1. Hiring and HR: CV screening, interview question generation, candidate ranking, reference summaries.
2. Admissions/selection programs: selecting farmers for credit, inputs, training, pilots, or subsidies.
3. Customer support and dispute handling: complaint triage, fraud labeling, “risk” language.
4. Marketing and creative: image generation of “typical farmer,” “laborer,” “village household.”

If your agritech product touches farmer onboarding or eligibility, you’re effectively doing social sorting. AI bias here becomes structural, fast.

A practical ethical AI checklist for SMEs (built for India)

Answer first: SMEs can reduce bias risk quickly by adding guardrails: policy, prompts, tests, human review, and logging—before scaling AI features.

Most SME teams don’t need a fairness lab to do better. You need a disciplined process.

1) Write a simple “no identity rewriting” rule

Your AI should not infer or alter sensitive identity attributes (caste, religion, ethnicity) or identity signals (names, surnames, community markers) unless the user explicitly asks.

Operationalize it:

• Add a pre-processing check: lock names and identifiers (don’t let the model rewrite them).
• Add a post-processing check: flag outputs that modify names, surnames, or demographic descriptors.

2) Create a mini bias test set from your real workflows

Don’t wait for industry benchmarks to include your context.

Build 30–60 prompts drawn from what your staff actually do:

• writing job posts for field roles,
• summarizing farmer interviews,
• translating messages into local languages,
• generating “typical customer persona,”
• describing “ideal borrower,”
• writing performance feedback for staff.

Then test across:

• different surnames and regions,
• different languages you serve,
• different role types (field agent vs office staff).

Track whether outcomes change when only identity signals change.

3) Force refusals on harmful stereotyping

A safe system should refuse or reframe caste-stereotyping prompts rather than complete them.

Implement:

• explicit refusal policies in system prompts,
• content filters for slurs and demeaning associations,
• “safe completion” patterns (e.g., explain why the prompt is harmful and offer a neutral alternative).

The goal isn’t moral theater. It’s preventing your product from outputting something that becomes a screenshot and a crisis.

4) Put humans where the harm is highest

Automation is fine; unreviewed automation is the problem.

Use human review for:

• shortlists,
• eligibility decisions,
• customer dispute outcomes,
• any content that represents real communities in visuals.

Even a 10% human audit sample can catch issues early.

5) Log outputs like you log payments

If you can’t observe it, you can’t fix it.

At minimum, log:

• prompt category (HR, support, marketing),
• language used,
• model version,
• refusal rate,
• flagged content rate,
• “identity rewrite” incidents.

Treat bias incidents as operational bugs, not “PR problems.”

People also ask: “Will this affect my small business if I’m not in HR?”

Answer first: Yes—because bias appears in customer interaction, marketing, and internal writing, not just hiring.

• Customer support: If your chatbot is warmer, faster, or more trusting with certain names or dialects, customers will feel it.
• Sales outreach: AI-written messages can mirror stereotypes about who is “serious” or “educated.”
• Agricultural advisory: If farmer personas and examples are biased, your advice content can become exclusionary.
• Brand visuals: Generative images can quietly encode class and caste stereotypes in housing, clothing, skin tone, and work.

And in late 2025, adoption is accelerating because AI tools are cheaper and easier to integrate than ever. That’s exactly when guardrails matter most.

Ethical AI is part of sustainable agritech—not a side project

Caste bias in AI is a reminder that models learn society’s patterns, not society’s values. If an SME deploys AI without testing, it can end up reinforcing the very inequities it claims it wants to reduce—especially in agriculture, where livelihoods and dignity are always close to the surface.

If you’re building or running an agribusiness SME, start small but be serious: protect identity fields, test your workflows, require refusals for stereotyping, and audit outputs like they’re financial transactions.

If you want AI to raise productivity in farming—better decisions, better pricing, better logistics—then it has to be trustworthy for the people doing the work.

What would change in your business if you treated AI bias incidents with the same urgency as a payment failure or a data breach?