Prove Your AI Spend: Evidence-Based IT Value

Most South African e-commerce and digital service teams don’t have an “AI problem”. They have a proof problem.

A board approves millions for a personalisation engine, a new CDP, a chatbot, fraud detection, or “AI-assisted marketing”. Six months later, the dashboard looks busy—more messages sent, more models trained, more tickets closed. Yet revenue is flat, churn hasn’t moved, and cost-to-serve is still painful. That gap between activity and outcomes is what I call AI theatre: impressive motion, weak business impact.

If you’re serious about using AI to grow an online store or digital service in South Africa, you need a way to put IT value “on trial”—with evidence that holds up under scrutiny. The good news: this is less about fancy reporting and more about governance discipline. You can build that discipline with a simple framework: define the business result, write a testable hypothesis, run 90-day proof cycles, and fund what’s proven.

AI theatre vs AI value: the difference is causality

AI value is measurable movement in a business value node (like margin or churn) caused by a specific intervention, within a defined time window. Anything else is storytelling.

E-commerce and digital services are especially vulnerable to theatre because AI tools generate endless metrics that feel meaningful but don’t pay the bills: “engagement”, “coverage”, “model accuracy”, “tickets deflected”. Those can matter, but only if they connect to a board-level outcome.

Here’s the stance I take: If you can’t explain how an AI initiative changes EBITDA drivers in plain language—and prove it—you don’t have a value case. You have a feature demo.

For South African businesses, the pressure is sharper in December and January: peak-season spend, returns, delivery bottlenecks, and high contact-centre loads. It’s the perfect time to demand evidence. If your AI investment can’t show impact through peak trading and the post-holiday churn window, that’s a signal.

The value nodes that matter most in SA e-commerce

Most board conversations eventually land on a small set of outcomes. Translate your AI work into these value nodes:

• Revenue (conversion rate, AOV, repeat purchase)
• Margin (discount leakage, returns cost, picking/packing efficiency)
• Churn / retention (subscription renewals, repeat rate, win-back)
• Cost-to-serve (contact centre minutes, delivery exceptions, payment failures)
• Cycle time (time to ship, time to resolve disputes, onboarding time)
• Risk (fraud losses, chargebacks, compliance incidents)

If your AI initiative can’t clearly target one node first, you’re already drifting toward theatre.

Start with one outcome, not a tool

The cleanest way to stop AI theatre is to ban tool-first proposals. No “we need GenAI for customer service” or “we should roll out a recommendation engine” without a single, board-level outcome attached.

What works is picking one strategic result from your 3–5 year plan and slicing it into a 12-month target. Then you name the exact value node you intend to move.

A practical example (online retail)

Instead of: “Implement AI personalisation across the site.”

Write: “Reduce churn among first-time buyers by 2.0 percentage points within 90 days of their first purchase.”

Now you’ve got something governable. From there, you can test personalisation, lifecycle messaging, customer service interventions, delivery promise accuracy, or even payment retry logic. The tech becomes a means, not the headline.

Build a value map with owners, baselines, and guardrails

A value map is a small table that makes governance possible. It forces clarity on what you’re moving, who owns it, and how far you’re allowed to push before risk becomes unacceptable.

For each value node, define:

• KPI (the number)
• Owner (a named business owner, not “IT”)
• Baseline (today’s reality)
• Target (what “better” means)
• Timeframe (by when)
• Guardrails (risk appetite: e.g., don’t increase refunds, don’t raise complaint rate)
• Decision rights (who can scale/stop)

Example value map row (digital services)

• Node: Cost-to-serve
• KPI: Cost per resolved support case
• Baseline: R48 per case
• Target: R40 per case
• Timeframe: Quarter 1
• Guardrails: CSAT must not drop below 4.2/5; complaint escalations cannot rise >10%
• Owner: Head of Customer Operations

This matters because AI can “look” like it’s saving cost while quietly damaging customer trust. Guardrails prevent false wins.

Turn AI ideas into testable hypotheses (with decision rules)

A hypothesis card is where your AI programme becomes serious. It’s a one-paragraph commitment to measurable impact.

Use this structure:

If we do X, then Y (value node) moves by Z within T.

Then add:

• Proof method (A/B test, matched before–after, stepped rollout)
• Decision rule (scale, pause, or kill based on results)

Example hypothesis cards you can steal

1) GenAI customer support (cost-to-serve)

• If we deploy GenAI agent-assist for the top 30 intents, then average handle time drops by 12% within 8 weeks, without reducing CSAT.
• Proof: stepped rollout by team (Team A vs Team B), matched for issue mix.
• Decision rule: scale if AHT improves ≥10% and CSAT holds; pause if escalations rise ≥15%.

2) Fraud AI (risk + margin)

• If we implement ML-based fraud scoring for card-not-present transactions, then chargeback rate drops by 0.15 percentage points within 60 days, while approval rate stays within guardrails.
• Proof: A/B by traffic split with manual-review sampling.
• Decision rule: scale if chargebacks fall and false declines don’t increase beyond threshold.

3) Personalised offers (margin)

• If we personalise discounting using predicted price sensitivity, then gross margin after discounts improves by 0.8% within 90 days.
• Proof: A/B at customer cohort level.
• Decision rule: scale if margin improves and refund rate doesn’t increase.

Notice what’s missing: “model accuracy”. Accuracy isn’t the business result.

Expose assumptions and run 90-day evidence plans

Every AI initiative rests on assumptions that can quietly destroy value. The fastest way to protect your budget is to force assumptions into the open and test them within 90 days.

Common AI assumptions in South African e-commerce and digital services:

• Adoption: Agents will actually use the assistant; merchandisers will trust recommendations.
• Behaviour change: Customers will respond to the new journey; they won’t feel “creeped out”.
• Data quality: Customer identities match; events are tracked correctly; product data is clean.
• Operational readiness: Fulfilment can handle a conversion lift; delivery promises are accurate.
• Integration: The AI can access real-time inventory, order status, pricing, and customer history.

Convert each assumption into a named test:

• Owner
• Method
• Sample size / cohort
• Success criteria
• Deadline (≤90 days)

Example: the assumption that breaks most recommendation projects

Assumption: “We can attribute uplift to recommendations.”

Test: A/B test with a holdout group, measured on incremental revenue per session and conversion, tracked for 4–6 weeks.

If attribution isn’t clean, you don’t scale. You fix tracking first.

Use a single board-level score: a Value Realisation Index for AI

Boards don’t need 40 charts. They need one clear signal that answers: Is AI delivering measurable value?

A practical approach is to adopt a Value Realisation Index (VRI)—a quarterly score out of 100 that only counts decision-grade evidence. The point isn’t the exact maths; it’s the discipline: what gets counted must be provable, traceable, and tied to outcomes.

A strong VRI design includes these dimensions and weightings:

• Strategic alignment (15%): AI spend maps to core drivers like EBITDA, churn, margin.
• Value-tree strength (15%): You’re targeting material nodes (not vanity metrics).
• Assumption discipline (20%): Critical assumptions tested within 90 days.
• Evidence quality (25%): Causal proof, clear method, timed KPI movement.
• Risk-adjusted outcomes (25%): Uplift achieved without violating guardrails.

Then you band it:

• Green: AI/IT is delivering measurable value.
• Amber: Progress, but evidence is incomplete or risk is rising.
• Red: Value isn’t being realised—stop funding theatre.

A useful rule: If your VRI is “green” but your P&L and customer outcomes are flat, your evidence standard is too low. Raise it.

The quarterly cadence that keeps everyone honest

Cadence beats motivation. If you want consistent AI returns, you need a rhythm that forces decisions.

Run a quarterly “court calendar”:

1. Board/CEO scorecard

   • VRI score (0–100) and trend
   • Top value nodes: baseline → target → delta
   • Proven vs assumed split (how much is actually evidenced)

2. 90-day plan

   • Every weak point becomes a test: owner, method, deadline

3. Decision and assumption ledger

   • Record every decision: fund/kill/scale/pause
   • Record which assumptions were proven or disproven
   • Reassign capacity based on evidence

Bring three questions to every meeting:

• Specificity: Which value node are we moving this quarter, and by how much?
• Evidence: What proof method did we agree, and when does the evidence land?
• Assumptions: What could break the case, and what are the ≤90-day tests?

If you can’t answer those, you’re not managing AI. You’re hosting a demo.

A quick self-audit for your next AI initiative

If you’re planning 2026 budgets right now (and most teams are), run this checklist before you approve the next AI line item:

1. Name one value node. If you list three, you’re avoiding the real choice.
2. Write a hypothesis with numbers. Baseline, target, timeframe.
3. Choose a proof method. A/B, matched before–after, stepped rollout.
4. Set guardrails. Decide what must not get worse.
5. Identify the top three assumptions. Turn them into 90-day tests.
6. Define the kill rule. Decide what failure looks like before you start.

This is how you stop AI spend from turning into an annual tradition of optimistic slides.

What to do next if you want AI that pays for itself

South African e-commerce and digital services are in a phase where AI is everywhere—marketing, service, fraud, logistics, content. The winners in this topic series won’t be the companies with the most tools. They’ll be the companies with the strongest evidence discipline.

If you want a practical next step: pick your single biggest in-flight AI initiative and put it on trial. Give it a value node, a baseline → target → delta, and one proof method. Then list the three assumptions most likely to break the case and assign 90-day tests.

When your next board meeting comes around, you’ll walk in with something rare: a number that means something, and a decision you can defend. What would change in your business if every AI project had to earn funding the same way—by proof, not theatre?