‘90% of questions don’t need a human’: Kogan's six-week sprint to agentic AI

Six weeks. That's how long it took Kogan to move from a one-day hackathon to putting AI agents in front of customers.

What began as an experiment has since developed into a suite of capabilities handling customer enquiries at a scale that would once have required significantly more human resources.

Goran Stefkovski, chief technology offer at Kogan, says that hack day was “ground zero” for a strategy that now sees the Australian online retailer resolve 67% of its AI-handled customer conversations, while 81% are not escalated to humans at all.

“One of the missions that drives us is to help Australians and our customers across the world live their best lives by delivering remarkable value," Stefkovski said at Salesforce’s annual Dreamforce conference in San Francisco.

“You need to have a great experience, a product at a very affordable price, and the support and quality you'd expect. Everything we do is about achieving that mission of getting products to customers.”

Using bots for basic customer enquiries wasn't new for Kogan. But making the jump to an agentic AI system has been something of a game-changer. And as the business looks towards even greater automation, Stefkovski believes the vast majority of customer questions no longer require a human to answer them.

SEE MORE: Nvidia boss rejects the idea AI safety means slowing down

“Ninety percent of questions don't need a human to actually respond. They've been asked before. We've got the answer. It's just providing information to customers quickly,” he told MARKETING-INTERACTIVE.

“What we wanted to do, once we'd seen how LLMs speak naturally like a human, is give customers something that feels natural to interact with. They can read between the lines. They also make mistakes like a human."

The kicker, though, is how customers have responded.

Stefkovski said customer satisfaction scores are seven percentage points higher among those using the bot. Internally, staff are increasingly freed from the repetitive task of answering enquiries one at a time.

But the more interesting story may be how Kogan got here, and what its experience says about the growing divide between businesses experimenting with AI and those putting it to work.

From hack day to production

Kogan's journey began with a relatively simple problem: customers wanted faster answers about their orders.
As an entirely digital retailer, Kogan had traditionally relied on forms, email and self-service to handle enquiries. These processes worked, but required customers to find information, submit requests and wait for a response.

The company's initial chatbot was basic, handling little more than “where is my order”, or WISMO, enquiries. Its resolution rate was just 6.2%. For Stefkovski, that wasn't necessarily a failure.

The use case was narrow, commercially relevant and sufficiently complex to test authentication and connections to Kogan's systems without trying to automate everything at once.

The breakthrough came when Kogan brought its customer care team together with Salesforce's forward-deployed engineers for a hack day. By the end of that day, they had a working prototype. Six weeks later it was live.

From there, Kogan expanded the same agent to include order tracking, product recommendations, returns, warranties and image verification.

The latter has also allowed the retailer to identify instances where customers submit images of damaged products that have previously been used in other claims.

Since then, speed of agent deployment time has been crunched down to two weeks, adapting the same workflows to different brands, systems and loyalty programmes. Kogan is now in the process of taking integrating that same architecture into its New Zealand retailer Mighty Ape.

A different equation for customer service

The commercial argument is particularly compelling for a business built around delivering low prices at scale.

With customer service demands increasing dramatically through Christmas, Black Friday and other major retail periods, meeting that demand meant traditionally meant recruiting more staff, often offshore and investing weeks in training. AI changes that equation.

“You can scale agents over Christmas in 30 seconds, but to scale people, your call centre team, you need weeks of training, lead time to hire people, training time to get them to understand the processes,” Stefkovski said.

The thing that stands out for me the most is that we've been able to scale, providing customer service and product question resolution really quickly, to as many people as needed at any time, whether it's Black Friday and there's a spike in volumes, or it's a normal day of the week.

Goran Stefkovski

Once the system is built, the same suite of agents can service one customer or thousands simultaneously.

“The thing that stands out for me the most is that we've been able to scale, provide customer service resolution to people, product question resolution to people, really quickly to as many people as we want, or as many people as needed at any time,” he said.

According to figures presented at Dreamforce, Kogan now resolves 67% of conversations with AI tools, compared with the original bot's 6.2% resolution rate. Some 78% of customers say they prefer interacting with an agent, while the system recorded more than 44,000 agent actions during a peak week.

Stefkovski said customer satisfaction scores are seven percentage points higher among customers using their bots. Perhaps more crucially, the company is also seeing its cost to serve these customers going down, as sales volumes increase.

“We're seeing an increase in sales volumes, but it's actually getting cheaper for us to serve that because we don't need to scale the headcount.”

The result is not yet a completely autonomous customer service operation. Kogan is still expanding the number of enquiries its agents can handle, with voice on the roadmap.

But the retailer is already reconsidering how much seasonal recruitment it needs. For Stefkovski, that's a more meaningful measure of success than the number of AI pilots a company can claim to be running.

What happens to the humans?

Perhaps the more significant change is taking place inside Kogan itself.

Asked whether the technology was replacing customer service employees or changing the company's culture, Stefkovski said the immediate impact was more nuanced.

The local team continues to handle complex enquiries, while employees previously responsible for repetitive work are increasingly involved in developing and improving the agents.

Kogan's customer care analysts, rather than its central engineering team, have been instrumental in building the system.
They understand the questions customers ask, the processes behind them and the problems that need solving. Having previously upskilled on Salesforce, they now work directly with the platform's engineers to develop agents.

“We're shifting work from a repetitive style of work to a more innovative style of work of okay, let's build flows and let's identify where new customer challenges are coming through,” Stefkovski said.

He expects that shift to continue, with employees spending more time on knowledge work, identifying unusual cases and proactively addressing customer problems.

How that plays out longer term remains uncertain, he acknowledged. It also changes who owns technology development.

Rather than every initiative passing through a central engineering roadmap, the people closest to the problem can increasingly build solutions themselves, supported by engineering teams responsible for security, authentication and scale.

“I think every leader has to work with it in the way it makes sense,” he said.

‘If it's not in production, it doesn't count’

There is a broader lesson in Kogan's approach, particularly for organisations struggling to move AI beyond experimentation.

Stefkovski argued that companies frequently spend too long polishing technology before allowing customers to use it. The risk, he said, is not simply to delay. It is investing heavily in a solution that may ultimately solve the wrong problem.

“I think a lot of businesses and a lot of organisations try and polish things too much before they get it out, and it means a lot of the time you've gone too far in the wrong direction,” he said.

“For a business to really get value in front of its customers, it has to be in front of its customers.”

Kogan's answer is to start with a focused use case, get it into production and use real customer interactions to determine what comes next. That philosophy also extends to investment.

Rather than committing millions of dollars upfront to a major transformation programme, Kogan has taken an incremental approach, building on its existing Salesforce investment and watching the costs and returns as each new capability is introduced.

“The investment was incremental. When we did our shift of platforms, we wanted a bot, so we did an RFP and we looked at all different solutions, and the bot was already included in that,” Stefkovski said.

While he didn’t disclose a total investment or dollar return, he said the company continually examines token expenditure, agent actions and operating costs as usage grows.

“It's not like here's millions of dollars to do this project and this is what we want to see in 12 months,” he said.

“That incremental approach means that you're doing small investments and you're watching the payback straight away.”

The approach also requires accepting that AI, like traditional software and human employees, will sometimes make mistakes.

Stefkovski argued that the guardrails provided by modern agent platforms have improved considerably, while suggesting AI errors attract disproportionate attention compared with mistakes made by humans.

“When the bot's offering you free tickets on a flight, the whole world hears about it,” he said. “But it happens regardless if it's a bot, or even traditional software with bugs in it. Humans make mistakes too.”