COVID-19 : What we’re doing

We are doing our best to play our part by adapting responsibly and responding in any way we can. Over the last few weeks, we’ve launched a quick deploy Chatbot Capability for COVID-19 Information that is being taken up by a variety of organisations. Find out more here

One of the great things about outsourcing your chatbot development to an agency is that you greatly reduce the risk and errors when building your chatbot as you’re paying for their experience and reliability to get the job done.

However, while some challenges are the same as outsourcing in other departments, chatbots come with their own challenges.

So what are the top 7 mistakes to avoid when outsourcing to a chatbot agency?

In a nutshell:

  1. Lack of fundamental knowledge of how chatbots work
  2. Mismatch of expectations of delivery vs cost
  3. Not spending enough time on defining your user demographic profiles and conversation design
  4. Hiring the cheapest workers and hoping it works
  5. Not asking the right questions to test the chatbot company
  6. Picking the wrong metrics to measure & setting unrealistic targets
  7. Not having a dedicated chatbot champion within the company

Which one of these has the biggest impact over the rest? Which are the easiest to avoid and which ones, even with your best efforts, have a chance of still occurring?

Let’s run through each one further to find out:

Lack of fundamental knowledge of how chatbots work

Start by getting to a state of known unknowns.

This is the root cause of the majority of mistakes most companies will make when outsourcing.

Unlike other verticals such as marketing or design and even development where you may have some understanding and experience, machine learning and chatbots are still very new.

To reduce the number of errors, you need to understand what the known unknowns are.

Otherwise, many estimates will be way off, such as underestimating costs, overestimating chatbot capability and not asking the right questions of the agency you plan to hire.

The simplest and cheapest solution is taking some time to understand this new sector. You’ve made a great start by checking out this article:

Mismatch of expectations of delivery vs cost

Without a doubt, the most common mistake we come across is:

The disparity between the chatbot capability and the pain you’re aiming to solve vs the cost and time it will take. 
Typically, management has a pain which could be solved with current technology, but to solve that challenge usually takes the very latest tech. Thus the cost is way higher than most anticipate.

We go through what it takes to build a chatbot and the costs involved in our article here.

The other bit of good news is a £40-50k chatbot project is roughly the same cost as an employee, meanwhile, it brings 6-10x of a full-time equivalent (FTE). Often a worthwhile investment.

Lastly, as you can see in the Gartner chart below, chatbots (or virtual assistants as it’s labelled) we’ve gone through the trough of disillusionment and being in 2019 to 2020, chatbots are certainly at the beginning of the slope of enlightenment.

The good news for you is, as both charts demonstrate, investing in the basic infrastructure now ensures you capture all the benefits and increased productivity in the years to come. 

Not spending enough time on defining your user demographic profiles and conversation design

Conversation flow example

As the old saying goes: Measure twice. Cut once. 

Most companies have a UX & UI process as part of the product development cycle and your chatbot should have the same. Doing rapid and low-cost interviews & tests with your target users will help validate the business case, find unforeseen intents and work out some early kinks that will save you a lot of pain and cost in the long run.   

Customer demographic profile example

There is an extra stage to chatbot design that chatbot agencies carry out, to take into consideration that didn’t exist in the design process before: “the bot persona”. 

Hiring the cheapest workers and hoping for the best

I’m sure you’ve heard the motto of “If you think it’s expensive to hire a professional to do a job, wait until you hire an amateur!”. As much as it makes me cringe to write it, it does hold truth. Let’s break the old cliché down:

Why hiring cheaper workers goes wrong

1. Poor communication

Cheap usually means going abroad. Going abroad equates to having some sort of language barrier. Building chatbots are difficult enough as it is, that has a lot of unique & complex terminology. Adding this extra challenge into the mix will only add extra mistakes and cost in the long run.

2. Culture differences & nuances with one language

Chatbot conversation design is critical to chatbot success. Language and culture greatly affect how you create that flow and train the intents.
For example, when designing a chatbot that had to cover multiple cities in the UK – the way of saying “hi” or “how are you?” changed from city to city. The responses those local communities expect from the bot also vary. 

3. Multitasking or non-specialist

Since they’re working at a lower rate, it’s likely they’ll be juggling more projects and few agencies are focused entirely on just AI. (usually, it’s a sub-branch of other offerings)
Simply: less focus = poorer results. 

4. They say “yes” to get the work and figure things out after. 

Sadly, we’ve seen many companies get taken advantage of, as they don’t know what the process should look like and incidentally get taken for a ride. 

We’ll elaborate on this section further in the “questions you should ask to test their knowledge” so you can be sure that the agency you hire will get things right first time around and create a chatbot that delivers results.

Not asking the right questions to test the chatbot company

We found the most effective questions to ask and test potential agencies are:

Hard skills

  • What does the typical chatbot development process look like? 
  • What success metrics do you recommend? 
  • How do the different skill sets of your team members complement our in-house resources? 
  • Can you give me a breakdown of who does what in your organisation?
  • What does it take to develop a contextual chatbot?

Soft skills

  • How do we stack up in terms of agency spend and production requirements compared to your other clients?
  • What makes a great client so good to work with? And what makes an awful client so bad?

Picking the wrong metrics to measure & setting unrealistic targets

This needs to be broken down into two parts which we go into further depth in our “Analytic techniques & tools to improve your chatbot performance” article where we cover the detailed reasons and data behind why we find certain metrics better than others. For now, though, we’ll cut to the answer you’re looking for:

Correct metrics to use

Here are our recommended top data points to keep a close eye on:

  • Customer Satisfaction Score (CSAT) / Net Promoter Score (NPS) score
  • Total Users
  • Chatbot accuracy/intent success rate.

The best data points to measure 

  • Conversation flow that highlights where it drops off
  • Successful completed journeys flow highlights
  • Containment rate
  • Average session per user
  • Full-time equivalent (FTE)
  • Successful & Failed intents
  • Intent confidence score
  • New & returning users

Common unachievable metric pitfalls

The above recommendations are when you have a level 2 chatbot or above. There are, of course, a lot of nuances and individual circumstances. 

The most common pitfall is expecting too high performance or ROI from a proof of concept. 

To elaborate on a couple of examples:

It’s commonly asked for an FTE / ROI of 6x or more

For a basic proof of concept chatbot, a full-time equivalent (FTE) / return of investment (ROI) of 4x or more is unrealistic. The purpose of a PoC is to prove integration and the bot build process is working, demonstrating how you reduce the risks of implementation and deployment.

When you start ramping up chatbot uses cases and increase the number of intents the bot can handle, THEN you can expect an FTE of 6-10x.

Containment rate of 30-40%

Containment rate on initial builds will usually be defined as a goal in themselves. For example: “this MVP will aim to answer 10% of all inbound FAQs that we’ve selected”

Sadly, some clients see the outstanding results of other chatbots and hear stats like 30-40% containment rate.

This only achievable after extensive A/B testing and after months of the bot running. As we just mentioned, your welcome message and following conversation flow even after initial wizard of oz / early user testing will be at 10-20%, especially if you only have a basic proof of concept or a chatbot that primarily uses buttons.

Not having a dedicated ‘chatbot champion’ within the company

When developing your budget, make sure you account for the fact one of your employees will have to take this on a full time, even when you’re outsourcing. Chatbot agencies like us, need a reliable port of call, someone with strong knowledge of the business and its needs. 

Having a chatbot champion has numerous benefits:

  • Helps build the relationship between the agency and the client
  • Someone to retain your IP and becomes the go-to for your chatbot when problems arise.
  • Improves the results of the chatbot output and performance.

This person champions and drives success metrics. As the chatbot function grows, this champion will be the person that grows the chatbot team.

(yes, the most successful chatbots have entire teams behind them, typically a chatbot developer, a product manager and an analyst.) 

Bonus mistake!: creating a poor quality RFP 

Honestly, 99% of all inbound enquiries of potential clients looking to outsource or partner with us have RFPs that’s missing a lot of information or aren’t specific enough. If you send us a message at hello@enterprisebotmanager.com we’ll gladly send you our version to use.

Related questions

What companies should use chatbots?

There are use cases for pretty much every vertical and industry. From HR to lead generation, chatbots are going to improve the sector.

There are 3 major factors that impact whether your company is right to use a chatbot:

  • Enough traffic & enquiries 
    • Without enough traffic, you’re unlikely to justify the ROI. As you won’t have enough enquiries to automate and thus make cost savings. 
  • You have enough budget
    • As discussed above, because the sector is still emerging the experience, and skillsets are still hard to come by, supply is low while demand is high. Thus, it’s expensive. It’s also because when implemented, they drive high-quality results and FTE. The challenge is whether you have enough capital to afford the work done.
  • Your leadership team is fully supportive and understand the process of the adoption of AI.
    • Artificial Intelligence (AI) is a hot potato to debate right now as there are a lot of uncertainties, so give some examples of the difficult topics that need to be discussed: 
      • Can we understand how the AI came to its decision?
      • What happens to the workers when parts of their job are automated?
      • How do manage data security and privacy? 

What are the best chatbot platforms?

Our usual go-to favourites are Dialogflow or Watson powering the natural language understanding, we then use our own analytics and bot management tool: EBM for the rest of it. 

Enterprise Bot Manager © 2020 Filament Consultancy Group. Registered in England and Wales -  Company Number 10180537.  © 2020 Filament Consultancy Group Canada Limited. Registered in Ontario, Corporation Number 1995332.