A leading entertainment tech company leverages Galileo to deliver “last mile” conversational AI precision

COMPANY OVERVIEW

Founded in 2016 and specializing in sports, entertainment, and tourism, a leading Entertainment Tech Company initially gained success by collaborating with organizations like the New York Mets, Macy’s, and the US Open, building virtual assistants and natural language processing (NLP) models that enable easy access to information often unavailable on websites. Their innovative solutions empower fans and customers to engage more deeply with their favorite teams, venues, and events.

CHALLENGE

In early 2020, the onset of COVID-19 forced many of the company's venue customers, including Madison Square Garden and Universal Studios Orlando, to shut down operations. In response, the team pivoted to focus on proactive fan engagement through NLP models that provided statistics and trivia. As venues began to reopen, they rekindled those connections and introduced their product, Context NLP.

By 2022, with the widespread availability of ChatGPT, they saw an opportunity to leverage Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to enhance their conversational AI solutions. While this approach quickly achieved 70% accuracy in responses, the remaining 30% presented significant challenges. Issues such as hallucinations, tone inconsistencies, and sensitivity needed to be monitored and mitigated to improve answer quality.

"We needed something that provided sophisticated, fine-tuned completion of answers," said the CPO and Co-founder. "Hallucinations and proper responses in terms of tone and sensitivity were areas we had to address to deliver precise answers."

Their manual scoring system hindered growth opportunities. Annotators and evaluators manually reviewed user-flagged questions, which was time-consuming and not scalable. They needed a solution to bridge the gap from 70% to 100% accuracy in a scalable way for their clients.

SOLUTION

The company partnered with Galileo to elevate their evaluation and monitoring processes. Galileo's comprehensive analytics, including metrics like Context Adherence, enabled their team to automate quality assessments and pinpoint areas requiring human intervention.

With Galileo Observe, they now manage hundreds of client deployments with ease. The integration of Galileo’s dashboard and APIs into their workflows provides real-time insights, allowing the team to monitor and optimize active implementations from a single, streamlined console. This deeper integration into their annotator workflows empowers their operations team to proactively address issues flagged by Galileo's metrics.

Additionally, the company has leveraged Galileo to streamline their business model by offering a self-service deployment option. Customers can now independently create, deploy, and monitor their conversational AI solutions, while their internal team oversees the health of these deployments using Galileo’s monitoring tools.

RESULTS

Since implementing Galileo in early 2023, the company has been able to mature their AI operations, scale their delivery model, and experienced significant benefits:

• Improved Accuracy: Increased conversational AI precision, raising accuracy from 70% to nearly 100%.
• Automated Quality Assessment: Automated scoring of bot responses for accuracy and context adherence, greatly reducing the need for manual evaluations.
• Real-Time Performance Monitoring: Efficiently manages over 400 installations with varying prompts, leveraging real-time monitoring to oversee all deployments seamlessly.
• Safe Prompt Changes: Utilizes Galileo to test and implement prompt changes, ensuring improvements without introducing regressions.
• Operational Efficiency & Cost Savings: Automation of monitoring and quality assessments has enabled their team to avoid the need for additional full-time hires, boosting efficiency and reducing costs.

Conclusion

By leveraging Galileo's Evaluate and Observe products, the Entertainment Tech Company significantly improved the precision and reliability of their conversational AI solutions. The partnership with Galileo has been instrumental in smoothly scaling operations, enhancing answer quality, and delivering accurate, trustworthy chatbot experiences to their end customers.

"Before Galileo, getting from 70% to 100% accuracy was a significant challenge. With Galileo, we've not only improved our responses but also scaled our services efficiently. Galileo's tools have truly filled the gap for us."

— CPO and Co-founder