Jun 2, 2025
How do you choose the right metrics for your AI evaluations?
Erin Mikail Staples
Senior Developer Advocate
Erin Mikail Staples
Senior Developer Advocate
When building AI systems—especially those that interact with users or make decisions—it's crucial to measure performance in ways that align with your goals (whether performance is speed, accuracy, tool selection, or quality). Galileo provides a comprehensive suite of evaluation metrics out of the box, as well as the ability to create custom metrics via LLM-as-a-Judge or code-based scoring. These metrics are designed to answer specific questions about your AI's behavior. But with so many options, how do you know which metrics to use, and when?
This guide walks you through the main categories of metrics, explains when to use each, and provides practical examples to help you make informed decisions.
Choosing the Right Metrics
Start with your goals: What matters most for your use case—accuracy, safety, confidence, progress, or style?
Mix and match: Real-world systems benefit from multiple categories. A support bot, for example, needs both accuracy and safety.
Establish baselines: Know where you're starting before making changes.
Track trends: Monitor changes as you iterate.
Set thresholds: Define "good enough" and build in alerts.
PSST — check out this YouTube video for a recap on how to choose the right metrics and what may be the most helpful for your needs.
🧠 Response Quality Metrics
What they measure:
Response Quality metrics serve as your AI's report card, evaluating how well the model understands and responds to prompts, particularly in terms of factual accuracy, completeness, and adherence to instructions.
They help answer questions like:
“Did the AI answer the question?”
“Did it follow the instructions—or just vibe its way through the prompt?”
“Is it making things up... again?”
They’re especially crucial for use cases where trust and clarity are non-negotiable—think customer service bots, educational tools, or retrieval-augmented generation (RAG) systems. Skip these metrics, and you risk deploying an AI that sounds confident while confidently delivering nonsense (a.k.a. hallucinations).
Risk if ignored:
Response quality issues can lead to incorrect financial advice, misinformed users, or loss of trust. A notable case: Google Bard sharing incorrect information in its launch demo, or even the Chicago Sun-Times publishing an AI-generated Summer Reads list, in which only 5 of the 15 books were real.
Key metrics:
🏡 Safety and Compliance Metrics
What they measure:
Safety and Compliance metrics are the guardian angels of your AI, watching for danger zones like leaked sensitive information, biased or toxic language, and attempts to manipulate your model via prompt injections.
Whether you're working in healthcare, finance, or just trying to keep your chatbot from going rogue on social media, these metrics flag the moments where things could get legally murky, ethically shady, or just plain awkward.
If you don’t implement them, you’re flying without a parachute. You might wake up to headlines you never wanted, audits you weren’t prepared for, or worse, losing user trust in a single bad interaction. Just ask Apple which found that the Apple Credit Card algorithm managed by Goldman Sachs would routinely offer women lower credit limits than men — even when women had higher credit scores.
Risk if ignored:
Ignoring safety and compliance metrics can lead to harmful outputs, legal exposure, and loss of user trust. It only takes one bad response to spark audits, headlines, or regulatory action.
Key metrics:
🎯 Model Confidence Metrics
What they measure:
Model Confidence Metrics about how sure your AI is of itself, and when it’s feeling shaky. They quantify uncertainty in responses and assess prompt complexity (a.k.a. how hard the question is).
They’re a lifesaver when building human-in-the-loop workflows or escalation systems. If your model is unsure, you can route that output to a human reviewer or flag it for deeper analysis. No one wants an AI answering medical queries with “I think maybe...” energy.
Skip these, and you risk presenting low-confidence outputs as facts, which is how misinformation, confused users, or a helpdesk full of angry or flustered users can occur. Take for example, Gmail’s smart reply feature that over-indexed on recommending users to tell someone they love them, or Waze confidently navigating people directly into wildfires.
Risk if ignored:
Ignoring model confidence metrics means your AI might present guesses as facts. This can confuse users, spread misinformation, or trigger unnecessary escalations—especially in high-stakes settings.
Key metrics:
🤖 Agentic Metrics
What they measure:
Agentic metrics track how well your AI agent navigates multi-step tasks, makes decisions, and uses tools. Think of these as performance reviews for your autonomous agents—are they moving toward the goal efficiently, or just poking around aimlessly? Are they choosing the right tools for the job, or breaking everything in sight?
This is vital if you're building AI that interacts with APIs, executes actions, or works autonomously in workflows, like auto-triaging support tickets or writing code.
Without these metrics, debugging becomes a nightmare. You won't know if your agent is failing because it used the wrong tool, misinterpreted the goal, or just decided to vibe. Microsoft recently released a white paper on how AI agents can be subsceptivble to ‘memory poisoning’, leading agents to be manipulated to act against organizational interests.
Risk if ignored:
Ignoring agentic metrics leaves you flying blind when it comes to how your AI agents make decisions, use tools, or complete multi-step tasks. Without them, you can’t tell if failures are due to tool misuse, poor goal understanding, or decision paralysis—making debugging guesswork and scaling risky. This can result in agents that appear functional but waste resources, break workflows, or generate unpredictable results in production.
Key metrics:
✍️ Expression and Readability Metrics
What they measure:
Expression and Readability Metrics measure the ✨vibes✨—aka your AI-generated content's tone, fluency, clarity, and human-likeness.
They're essential for any system where words matter, such as marketing copy, brand voice, educational materials, or any other user-facing content. You want your AI to sound smart, not robotic; clear, and condescending.
Without these metrics, you risk generating content that’s technically accurate but emotionally tone-deaf, or just plain unreadable. It’s like dressing your model in a suit of jargon when the user just wanted a friendly chat. Case in point — the National Eating Disorder Association (NEDA)’s chatbot was removed after it repeatedly gave dangerous and insensitive advice to users with eating disorders, and delivery firm DPD had their chatbot swear at customers and expressed hatred for the company when prompted.
Risk if ignored:
Ignoring expression and readability metrics can hurt your brand voice, confuse users, and erode trust—especially in customer-facing apps. Poorly worded or off-tone responses can lead to higher churn, reduced engagement, and even viral backlash. What sounds robotic or awkward to users can directly translate into lost revenue and reputational damage.
Key metrics:
🧪 Custom Metrics: Tailoring Evaluation to Your Needs
While Galileo offers a robust set of built-in metrics, we understand that every AI application has unique requirements. That's why Galileo supports the creation of custom metrics via LLM-as-a-Judge or code-based scoring, allowing you to define and register your own evaluation criteria.
You can choose between Custom Scorers and Registered Scorers, depending on where, when, and how you want these metrics executed.
Explore the documentation to learn how to get started with your own custom metrics, tailored to your specific needs.
🛠️ Sample Applications: Putting Metrics into Practice
To help you get started, we’ve curated a collection of sample applications demonstrating how to implement and utilize various metrics in real-world scenarios. Check out our GitHub repository: Galileo SDK Examples
Final Thoughts
Choosing the right metrics is foundational to building trustworthy, effective, and user-friendly AI systems. By understanding what each metric category measures and when to use it, you can tailor your evaluation strategy to your specific goals and deliver more effective AI experiences.
Sign up for Galileo to get started on your journey to building reliable AI applications and/or explore the Galileo documentation to dive deeper into each metric and start measuring what matters.
When building AI systems—especially those that interact with users or make decisions—it's crucial to measure performance in ways that align with your goals (whether performance is speed, accuracy, tool selection, or quality). Galileo provides a comprehensive suite of evaluation metrics out of the box, as well as the ability to create custom metrics via LLM-as-a-Judge or code-based scoring. These metrics are designed to answer specific questions about your AI's behavior. But with so many options, how do you know which metrics to use, and when?
This guide walks you through the main categories of metrics, explains when to use each, and provides practical examples to help you make informed decisions.
Choosing the Right Metrics
Start with your goals: What matters most for your use case—accuracy, safety, confidence, progress, or style?
Mix and match: Real-world systems benefit from multiple categories. A support bot, for example, needs both accuracy and safety.
Establish baselines: Know where you're starting before making changes.
Track trends: Monitor changes as you iterate.
Set thresholds: Define "good enough" and build in alerts.
PSST — check out this YouTube video for a recap on how to choose the right metrics and what may be the most helpful for your needs.
🧠 Response Quality Metrics
What they measure:
Response Quality metrics serve as your AI's report card, evaluating how well the model understands and responds to prompts, particularly in terms of factual accuracy, completeness, and adherence to instructions.
They help answer questions like:
“Did the AI answer the question?”
“Did it follow the instructions—or just vibe its way through the prompt?”
“Is it making things up... again?”
They’re especially crucial for use cases where trust and clarity are non-negotiable—think customer service bots, educational tools, or retrieval-augmented generation (RAG) systems. Skip these metrics, and you risk deploying an AI that sounds confident while confidently delivering nonsense (a.k.a. hallucinations).
Risk if ignored:
Response quality issues can lead to incorrect financial advice, misinformed users, or loss of trust. A notable case: Google Bard sharing incorrect information in its launch demo, or even the Chicago Sun-Times publishing an AI-generated Summer Reads list, in which only 5 of the 15 books were real.
Key metrics:
🏡 Safety and Compliance Metrics
What they measure:
Safety and Compliance metrics are the guardian angels of your AI, watching for danger zones like leaked sensitive information, biased or toxic language, and attempts to manipulate your model via prompt injections.
Whether you're working in healthcare, finance, or just trying to keep your chatbot from going rogue on social media, these metrics flag the moments where things could get legally murky, ethically shady, or just plain awkward.
If you don’t implement them, you’re flying without a parachute. You might wake up to headlines you never wanted, audits you weren’t prepared for, or worse, losing user trust in a single bad interaction. Just ask Apple which found that the Apple Credit Card algorithm managed by Goldman Sachs would routinely offer women lower credit limits than men — even when women had higher credit scores.
Risk if ignored:
Ignoring safety and compliance metrics can lead to harmful outputs, legal exposure, and loss of user trust. It only takes one bad response to spark audits, headlines, or regulatory action.
Key metrics:
🎯 Model Confidence Metrics
What they measure:
Model Confidence Metrics about how sure your AI is of itself, and when it’s feeling shaky. They quantify uncertainty in responses and assess prompt complexity (a.k.a. how hard the question is).
They’re a lifesaver when building human-in-the-loop workflows or escalation systems. If your model is unsure, you can route that output to a human reviewer or flag it for deeper analysis. No one wants an AI answering medical queries with “I think maybe...” energy.
Skip these, and you risk presenting low-confidence outputs as facts, which is how misinformation, confused users, or a helpdesk full of angry or flustered users can occur. Take for example, Gmail’s smart reply feature that over-indexed on recommending users to tell someone they love them, or Waze confidently navigating people directly into wildfires.
Risk if ignored:
Ignoring model confidence metrics means your AI might present guesses as facts. This can confuse users, spread misinformation, or trigger unnecessary escalations—especially in high-stakes settings.
Key metrics:
🤖 Agentic Metrics
What they measure:
Agentic metrics track how well your AI agent navigates multi-step tasks, makes decisions, and uses tools. Think of these as performance reviews for your autonomous agents—are they moving toward the goal efficiently, or just poking around aimlessly? Are they choosing the right tools for the job, or breaking everything in sight?
This is vital if you're building AI that interacts with APIs, executes actions, or works autonomously in workflows, like auto-triaging support tickets or writing code.
Without these metrics, debugging becomes a nightmare. You won't know if your agent is failing because it used the wrong tool, misinterpreted the goal, or just decided to vibe. Microsoft recently released a white paper on how AI agents can be subsceptivble to ‘memory poisoning’, leading agents to be manipulated to act against organizational interests.
Risk if ignored:
Ignoring agentic metrics leaves you flying blind when it comes to how your AI agents make decisions, use tools, or complete multi-step tasks. Without them, you can’t tell if failures are due to tool misuse, poor goal understanding, or decision paralysis—making debugging guesswork and scaling risky. This can result in agents that appear functional but waste resources, break workflows, or generate unpredictable results in production.
Key metrics:
✍️ Expression and Readability Metrics
What they measure:
Expression and Readability Metrics measure the ✨vibes✨—aka your AI-generated content's tone, fluency, clarity, and human-likeness.
They're essential for any system where words matter, such as marketing copy, brand voice, educational materials, or any other user-facing content. You want your AI to sound smart, not robotic; clear, and condescending.
Without these metrics, you risk generating content that’s technically accurate but emotionally tone-deaf, or just plain unreadable. It’s like dressing your model in a suit of jargon when the user just wanted a friendly chat. Case in point — the National Eating Disorder Association (NEDA)’s chatbot was removed after it repeatedly gave dangerous and insensitive advice to users with eating disorders, and delivery firm DPD had their chatbot swear at customers and expressed hatred for the company when prompted.
Risk if ignored:
Ignoring expression and readability metrics can hurt your brand voice, confuse users, and erode trust—especially in customer-facing apps. Poorly worded or off-tone responses can lead to higher churn, reduced engagement, and even viral backlash. What sounds robotic or awkward to users can directly translate into lost revenue and reputational damage.
Key metrics:
🧪 Custom Metrics: Tailoring Evaluation to Your Needs
While Galileo offers a robust set of built-in metrics, we understand that every AI application has unique requirements. That's why Galileo supports the creation of custom metrics via LLM-as-a-Judge or code-based scoring, allowing you to define and register your own evaluation criteria.
You can choose between Custom Scorers and Registered Scorers, depending on where, when, and how you want these metrics executed.
Explore the documentation to learn how to get started with your own custom metrics, tailored to your specific needs.
🛠️ Sample Applications: Putting Metrics into Practice
To help you get started, we’ve curated a collection of sample applications demonstrating how to implement and utilize various metrics in real-world scenarios. Check out our GitHub repository: Galileo SDK Examples
Final Thoughts
Choosing the right metrics is foundational to building trustworthy, effective, and user-friendly AI systems. By understanding what each metric category measures and when to use it, you can tailor your evaluation strategy to your specific goals and deliver more effective AI experiences.
Sign up for Galileo to get started on your journey to building reliable AI applications and/or explore the Galileo documentation to dive deeper into each metric and start measuring what matters.
When building AI systems—especially those that interact with users or make decisions—it's crucial to measure performance in ways that align with your goals (whether performance is speed, accuracy, tool selection, or quality). Galileo provides a comprehensive suite of evaluation metrics out of the box, as well as the ability to create custom metrics via LLM-as-a-Judge or code-based scoring. These metrics are designed to answer specific questions about your AI's behavior. But with so many options, how do you know which metrics to use, and when?
This guide walks you through the main categories of metrics, explains when to use each, and provides practical examples to help you make informed decisions.
Choosing the Right Metrics
Start with your goals: What matters most for your use case—accuracy, safety, confidence, progress, or style?
Mix and match: Real-world systems benefit from multiple categories. A support bot, for example, needs both accuracy and safety.
Establish baselines: Know where you're starting before making changes.
Track trends: Monitor changes as you iterate.
Set thresholds: Define "good enough" and build in alerts.
PSST — check out this YouTube video for a recap on how to choose the right metrics and what may be the most helpful for your needs.
🧠 Response Quality Metrics
What they measure:
Response Quality metrics serve as your AI's report card, evaluating how well the model understands and responds to prompts, particularly in terms of factual accuracy, completeness, and adherence to instructions.
They help answer questions like:
“Did the AI answer the question?”
“Did it follow the instructions—or just vibe its way through the prompt?”
“Is it making things up... again?”
They’re especially crucial for use cases where trust and clarity are non-negotiable—think customer service bots, educational tools, or retrieval-augmented generation (RAG) systems. Skip these metrics, and you risk deploying an AI that sounds confident while confidently delivering nonsense (a.k.a. hallucinations).
Risk if ignored:
Response quality issues can lead to incorrect financial advice, misinformed users, or loss of trust. A notable case: Google Bard sharing incorrect information in its launch demo, or even the Chicago Sun-Times publishing an AI-generated Summer Reads list, in which only 5 of the 15 books were real.
Key metrics:
🏡 Safety and Compliance Metrics
What they measure:
Safety and Compliance metrics are the guardian angels of your AI, watching for danger zones like leaked sensitive information, biased or toxic language, and attempts to manipulate your model via prompt injections.
Whether you're working in healthcare, finance, or just trying to keep your chatbot from going rogue on social media, these metrics flag the moments where things could get legally murky, ethically shady, or just plain awkward.
If you don’t implement them, you’re flying without a parachute. You might wake up to headlines you never wanted, audits you weren’t prepared for, or worse, losing user trust in a single bad interaction. Just ask Apple which found that the Apple Credit Card algorithm managed by Goldman Sachs would routinely offer women lower credit limits than men — even when women had higher credit scores.
Risk if ignored:
Ignoring safety and compliance metrics can lead to harmful outputs, legal exposure, and loss of user trust. It only takes one bad response to spark audits, headlines, or regulatory action.
Key metrics:
🎯 Model Confidence Metrics
What they measure:
Model Confidence Metrics about how sure your AI is of itself, and when it’s feeling shaky. They quantify uncertainty in responses and assess prompt complexity (a.k.a. how hard the question is).
They’re a lifesaver when building human-in-the-loop workflows or escalation systems. If your model is unsure, you can route that output to a human reviewer or flag it for deeper analysis. No one wants an AI answering medical queries with “I think maybe...” energy.
Skip these, and you risk presenting low-confidence outputs as facts, which is how misinformation, confused users, or a helpdesk full of angry or flustered users can occur. Take for example, Gmail’s smart reply feature that over-indexed on recommending users to tell someone they love them, or Waze confidently navigating people directly into wildfires.
Risk if ignored:
Ignoring model confidence metrics means your AI might present guesses as facts. This can confuse users, spread misinformation, or trigger unnecessary escalations—especially in high-stakes settings.
Key metrics:
🤖 Agentic Metrics
What they measure:
Agentic metrics track how well your AI agent navigates multi-step tasks, makes decisions, and uses tools. Think of these as performance reviews for your autonomous agents—are they moving toward the goal efficiently, or just poking around aimlessly? Are they choosing the right tools for the job, or breaking everything in sight?
This is vital if you're building AI that interacts with APIs, executes actions, or works autonomously in workflows, like auto-triaging support tickets or writing code.
Without these metrics, debugging becomes a nightmare. You won't know if your agent is failing because it used the wrong tool, misinterpreted the goal, or just decided to vibe. Microsoft recently released a white paper on how AI agents can be subsceptivble to ‘memory poisoning’, leading agents to be manipulated to act against organizational interests.
Risk if ignored:
Ignoring agentic metrics leaves you flying blind when it comes to how your AI agents make decisions, use tools, or complete multi-step tasks. Without them, you can’t tell if failures are due to tool misuse, poor goal understanding, or decision paralysis—making debugging guesswork and scaling risky. This can result in agents that appear functional but waste resources, break workflows, or generate unpredictable results in production.
Key metrics:
✍️ Expression and Readability Metrics
What they measure:
Expression and Readability Metrics measure the ✨vibes✨—aka your AI-generated content's tone, fluency, clarity, and human-likeness.
They're essential for any system where words matter, such as marketing copy, brand voice, educational materials, or any other user-facing content. You want your AI to sound smart, not robotic; clear, and condescending.
Without these metrics, you risk generating content that’s technically accurate but emotionally tone-deaf, or just plain unreadable. It’s like dressing your model in a suit of jargon when the user just wanted a friendly chat. Case in point — the National Eating Disorder Association (NEDA)’s chatbot was removed after it repeatedly gave dangerous and insensitive advice to users with eating disorders, and delivery firm DPD had their chatbot swear at customers and expressed hatred for the company when prompted.
Risk if ignored:
Ignoring expression and readability metrics can hurt your brand voice, confuse users, and erode trust—especially in customer-facing apps. Poorly worded or off-tone responses can lead to higher churn, reduced engagement, and even viral backlash. What sounds robotic or awkward to users can directly translate into lost revenue and reputational damage.
Key metrics:
🧪 Custom Metrics: Tailoring Evaluation to Your Needs
While Galileo offers a robust set of built-in metrics, we understand that every AI application has unique requirements. That's why Galileo supports the creation of custom metrics via LLM-as-a-Judge or code-based scoring, allowing you to define and register your own evaluation criteria.
You can choose between Custom Scorers and Registered Scorers, depending on where, when, and how you want these metrics executed.
Explore the documentation to learn how to get started with your own custom metrics, tailored to your specific needs.
🛠️ Sample Applications: Putting Metrics into Practice
To help you get started, we’ve curated a collection of sample applications demonstrating how to implement and utilize various metrics in real-world scenarios. Check out our GitHub repository: Galileo SDK Examples
Final Thoughts
Choosing the right metrics is foundational to building trustworthy, effective, and user-friendly AI systems. By understanding what each metric category measures and when to use it, you can tailor your evaluation strategy to your specific goals and deliver more effective AI experiences.
Sign up for Galileo to get started on your journey to building reliable AI applications and/or explore the Galileo documentation to dive deeper into each metric and start measuring what matters.
When building AI systems—especially those that interact with users or make decisions—it's crucial to measure performance in ways that align with your goals (whether performance is speed, accuracy, tool selection, or quality). Galileo provides a comprehensive suite of evaluation metrics out of the box, as well as the ability to create custom metrics via LLM-as-a-Judge or code-based scoring. These metrics are designed to answer specific questions about your AI's behavior. But with so many options, how do you know which metrics to use, and when?
This guide walks you through the main categories of metrics, explains when to use each, and provides practical examples to help you make informed decisions.
Choosing the Right Metrics
Start with your goals: What matters most for your use case—accuracy, safety, confidence, progress, or style?
Mix and match: Real-world systems benefit from multiple categories. A support bot, for example, needs both accuracy and safety.
Establish baselines: Know where you're starting before making changes.
Track trends: Monitor changes as you iterate.
Set thresholds: Define "good enough" and build in alerts.
PSST — check out this YouTube video for a recap on how to choose the right metrics and what may be the most helpful for your needs.
🧠 Response Quality Metrics
What they measure:
Response Quality metrics serve as your AI's report card, evaluating how well the model understands and responds to prompts, particularly in terms of factual accuracy, completeness, and adherence to instructions.
They help answer questions like:
“Did the AI answer the question?”
“Did it follow the instructions—or just vibe its way through the prompt?”
“Is it making things up... again?”
They’re especially crucial for use cases where trust and clarity are non-negotiable—think customer service bots, educational tools, or retrieval-augmented generation (RAG) systems. Skip these metrics, and you risk deploying an AI that sounds confident while confidently delivering nonsense (a.k.a. hallucinations).
Risk if ignored:
Response quality issues can lead to incorrect financial advice, misinformed users, or loss of trust. A notable case: Google Bard sharing incorrect information in its launch demo, or even the Chicago Sun-Times publishing an AI-generated Summer Reads list, in which only 5 of the 15 books were real.
Key metrics:
🏡 Safety and Compliance Metrics
What they measure:
Safety and Compliance metrics are the guardian angels of your AI, watching for danger zones like leaked sensitive information, biased or toxic language, and attempts to manipulate your model via prompt injections.
Whether you're working in healthcare, finance, or just trying to keep your chatbot from going rogue on social media, these metrics flag the moments where things could get legally murky, ethically shady, or just plain awkward.
If you don’t implement them, you’re flying without a parachute. You might wake up to headlines you never wanted, audits you weren’t prepared for, or worse, losing user trust in a single bad interaction. Just ask Apple which found that the Apple Credit Card algorithm managed by Goldman Sachs would routinely offer women lower credit limits than men — even when women had higher credit scores.
Risk if ignored:
Ignoring safety and compliance metrics can lead to harmful outputs, legal exposure, and loss of user trust. It only takes one bad response to spark audits, headlines, or regulatory action.
Key metrics:
🎯 Model Confidence Metrics
What they measure:
Model Confidence Metrics about how sure your AI is of itself, and when it’s feeling shaky. They quantify uncertainty in responses and assess prompt complexity (a.k.a. how hard the question is).
They’re a lifesaver when building human-in-the-loop workflows or escalation systems. If your model is unsure, you can route that output to a human reviewer or flag it for deeper analysis. No one wants an AI answering medical queries with “I think maybe...” energy.
Skip these, and you risk presenting low-confidence outputs as facts, which is how misinformation, confused users, or a helpdesk full of angry or flustered users can occur. Take for example, Gmail’s smart reply feature that over-indexed on recommending users to tell someone they love them, or Waze confidently navigating people directly into wildfires.
Risk if ignored:
Ignoring model confidence metrics means your AI might present guesses as facts. This can confuse users, spread misinformation, or trigger unnecessary escalations—especially in high-stakes settings.
Key metrics:
🤖 Agentic Metrics
What they measure:
Agentic metrics track how well your AI agent navigates multi-step tasks, makes decisions, and uses tools. Think of these as performance reviews for your autonomous agents—are they moving toward the goal efficiently, or just poking around aimlessly? Are they choosing the right tools for the job, or breaking everything in sight?
This is vital if you're building AI that interacts with APIs, executes actions, or works autonomously in workflows, like auto-triaging support tickets or writing code.
Without these metrics, debugging becomes a nightmare. You won't know if your agent is failing because it used the wrong tool, misinterpreted the goal, or just decided to vibe. Microsoft recently released a white paper on how AI agents can be subsceptivble to ‘memory poisoning’, leading agents to be manipulated to act against organizational interests.
Risk if ignored:
Ignoring agentic metrics leaves you flying blind when it comes to how your AI agents make decisions, use tools, or complete multi-step tasks. Without them, you can’t tell if failures are due to tool misuse, poor goal understanding, or decision paralysis—making debugging guesswork and scaling risky. This can result in agents that appear functional but waste resources, break workflows, or generate unpredictable results in production.
Key metrics:
✍️ Expression and Readability Metrics
What they measure:
Expression and Readability Metrics measure the ✨vibes✨—aka your AI-generated content's tone, fluency, clarity, and human-likeness.
They're essential for any system where words matter, such as marketing copy, brand voice, educational materials, or any other user-facing content. You want your AI to sound smart, not robotic; clear, and condescending.
Without these metrics, you risk generating content that’s technically accurate but emotionally tone-deaf, or just plain unreadable. It’s like dressing your model in a suit of jargon when the user just wanted a friendly chat. Case in point — the National Eating Disorder Association (NEDA)’s chatbot was removed after it repeatedly gave dangerous and insensitive advice to users with eating disorders, and delivery firm DPD had their chatbot swear at customers and expressed hatred for the company when prompted.
Risk if ignored:
Ignoring expression and readability metrics can hurt your brand voice, confuse users, and erode trust—especially in customer-facing apps. Poorly worded or off-tone responses can lead to higher churn, reduced engagement, and even viral backlash. What sounds robotic or awkward to users can directly translate into lost revenue and reputational damage.
Key metrics:
🧪 Custom Metrics: Tailoring Evaluation to Your Needs
While Galileo offers a robust set of built-in metrics, we understand that every AI application has unique requirements. That's why Galileo supports the creation of custom metrics via LLM-as-a-Judge or code-based scoring, allowing you to define and register your own evaluation criteria.
You can choose between Custom Scorers and Registered Scorers, depending on where, when, and how you want these metrics executed.
Explore the documentation to learn how to get started with your own custom metrics, tailored to your specific needs.
🛠️ Sample Applications: Putting Metrics into Practice
To help you get started, we’ve curated a collection of sample applications demonstrating how to implement and utilize various metrics in real-world scenarios. Check out our GitHub repository: Galileo SDK Examples
Final Thoughts
Choosing the right metrics is foundational to building trustworthy, effective, and user-friendly AI systems. By understanding what each metric category measures and when to use it, you can tailor your evaluation strategy to your specific goals and deliver more effective AI experiences.
Sign up for Galileo to get started on your journey to building reliable AI applications and/or explore the Galileo documentation to dive deeper into each metric and start measuring what matters.