How Kaiya Search Works: Uncovering the Magic Behind Our Supercharged Search

The ability to effortlessly extract insights from complex datasets has become a critical differentiator for businesses. But here at Tellius, we thought, “There’s got to be a better way.” Enter Kaiya Search—our AI-powered search that understands your data.

Now, I know what you’re thinking. “Great, another AI thing that’ll probably just misunderstand me and tell me to reboot.” But hang on a second. Kaiya’s different, and here’s why:

At Tellius, we’ve been tinkering with natural language processing and AI since 2016. (Yes, we’re that kind of nerdy.) We’ve been on a mission to make data work for you, not the other way around. Kaiya actually understands context, learns your business-specific terms, and translates your questions into something databases understand. But as the old saying goes, “With great data comes great complexity” (or something like that).

But how does it really work under the hood? That’s why we’re here to dive deep today. We’re going to break down how Kaiya takes a question like “How’s CoughPill doing compared to other brands last quarter?” and turns it into meaningful insights.

We’re talking about some seriously cool tech—machine learning models, semantic knowledge, indexing strategies, understanding of context—and Kaiya makes it look easy.

Written by: Ramya Priya,

Product Content and Strategy Lead at Tellius

Cracking the code: Why natural language to SQL conversion is challenging

Before we dive into how Kaiya works its magic, let’s talk about the elephant in the room: Converting natural language into SQL queries is hard. Like, really hard. And here’s why:

• The intent isn’t always clear

When you ask, “Show me last quarter’s sales,” do you mean the previous calendar quarter or the last three months? It’s tricky to know exactly what you’re after, and that ambiguity makes it challenging to get you the right information.

• Using implicit information

A request like “Compare iPhone sales to Samsung” might mean that you’re interested in smartphone sales across multiple models, not just a single product.

• Vague questions lead to vague answers

Questions like “Show me top performers” could mean employees, products, or marketing campaigns. It has to determine which “performers” you’re interested in based on context.

• Date formats can be confusing

“Sales since 03/04/22″—is that March 4th or April 3rd? Your locale might change the interpretation, and it might be confusing to fetch accurate data.

• SQL dialects differ across systems

Functions like getting the current date vary between SQL dialects—GETDATE() in SQL Server vs. CURRENT_DATE in PostgreSQL. Queries must be generated that match the specific database you’re using.

• Huge queries can overwhelm your database

A broad request like “List all transactions” could overwhelm your database if not handled properly. Queries need to be optimized to provide the information you need without causing performance issues.

• Misunderstandings lead to errors

“Show data for O’Brien” could break a query due to the apostrophe. Special characters need to be handled safely to prevent SQL injection attacks and ensure accurate results.

• Your business keeps evolving

Your business isn’t static—new product names, metrics, and industry terms pop up all the time. The system needs to adapt to these changes and learn from your interactions to stay relevant.

These challenges make natural language to SQL conversion a tough nut to crack. But that’s exactly where Kaiya shines.

What sets Kaiya apart

So, what makes Kaiya different from other natural language query interfaces? It’s not just another AI tool that gives out canned responses. Kaiya is a sophisticated self-service analytics system that leverages advanced machine learning techniques to bridge the gap between your questions and data-driven insights. We’ve revamped the Kaiya Search interface to be more intuitive and user-friendly, transforming your data queries into a conversational AI-powered search experience.

Kaiya understands what you mean

Kaiya uses deep learning models fine-tuned on domain-specific datasets to accurately capture your AI-powered analytical goals. It understands the nuances in your questions, even when they involve complex, multi-dimensional queries.

It recognizes your key terms

By implementing state-of-the-art algorithms, Kaiya identifies and categorizes key elements within your queries. It knows that “CoughPill” is a specific product and “brands” is a category for comparison.

Context is king

Kaiya employs knowledge graphs to understand relationships between entities and map them to the appropriate data structures. It recognizes temporal expressions like “last quarter” and translates them into the correct date range in your data.

It generates precise queries for you

Using advanced techniques, Kaiya constructs precise SQL queries that accurately reflect your intent. It can join necessary tables, apply correct filters, and calculate the required metrics—all without you writing a single line of code.

Complexity doesn’t scare Kaiya

Kaiya goes beyond simple keyword matching. It can handle queries that involve multiple conditions, aggregations, or nested structures, providing you with meaningful insights.

Smart embedding strategies accelerate access

This is Kaiya’s recipe for rapid data access. It uses advanced indexing methods, including column, table, and bitmap indexes, alongside vector stores. These strategies enable quick retrieval and processing of relevant data.

It adapts as your data changes

Kaiya’s dynamic schema mapping adapts to changes in your data structure. If your database evolves, Kaiya can quickly re-index the new schema, ensuring minimal downtime or loss of accuracy.

Flexibility with your data infrastructure

Kaiya is bilingual in data warehouses and data lakehouses, making it a bridge between your historical data analysis and future goals.

Delivering high accuracy

All these features come together to make Kaiya a powerhouse in delivering accurate and precise answers, adapting to various business needs and data types. In fact, Kaiya leverages the power of advanced semantic models and continuous user learning to achieve over 90% accuracy on real-world complex use cases. This means you receive reliable insights every time, empowering you to make informed decisions.

The anatomy of a Kaiya query: A behind-the-scenes peek

Alright, time to lift the hood and see how Kaiya works its magic. Let’s walk through how Kaiya processes a typical business question like, “How is CoughPill performing compared to other brands last quarter?“

How Kaiya Search Works

1. You ask a question

This is where you push the button to set Kaiya into action. You type your natural language query into our AI-powered search.

2. Kaiya prepares to understand

Kaiya starts by putting your query through several crucial steps:

• Validation: It ensures your query makes sense. If you ask something like, “How many Ronaldo fans bought our product?,” Kaiya might politely inform you that unless you have data for that, it can’t help (but nice try!).
• Intent detection: Kaiya figures out what you’re really after. In this case, it’s a performance comparison.
• Entity extraction: It identifies the key players—”CoughPill,” “brands,” and “last quarter.”
• Schema retrieval: Kaiya matches your query to the relevant database and tables. This ensures it knows exactly where to look in the database.
• Breaking down your words: Kaiya tokenizes your query into smaller, digestible chunks and converts them into numerical representations called embeddings that capture the semantic meaning.

3. Finding the right information quickly

Kaiya compares the embeddings of your query against pre-existing embeddings stored in a vector database. This quickly identifies the most similar items by measuring how close these numerical representations are to each other, enabling the efficient retrieval of relevant content, even across very large datasets.

4. Leveraging advanced language models

This is where Kaiya gets supercharged with large language models (LLMs) like GPT. It sends your processed query to an LLM for deep natural language understanding. It’s not just looking at keywords; it’s grasping context, nuance, and intent.

5. Double-checking and refining

Kaiya doesn’t just trust the LLM blindly (it’s smart, not gullible). It goes through a few more steps:

• Validating interpretation: It checks to ensure the LLM’s understanding aligns with your intent.
• Adding missing pieces: Kaiya enriches the response by adding any missing fields, metadata, or contextual parameters.
• Turning it into SQL: It converts the LLM’s output into an actual database query (SQL, in most cases).

6. Getting the data

The Tellius engine is a powerhouse that handles the nitty-gritty of query execution. It determines the most efficient way to retrieve the requested data.

Kaiya’s query optimization engine balances speed and accuracy, choosing between cached results and live data queries as required. It offers two modes and executes the query across the relevant data sources.

• Fast query engine: For rapid responses using cached or pre-processed data.
• Live mode: For accessing real-time data, ensuring you get the most up-to-date information.

7. Presenting the results

Finally, Kaiya presents the results visually in the best-fit chart because, let’s face it, a wall of numbers is about as exciting as watching paint dry.

Kaiya learns from you: Tailored to your business

When it comes to AI-powered analytics, one size doesn’t fit all. That’s why Kaiya lets you teach to understand the unique language of your business. Through its learning and feedback loop, Kaiya becomes smarter and more accurate with every interaction, helping you get the most relevant and precise answers over time.

Customize how Kaiya understands your world

Kaiya’s learning system allows you to customize how it interprets your specific terms and queries at different levels:

• System-level: Define terms that apply across all business views in your instance.
• Domain-level: Train Kaiya to understand industry-specific language unique to your field.
• Business view-level: Teach Kaiya how to respond to specific questions within a business view.

This modular approach lets you update definitions and meanings without disrupting core functionality or processing speed. By separating learning from the core prompts, Kaiya ensures the flexibility to refine knowledge as your business evolves.

Making your organization smarter over time

Kaiya encourages you to provide feedback on the results of your queries. This feedback loop means Kaiya continually learns from you, fine-tuning its results to give you more accurate answers the next time you search, and in turn, your organization gains sharper insights and a deeper understanding of your data.

• If a query delivers unsatisfactory results, you can click the thumbs-down button, prompting you to refine the query. This mechanism allows you to map keywords to datapoints that will be remembered for future interactions.
• On the flip side, when Kaiya gets it right, a thumbs-up ensures it remembers your preferences, making it smarter with each interaction. 

Keeping it real: How Kaiya fights against hallucinations

Let’s face it: AI models can sometimes “hallucinate,” generating answers that sound plausible but are actually false. Kaiya has built-in systems to keep these inaccuracies in check, so you can trust the information it provides.

Kaiya double-checks itself

Kaiya employs self-checking mechanisms. It generates multiple responses to the same query and compares them. If the answers are consistent, that’s a good sign. If they’re all over the place, Kaiya flags it as a potential hallucination and adjusts accordingly.

It ensures proper formats

Using tools like Pydantic Output Parser, Kaiya ensures that answers follow a specific format, especially when dealing with structured outputs like JSON. If there’s a formatting mistake, the parser catches and corrects it.

It stands up to tricky questions

Kaiya is put through tough tests to see how it handles tricky or misleading queries. It’s designed to recognize and block harmful or unethical information requests.

• Protecting sensitive data: Prevents revealing private or confidential information, such as API keys, even if prompted.
• Preventing exploits: Shuts down unethical or dangerous prompts, such as asking for illegal or harmful information.
• Maintaining safety protocols: Handles unusual inputs without bypassing safety measures.

Advanced evaluation keeps it accurate

Kaiya uses advanced evaluation techniques like BERTScore and ROUGE. BERTScore checks if the AI’s answers make sense semantically, while ROUGE focuses on exact word matches. This helps Kaiya stay accurate, even with more nuanced questions.

Grounding responses in reality

To keep responses grounded, Kaiya uses a retrieval-augmented generation (RAG) approach. Instead of making things up, it pulls from real, reliable documents to build its answers, ensuring you get accurate, up-to-date information.

• Context relevance: Ensures the response directly addresses your question.
• Groundedness: Confirms the response is based on verifiable information, reducing the likelihood of fabricated content.
• Answer relevance: Evaluates whether the response is contextually appropriate.

Your feedback helps keep it real

Kaiya goes beyond internal mechanisms by integrating your feedback into its hallucination detection process. When you report inaccuracies through feedback tools (like giving a thumbs-down for incorrect responses), Kaiya updates its context embeddings and knowledge graphs, enhancing future responses and minimizing the risk of hallucinations.

Kaiya vs. generic LLMs: Why specialized AI makes the difference

You might be wondering, “Why should I use Kaiya when ChatGPT can generate data insights?” Great question. Let’s break it down.

Accuracy matters

• LLMs: While LLMs are generally good at understanding language, they might occasionally mix up terms or provide generic answers. They don’t have direct access to your specific datasets unless you provide all the context every time.
• Kaiya Search: Kaiya connects directly to your data sources, ensuring that the insights you get are accurate and up-to-date. It knows that when you say “revenue,” you mean your company’s revenue, not some generalized concept.

Tailored to your business

• LLMs: These models are trained on a wide range of internet data. They’re jacks-of-all-trades but masters of none when it comes to your specific business needs.
• Kaiya Search: Kaiya is tailor-made for your business. It understands your unique terminology, datasets, and industry nuances.

Seamless integration with your systems

• LLMs: They require manual data input and don’t integrate with your existing data infrastructure out of the box.
• Kaiya Search: Kaiya seamlessly integrates with your data warehouses and lakehouses. It knows where your data lives and how to access it efficiently.

Continuous learning specific to you

• LLMs: They don’t learn from your interactions in a way that improves their understanding of your specific business over time.
• Kaiya Search: With its feedback loop, Kaiya learns from every interaction you have. It becomes more attuned to your preferences, terminology, and needs, improving accuracy and relevance over time.

Visualization and actionable insights

• LLMs: They can generate text-based answers but often lack the ability to provide the perfect visualizations.
• Kaiya Search: Kaiya doesn’t just give you answers; it presents them visually in the best-fit charts and dashboards with digestible summaries, making insights immediately actionable.

Limited awareness of your data

• LLMs: GPT-4o operates based on its training data up to October 2023. It doesn’t have access to your real-time enterprise data.
• Kaiya Search: Kaiya connects securely to your enterprise data sources, accessing real-time information across multiple systems. It adapts to your unique data structures and evolves with your business, ensuring consistent accuracy.

Speaks your business language

• LLMs: They may not recognize industry-specific terms, abbreviations, or metrics unique to your business.
• Kaiya Search: Kaiya recognizes domain-specific terms, abbreviations, and metrics, mapping them accurately to your data schema.

Built-in data prep

• LLMs: They can’t assist with preparing or cleaning your data for analysis.
• Kaiya Search: With Tellius, you can import datasets and apply/automate data preparation and cleaning operations. This ensures your data is accurate, consistent, and ready for any type of search functionality and in-depth drill-down analysis.

Real-world queries that stump GPT-4o but not Kaiya Search

• Can you display Xycoran’s monthly calls and sales, focusing on CA, since 2023?
• Can you compare the regional performance of new prescriptions for Xycoran and Ventrixa over the past three months?
• Which ten states have had the highest number of prescriptions and break it down by product?
• Could you reveal the Subs Profit and GMV for refurbished items in California by category division and seller subset for 2023?
• What are the percentage changes in shipment quantity for this year compared to last year for California, with breakdown by Category?

Unlike generic LLMs that might excel at generating data or crafting queries in a vacuum but stumble when faced with your actual datasets, Kaiya is designed to work with the complexity of real-world enterprise data. We’re talking about datasets with hundreds of columns—not all of which have straightforward names—and datasets that can easily contain over billions of rows.

Generic LLMs often struggle in these scenarios because they can’t handle the specific nuances and scale of your data without extensive customization. Kaiya, on the other hand, thrives in this environment. It combines the strengths of advanced language models with deep integration into your data ecosystem, providing a level of precision and relevance that generic models can’t match.

Why Kaiya Search is more than just another search interface

Our secret sauce is a proprietary blend of deep data understanding, advanced LLM reasoning, and tailored workflows that combine to create highly contextual, accurate, enterprise-grade AI-powered analytics for your data. Kaiya securely connects to all of your organization’s data—across multiple systems and sources—from sales and finance to marketing campaigns and customer behavior. It builds a comprehensive data ontology and knowledge graph that considers the intent behind your key business questions.

It’s the difference between asking, “What were our sales last quarter?” and “How did our product launches impact customer sentiment across different market segments, and how does this correlate with our sales performance?” Kaiya can handle both with equal ease, making it possible for everyone in your organization—from data analysts to business users to the CEO—to get meaningful insights from your data.

So the next time you have a burning question about your data, you know who to ask. Give Kaiya a try and see how effortless data analysis can be.