Enhancing Agent Retrieval with Structured Chart Extraction

Enhancing Agent Retrieval with Structured Chart Extraction | Databricks Blog Skip to main content

The Motivation More and more enterprises are now asking agents to work with their proprietary documents and answer questions about their contents. However, much of the important information lives inside figures and charts. Many customers have been finding that agents struggle to answer questions that require reading and counting values in charts. For agents to work reliably in diverse enterprise settings, we need to make charts more interpretable. How can we make a chart easier for agents to understand? We ran a simple, quick test: we asked different agents, “How many local maxima are on this chart?” Below is a comparison of a frontier agent and Databricks Genie at answering this question. The frontier agent was passed just the image, spent 50 seconds reasoning, but still got an incorrect answer of 17. Meanwhile, Databricks Genie used a structured extraction of the chart through ai_parse_document, and got the correct answer 18.

Response from a frontier agent with just the image (incorrect): Response from Genie Agent with a structured extraction of the chart (correct):

We noticed these shortcomings in our OfficeQA Pro benchmark , where models performed worse on chart-based and multimodal questions than on questions that did not require chart understanding. We see the same gaps in customers’ information retrieval systems, particularly in financial services. A text-based retrieval system can only search the text space. A common solution is to generate a caption to describe what a chart is about; however, that may miss the data needed for fine-grained questions on the numbers inside the chart. As a result, the system may retrieve the wrong page, or retrieve the right page without having enough information to answer the question. In this post, we show that structured extraction from charts improves both retrieval and answer quality on chart-based questions. We evaluate our approach on two datasets: a chart-heavy subset of ViDoRe V3, a benchmark for retrieval and question answering over visually rich documents, and a synthetic chart-focused dataset we call Chart-RAG. Our approach performs competitively with large single-vector and multi-vector multimodal embedding models.

Figure 1: Answer accuracy for description-only parsing, enriched ai_parse_document with the top three retrieved images, and the strongest multimodal embedding baseline when the top five retrieved pages are used for answering. Results are averaged across three runs.

We build a chart-aware retrieval pipeline end-to-end with Databricks’s AI functions , a set of composable functions that are optimized with state-of-the-art research techniques (see Figure 2 ). We used ai_parse_document to extract document content, including charts represented as structured JSON, and ai_prep_search to transform the content into retrieval-ready chunks, indexed with a lightweight 300M-parameter text embedding model. We created an index from the chunks using ai_search and connected the index to Genie for retrieval and answering.

Figure 2: Chart extraction and retrieval pipeline implemented using Databricks AI functions.

Evaluation Methodology We compared two indexes, created using a 300-million parameter BGE text embedding model, built from the same source PDFs, differing only in chart figure representation: Description-only (baseline): figures represented by captions only JSON-enriched: chart figures represented by captions and structured chart JSON, embedded inline in the figure's chunk.

An example chart extraction:

Grouped bar chart comparing five economic forecast scenarios for GDP growth and headline inflation in 2026 and 2027.

We evaluated 310 chart and infographic-heavy questions from the ViDoRe V3 benchmark. The benchmark evaluates retrieval and answering across seven domains: employment, energy, pharmaceuticals, physics, finance, computer science, and industrial documents. For each experiment, we parsed and chunked the entire 16K-page English corpus and generated answers to every query, searching over the full index. Although chart-focused questions were selected, many could still be answered using the surrounding text. To isolate the impact of the chart content, we created a second benchmark that focused only on chart-based questions created from three complex, chart-heavy reports ( BIS Quarterly Review , IMF World Economic Outlook , J.P. Morgan Long-Term Capital Market Assumptions ). We wrote 114 visually grounded questions from these 3 documents totaling 378 pages to build the synthetic Chart-RAG dataset. Grading: We scored each answer as Correct / Partially Correct / Incorrect using an LLM judge (gemini-3-flash) against its gold answer. Retrieval: We report Hit Rate@10 and nDCG@10. Hit Rate@10 checks whether at least one gold page appears in the top 10 retrieved results. The questions from the ViDoRe benchmark may have multiple gold pages, each with a relevance score of 1 or 2. For Hit Rate@10, we convert graded relevance to binary relevance by allowing pages with either score to count as a hit. For nDCG@10, we keep the original graded relevance. In the Chart-RAG dataset, each query has one gold page. To achieve stable measurement, we ran each configuration three times and report the results with confidence intervals. Structured Chart Data Improves Retrieval and Answering

Figure 3: Answer correctness, Hit@10, and nDCG@10 for description-only and Chart-JSON-enriched retrieval across ViDoRe V3 subset and synthetic Chart-RAG datasets. Results are averaged across three runs.

Structured Chart JSON improves both answer quality and retrieval across both datasets. The question-level analysis below shows when corrected answers coincide with better retrieval.

Figure 4: Adding chart JSON corrected answers that were previously incorrect. Amongst those corrected answers, the figure shows how often JSON also improved retrieval (Hit@10) or ranking (nDCG@10). Results are averaged across the three runs.

The following example from the Chart-RAG dataset demonstrates how JSON representations improve retrieval and answer quality: Question Answer Before Answer with chart-JSON enriched ai_parse_document Roughly what peak level (% pts) did the Oil VIX reach toward Q1 2026? Refers to the following chart:

"I cannot find specific information about the exact peak level of the Oil VIX in Q1 2026 in the provided search results...."...