Artificial agents, applications, and bots are popping up across every enterprise landscape, and data is rolling through at a fast and furious pace.
While there has been plenty of hype, excitement, and fear about AI’s possibilities, there has been scant attention regarding the data infrastructure needed to make it all work—especially at the enterprise level. AI requires a strong and well-considered data foundation, whether it’s extended out of legacy infrastructures or part of the next generation of data technology.
“This trend is going to continue in almost every industry, and as AI moves from experimentation to core business operations, the pressure on the data layer also intensifies,” said Karthik Ranganathan, co-CEO and co-founder of Yugabyte. To explore the shape of data foundation in the AI age, we canvassed experts and leaders across the industry who weighed in on the thinking and work required to build a well-functioning data environment responsive to today’s and tomorrow’s enterprise AI initiatives. Remember, it’s all about the data.
ARE WE READY?
There is general agreement among data leaders and experts speaking to BDQ that today’s data infrastructures are not ready for the demands of AI. “Enterprises can power impressive AI demos, but underneath there is still disordered data, identity sprawl, and fragmented platforms,” said Mark Gowdy, chief partner technologist at Quest Software.
Most enterprise data infrastructures “have been built and deployed during the analytics era,” according to Vikram Venkat, principal at Cota Capital. “Most data was structured, row-column-optimized tables were the predominant format, and batch processing was sufficient for most large-scale requirements.”
What’s missing for most enterprises is “a clear, consolidated, well-defined data model,” said Cole Bowden, developer advocate at InfluxData. “When you’re trying to use AI to leverage your data, it will explore your tables, your schemas, your column names, and it has to make inferences about how all of the data is related and how plain-text concepts relate to all of the data you have.”
Along these lines, “any column named ‘col1’ is functionally invisible and useless to any AI agent,” Bowden illustrated. “If you have multiple similarly named columns that contain different data or different concepts, AI will struggle heavily. Your data model is your documentation for AI, and most data models are not clean enough to power AI fully.”
The result is an “AI that can’t be fully trusted, traced, or secured in production,” Gowdy pointed out. “A small minority of enterprises are closing that gap, but only where they’ve deliberately invested in finding, understanding, and governing their data while consolidating onto modern platforms with strong identity at the center. Until companies fix that housekeeping, it’s akin to dumping a junk drawer of random parts and information into a scalable environment and calling it AI-ready.”
While creating AI-powered environments gets easier every year, the underlying data architecture can make or break an AI application. This is where things get complicated. “Building on top of existing traditional databases becomes painfully complex,” said Ranganathan. “It requires multiple databases, each supporting a different data model—typically SQL, vector, graph, search, and time/series. Hacking all these databases together creates a huge problem of data silos, which drastically slows down the development, leads to poor performance, and a lack of observability into the system.”
Chetas Joshi, a software engineer at Robinhood, said he has seen firsthand “that most data infrastructure is unprepared to support AI and machine-learning use cases.” AI systems—especially large language model (LLM)-powered applications, real-time decisioning systems, and copilots—“bring very different requirements,” he explained. “Use cases like fraud detection, personalization, recommendations, and RAG [retrieval-augmented generation] rely on continuous streams of high-volume, often unstructured data. They need fresh signals—high-throughput, low-latency data-processing requirements—and the right context.”
The gap between AI capability and enterprise readiness “is less about hardware and more about hygiene,” advised Anusha Kovi, business intelligence engineer with Amazon. “Most enterprise data environments were built to store and report, not to feed models. That means the data is siloed, inconsistently labeled, poorly documented, and not structured in a way that makes it usable for AI without significant cleanup work first. The infrastructure exists, but the foundation underneath it was never designed with this use case in mind.”
Practical challenges also need to be considered—“managing costs as data volumes grow, keeping data fresh enough for real-time use cases, handling model drift, and maintaining governance across distributed systems,” Joshi added. “Supporting AI at scale requires high-throughput streaming ingestion, online serving layers backed by intelligent caching layer, vector indexing, and reliable object storage all working together.”
“Enterprise data is distributed across many systems without a uniform means of access,” said Pascal Van Hentenryck, A. Russell Chandler III Chair and professor at Georgia Technology and recently appointed head of the Gurobi AI Innovation Lab. “Security, privacy, and the need to map roles and users for access control purposes collectively create another layer of complexity. So, the challenge is to move from data to workflows—moving from data infrastructures to workflow infrastructures. It creates a paradigm shift.”
APPLICATIONS FOR AI
One thing is certain: AI-based applications are hungry for data—and lots of it. “From forecasting to optimization to generative and agentic solutions, AI is data-driven and thus data-hungry by definition,” said Van Hentenryck. At this point, AI is showing up in “real-time decisioning, fraud detection, personalization, and operational analytics,” said Gowdy. “You also see LLM-based copilots and assistants sitting on top of warehouses and lakehouses, answering questions over documents, logs, and reports, plus heavy regulatory and risk analytics that blend structured and unstructured data.”
LLMs are being used to “power better decision-making through more informed analytics, while agentic AI enables the automation of end-to-end processes to increase speed and efficiency,” said David Thoumas, co-founder and CTO of Huwise (previously called Opendatasoft). “Both of these rely on access to trustworthy data that is reliable, understandable, and provides context and consistency.”
AI is being drawn toward the richest, most established datasets. These consist of “data warehouses and back-office systems like CRMs that have grown up alongside the organization itself,” according to Graham McMillan, CTO of Redgate. “These contain the messiest but most valuable data an enterprise holds, and that’s precisely why people want to deploy AI against them.”
AI is being employed against datasets, “spot trends in customer behavior, identify churn before it happens, and find cross-sell opportunities hiding in plain sight,” McMillan said. Additional use cases include “real-time fraud prevention, personalized digital banking, telecom network optimization, logistics coordination, ecommerce recommendation engines, and intelligent retail pricing,” said Ranganathan. For instance, an example is “an AI-driven shopping concierge on an ecommerce site that may depend on past transactional data, website behavior, locale information, and current conversation context to make accurate decisions instantly, often across globally distributed environments.”
SCALING CHALLENGES
AI requires moving from static data to continuous streams of data, which calls for “very different computing infrastructures than other solutions,” said Van Hentenryck. “Most enterprises are not yet technologically ready for this, although many options exist at this point.” In addition, enterprises need skilled talent to build and maintain these infrastructures. “Scaling AI doesn’t mean you have to fix everything up front, but identifying which databases are in good shape, fixing those that aren’t, and then maintaining them consistently going forward is critical. Without that foundation, expanding AI initiatives introduces risk rather than value,” McMillan said.
Even with the help of AI to write queries or a user interface to select options from, “a human user isn’t going to make requests or leverage your infrastructure more than a few times a minute,” said Bowden. “An AI agent can run 10 queries in a second, hundreds per minute, and more agents can be working in parallel than humans. Infrastructure that worked fine for a few humans working at human pace is going to be woefully underequipped to handle dozens of agents retrieving data out of the database every second of every day.”
Data governance also needs the full attention of data leaders and their business counterparts. “As data volumes grow and more systems get connected, the question of who owns what data, what’s allowed to be used for what purpose, and how you audit that becomes genuinely hard,” said Kovi. “Latency is another issue. Real-time AI applications need data that is current, not hours old. Most enterprise pipelines weren’t built for that speed.”
What is needed is “a ground-up redesigned data layer that brings together data across the inference layer, memory layer, and database layer,” said Ranganathan. Applications need to be able to “scale and survive any downtimes and failures.” Importantly, all of this must scale, Ranganathan emphasized. “If every new feature requires you to spin up a specialized database, development slows down, operations become complicated, maintenance costs increase, and the data layer becomes a silent productivity killer.”
To address these problems, data managers “need to radically simplify your stack by converging data capabilities into a single, scalable system,” said Ranganathan. “Instead of managing traditional databases, vector stores, graph systems, scalable key-value stores, and search engines all separately, you need to unify them within a single familiar, PostgreSQL-compatible infrastructure.” The bottom line is that “data needs to be made available to AI and human users,” Thoumas said. “Companies need to move beyond simply cataloging their data to making it accessible and consumable through self-service data product marketplaces.”
ESSENTIAL INGREDIENTS
Successful AI applications sit on top of “a complicated stack of databases, including relational databases, vector stores, graph databases, RAG pipelines, and indexing systems,” said Ranganathan. “Each component often runs as a separate system and must be deployed, scaled, secured, and monitored, adding additional layers of operational complexity.”
To address this complexity, consider the following measures:
- Work toward greater cohesion across all data in the enterprise—“Structured and unstructured multimodal data live in separate systems, and metadata is incomplete or stale,” said Joshi. “A strong semantic layer—capturing business definitions, lineage, and relationships—becomes critical. And it can’t be static. It needs to evolve through feedback loops, human input, and model interactions so that the system’s understanding of the business improves over time. For LLM-powered systems, metadata and context are part of the reasoning process itself.”
AI-ready database infrastructure “should be adept at handling both structured and unstructured data,” according to Venkat. “It should include polyglot persistence by default, enabling efficient normalization across batch and streaming data from relational, graph, vector, and other databases.”
- Seek enhanced support for AI workloads—AI workloads present new challenges and requirements for data sites. This starts with “coordinating heterogeneous, accelerator-driven workloads,” said Jaikumar Ganesh, head of engineering at Anyscale. “Modern AI pipelines combine CPU-bound preprocessing, GPU-bound training and inference, and, increasingly, reinforcement learning or evaluation loops. These stages have different scaling characteristics and resource requirements. Pipelines often fan out into thousands of fine-grained tasks connected by data and control dependencies.”
- Implement monitoring and observability for greater transparency—The most critical technologies to AI “are those that give you visibility and control over your data estate as it evolves,” McMillan said. “You can’t govern what you can’t see, so monitoring and observability tools are foundational. They allow teams to understand how their data environments are actually behaving, spot anomalies, and identify issues before they affect downstream systems or, worse, before AI starts producing outputs based on data that’s quietly degraded. As environments grow larger and more complex, that early warning capability becomes increasingly important.”
Data catalogs and observability tools are also key ingredients to an AI-ready data foundation. “Vector databases are becoming essential for anything involving unstructured data or language models,” said Kovi. “And honestly, solid data pipelines with real monitoring matter more than whatever the newest platform is. The boring infrastructure stuff is what actually determines whether AI projects work or fail.”
- Build a data layer—The long-term success of AI, Ranganathan said, “depends on consolidating and strengthening the data layer first, starting with a unified data foundation for AI that supports a distributed architecture for resilience, high availability, and seamless scaling.”
- Look to AI to better manage AI—While AI taxes data infrastructures, it also provides a solution. Today’s data infrastructures “are under constant pressure. They’re being continuously updated with new operational data flowing in from the transactional platforms, SQL Server, Oracle, MySQL, PostgreSQL, which keep the business running day to day,” McMillan said. “Keeping that data reliable, well-governed, and accessible only to those with the right to see it is not a small task, but it’s one that AI is increasingly being applied to help manage.”
AI usage in database management “rose from 15% to 44% in a single year within a survey conducted by Redgate,” McMillan relayed. “This steep change … reflects just how quickly organizations are recognizing that managing the data for AI is as important as deploying AI on the data.”
