In April 2025, MLtwist and Vectara hosted the AI Data Readiness Forum—a focused, in-person event in Palo Alto that brought together leaders at the forefront of generative AI data strategy.
In this panel, executives from Overstock, Adobe, and MLtwist share how their teams are tackling the messy realities of AI data preparation, from building scalable pipelines to managing RAG sprawl and aligning infrastructure with business goals. Whether you’re refining your data stack or just starting your AI journey, the insights shared here offer a grounded look at what it really takes to operationalize AI today.
Title: AI Data Readiness Forum: AI Data Preparation Insights
Hosts: Vectara / MLtwist
Ravi Shankar, Manager of Machine Learning, Overstock
Aditya Gupta, Machine Learning Engineer, Google
Audrey Smith, COO, MLtwist
This session centers on the evolving challenges and strategies in evaluating and labeling data for machine learning (ML) systems, particularly in the context of generative AI, foundation models, and agentic workflows. The panel explores synthetic data, KPI trade-offs, labeling automation, and the complexities of delivering high-quality, scalable data in real-world applications. Topics include:
– The rise of synthetic data for training and evaluation.
– The shift from manual to AI-assisted data labeling.
– Balancing quality, cost, and timeline in large-scale ML projects.
– Best practices in evaluation frameworks and pipeline scalability.
– Creating useful evaluation (eval) sets requires tight alignment with use cases and customer needs.
– Enterprises now need comprehensive, scalable evals to support complex agentic workflows.
– Maintenance and iteration of these evals are crucial as benchmarks and models evolve.
Data Quality: Always the top priority.
Cost: Impacts volume and feasibility of evaluations.
Timeline: Influences how thorough evaluations can be, especially under production deadlines.
Synthetic data helps fill gaps in real-world datasets, especially for rare events, underrepresented languages, or instruction tuning.
Used heavily in RAG (retrieval-augmented generation) systems to avoid hallucinations and improve alignment.
Challenges include data balancing and engineering risks—overrepresentation of edge cases can lead to distorted model behavior.
– Most teams now use hybrid approaches: AI-assisted labeling plus human validation.
– Tools and models have improved, but complex or noisy data (e.g., drone videos, geospatial imagery) still require human-first approaches.
– Common pitfalls include overestimating model reliability and underinvesting in instruction quality for human labelers.
The session concluded with a Q&A that emphasized:
– There’s no “set-and-forget” solution—labeling and eval remain iterative, judgment-driven processes.
– Synthetic and AI-assisted methods are powerful but not yet replacements for human oversight.
– High-quality data starts with clear instructions, domain understanding, and strong feedback loops.
– The future points toward more automation, but for now, expert involvement is critical—especially in high-stakes domains.
The Ultimate Guide to AI Data Pipelines: Learn how to Build, Maintain and Update your pipes for your unstructured data
