🎙️ Better Tech Podcast Episode: Balancing Automation and Human Expertise in Data Operations
Host: Peggy S.
Guest: Audrey Smith, COO of MLtwist
Introduction
Peggy:
Hello and welcome back to another episode of Better Tech! I’m Peggy S., and today we’ll be diving into a crucial topic: balancing automation with human expertise in data operations. I’m thrilled to be joined by Audrey Smith, the Chief Operating Officer of MLtwist.
Audrey:
Hi Peggy, thank you for having me. I’m really excited to be here.
Career Journey
Peggy:
Let’s start with your background. Can you tell us how you became COO of MLtwist?
Audrey:
Sure! My career path into data operations has been quite unconventional. I began as an in-house lawyer in France and later moved to the UK for six years. When I relocated to the U.S. a decade ago, I wanted to pivot into tech. I essentially restarted my career and stumbled into the world of AI data operations while working on a voice recognition app. As a native French speaker, I reviewed transcriptions for quality, and that’s when I discovered data labeling and its importance in AI model performance.
I later joined Google as a contractor focusing on quality control for projects like Google Shopping and YouTube. Then I spent four years at Amazon, primarily working on visual search data operations across different data types. This is where I learned to manage data quality from raw collection to final annotation.
After Amazon, I joined Labelbox as the first hire in data operations, building a team and helping customers navigate their labeling needs. Two years ago, I became COO at MLtwist, where we focus on automating what can be automated in data operations while ensuring humans stay in the loop for quality assurance.
On Balancing Automation and Human Input
Peggy:
What do you see as the key to balancing automation with human input in data workflows?
Audrey:
There are three key factors to balance: data quality, timeline, and budget. You need to decide what level of accuracy is acceptable given your constraints. For example, some projects prioritize speed over perfection, while others can afford to aim for near-perfect quality.
No matter how advanced your tools are, I firmly believe human input should always remain part of the process. Tools can’t catch bias, contextual nuances, or cultural subtleties the way a human can. Responsible AI requires that we keep people in the loop—especially when tackling challenges like fairness and data sourcing bias.
Real-World Lessons
Peggy:
Can you share an example where this balance really made a difference?
Audrey:
Definitely. One standout case involved drone-recorded video data. Due to drone instability—wind, piloting variation—each frame can differ significantly. Pre-labeling and interpolation tools don’t work well here; they can actually increase costs and errors. We found that fully manual labeling was more efficient and accurate for this type of data. This experience reinforced the importance of evaluating tools based on each use case rather than adopting them blindly.
Staying Relevant in Data Operations
Peggy:
With AI tools evolving so fast, what should professionals do to stay current?
Audrey:
First, keep up with technology trends. New data labeling tools appear constantly, each offering different capabilities. Try them out on a sample dataset and assess their performance—many offer free trials.
Second, be cautious about hype. There’s a big gap between flashy LinkedIn posts and what actually works in real-world AI deployments. Companies are becoming more thoughtful about ROI before investing in new tech.
Finally, understand that responsible AI is not just a buzzword. It’s about being mindful of your tooling, workforce diversity, and compliance—especially in sensitive fields like healthcare.
MLtwist’s Approach
Peggy:
How does MLtwist support data scientists and streamline this process?
Audrey:
Our mission is to eliminate the janitorial work that takes up 80% of a data scientist’s time. We automate data intake, reformatting, and tool integration so that scientists can focus on model development.
Our platform connects directly to a customer’s cloud storage, selects the best tool for the use case, reformats the data as needed, and handles human-in-the-loop labeling. Once labeled, we reformat it again for model training. It’s all about reducing friction and maximizing consistency.
Responsible AI and Data Cards
Peggy:
You’ve mentioned responsible AI—how is MLtwist approaching that?
Audrey:
We’ve introduced “data cards,” which act like IDs for datasets. They track where data came from, which tools were used, and who labeled it. For example, Stanford’s AI team used our data cards in a research project and cited them in their publication.
The next step is disclosing more metadata—like annotator demographics—to help customers identify potential biases. If your dataset is labeled only by young men, for instance, that can skew model behavior. Our goal is transparency so AI systems better reflect the people they serve.
Regulation and Compliance
Peggy:
How are you addressing regulatory frameworks like the EU AI Act or the U.S. Executive Order on AI?
Audrey:
Even though we don’t have European clients yet, we’re preparing for it. We’re fully aligned with compliance requirements and data sovereignty principles. Some customers insist their data stay within the EU, and we respect that. Transparency and quality assurance are embedded in everything we do.
Quality Assurance Systems
Peggy:
How do you ensure high data quality?
Audrey:
We’ve built QA features into our platform. For example, when 90% of a dataset is labeled one way but the last 10% deviates, we flag it for human review—not necessarily because it’s wrong, but because it needs a second look.
We’ve also developed tools to help with video QA, where we reformat labeled footage into movie clips that customers can quickly review. This is especially helpful when the labeling tool’s interface lags or isn’t optimized for large files. These small features significantly boost final dataset quality.
Final Thoughts
Peggy:
Any final advice for our listeners?
Audrey:
Yes—always balance data quality, time, and budget. It’s a dance. Also, I co-founded a LinkedIn group called Data Ops for AI. We’re about 250 members strong and growing. It’s a great place for professionals to exchange tools, ideas, and best practices—especially for emerging areas like 3D annotation.
Peggy:
Fantastic. Thank you, Audrey, for sharing your expertise. I’m sure our listeners have gained valuable insights. And to everyone tuning in, we’ll see you next time at the crossroads of technology and innovation!