Raw data, cleaned and ready to label

Raw data rarely arrives ready to label. It comes from different teams, devices, and vendors, in mixed formats, with duplicates, corrupt files, and naming that made sense to whoever created it. The details that give a file meaning — a sensor reading, a capture location, a prior label — usually sit in another system. Before anyone labels anything, someone has to find the files, clean them, convert them, and match them with their context. That work is slow, it’s easy to get wrong, and it’s rarely anyone’s full-time job.

Good preparation does three things. It removes what shouldn’t be labeled, so budget goes to files that will improve the model. It puts every file in a form your tools can open, at a size and format that fits the work. And it brings context along, so annotators and reviewers see the whole picture instead of guessing. Done well, it also repeats: the next batch goes through the same steps, the same way, without an engineer rewriting a script.

MLtwist runs preparation as a pipeline of Twists — versioned steps that read from your Google Cloud Storage, Amazon S3, or Azure bucket and write the results back. Files are cleaned, split, resized, or transcoded, companion files and metadata are attached, and foundation models draft labels so people correct instead of starting from zero. The prepared files are organized into datasets and batches and sent to the labeling tool you choose. At Sandia National Laboratories, that approach cut screening-data preparation from eight weeks to three.

Use data preparation when raw files are slowing your program down, when annotators keep asking for context they can’t see, or when every new batch means another round of one-off scripts. It pairs with labeling — our team, yours, or both — but it also stands on its own when what you need is a clean, organized, well-documented dataset.