The Data Labeling and Annotation Process

Shovels employs a rigorous data labeling and annotation process to ensure high-quality, accurately classified permit data. Our approach involves multiple independent annotators labeling each record, with manual review and resolution in cases where their responses diverge. The validation sample size is proportionate to the representation of each category within the total dataset, typically including between 1-5% of the overall data to ensure adequate validation points for every category. A key aspect of our methodology is having annotators independently solve the task rather than validating model outputs. This prevents annotator bias and creates a "golden dataset" of correct answers that can be used to benchmark any new model outputs across iterations without requiring fresh human validation each time. This approach has proven particularly effective for accurately classifying permit descriptions, which often contain industry-specific terminology, abbreviations, and inconsistent formatting. Our case study on using specialist participants for data labeling (available on our blog) details how we achieved 98% accuracy in our classifications by incorporating a panel of experts from the construction industry.