How Monthly EDL Data Deliveries Work

For Enterprise Data License (EDL) customers, Shovels employs an overwrite approach when pushing data into data lakes such as Snowflake, Databricks, or BigQuery. Each monthly delivery is treated as a fresh table, completely replacing the previous month's data with the updated dataset. This approach simplifies data management by eliminating the need to merge incremental updates. Importantly, our permit IDs, address IDs, and contractor IDs remain static and don't change from release to release. This consistency ensures that any scripts, analyses, or processes that rely on these IDs won't be impacted by the monthly refresh. The stability of our ID system allows customers to build reliable data pipelines and maintain referential integrity in their analyses. Each monthly delivery includes all historical data plus new permits and updates, providing a complete and current view of the construction permitting landscape. For customers who need to track changes between deliveries, we recommend using the first_seen_date field in combination with status change indicators to identify new and updated records since the previous delivery.