Sustainability reporting has moved from voluntary practice to regulatory obligation. CSRD in Europe, SEC climate disclosure rules, and investor-driven ESG frameworks mean that engineering teams are now responsible for producing figures that can be traced, audited, and defended. This is a data engineering problem as much as a sustainability problem.
This guide covers the architectural decisions that separate an ESG dashboard from an attestable ESG disclosure.
The Core Challenge: Provenance at Scale
An ESG report is only as trustworthy as the trail from metric to source. The typical failure mode looks like this: a sustainability team assembles a Scope 2 carbon figure from three spreadsheets, one utility API, and a manual entry from facilities management. Six months later, an auditor asks to see the source. Nobody can reproduce exactly how the number was calculated.
The engineering goal is to make every reported metric the deterministic output of a versioned, documented, and immutable transformation of ingested source data — not an opaque aggregate.
Architecture: The Four-Layer ESG Pipeline
- Raw ingestion layer: Ingest all source data — energy bills, fleet GPS, utility APIs, supplier emission factors, HR headcount — into an append-only raw store (S3 with Object Lock, or equivalent). Never transform in place. Every file carries an ingestion timestamp and source identifier.
- Transformation layer: All emission factor lookups, unit conversions, and aggregations are performed in versioned, testable transformation code (dbt models, Spark jobs, or SQL scripts committed to version control). Every transformation references the emission factor version it used (e.g. DEFRA 2025 Scope 3 Category 6 factors).
- Reporting period layer: When a reporting period closes, snapshot the canonical output into an immutable, period-locked dataset. Any restatement creates a new named version — never overwrites the original. This is your audit evidence.
- Disclosure layer: Map canonical metrics to framework-specific output (GRI 305 energy, SASB sector KPIs, CSRD ESRS indicators) via configuration, not hardcoded queries. This means adding a new framework is a configuration change, not a code change.
Data Collection: The Hardest Part
Most ESG metric errors originate at ingestion, not in calculations. Common collection challenges:
- Supplier data: Scope 3 emissions require emissions data from your supply chain. Few suppliers provide machine-readable data. Build a supplier data portal or structured intake form that captures data in a consistent schema — avoid free-text responses that must be manually interpreted.
- Utility metering: Monthly utility bills are often the source for Scope 1 and 2 energy. Where possible, integrate with smart meter APIs or AMI systems for monthly or weekly automated ingestion rather than manual bill entry.
- Fleet and travel: Telematics APIs (FMS providers, GPS fleet management tools) can provide fuel consumption and distance by vehicle. For business travel, integrate with expense management or travel booking systems.
- Boundary setting: Document your organisational and operational boundary explicitly in code — which legal entities, facilities, and activities are in scope for each reporting period. Store this as versioned configuration.
Emission Factor Management
Emission factors change annually (DEFRA, EPA, IEA, IPCC updates). Your pipeline must:
- Store all emission factor tables in version-controlled reference data, tagged by source and year
- Ensure transformations explicitly reference the factor version used — so the same raw data re-run with 2024 vs 2025 factors produces labelled, distinguishable outputs
- Alert the sustainability team when new factor versions are published so restatement decisions can be made deliberately
Lineage and Audit Trail
For each disclosed metric, you should be able to answer:
- Which source records contributed to this number?
- Which transformation version was applied?
- Which emission factor version was used?
- Who approved this figure for disclosure and when?
Tools like dbt's lineage graph, Apache Atlas, or OpenLineage can provide this automatically if your pipeline is structured correctly. At minimum, log transformation inputs, outputs, and factor references to a queryable audit table.
Assurance-Readiness Checklist
- Raw source data retained in immutable storage for the reporting period plus 7 years
- All transformations committed to version control with change history
- Emission factor versions explicitly referenced in all calculations
- Period-locked snapshots created at close of each reporting period
- Restatements documented as named versions, not overwrites
- Disclosure mapping table linking framework indicators to canonical metric IDs
- Data quality checks (completeness, plausibility) run and logged at ingestion
ESG reporting is no longer a sustainability team's spreadsheet problem — it is a data engineering challenge that requires the same rigour as financial reporting. We help organisations in sustainability-focused industries build reporting infrastructure that survives assurance review. Explore our sustainability solutions or talk to our team about your reporting timeline.
