Challenges
Renewable telemetry is not a high-volume problem so much as an awkward-shape problem. A single inverter reports a few dozen signals every second, a turbine reports more, and a revenue meter reports once per settlement interval. All three describe the same asset at the same moment, at three different resolutions, and the business needs them reconciled.
The first difficulty was that no two vendors describe the same thing the same way. Active power arrives under a different tag name from each inverter manufacturer, sometimes in kilowatts and sometimes in watts, occasionally with a scaling factor documented only in a commissioning spreadsheet. Fault codes are worse: each vendor maintains its own vocabulary, so "why did this asset stop" could not be answered across the portfolio without a per-vendor lookup written into every query.
The second was connectivity. Remote sites run on cellular or satellite backhaul that drops out. When a link is restored, the site's collector flushes everything it buffered — hours or occasionally days of readings, arriving out of order and overlapping data already received. An append-only ingestion path treats that as new generation, and the portfolio total quietly inflates. The team had been correcting these by hand after noticing that a month's output did not reconcile against the meter.
The third was that settlement data does not stay still. Meter readings are revised by the metering agent after the fact, sometimes weeks later. The operator has to be able to answer what it reported on a given date, what it would report now, and what changed in between — a question the existing warehouse could not answer because each load overwrote the previous figure with no record of it.
Time handling compounded all of this. Settlement periods are defined in market local time while telemetry arrives in UTC, and daylight saving transitions produce days with 23 and 25 hours. A model that stores only local timestamps loses an hour twice a year and duplicates one; a model that stores only UTC cannot align to settlement periods without recomputing the offset for every row.
Finally, the numbers that matter commercially — availability, performance ratio, curtailment — are not raw telemetry. They require joining equipment state against irradiance and wind resource data, and against an asset register that itself changes as sites are re-powered or inverters replaced. Those joins were being done in spreadsheets, so operations and commercial reporting routinely disagreed about the same asset's performance.
Solution
Lucent Innovation built the pipeline around a principle that suits this data specifically: raw telemetry is never edited, and every derived figure is reproducible from it.
Site collectors publish over MQTT into Kafka, and Databricks Auto Loader streams that into a Bronze layer in Delta Lake. Bronze is append-only and keeps the original vendor payload intact alongside ingestion metadata — when it arrived, from which site, through which collector. Nothing is normalised at this stage, which is what makes it possible to reprocess history later when a mapping turns out to have been wrong.
Vendor differences are resolved through a governed mapping table rather than in code. Each vendor tag maps to a canonical signal name, a unit and a scaling factor, and the mapping is versioned with validity dates. Reprocessing a period from two years ago applies the mapping that was correct then, not today's. Fault codes map into a shared severity and cause taxonomy the same way, which is what makes a portfolio-wide question about downtime answerable in a single query.
The Silver layer is where late data stops being a problem. Rather than assuming ordered arrival, readings are upserted with a Delta MERGE keyed on asset, signal and event time, so a buffered flush from a site that has been offline reconciles against what is already stored instead of adding to it. Re-delivering the same reading is a no-op. A separate scheduled backfill path re-runs recent windows to pick up anything that arrived after its window first closed, so no manual correction is needed when a site returns.
Settlement data is modelled bi-temporally: each figure carries both the interval it describes and the period during which it was believed to be correct. A revision closes the previous version and opens a new one rather than overwriting it. That is what allows the operator to reproduce any past submission exactly as filed and show the revision history behind the current number — with Delta time travel retained across the dispute window as a second, independent check.
Timestamps are stored in UTC and carry an explicit market-local settlement period index computed at ingestion, so alignment is a stored fact rather than a calculation repeated in every query. The index accounts for short and long days directly, which is what keeps totals correct through the transitions.
The Gold layer then produces the figures the business actually uses: availability and performance ratio computed against resource data rather than nameplate assumptions, and curtailment attribution that separates grid-instructed reduction from equipment fault and from genuinely low wind or irradiance. Because settlement, operations reporting and forecasting all read from the same normalised signal layer, the disagreement between commercial and operational views resolved itself — they are now the same numbers, derived once.
Unity Catalog governs the settlement tables with column-level access control and end-to-end lineage, so any reported figure can be traced back through the Gold aggregation, the Silver normalisation and the mapping version that produced it, to the raw vendor payload in Bronze.
