Helio Grid Analytics keeps meter data flowing through outages
An energy analytics team replaced hand-tuned meter ingestion jobs with durable pipelines that resume after failures and respect each utility's limits.
The challenge
Helio Grid Analytics turns smart meter readings into usage forecasts for utilities. Readings come from several utility platforms, each with its own API, its own rate limit and its own way of being late.
Ingestion jobs were tuned by hand. A slow platform could hold up everything behind it, and a worker crash mid-run meant partial loads. Gaps in readings were only found when forecasts looked wrong.
The solution
Each utility platform became a source with its own request budget, so a slow or strict provider no longer affects the others. Runs are durable: when a worker is replaced, the run continues from the last finished step. Steps pass references to stored data, so large batches of readings do not slow runs down.
Automatic retries handle short outages. A source that keeps failing is paused and the data team is alerted in Slack, which stopped gaps from spreading into forecasts. Cost insights show which sources use the most compute.
From first source to production
Weeks 1 to 2
First three utility platforms connected with per-source budgets.
Weeks 3 to 5
Partitioned backfills of historical readings within each provider's limits.
Weeks 6 onward
Alerts, pausing and cost insights adopted by the on-call rota.
What the pipelines look like
A simplified example of the pattern the team uses. Names and values are illustrative.
utility_a = sy.Source(
name="utility_a_meters",
connector="http_json",
base_url="https://meters.utility-a.example/api",
rate_limit=sy.RateLimit(per_minute=90),
)
@sy.asset(layer="staged", source=utility_a,
retry=sy.Retry(max_attempts=5, backoff="exponential"))
def meter_readings(ctx, raw):
for r in raw.records():
yield {"meter_id": r["meter"], "kwh": float(r["kwh"]), "ts": r["ts"]}What changed
Early results from the beta period, as reported by the Helio Grid Analytics team.
“Meter data is never tidy. Platforms go down, readings arrive late, and a backfill can easily overload a provider. Durable runs and per-source budgets mean we stop thinking about those problems. When a source breaks, it is paused, we get a message, and nothing downstream is corrupted. We are still early with the beta, but our on-call nights have been much quieter.”
Beta customer
Plans for the coming months
Helio plans to connect more utilities and to send cleaned readings to its forecasting models through a change feed. The team also wants to use data quality checks to flag meters with suspicious readings.
Helio has asked for finer controls on backfill scheduling, which is being considered for a future release.
Build a pipeline like Helio Grid Analytics's
Synchronoid is in private beta. Tell us about your data and we will get you a workspace.