Calculated and Virtual DP range cut-off bug

Status: diagnosed, not yet fixed

For Calculated and Virtual data points, the request from/to range filter is not enforced on the returned time series. Rows can appear past to. All four retrieval methods are affected: retrieve, retrieve_at, retrieve_history, retrieve_client. Normal data points are correct.

Bug investigation captured on develop branch (2026-05-18, andras.lederer) for the mando-lib retrieval path. No fix shipped yet — this note records the diagnosis, mechanisms, and proposed fix shape.

Symptom

The [from, to) range filter applied at the SQL boundary is correct for Normal DPs, but the Virtual and Calculated branches perform in-memory Polars transformations downstream that can expand the time domain past to. The final DataFrame returned to the caller therefore contains timestamps >= to.

Convention recap

All ranges in mando are [from, to) — inclusive start, exclusive end. Verified at:

• mando-core/src/util/data_frame/filter_by_range.rs:10,14 — gt_eq / lt
• mando-lib/src/repo.rs:477,582,621,650,686 — SQL and :from <= value_time and value_time < :to

Root cause

MandoServiceBase::handle_data_point_types (mando-lib/src/service_base.rs:94-159) applies the range correctly at the SQL boundary for the Normal branch, but the Virtual and Calculated branches perform in-memory Polars transformations downstream that can expand the time domain past to. There is no final filter_data_frame_by_range cut-off after those transformations.

Defense-in-depth missing

EvaluationMetaData { range } is plumbed through the calculation pipeline but the only consumer is FillMissing. The range is never enforced on the final calculated/virtual output.

Mechanism A — convert_to_metadata upsampling

• File: mando-lib/src/util/resolution/convert_resolution.rs:39-95
• Call sites:
  • service_base.rs:116 (Virtual branch)
  • service_base.rs:128 (every Calculated dependency)
• Signature: convert_to_metadata(source, target, data_frame) takes no request range.
• Upsampling converters (e.g., simple_convert_1hour_to_15min at line 134) explode one source row into N rows:
  • Hour1 → Min15: 1 → 4 rows
  • Hour1 → Min1: 1 → 60 rows

Concrete failure

Request [10:00, 10:30) on a Virtual Hour1 → Min15 DP:

1. SQL returns the 10:00 row (inside the range).
2. convert_to_metadata emits 10:00, 10:15, 10:30, 10:45.
3. Rows 10:30 and 10:45 leak past to.

Even a single source row at from plus pure upsampling (no joins, no evaluation) is enough to trigger the leak.

Mechanism B — evaluate_expression joins/concats

• File: mando-lib/src/util/data_frame/calculation/evaluation.rs

Op	Lines	Join semantics	Leak behavior
Override	:62-72	JoinType::Full + coalesce of value_time_right/value_time	Output = union of both deps’ timestamps. Any leaked row on either side appears.
Sum / Max / Min	:175-178	concat(...) + group_by([col("value_time")])	Output timestamps = union of all deps.
Mul / Div / Zip	:91-105, :144	JoinType::Inner	Intersection only; leaks only when both sides share the same leaked row (which Mechanism A can produce).

EvaluationMetaData { range } is plumbed through (:30-34) but consumed only by FillMissing (:74-90) — never enforced on the final calculated output.

Affected call sites

All four retrieval methods funnel through handle_data_point_types:

Method	Call site
retrieve	service_base.rs:207
retrieve_at	service_base.rs:362
retrieve_history	service_base.rs:439
retrieve_client	service_base.rs:590

Unaffected paths

• Normal DPs (service_base.rs:104-113) — returned directly from the SQL-bounded result, no post-processing that expands time domain.
• post_process_retrieved_data (:161-193) — only handles update_info / generation_time / fetch_time rewriting; no range trim.

Proposed fix shape (not yet implemented)

Add a final filter_data_frame_by_range(&df, Some(range.from), Some(range.to)) in the Virtual and Calculated branches of handle_data_point_types. The per-DP range is already available via the evaluation_metadata: HashMap<DataPointId, EvaluationMetaData> parameter — keyed by id, so each calculated/virtual DP can be trimmed by its own request range.

// pseudocode, Virtual / Calculated branches in service_base.rs
let trimmed = filter_data_frame_by_range(
    &df,
    Some(evaluation_metadata[&dp_id].range.from),
    Some(evaluation_metadata[&dp_id].range.to),
);

Why defense-in-depth at the boundary

Trimming at the leaf (pushing range into convert_to_metadata) would fix Mechanism A but not Mechanism B — evaluate_expression joins/concats independently produce union timestamps regardless of converter behavior. Trimming at the final boundary catches both with one call and minimal blast radius.

Verification

Diagnosis independently confirmed by a second investigation that started cold with only the symptom — same root cause, same leak mechanisms, same call-site map. No alternative explanation found.

Related prior work

Commit	Description	Relevance
d8855f02	”fix: convert parts of calculated data points” (BE-2225)	Fixed unit/resolution conversion of deps but did not address range cut-off.
c9e97b5d	Virtual Imbalance / Realised DP with Now(15Min, Floor) capping	Config-level workaround at the DP definition level, not an engine fix.
66fe82c5	”fix: add inclusive versioned datapoint metadata getter”	Adjacent boundary-handling work.

Related

• Agent Context
• mando-lib
• Arrow Flight Streaming Design