Displayr

Anonymous

Live query pushdown (DirectQuery-style) to remove Displayr's data size ceiling
Problem: Displayr's existing SQL data source (see How to Import Data by Connecting to an SQL Database) pulls one query's result into the document as a static, in-memory dataset, refreshed only on schedule or on open, not re-queried per interaction. Documents still have to load within Displayr's 2-minute limit and fit in the R engine's memory, so customers with very large sources still need to pre-filter or pre-aggregate every query outside Displayr rather than working with the dataset live. This will mean we can support a live data source mode where individual tables/charts/filters push their query down to the source at render time, returning only the aggregated result needed, instead of requiring one static pull of the whole result set upfront. Why it matters: Power BI's DirectQuery connects directly to the source and runs a live query per visual instead of importing data, which is exactly how it bypasses Power BI's 1GB import limit for Pro (and works with data far too large to import at all). Tableau's Live Connection works similarly, though both trade off performance for scale; report speed becomes dependent on the source. We have customers with source files that exceed Displayr's manual-upload limit (8GB) today; without this, we're at a real disadvantage against both tools for large-data use cases. Trade-offs to weigh (based on how Power BI's DirectQuery and Tableau's Live Connection behave in practice): Performance becomes dependent on the source database and network; slow sources mean slow reports, and both competitors are explicit about this trade-off Not every transformation/calculation can be pushed down; some computations that can't translate into the source's native query language would need workarounds Built-in row limits are typically needed per query/visual to prevent overload (Power BI caps cloud sources around 1M rows per visual) Power BI offers a hybrid "Dual" mode combining Import and DirectQuery (e.g., import historical data, live-query only recent data), worth considering as a lower-risk first step rather than full live mode everywhere Optional enhancement — AI-assisted querying: Most Displayr customers (market researchers) won't hand-write SQL for every table. AI could translate a Displayr table/crosstab/banner definition (or a plain-language request) directly into the optimized query run against the source, automating the push-down step so non-technical users benefit without needing SQL skills. AI could also generate a representative sample or pre-computed summary when full live querying isn't practical (e.g., slow source, exploratory analysis), preserving the statistical properties needed without manual file trimming. This wouldn't replace the core live-query architecture, but would make it usable for Displayr's actual audience. Open questions for eng: Which sources to support first (SQL Server / Postgres / MySQL / Redshift / Snowflake already have providers), how this interacts with the R-based compute engine, latency of per-interaction queries, caching strategy, row-limit/safety guards, and fallback if the source is unreachable.
0
·
Data Import
Load More