Ana Klimovic (ETH) - End to End Declarative Data Analytics in the Cloud
Summary written by Simon Cyrani
Modern cloud-based data platforms such as Snowflake and Databricks were developed with a strong focus on abstracting away the complexities of cluster management and query execution. Their promise is to allow users to focus on analytical needs rather than infrastructure. Yet, despite this aim, these systems still rely heavily on static resource allocation models. For example, consider Snowflake’s warehouse sizing: users must choose from predefined “T-shirt sizes” (S, M, L, etc.) to create a warehouse to run any queries on. The underlying assumption of resource needs being static is fundamentally mismatched with the dynamic nature of analytical workloads. As a result, users either suffer slowdowns during peak demand or overpay for capacity they do not fully use.
In general, cloud platforms deploy engines inside fixed-size VMs offered by providers like AWS or Azure. These VMs provide secure tenant isolation and a familiar abstraction for system designers, but at the cost of fine-grained flexibility. They operate like black boxes: cloud providers do not expose low-level, declarative descriptions of resource needs. Instead, they run the database vendor’s compiled binaries without insight into execution behavior. The result is an inability to optimize resource allocation at run-time in the same way that query optimizers dynamically optimize execution plans.
Even serverless databases like Google BigQuery,AWS Athena, and Microsoft Fabric inherit this rigidity implicitly. While they remove the need for user-controlled sizing and charge based on data scanned, their execution engines still rely on opaque VM-based environments under the hood. Moreover, VM cold-start latency (typically in the order of hundreds of milliseconds) becomes a substantial performance barrier for short queries.
Ana Klimovic and her research team, in collaboration with Gustavo Alonso and his research group at ETH Zurich, propose a prototype to tackle these resource allocation problems by rethinking the interface between database engines and cloud infrastructure. Their approach takes inspiration from the philosophy that query processing is declarative: In SQL, users specify what they want, not how to produce it. The optimizer then decides on things like join orders, degrees of parallelism, and operators. By extending this declarative nature to include hardware and resource allocation, Klimovic’s team hopes to enable system-wide optimization and smarter resource usage. Achieving this requires two components: (1) a composable, backend-flexible query engine called Maximus, and (2) a cloud-native elastic execution platform named Dandelion.
Maximus: A Multi-Backend Composable Query Engine
Maximus is designed around the principle of hardware heterogeneity. Queries written in SQL are parsed into an abstract query plan, which is then instantiated for specific execution backends such as CPUs, GPUs, or even FPGAs. Operators are implemented across several execution engines like ACERO, CUDF, EIGER, such that Maximus automatically manages data transfer and format conversion between them. This modularity makes it possible to assign the right hardware to the right operation, such as offloading joins or aggregations to specialized hardware when beneficial.
However, to extend Maximus effectively into the cloud, an execution substrate is needed that is both elastic and secure, capable of fine-grained scaling without reliance on monolithic VMs. This is where Dandelion comes in.
Dandelion: A Declarative, Elastic Cloud Execution Platform
Dandelion executes directed acyclic graphs (DAGs) of two kinds of functions as nodes:
Pure compute functions: Untrusted user-defined, stateless, and isolated functions. In particular, these only work on predefined input and output and are not capable of using I/O-based operations or other system calls.
Communication functions: Trusted operations implemented by Dandelion that are allowed to perform system calls, such as invoking HTTP calls to e.g. retrieve data.
This division makes Dandelion fundamentally different from systems like Spark or Hive that at their core also execute DAGs. Those systems are typically deployed as single-tenant clusters per user, minimizing cross-tenant risks at the expense of flexibility and resource sharing. Dandelion, in contrast, is explicitly designed for multi-tenancy and fine-grained isolation.
Furthermore, Dandelion also aims to significantly increase performance and decrease cost usage. Standard VM-based sandboxing generally is too expensive: Although VMs can be launched in tens of microseconds, the real delay comes from initializing guest operating systems, which takes hundreds of milliseconds, among others needed for providing a filesystem or establishing sockets for communication. This delay is unsustainable for short-lived queries. Moreover, cloud providers often delay shutting down VMs, hoping to reuse them and thus reducing cold start delays. This leads to chronic overprovisioning and inflated memory consumption.
Dandelion avoids these problems by eliminating the need for guest operating systems in the common case. Since compute functions are pure, they can be isolated within their own lightweight sandbox (i.e. KVM MicroVM), skipping OS boot entirely. Communication functions run on the dedicated Dandelion host service allowing for cooperative scheduling or automatic multiplexing of e.g. HTTP requests.
Preliminary p99 tail latency measurements show Dandelion achieving 1ms VM initialization, up to two orders of magnitude faster than AWS Firecracker or Google gVisor. Teardown is equally efficient, enabling aggressive scaling down without penalties. Because the system does not rely on speculative VM retention, Dandelion halves memory usage compared to current serverless approaches.
Integrating Maximus and Dandelion
Integration only requires implementation of the Dandelion operators within Maximus. The workflow then automatically handles execution, requiring no further user involvement:
SQL is parsed into an abstract query plan.
Maximus translates this into an engine-specific plan.
This is then transformed into a Dandelion operator plan.
Dandelion decomposes the plan into DAGs of pure and communication functions.
Execution occurs elastically across shared multi-tenant hardware.
Initial experiments on TPC-H (scale factor 0.01, Q1 and Q2) show promising results. The first execution incurs overhead from loading operator binaries from disk. Subsequent queries benefit from operator caching. The experiments also reveal that context setup takes up a significant portion of time with the largest potential for future optimization: Currently, context is copied across pure functions; a zero-copy approach should cut processing times considerably, but requires careful engineering.
Early comparisons to AWS Athena, though not perfectly controlled since hardware differs, indicate 40% lower latency and 67% lower cost on average.
Further improvement of the prototype will be put in:
Caching and locality awareness: Since Dandelion sees all data accesses, it can predictively cache frequently accessed data.
Elasticity: Compute resources can scale smoothly based on demand, without VM-level granularity.
Heterogeneous hardware utilization: Maximus can route operations to CPUs, GPUs, or FPGAs depending on operator characteristics, and Dandelion can adaptively schedule them.
