L3 columnar windowing at sub-millisecond latency. Arrow-compatible. High-level Rust your researchers read and your production runs.
In markets, the value of a signal depends on how quickly it can be tested, trusted and acted on. The longer the path from research to live execution, the more opportunity is lost to delay, duplication and operational friction.
Most analytics stacks divide that path across separate systems and tools for research, streaming, statistics and production deployment. Each handoff adds latency, engineering work and another point where live behaviour can diverge from the model that was validated.
SpaceCell Lightning brings the analytical path into one live engine, so firms can move from signal to decision with less delay and infrastructure between them. Its high-precision statistical engine supports per-tick risk and market monitoring to identify anomalies and respond quickly.
A historical dataset is a bounded stream. The analytical logic stays the same; only the source and windowing change. No separate live implementation is required.
Per-symbol realised volatility over the full dataset.
let vol: Table = ticks .group_by(&["symbol"]) .select(&["mid_return"]) .std() .collect()?;
for row in &vol { // process as needed }
The same pipeline, windowed per second.
let vol: LiveStream<Table> = feed .dam("1s") .group_by(&["symbol"]) .select(&["mid_return"]) .std() .live()?;
for window in vol { // every second, per symbol }
Count-based windows over L3 order book updates.
let imbalance: LiveStream<Table> = book_events .dam(1000) .group_by(&["symbol"]) .select(&["bid_qty", "ask_qty"]) .mean() .live()?;
Route the result while it is still actionable.
for window in imbalance { // every 1000 book events, per symbol if requires_action(&window) { route_signal(&window)?; } }
The Rust API is deliberately high-level and intuitive, while retaining the full performance stack: SIMD-accelerated kernels, adaptive parallelism, and extensive memory, cache and operator-fusion optimisations.
Researchers who would rather stay in Python keep their model. The Liquid bridge embeds it inline, scores the live window, and returns the result over Arrow on the same pipeline. The bridge confines Python to the model call, leaving the rest of it in Rust.
// This is your Rust service let scored: Value = rt .with_python(&window, |py, table| { let scope = PyDict::new(py); scope.set_item("table", table)?; py.run(cr#" # This is your model pasted in verbatim, which receives the # data from Rust 'zero-copy' through direct memory access. import polars as pl from joblib import load model = load("alpha.pkl") df = table.to_polars() result = df.with_columns( score = model.predict(df.to_numpy()) ) "#, Some(&scope), Some(&scope))?; scope.get_item("result") })?;
| Bridge Option | Round trip | Availability | Notes |
|---|---|---|---|
| Minarrow PyLiquid | < 10μs, in-process | Apache 2.0 | Best for local development and non-critical services. |
| SpaceCell Liquid | < 30μs, cross-process | Licensed. Included with Lightning |
Process isolation keeps Python failures outside the Rust service boundary for production workloads. |
Measured on an Intel Core Ultra 7 155H consumer laptop (16 cores, 32 GB RAM).
The standard desk workflow maintains two versions of every model: the Python research implementation and the production port in C++ or Rust. Every change must be reproduced, divergence becomes an ongoing audit risk, and the time spent porting is time lost from a validated signal.
Lightning removes the duplicate implementation. The pipeline tested on historical data runs against the live feed on the same engine and code path, reducing the gap between validation and deployment.
Lightning runs risk calculations directly on the live stream: rolling covariance and correlation over tick- or millisecond-scale windows, factor exposure through in-pipeline regression, and multivariate breach detection using measures such as Mahalanobis distance and Hotelling’s T².
Thresholds can route breaches immediately to alerts, dashboards or trading-halt hooks. High-precision statistical implementations reduce cancellation error and preserve numerical accuracy over long-running streams.
The statistical suite is validated against high-precision reference results, with documented comparisons to SciPy, R and statsmodels. Results typically agree within 1e-15 and often exceed the numerical accuracy of standard library implementations.
Lightning includes a live console for the running process. The pipeline is shown as a flowing DAG, with per-block metrics for batch rate, row rate, mean latency, drops and errors.
When live behaviour deviates, the latency view helps isolate the cause quickly. It combines end-to-end latency distributions with searchable traces, filterable by block, time window and percentile. Each trace shows the event’s path through the pipeline, including lineage, exceptions and per-hop timing, making it easier to distinguish a transient market event from a fault in the processing path.
Metrics are also exposed in OpenMetrics format for Prometheus and Grafana.
Rolling covariance, factor exposure and multivariate breach detection across the portfolio, with threshold events routed directly to alerts and controls.
Toxic-flow, imbalance and adverse-selection metrics over normalised L3 order-book data.
Windowed momentum, mean-reversion and other signals running live on the same engine used for historical validation.
Slippage, fill quality and benchmark performance calculated directly from live orders and market data.
Continuous analytics and risk across 24-hour exchange feeds, without splitting the workload across separate systems.
Keep your notebooks and models. Liquid runs them in the live pipeline at a ~30μs roundtrip with full process-level isolation.
Lightning owns the analytical path between your data infrastructure and the systems that act on its output.
| Layer | Lightning | Remains in your stack |
|---|---|---|
| Market-data feeds and normalisation | Your feed handlers | |
| Transport, durability and replay | Kafka, Aeron or your existing bus | |
| Historical and tick storage | kdb+, ClickHouse, Parquet or your lakehouse | |
| Research and backtesting | The same engine used for live execution | Your notebooks and Python tools, through the bridge |
| Signal and feature computation | One pipeline across historical and live data | |
| Statistical modelling | Regression, hypothesis tests, distributions and time series | |
| ML training | Your Python and ML stack | |
| ML inference in the live path | Through the Liquid bridge at roughly 30μs round trip | |
| Live risk, TCA and surveillance | Computed directly on the stream | |
| Order and execution management | Your OMS and EMS | |
| Tick-to-trade order placement | Your latency-critical C++ or FPGA path | |
| Pipeline observability | Native console, plus Prometheus and Grafana integration | Your existing Prometheus and Grafana stack |
Lightning consolidates live analytics, statistical computing, model inference and observability without replacing the infrastructure you already rely on.
Keep them. Lightning reads history from Parquet and Arrow IPC and runs beside your store as the live compute and statistics layer. You can adopt one open layer or the whole engine, alongside what you already operate.
No. Lightning is the analytical and statistical plane beside your execution stack. Order placement and the microstructure hot path stay on your own fast path.
Yes. Your statistical models and machine-learning code run inside the live pipeline through the Liquid bridge, unchanged. Researchers stay in Python, and the engine runs it live.
No. Demos run on synthetic data, and evaluation stays within your walls. We do not ask for production data, proprietary detail, or trading strategies at any point.
Yes. Lightning is a single-process executable binary with a small, auditable set of dependencies, largely built in-house. It runs in your colo, your cloud, or on a workstation, entirely within your environment.
A live engine with full statistical modelling, parallelism, streaming, windowing semantics, and an observability surface is a large undertaking, and the engineers at firms in this space are capable of building it - some already have. Where Lightning earns its place is cost, speed of onboarding and deployment, and a team whose full-time focus is this engine and your support. Because it is composable, you adopt it at the level that suits you, from a single open foundation to the whole engine, so it plugs into your existing architecture and your own work keeps compounding.