SpaceCell processes live data and performs sub-millisecond computation within your existing environment. Its components are composable and built on open-source foundations, so teams can adopt only the parts they need.
SpaceCell is built from first principles to reduce dependency overhead and make better use of the underlying hardware. Its components are modular and use the Apache Arrow format, so teams can adopt only what fits their stack without lock-in.
A live analytics engine that connects to live and historical sources and runs the same pipeline from backtesting to production feeds.
A process-isolated Python bridge for running models and inference inside Rust pipelines, without rewriting Python research code.
High-performance Arrow IPC and Protobuf streaming, with SIMD-aligned, zero-copy paths from network buffers to the data structures used for computation.
A Rust-native, Apache Arrow-compatible columnar library with fully typed data structures and a small dependency footprint.
Lightning is the compute core of the SpaceCell stack. It connects to live sources, converts incoming data into typed Apache Arrow tables, and runs transformations, statistics and controls on that data as it arrives.
The engine manages operators, state, scheduling, concurrency and backpressure. Pipelines run continuously for weeks or months as part of a deployed application, and each processing stage typically completes in under a millisecond.
Network feeds, market data from providers such as Databento and Coinbase, and files including Parquet, Arrow, CSV and Excel, all arriving as typed columnar tables.
Transformation, joins, reshaping, text, datetime and mathematics, with financial functions for risk, rates, bonds and market microstructure.
Tests, regression, multivariate, survival and time-series analysis, implemented for numerical stability in long-running processes.
Windows, joins, broadcasts, triggers and barriers that compose into a live DAG, with event-time windowing and watermarks.
Captured feeds replay on a shared clock at any multiple of real time, against the same pipeline that runs in production.
Per-stage throughput, latency and queue depth in Prometheus, with tracing and data provenance for audit.
.collect() runs a pipeline over a finite dataset from beginning to end. .live() runs the same pipeline continuously against a live feed. The analytical logic stays the same, and only the source and windowing change.
Per-symbol realised volatility over the full dataset.
let vol: Table = ticks .group_by(&["symbol"]) .select(&["mid_return"]) .std() .collect()?;
The same pipeline, windowed per second.
let vol: LiveStream<Table> = feed .dam("1s") .group_by(&["symbol"]) .select(&["mid_return"]) .std() .live()?;
Three interfaces over the same engine, so analysts and engineers work on the same pipelines in the language that suits them.
A declarative pipeline API, embedded as a library in your application.
ticks.group_by(&["symbol"]) .std().collect()?;
The same engine from Python, returning Arrow tables compatible with Polars, DuckDB and pandas.
ticks.group_by(["symbol"]) .std().collect()
SQL with Python-style method chains, with no packages to manage. Well suited to LLM-generated queries.
SELECT symbol, mid_return.std() AS vol FROM ticks GROUP BY ticks.symbol
Performance comes from the full stack. SIMD-aligned buffers remain aligned across network, storage and memory boundaries. The columnar layout improves cache locality and reduces unnecessary data movement. Together, these choices deliver columnar throughput without giving up sub-millisecond latency.
Lightning's kernels are SIMD-first with scalar fallbacks. The scheduler manages analytical workloads to make effective use of the CPU's caches and hardware capabilities, and consecutive compatible operations are fused into a single pass with no intermediate materialisation.
Execution adapts to the workload. When the backlog is small, the runtime prioritises latency. When a large burst arrives, it scales across the available cores to clear it, and individual stages can be pinned to specific hardware to protect latency-sensitive paths.
Use individual components or the full platform. Lightning reads historical data from Parquet, Arrow IPC, CSV and Excel, and queries databases through ADBC, including Snowflake, Databricks, Postgres and BigQuery. It consumes live data from your existing transport or market data provider, and sends results back into the tools you already operate.
Because Lightning uses the Apache Arrow format, data moves to and from DuckDB and Polars within the same pipeline without conversion.
Lightning is an excellent fit for teams in quantitative finance, digital assets, payments, fraud detection and industrial monitoring.
We focus on latency-sensitive organisations closing the loop on event to outcome.