High-performance data transport benchmark

SpaceCell Lightstream vs Apache Arrow Flight

This benchmark compares the in-memory, across-the-wire network throughput of SpaceCell's Lightstream data transport protocol and the established industry standard Apache Arrow Flight, measuring throughput and delivery consistency between nodes, timed from request to final verified arrival.

Both systems serve the same RAM-resident Apache Arrow table, so the measured path is limited to transport and network delivery, and no storage reader is involved. Throughput is reported as receiver-verified logical payload in GiB/s. Details are available in the methodology and appendix.

50 Gbit/s network · four schema shapes · warm median of five runs

Headline numbers

1.11 GiB/s

Saturated TCP

Single-stream throughput, identical on every workload shape.

5.79 GiB/s

Saturated NIC

Peak parallel throughput, at the 50 Gbit/s network ceiling.

6–71%

Led every result

Throughput advantage from worst lead to best lead, across all sixteen measured combinations.

≤ 1%

Comparatively stable

p99 sits within 1% of its median on every schema.

Ordering Guarantees

Both systems provide the strongest ordering guarantees supported by their respective architectures. SpaceCell's Lightstream protocol reconstructs one globally ordered stream across all connections and verifies that ordering at the receiving endpoint, and that reconstruction work counts against its own reported throughput. Apache Arrow Flight, by contrast, orders only within each ticketed endpoint. Because no native capability to combine those parallel streams is supplied, they are left separate, so when multiple streams are processed in parallel their combined output carries no global ordering guarantee.

Note: enforcing global ordering for Arrow Flight would require either proxying the parallel connections through a single ordered TCP stream or reordering the batches after receipt. The first approach would artificially constrain Arrow Flight's parallelism, while the second would introduce head-of-line blocking. Both approaches would materially affect reportable performance and were therefore excluded from the comparison rather than introducing external implementations.

Therefore, for the closest comparison, see the single-stream example below, where both sides deliver an ordered stream. For the parallel benchmark, Lightstream delivers those additional guarantees.

Finally, Lightstream provides 64-byte alignment guarantees across the wire, ensuring that data on the receiving side remains SIMD-compatible once introduced through the Minarrow crate and its custom Vec64 aligned allocator.

1 · Raw speed

Lightstream surpassed Arrow Flight throughput in every measured benchmark combination. With a single stream, Lightstream reaches approximately 1.11 GiB/s for every tested schema, which is consistent with the single-flow ceiling of the environment - its fast decoding makes the single TCP flow the bottleneck rather than the transport. Apache Arrow Flight records between 15% and 41% less throughput on a single stream and varies more noticeably with schema shape. As parallel streams are added, serving the same data workload and, in Lightstream's case, still guaranteeing fully ordered delivery, both systems move towards the aggregate network ceiling, while Lightstream remains ahead across every tested schema and stream count.

Single-stream, in-memory throughput. Lightstream remains at approximately 1.11 GiB/s across all four schema shapes, saturating one TCP flow in each case. Apache Arrow Flight's throughput varies with the structure of the payload.

In-memory throughput across 1, 4, 8, and 16 streams. Navy represents Lightstream and gold represents Apache Arrow Flight, and deeper shades indicate higher stream counts. Lightstream reconciles the parallel streams into one globally ordered stream and still leads, while Arrow Flight orders only within each stream. Both systems approach the approximately 5.8 GiB/s aggregate NIC ceiling at 16 streams. Hover over a group to view the exact values.

2 · Consistency

Lightstream's result is highly stable: p99 lands within 1% of p50 on every schema. With a single stream, the interval between batch arrivals provides a direct measure of per-batch delivery time. Apache Arrow Flight develops a more pronounced tail, particularly for the wide schema, where its p99 reaches approximately 1.4 times its median.

Delivery consistency expressed as p99 divided by p50 for each batch on a single stream. Lower values indicate steadier delivery, with 1.00 representing effectively identical median and tail performance. Lightstream remains between 1.00 and 1.01 across all schema shapes.

Per-batch delivery time at p50, p95, and p99 on a single stream, measured in milliseconds. Lightstream remains tightly grouped across the measured percentiles for every schema. Apache Arrow Flight develops a wider tail, with the largest separation appearing on the wide schema.

3 · Beyond speed

The two systems also differ in the formats they can carry, the transports they support, the languages from which they can be used, and the amount of infrastructure required before data begins to move. The following comparison unpacks where each system is strong.

Where each system is strong
Capability	Lightstream	Arrow Flight
Payload formats	Arrow IPC, Protobuf, and MessagePack, with custom formats available through the `FrameEncoder` trait.	Apache Arrow record batches, Arrow-native by design.
Transports	TCP, QUIC, WebSocket, WebTransport, Unix domain sockets, and standard input/output. Optional io_uring async is available and was not included in the published benchmarks.	gRPC over HTTP/2 and TCP, with Flight SQL for database-oriented workloads.
Client languages	Rust and Python.	C, C++, C#, Go, Java, JavaScript, Python, R, Ruby, and Rust.
Setup	Add the crate or Python package, register the table, and begin reading or writing. No schema compiler or generated code is required.	Define a gRPC service, run the Protobuf toolchain during the build, and implement `DoGet`, `DoPut`, and `GetFlightInfo` over the HTTP/2 stack.
Compatibility	Leverages Minarrow, a native Rust implementation of the public Apache Arrow standard. Implements PyCapsule and PyO3 bindings, with `.to_python()` enabling zero-copy hand-off to Python in nanoseconds.	Built on Arrow-RS, the official Apache Arrow Rust implementation. Implements PyCapsule and provides FFI bindings to Python. Does not include Python closure callbacks.
Maturity	A new open-source SpaceCell product launching in 2026, with Rust and Python support. Currently requires Nightly Rust.	An established Apache Arrow component available since 2019, with broad adoption and Apache Software Foundation governance.
Licence	Mozilla Public License 2.0.	Apache License 2.0.

Apache Arrow Flight is Arrow-native by design: control operations use Protobuf actions, while the data payload is Apache Arrow. Lightstream can carry Arrow data alongside other typed message formats on the same connection. Arrow Flight's breadth of language bindings remains an important advantage of an established Apache project.

Methodology

Every avenue was matched as closely and fairly as possible in good faith, and the full benchmark is available to view and launch from the official Lightstream GitHub repository. It was designed to isolate transport behaviour while keeping the source data, payload representation, and receiver-side validation as comparable as possible. The principal controls and measurement choices are set out below.

Baseline. Arrow Flight ran the official arrow-flight crate over gRPC and HTTP/2, following the Arrow-RS default configuration in Rust without modifying the implementation. Record batches were served through DoGet. Official public guidance was followed, including keeping dictionary-encoded data in its encoded form on the wire so string data was not re-materialised, and using multiple connections rather than throttling a single multiplexed connection.
Lightstream. Lightstream used its parallel transport protocol with one TCP connection per stream, zero-copy data handling, and an ordered global merge at the receiver that reconstructs one globally ordered stream across all connections. That reconstruction work counts against Lightstream's own reported throughput. The io_uring feature was switched off.
Isolation. The source materialised a single Apache Arrow table in memory and resent it by increasing reference counts for each send, with black-boxing to prevent the work being elided. No file, disk, storage reader, or external data source appeared in either transport path.
Hardware. The 50 Gbit/s tests used two AWS i3en.12xlarge instances in one placement group and availability zone, communicating over host networking on private IP addresses.
Workload. Each table batch contained one million rows, drawn from the full benchmark pool of 300 GB until empty and constructed before timing started. Four schema shapes - numeric, mixed, string-heavy, and wide - cover plain numeric columns, offset-based arrays, dictionary encoding, and wide schemas, and all reside in RAM.
Measurement. Timing was recorded at the receiver from the dataset request until the final verified arrival, so connection establishment fell outside the measurement window for both systems. Throughput represents logical Apache Arrow payload in GiB/s and excludes framing overhead, though framing itself sits within the timed region. Every reported cell is the warm median of five runs, which smooths transient cloud network variance and scheduling noise.
Payload framing. Lightstream used 8 MiB framing and Arrow Flight its recommended 2 MiB default, unchanged.
Delivery verification. The receiver asserted the expected ordered row sequence. Any reordered, incomplete, or malformed delivery caused the run to fail.
Constraints. Both systems were tested without transport encryption. Within the AWS placement group, a single TCP flow was capped near 10 Gbit/s, so the single-stream results are flow-limited while the multi-stream configurations aggregate towards the approximately 5.8 GiB/s NIC ceiling.

Test instance specifications

Both benchmark hosts used the same AWS instance configuration. Each instance provided 48 vCPUs, 384 GiB of memory, and up to 50 Gbit/s of network bandwidth. Keeping the hardware configuration identical ensured that the comparison reflected the behaviour of the two transport implementations under equivalent compute, memory, and network conditions.

Compute and network configuration
Compute	Value
vCPUs	48
Memory	384 GiB
Memory per vCPU	8 GiB
Physical processor	Intel Xeon Platinum 8175
Clock speed	3.1 GHz
CPU architecture	x86_64
Network bandwidth	Up to 50 Gbit/s

Workload schema shapes

The four shapes span plain numeric columns, dictionary encoding, variable-length strings, and wide schemas, to cover common representative workloads.

What each schema shape contains
Shape	Columns (one million rows per batch)
Numeric	Four fixed-width numeric columns: `i32`, `i64`, `f32`, and `f64`.
Mixed	An `i32`, an `f64`, a short UTF-8 string, and a dictionary-encoded categorical column with three distinct values.
String-heavy	An `i32` identifier, a long UTF-8 string, a short UTF-8 string, and a dictionary-encoded categorical column with one hundred distinct values.
Wide	One hundred numeric columns, split evenly across `i32`, `i64`, `f32`, and `f64`.

Appendix · Full 50 Gbit/s network results

The tables below report warm median throughput as logical payload in GiB/s. The ratio is calculated as Lightstream divided by Apache Arrow Flight, and navy indicates a Lightstream lead.

Matched framing check

The results above run Lightstream at its recommended 8 MiB batch size and Apache Arrow Flight at its 2 MiB default. To keep the comparison fair regardless of frame size, this check frames both transports at Arrow Flight's own 2 MiB batch size on the same 50 Gbit/s hardware instance pair. The results are consistent with those above.

More information

The open benchmark is available to view and launch from the official Lightstream GitHub repository at github.com/SpaceCell/lightstream.

Apache Arrow, Arrow Flight, and Apache are trademarks of the Apache Software Foundation. SpaceCell is not affiliated with or endorsed by the ASF. Results were measured in July 2026 on the configuration described. Performance varies with workload and environment.