Errow

Errow (Erlang + Arrow) provides zero-copy, in-memory columnar arrays for Elixir and the BEAM, fully integrated with Elixir protocols (Enumerable, Collectable, Access) and 100% byte-for-byte compliant with the Apache Arrow Columnar Specification and Arrow C Data Interface.

Why Errow?

Standard Elixir collections are linked lists ([head | tail]) or tree structures. While excellent for general functional programming, they have limitations for data-intensive and interop workloads:

• High Memory Overhead: Each list element requires 2 machine words (16 bytes on 64-bit BEAM) plus heap terms.
• O(N) Random Access: Indexing Enum.at(list, 500) requires traversing 500 pointers.
• Complex Native Interop: Passing lists to C, Rust, Zig, or DuckDB requires allocating memory and copying every single item across the NIF boundary.

Errow solves this by storing data in continuous BEAM binaries:

• O(1) Random Access: Slicing and indexing (arr[i]) use BEAM's native binary_part/3 sub-binary references with zero memory copying.
• Idiomatic Elixir: Seamlessly use arr[0], for x <- arr, into: Errow.string(), Enum.map/2, put_in/2, and update_in/2.
• Zero-Copy Native Interop: Native NIFs (C, Rust, Zig) and analytical engines (DuckDB, Polars, PyArrow) can read Errow buffers directly via the Apache Arrow C Data Interface without serialization or copying!

Installation

Add errow to your list of dependencies in mix.exs:

def deps do
  [
    {:errow, "~> 0.1.0"}
  ]
end

Quickstart

1. Strings (Errow.String)

# Build in comprehensions with Collectable
names = for name <- ["Elixir", "BEAM", "Arrow", "Errow"], into: Errow.string() do
  String.upcase(name)
end

# O(1) Indexing via Access
names[0]    # => "ELIXIR"
names[-1]   # => "ERROW" (negative indices supported!)
names[1..2] # => ["BEAM", "ARROW"]

# In-place updates via Access
updated = put_in(names[1], "ZIG")
updated[1]  # => "ZIG"

# Enumerable & Stream
Enum.map(names, &byte_size/1) # => [6, 4, 5, 5]

2. Numerics with 128-bit and 256-bit BigInts (Errow.Numeric)

Errow.Numeric supports fixed-width primitive types:

• Signed: :s8, :s16, :s32, :s64, :s128, :s256
• Unsigned: :u8, :u16, :u32, :u64, :u128, :u256
• Floats: :f32, :f64

# 32-bit signed integers (compact: 4 bytes per item)
ints = Errow.numeric([10, 20, 30, 40], :s32)
Enum.sum(ints) # => 100

# 128-bit BigInts (Arrow Decimal128 compatible, exactly 16 bytes per item)
big128 = Errow.numeric([300_000_000_000_000_000_000_000_000_000_000_000], :s128)
big128[0] # => 300000000000000000000000000000000000

# 256-bit BigInts (Arrow Decimal256 / Web3 uint256, exactly 32 bytes per item)
u256 = Errow.numeric([115_792_089_237_316_195_423_570_985_008_687_907_853_269_984_665_640_564_039_457_584_007_913_129_639_935], :u256)

3. Raw Binary Blobs (Errow.Binary)

For raw, opaque byte buffers (hashes, UUIDs, PNG/JPEG payloads, Protobuf messages):

blobs = Errow.binary([<<1, 2, 3>>, <<255, 254>>, nil])
blobs[0] # => <<1, 2, 3>>
blobs[2] # => nil

Apache Arrow C Data Interface (Zero-Copy Interop)

Exporting your Elixir arrays to native libraries in C, Rust, Zig, or DuckDB requires zero memory allocations:

arr = Errow.string(["Daniel", "Kuku", "Elixir"])

# 1. Export schema metadata
schema = Errow.CData.export_schema(arr, name: "username")
schema.format     # => "u" (Arrow Utf8 format)
schema.n_buffers  # => 3

# 2. Export 8-byte aligned raw binary buffers
%{length: 3, null_count: 0, buffers: [validity, offsets, data]} = Errow.CData.export_buffers(arr)

Consuming in Rust (Rustler + arrow-rs):

use rustler::{Binary, NifResult};
use arrow::array::StringArray;
use arrow::buffer::{Buffer, OffsetBuffer};

#[rustler::nif]
fn inspect_arrow<'a>(_validity: Binary<'a>, offsets_bin: Binary<'a>, data_bin: Binary<'a>) -> NifResult<usize> {
    // Zero-copy wrap BEAM binaries in Arrow buffers:
    let data_buf = Buffer::from_slice_ref(data_bin.as_slice());
    let offsets_slice: &[i32] = bytemuck::cast_slice(offsets_bin.as_slice());
    let offset_buf = OffsetBuffer::new(offsets_slice.to_vec().into());

    let string_array = unsafe {
        StringArray::new_unchecked(offset_buf, data_buf, None)
    };

    Ok(string_array.len())
}

Consuming in C (erl_nif.h):

#include <erl_nif.h>
#include <stdint.h>

static ERL_NIF_TERM get_string_len(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
    ErlNifBinary offsets_bin, data_bin;
    enif_inspect_binary(env, argv[0], &offsets_bin);
    enif_inspect_binary(env, argv[1], &data_bin);

    const int32_t* offsets = (const int32_t*) offsets_bin.data;
    uint32_t start = offsets[0];
    uint32_t end = offsets[1];
    
    // Direct pointer to UTF-8 characters without copying!
    const char* str = (const char*) (data_bin.data + start);
    return enif_make_uint(env, end - start);
}

Consuming in Python (pyarrow):

import pyarrow as pa

# Zero-copy wrap BEAM binary buffers exported by Errow.CData.export_buffers/2:
buf_offsets = pa.py_buffer(offsets_bytes)
buf_data = pa.py_buffer(data_bytes)
buf_validity = pa.py_buffer(validity_bytes) if validity_bytes else None

arrow_array = pa.Array.from_buffers(
    pa.string(),
    length,
    [buf_validity, buf_offsets, buf_data],
    null_count=null_count
)

Testing & Specification Compliance

Errow is verified against official Apache Arrow test fixtures from apache/arrow-testing:

• Columnar IPC Files (.arrow): Verified against official generated_primitive.arrow, generated_decimal.arrow, and generated_decimal256.arrow with "ARROW1" magic encapsulation and FlatBuffers footers.
• IPC Streams (.stream): Conforms to official Arrow IPC framing and 8-byte buffer alignment.
• LSB-First Validity Bitmaps: Exports null bitmaps matching the Arrow specification (bit 0 = slot 0, bit 1 = slot 1).
• Cross-Language Integration: End-to-end integration tests confirm that pyarrow directly reconstructs Errow arrays with zero deserialization errors.

License

Errow is licensed under the Apache-2.0 License.