The Investors Exchange (IEX) publishes daily historical market data as raw network packet captures (.pcap / .pcapng). These files capture every UDP packet transmitted across the exchange’s market data distribution network.

While PCAP files are great for network debugging, querying hundreds of gigabytes of raw binary packet captures for quantitative analysis or order book reconstruction is painfully slow.

iex_to_parquet is a high-performance C++ parser and data pipeline that ingests raw PCAP network dumps and streams them directly into compressed, columnar Apache Parquet datasets.

Ingestion Pipeline

  +--------------------+      +--------------------+      +--------------------+
  |  Raw PCAP / PCAPNG | ---> |  IEX-TP & DEEP     | ---> |  Apache Arrow      |
  |  Network Capture   |      |  Protocol Decoder  |      |  Parquet Writers   |
  +--------------------+      +--------------------+      +--------------------+
                                                                     |
                                      +------------------------------+
                                      |              |               |
                                      v              v               v
                                 [buys.parquet] [sells.parquet] [trades.parquet]

Protocol Decoding Engine

The C++ parser works directly from binary network buffers without third-party packet analysis tools like Wireshark:

  1. PCAP / PCAPNG Stream Processing: Reads packet headers, validates link-layer framing, and traverses IPv4 UDP datagrams.
  2. IEX-TP (Transport Protocol): Parses the IEX session protocol layer, handling payload sequences, channel IDs, and nanosecond timestamp headers.
  3. DEEP & TOPS Specification Parsing: Decodes high-frequency market events into strongly typed structs:
    • Order Book Updates: Price level additions, size changes, and cancellations (PriceLevelUpdateMessage).
    • Trade Reports: Completed executions and volume prints (TradeReportMessage).
    • Auctions: Opening, closing, and halt auction collar updates (AuctionInformationMessage).
    • Administrative: Security directory entries, trading halts, short-sale circuit breaker statuses, and system events.

Columnar Storage with Apache Arrow

Rather than dumping millions of rows into disk-heavy CSVs or relational databases, the parser writes directly into partitioned Parquet files using the C++ Apache Arrow APIs:

  • Entity Splitting: Separates discrete event types into dedicated tables (buy_writer, sell_writer, trades_writer, auctions_writer).
  • Snappy & ZSTD Compression: Reduces raw PCAP footprint by 80–90% while enabling sub-second analytical queries using DuckDB or Polars.
  • Microsecond Precision: Preserves exact exchange matching-engine nanosecond timestamps for tick-level order book reconstruction.

Python Bindings & Parallel Cloud Processing

To support scalable batch processing across months of historical exchange archives, the core parser is wrapped in Python using pybind11:

import py_iex_converter as iex

# Fast C++ conversion of compressed PCAP to columnar Parquet
converter = iex.IexConverter()
converter.process_file(
    input_pcap="2025-12-15_sample_deep.pcapng.gz",
    output_directory="./parquet_out/"
)

A companion multi-processing script enables parallel batch conversion across multi-core servers, automatically downloading historical daily dumps from cloud storage buckets and exporting partitioned Parquet partitions.