hftbacktest.data.utils.tardis module

convert(input_files, output_filename=None, buffer_size=100000000, ss_buffer_size=1000000, base_latency=0, snapshot_mode='process')[source]

Converts Tardis.dev data files into a format compatible with HftBacktest.

For Tardis’s Binance Futures feed data, they use the ‘E’ event timestamp, representing the sending time, rather than the ‘T’ transaction time, indicating when the matching occurs. So the latency is slightly less than it actually is.

If you encounter an IndexError due to an out-of-bounds, try increasing the buffer_size and ss_buffer_size.

Parameters:

  • input_files (List[str]) – Input filenames for both incremental book and trades files, e.g. [‘incremental_book.csv.gz’, ‘trades.csv.gz’].

  • output_filename (str | None) – If provided, the converted data will be saved to the specified filename in npz format.

  • buffer_size (int) – Sets a preallocated row size for the buffer.

  • ss_buffer_size (int) – Sets a preallocated row size for the snapshot.

  • base_latency (float) – The value to be added to the feed latency. See correct_local_timestamp().

  • snapshot_mode (Literal['process', 'ignore_sod', 'ignore']) –

    • If this is set to ‘ignore’, all snapshots are ignored. The order book will converge to a complete order book over time.

    • If this is set to ‘ignore_sod’, the SOD (Start of Day) snapshot is ignored. Since Tardis intentionally adds the SOD snapshot, not due to a message ID gap or disconnection, there might not be a need to process SOD snapshot to build a complete order book. Please see https://docs.tardis.dev/historical-data-details#collected-order-book-data-details for more details.

    • Otherwise, all snapshot events will be processed.

Returns:

Converted data compatible with HftBacktest.

Return type:

ndarray[Any, dtype[_ScalarType_co]]

convert_depth(out, inp, row_num, ss_bid, ss_ask, snapshot_mode)[source]