I have been developing a tool wrapped around tshark. The first blog post was about hitting a wall on a 2.5 GB file. Later, I talked about parallelizing the PCAP processing in my second blog post where I ran into a file corruption bug. I was using PcapSplitter from PcapPlusPlus in connection mode and output started coming back corrupted.

The initial signal came from "Total Block Length" errors thrown by tshark on some output files. The actual signal came when I built a small reproduction. PcapSplitter reported 12 files and 48 packets, but on disk, there were 11 files and 44 packets. Exit code zero and printed "Finished" on standard output.

Two wrong theories first. Someone on Reddit suggested file descriptor exhaustion. It was plausible but I found that at low ulimit -n it silently drops most packets and still exits zero. A real bug, but not mine. Then, I found that the PcapSplitter library has a hardcoded MAX_NUMBER_OF_CONCURRENT_OPEN_FILES = 250 with an LRU that closes and reopens handles past it. My minimal reproduction was 13 connections, nowhere near the cap.

The real culprit: the PcapSplitter's FiveTupleSplitter caused file truncation/corruption on TCP session reuse (i.e., a new SYN packet arrives for an already tracked 5-tuple hash).

The correct part of the splitter was assigning a new file number when a TCP session reuses a 5-tuple, but the filename function builds the name from IP and port only. So, in this case, both sessions get the same filenames. Then main.cpp sees a file number it has never seen, and opens that file fresh, without append. This truncates the existing file or causes a race condition between two active file writer handlers.

I filed the issue with a 13-connection reproduction, the maintainer and I worked out a fix that only suffixes on an actual collision, tested on the same corruption, and now the fix is merged.