If you get something like this in your Cisco’s IOS firewall log:
Mar 12 15:05:33 192.168.1.1 3129: 003121: *Mar 12 15:03:03.195 EST: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:525214740 1415 bytes is out-of-order; expected seq:525170856. Reason: TCP reassembly queue overflow – session 192.168.1.5:53022 to 208.79.250.63:80 on zone-pair ccp-zp-in-out class ccp-protocol-http
sometimes accompanied by hangs in downloads, then what is happening is you are blowing out the buffers used to reassemble TCP segments when the segments have arrived “out-of-order” (also abbreviated “OoO”).
The problem for a stateful firewall or IDS/IPS is it often needs to see more of the packet stream than just the initial segment to make a forwarding/block decision. Thus it has to collect these segments together, however sometimes the segments don’t arrive “in order”. This can particularly happen when VPN is used.
In order to get around this, it essentially collects the streaming segments going by in a queue until it can find the missing segment (assumed to be “out-of-order”). It queues those packets in memory, but for obvious reasons it cannot have infinitely sized queues – it would run out of resources. In fact if it did, this would offer a very effective DoS (Denial of Service) attack.
Thus, there are defined limits set to the TCP reassembly queue. Those limits are actually fairly small to start (16 entries and 1 mb), thus you may want to adjust them if you are regularly seeing messages like above.
Using the old CBAC method of inspection, you could insert the following command:
ip inspect tcp reassembly {[queue length packet-number] [timeout seconds] [memory limit size-in-kb] [alarm {on | off}]}
However with the newer Zone Firewall inspection methods don’t use the same settings. Instead the new command format is:
parameter-map type ooo global
tcp reassembly alarm {on | off}
tcp reassembly memory limit memory-limit-kb
tcp reassembly queue length queue-length
tcp reassembly timeout time-limit-secs
To note the defaults are as follows:
parameter-map type ooo global
tcp reassembly alarm off
tcp reassembly memory limit 1024
tcp reassembly queue length 16
tcp reassembly timeout 5
So, if say you wanted to quadruple the default queue/memory lengths:
parameter-map type ooo global
tcp reassembly memory limit 4096
tcp reassembly queue length 64
Note it’s not clear if a dropped segment appears the same as an “out-of-order” segment to the router – that is with a dropped/lost segment the router keeps expecting it to arrive, just out of order. Thus the error could be telling you more that you’re dropping packets than you’re blowing out your “out-of-order” queues. Unfortunately I cannot find documentation one way or another on this.
Also to note if you’re increasing the queue length, you might want to increase the timeout (“tcp reassembly timeout time-limit-secs“), however 5 seconds is an awful long time for a segment that might be out-of-order not to arrive. As bandwidth increases, while it is likely that more packets/bytes might come in to blow out the queue, it’s unlikely they would take more time to do so (quite the opposite – an out-of-order packet at higher bandwidth is if anything likely to show up sooner, not later), thus I wouldn’t expect this to need adjustment.
Leave a Reply