Zone Firewall TCP reassembly size

If you get something like this in your Cisco’s IOS firewall log:

Mar 12 15:05:33 3129: 003121: *Mar 12 15:03:03.195 EST: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:525214740 1415 bytes is out-of-order; expected seq:525170856. Reason: TCP reassembly queue overflow – session to on zone-pair ccp-zp-in-out class ccp-protocol-http

sometimes accompanied by hangs in downloads, then what is happening is you are blowing out the buffers used to reassemble TCP segments when the segments have arrived “out-of-order” (also abbreviated “OoO”).

The problem for a stateful firewall or IDS/IPS is it often needs to see more of the packet stream than just the initial segment to make a forwarding/block decision. Thus it has to collect these segments together, however sometimes the segments don’t arrive “in order”. This can particularly happen when VPN is used.

In order to get around this, it essentially collects the streaming segments going by in a queue until it can find the missing segment (assumed to be “out-of-order”). It queues those packets in memory, but for obvious reasons it cannot have infinitely sized queues – it would run out of resources. In fact if it did, this would offer a very effective DoS (Denial of Service) attack.

Thus, there are defined limits set to the TCP reassembly queue. Those limits are actually fairly small to start (16 entries and 1 mb), thus you may want to adjust them if you are regularly seeing messages like above.

Using the old CBAC method of inspection, you could insert the following command:

ip inspect tcp reassembly {[queue length packet-number] [timeout seconds] [memory limit size-in-kb] [alarm {on | off}]}

However with the newer Zone Firewall inspection methods don’t use the same settings. Instead the new command format is:

parameter-map type ooo global
tcp reassembly alarm {on | off}
tcp reassembly memory limit
tcp reassembly queue length queue-length
tcp reassembly timeout

To note the defaults are as follows:

parameter-map type ooo global
tcp reassembly alarm off
tcp reassembly memory limit 1024
tcp reassembly queue length 16
tcp reassembly timeout 5

So, if say you wanted to quadruple the default queue/memory lengths:

parameter-map type ooo global
tcp reassembly memory limit 4096
tcp reassembly queue length 64

Note it’s not clear if a dropped segment appears the same as an “out-of-order” segment to the router – that is with a dropped/lost segment the router keeps expecting it to arrive, just out of order. Thus the error could be telling you more that you’re dropping packets than you’re blowing out your “out-of-order” queues. Unfortunately I cannot find documentation one way or another on this.

Also to note if you’re increasing the queue length, you might want to increase the timeout (“tcp reassembly timeout time-limit-secs“), however 5 seconds is an awful long time for a segment that might be out-of-order not to arrive. As bandwidth increases, while it is likely that more packets/bytes might come in to blow out the queue, it’s unlikely they would take more time to do so (quite the opposite – an out-of-order packet at higher bandwidth is if anything likely to show up sooner, not later), thus I wouldn’t expect this to need adjustment.

6 comments to Zone Firewall TCP reassembly size

  • Tom

    Thanks! Just what I was looking for.
    I had been trying to use the CBAC commands on my ZBF to no avail. (I only just started CCNA Security…)
    This was useful as well as educational.

  • Huzah!

    Thanks a million!
    Quadrupling using your values fixed it immediately.

  • Sheri Toler

    Over a year ago, we ran into errors regarding TCP Reassembly Queue Overflows:

    %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:-164305571 1320 bytes is out-of-order; expected seq:4130637405. Reason: TCP reassembly queue overflow

    We found that by setting the parameters to the following, much of this was mitigated.

    parameter-map type ooo global
    tcp reassembly timeout 5
    tcp reassembly queue length 128
    tcp reassembly memory limit 8192
    tcp reassembly alarm off

    We are starting to experience high volumes of these errors again. It appears we are seeing this only from systems that have been upgraded for bandwidth, 10Mb Ethernet and where we can’t get ethernet, bundled T1s or throttled DS3. Given the increased bandwidth, should we be changing these parameters?

    Most of our routers should be running with 2GB of memory. I was thinking about setting to the following:

    parameter-map type ooo global
    tcp reassembly timeout 5
    tcp reassembly queue length 512
    tcp reassembly memory limit 32768
    tcp reassembly alarm off

    What do you think?

    Thanks, Sheri

  • I think it’s worth a try. I believe I did something similar on my router in the last year or so because I too was getting more “out-of-order” packets again. The key limitation is router memory, but you are still well below that value.

  • Matt

    Just what I needed. The following worked:

    parameter-map type ooo global
    tcp reassembly timeout 10
    tcp reassembly queue length 1024
    tcp reassembly memory limit 4096
    tcp reassembly alarm off

    My Cisco 877W was constantly getting these errors.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">