After upgrading one of our Dell PowerEdge 1950 servers to the latest Broadcom drivers, and installing the Broadcom Advanced Control Suite 2 (BACS), I started having issues with our nightly Arcserve r11.5 backups. Our server would drop offline, CPU usage would spike to 50%, all disk activity would stop, and the server would effectively drop off the face of the earth for 6-7 minutes, then recover like nothing had happened.
Our Arcserve Job Log reported the following error:
E3392
Backup server TCP reconnection timeout.
Not a lot to go on, but at least a start. While investigating, I noticed the BACS2 had the TCP Offload Engine (TOE) enabled and it was licensed for 1024 connections. I knew the backup wasn't using more than 1024 simultaneous connections and we had several other PowerEdge 1950 servers that were running backups just fine - but they didn't ship with TOE enabled!
Ah ha, now we're on to something. A quick Google search revealed several posts detailing problems with Windows 2003 SP1+ and TOE, especially the PowerEdge 195x series of servers. The recommendation of everyone?
Simply disable it.
So, without further ado, here is how you disable TOE on a Windows 2003 SP2 server, with the Broadcom Advanced Control Suite 2 installed via software.
Note: You can also remove an internal hardware key to disable it, but I hate going to the data center at 11PM at night. Here's a post that has a couple pictures of what the TOE hardware key looks like in case you'd rather take that approach.
-
Open a command prompt and disable the TCP Offload Engine in the Windows 2003 SP2 TCP stack using the following command:
-
-
Click the link for the Microsoft KB Article.
-
Open the BACS2 and disable TOE:
-
Start -> All Programs -> Broadcom Advanced Control Suite 2
-
Select the adapter you wish to disable TOE on.
-
Scroll over until you see the Resource Reservations tab:
-
Highlight TCP Offload Engine
-
Click Configure
-
Check and Deselect the TCP Offload Engine under the NDIS section.
-
You should see the bar at the top go from 83% to 0%.
-
Click Apply - this will disconnect all network sessions for a few seconds.
-
After you've disabled it on each adapter, click OK.
-
Reboot your server.
You may only need to use the Netsh command to disable it, as upon reboot you can not re-enable TOE through the BACS2 - all the options are greyed out.
Good luck!