Yesterday started out well enough… my TMJ was healing nicely, the sun was shining, the humidity was low, and the AC was on. It was all going great… until I changed one tiny thing on two of my Dell PowerEdge 850 boxes. I updated the Large Segment Offload setting for the broadcom NIC team on both boxes to “Disabled”. It was 2pm… it was the end of my productive day.
After making the change and rebooting both boxes, they both came up, connected to the network for about 20 seconds (long enough to page me that they had rebooted), and then went MIA. Changing the setting back to Enabled did nothing. Suddenly I was racing to try and get it all figured out. In the end, it took me 4 and a half hours to get these boxes back up and running on the network.
I tried removing the team, that refused to work… locking up the box each time I attempted it. I tried removing the physical NICs with the same result. I also tried just removing the teaming software and drivers, but still both boxes refused to let go and locked up mid-uninstall. Ultimately I had to dive into the registry and manually change everything.
First off, all the GUIDs change for each install, so if you need to do something similar, just know the GUIDs will not be the same on your box, except for a couple standard GUID locations like the class GUID name in step 2.
1) I first went to
and checked the subkeys to find the physical network adapter GUIDs and wrote them down. They were {297C2ABE-030B-4A38-B2C0-A68A7AD901B0} and {57967509-D4FD-4D17-93C5-C72C3424BC6E}.
2) Then I went to
{4D36E972-E325-11CE-BFC1-08002bE10318}\
and looked for a reference to the broadcom netxtream team virtual adapter. I found it with a GUID of {F5E85D39-D368-4D76-AB7C-99383CD7019A}. I then removed the key and all it’s subkeys.
3) I continued to HKLM\SYSTEM\CurrentControlSet\Control\Network\ and removed references to the team nic GUID
4) I also removed the team nic GUID from HKLM\SYSTEM\CurrentControlSet\Services\
5) And finally, I removed the team nic GUID from
and
After I removed those references, I rebooted. When the box came back up, I went to add\remove programs and removed the Broadcom Network Drivers and Management Tools software from the box and rebooted. At this point, the two network interfaces were no longer installed. Windows automatically found them and re-installed them… however the registry was a mess still and some things didn’t work correctly… so back into the registry I went.
1) Now I had to clean up the mess. I re-traced my steps, going first to
and writing down the new nic GUIDs {2B5B1CA3-64B1-4D74-B9AA-68A373690AD8} and {9CF9C479-1FE3-47BF-A483-731D11221BF2}.
2) I then went through the whole registry and
– where there were only the old physical NIC GUIDs, I replaced them with the new ones
– where there were both old and new physical NIC GUIDs, I simply removed the old ones
– I removed all references to the old teaming nic GUID
– I deleted the Config binary value from HKLM\SYSTEM\CurrentControlSet\Control\Network
Once that was all completed (there were a lot of references), I rebooted. Once the box came up, I configured both NICs, gave them IP addresses, and everything was perfect. I then repeated the process on the second box, which went a lot faster now that I knew what I had to do.
I’ll never go back to the broadcom NIC teaming software again, and probably won’t buy any new server hardware with broadcom in it.
