[TECHNICAL] Weblogic multicasting breaking your network

To all offices out there that have clustered weblogic servers and are experienced network problems: they're related.

My office has been experiencing serious network issues for a few months now. Problems like network connections resetting or extended outages were happening with increasing frequency, eventually reaching the point where there were practically hourly outages. This is especially bad since we use VoIP phones, so our client services people could be on a critical call and suddenly get dropped. We got to the point where we tore out all of our network infrastructure and replaced it with new hardware, but with no results.

After a ton of work by the CTO, our head of IT, the phone company's consultants and a Weblogic representative, we believe we have found the problem: clustered Weblogic servers using multicast. Basically, they constantly broadcast themselves over the network and clutter the system. The reason why it was getting worse over time was because the development teams all have been building their own clustered environments recently to act as production replicas.

A link to the problem described is here. FYI, we were using Weblogic 8 as well.