Failure resolution of layer 2 load balancer service access in F5 private network (CISCO OTV+)
I. problem phenomenon
Recently, due to the hardware failure and crash of core CISCO 6509 in a customer during the holiday, it was found that there were three application access problems released by F5, and some people were unable to access the application, sometimes good and sometimes bad. The internal network uses F5 GTM+LTM for domain name double activation, and the internal dual active DC is constructed by using CISCO's layer 2 technology OTV+LISP technology through three-layer routing.
F5 checks that both VS and pool member are normal. The health check or monitor algorithm uses TCP; to direct the cross-DC member disable resolution of the member of the DC business facing the end of the LTM to the main DC, and the business access is normal at this time, forming a single job for troubleshooting.
The problem appears to be that business access is abnormal after cross-DC access, but the magic is that only part of the vlan has a problem, and most of the cross-DC vlan has no problem!
Through preliminary investigation, the application personnel said that there was no problem with the application, and the network personnel said that there was no problem with the network (you can apply IP from the ping standby center of the main center, you can connect the business application port across the DCtelnet, and there is no problem with other vlan). F5 personnel also said that all aspects of F5 log are normal and there is no abnormal log!
Second, the cause of the problem
F5 personnel suggest direct access to port 443 accessed across DC (without F5 load) testing and packet grabbing to check packet communication.
By grabbing the packet, it is found that the TCP three-way handshake is normal, but the SSL protocol handshake is abnormal. After the client sends client hello, the server sends back a ssl data (non-server hello) packet of about 1050byte and prompts that the leading segment is missing! Then the client FIN lost the connection!
Then, by visiting the normal application of this DC, it is clear that the SSL negotiation is normal, and the SSL handshake package is hundreds of byte at most, so this is an abnormal problem at the application level, not a simple network level problem.
But whether it is the problem of the application, let the application staff change a vlan, the access is normal! It is proved that it is not an abnormal configuration problem at the application level! It is probably a problem that the network affects the application!
In view of the hardware failure, the path change, the abnormal size of the interactive packet using the ssl protocol, and the network problems such as the loss of the leading segment of previos fragment, the F5 personnel suggested to check the MTU setting, and then the customer management and network personnel said that there had been a MTU problem before and asked CISCO TAC to check. After several hours of inspection, it was finally confirmed that part of the VLAN OTV path change was caused by the CISCO 6509 crash. MTU did not change to 9216 bytes of MTU caused!
Business access is normal after the change!
III. Solutions
After changing the OTV MTU in the path, the F5 related configuration is restored, and the application test is normal!
