As per Azure documentation - [https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#limitations](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#limitations) – the Azure Internal Load Balancer default behaviour is as follows
> ..if an outbound flow from a VM in the backend pool attempts a flow to frontend of the internal Load Balancer in which pool it resides and is mapped back to itself, both legs of the flow don't match and the flow will fail.
So **what happens if your application design requires backend pool members to make calls to the private frontend of the same load balancers they are associated with?**
In the above example, if VM-WE-02-Web01 initiates a connection to 10.2.1.100:80 (ILB VIP) there is a 100% chance this connection will fail. If the backend pool happened to contain other VMs (E.g. backend pool with 2 instances) then there is a chance (50/50) the frontend request would get mapped, successfully, to another backend member. As shown below:
> _When the flow maps back to itself the outbound flow appears to originate from the VM to the frontend and the corresponding inbound flow appears to originate from the VM to itself._
Let's take a look at the default NAT behaviour of the ILB to understand the problem in more detail.
* The Azure ILB does not perform inbound Source-NAT (SNAT) and therefore the original source IP is preserved.
* When using the _default_ LB rule setting of DSR (aka floating IP) disabled, we **do** perform Destination-NAT (DNAT)
All of the above results in the following, again from the original documentation link:
> From the guest OS's point of view, the inbound and outbound parts of the same flow don't match inside the virtual machine. The TCP stack will not recognize these halves of the same flow as being part of the same flow as the source and destination don't match
We can confirm this behaviour using WireShark. Firstly, for a flow that does work, showing a successful 3-way TCP handshake. (FYI this flow is sourced from the on-premise location, see topology diagram in the next section)
Now for a flow that does not work, showing a failure of the TCP handshake. We do not get past the SYN stage. As the Azure ILB performs DNAT (see frame number 7647 on the screenshot below for confirmation of this) on the return traffic, the O/S is unable to reconcile the flow and we therefore fail to observe a TCP SYN ACK.
However, from the backend VM (10.2.1.4) we are only able to load the web page using the local VM IP address. Access via the frontend ILB VIP fails, due to the condition described in section 1.
```
**// show single NIC**
c:\\pstools>ipconfig | findstr /i "ipv4"
IPv4 Address. . . . . . . . . . . : 10.2.1.4
**// show working connectivity using localhost address**
* Add a second NIC to the Virtual Machine (from within the Azure VM config) with different IP address (we use .5 in the diagram above)
* Configure local (O/S level) static route forcing traffic destined to the LB VIP out of the secondary NIC
This works as the packet from backend-to-frontend now has a different source (10.2.1.5) and destination IP address (10.2.1.100 > DNAT > 10.2.1.4). With verification as per below:
* Re-create the Load Balancer rule with DSR enabled, Enabling DSR causes the packet to be delivered to the destination VM with the original destination IP address intact. In our case this is the frontend IP of the ILB (10.2.1.100). With DSR disabled (the default), the packet delivered to the backend VM would have a destination IP address of the backend IP itself (10.2.1.4 in our case). Further reading:
* Configure backend VM applicaton (IIS in our case) to listen on additional IP address. _NB. If using Windows Server 2016, enable weakhostsend on both the NIC and Loopback interface. See RFC1122._
Minimum = 19.39ms, Maximum = 20.17ms, Average = 19.78ms
Connectivity from the web server itself is also working when accessing the service on 10.2.1.100, **as this exists locally on the server, aka on-link.**
Minimum = 0.11ms, Maximum = 0.20ms, Average = 0.14ms
```
There are three key call outs here:
1. The backend call to the frontend VIP never leaves the backend VM. This may or may not suit your application requirements, the request can only be served locally.
2. DSR is optional, but allows the backend VM to listen on a common IP (The ILB VIP) for all connecitons, locally originated and remote.
3. You must continue to listen on the physical primary NIC IP address for application connections, otherwise LB health probes will fail
A simple option for HTTP/S traffic is to utilise Azure Application Gateway instead. Note, use of either APGW or an NVA has cost, performance and scale limitations as these are fundamentally different products. These solutions are based on additional compute resources that sit inline with the datapath, where as the Azure Load Balancer can be thought of more as a function of the Azure SDN.
Application Gateway only supports HTTP/S frontend listeners, therefore, if a LB solution for other TCP/UDP ports is required an NVA (Network Virtual Appliance) is required. NGINX is one of these 3rd party NVA options.
See [https://github.com/jgmitter/nginx-vmss](https://github.com/jgmitter/nginx-vmss) for fast start NGINX config on Azure including ARM template. Also see https://github.com/microsoft/PL-DNS-Proxy for a similar NGINX template with ability to deploy to custom VNet.
Two NGINX instances are used for high availability. Each instance contains the same proxy/LB configuration. These instances are fronted by an Azure internal load balancer themselves to provide a single front end IP address for client access. This front end IP also works for backend members specified in the NGINX config file, as shown on the diagram above.
The simple NGINX proxy configuration is shown below.
```
upstream backend {
server 10.2.1.4;
}
server {
listen 80;
location / {
proxy\_pass http://backend;
}
}
```
The number of backend servers in my example is a single VM, in production there would be multiple nodes and additional lines within the upstream module. E.g.
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
# Legal Notices
Microsoft and any contributors grant you a license to the Microsoft documentation and other content
in this repository under the [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/legalcode),
see the [LICENSE](LICENSE) file, and grant you a license to any code in the repository under the [MIT License](https://opensource.org/licenses/MIT), see the
[LICENSE-CODE](LICENSE-CODE) file.
Microsoft, Windows, Microsoft Azure and/or other Microsoft products and services referenced in the documentation
may be either trademarks or registered trademarks of Microsoft in the United States and/or other countries.
The licenses for this project do not grant you rights to use any Microsoft names, logos, or trademarks.
Microsoft's general trademark guidelines can be found at http://go.microsoft.com/fwlink/?LinkID=254653.
Privacy information can be found at https://privacy.microsoft.com/en-us/
Microsoft and any contributors reserve all other rights, whether under their respective copyrights, patents,
or trademarks, whether by implication, estoppel or otherwise.
