Enterprise Mobile

When installing System Center Mobile Device Manager (aka MDM) in a customer environment recently, I encountered a scenario where VPN connectivity worked on all but one of the mobile networks we tested. Here’s a recap of the challenges I encountered, and the eventual solution.
Summary
After installing all MDM components, we were able to successfully enroll and connect two T-mobile devices without any problems. Next, we tried connecting AT&T devices (using the recommended isp.cingular APN), and the tunnel would come up but the device could not access any internal systems. Policies would get pushed down to T-mobile (or even WiFi) connected devices flawlessly, yet isp.cingular would always fail. This happened consistently across a variety of devices, SIMs and regions of the country. We were also able to take the same device, SIM and APN and connect fine to our MDM lab here at Enterprise Mobile.
Troubleshooting
To troubleshoot this issue, we tried quite a few different tests. For each test, we would open up IE Mobile on the device and try to access both the FQDN and IP of the Device Management (DM) server. Keep in mind that the DM service typically run on port 8443, so to properly test connectivity, you need to access a URL in the format of https://<IP or FQDN>:8443/. When connectivity is working properly, you will get prompted to select a client certificate for authentication with the DM server.

The tests we attempted were:

• Enroll and connect a device using a T-Mobile SIM, then swapping in the AT&T SIM (failed).
• Run network captures on devices connected to both isp.cingular and wap.voicestream and compare the results. Behavior looked identical: the tunnel came up fine, and DNS requests were sent out over the internal network. However, the isp.cingular device never received the query response, while the wap.voicestream device did.
• Run network captures on the firewall between the Gateway (GW) and DC/DM servers. DNS requests from devices made it in/out, but never made it back to isp.cingular devices.
• Verified that the SIMs were using were properly provisioned for isp.cingular.
• Through all this, wap.cingular worked fine (although that is not recommended) while isp.cingular continued to fail.

Solution
After speaking with a number of people at EM and MS regarding this issue, we discovered that the MDM mobile VPN operates over slightly different ports depending on whether the client network uses NAT. When there is no NAT in use, Protocol 50 is used for encapsulation of data in the VPN (IPsec) tunnel. However, when NAT is in use, UDP 4500 is used for this encapsulation instead.

The fact that data was flowing over the tunnel fine for some connections but not other likely indicated a problem with connectivity over either Protocol 50 or UDP 4500 from the Internet to the GW. As it turns out, all of the networks were testing over used NAT, with the exception of isp.cingular. That narrowed our focus down to Protocol 50.

Now that we knew the likely issue was with Protocol 50, we focused on the firewall (a Sidewinder 1150) that was between the GW server and the internet. We eventually found the following entries in the FW logs:

Apr 4 14:35:46 192.168.122.10 auditd: date="Apr 4 18:35:46 2008 GMT",
fac=f_kern_ipfilt,area=a_general_area,type=t_info,pri=p_minor,pid=0,ruid=0,euid=0,pgid=0,logid=0,cmd=kernel,domain=Abot,edomain=Abot,hostname=<fw>,
rule_name=<ESP-rule-name>,srcip=<client-IP>,dstip=<GW-IP>,protocolname=esp,information=”=IP Filter: 1 packets successfully forwarded by rule”
Apr 4 14:35:46 192.168.122.10 auditd: date="Apr 4 18:35:46 2008 GMT"
,fac=f_kernel,area=a_nil_area,type=t_netprobe,pri=p_minor,hostname=<FW>,event=IPv4 netprobe,srcip=<GW-IP>,dstip=<client-IP>,
protocol=50,srcburb=<srcnet>,interface=vlan2,reason=”An IPv4 packet was received for a protocol not supported over IPv4.”

It turns out that the FW was allowing Protocol 50 inbound, but blocking it on the way back out. The rule that was defined for this traffic specified bi-directional, but the FW still blocked the outbound.

The final solution was to change the existing rule to inbound, and create a separate IP filter rule for outbound Protocol 50. With that change, outbound traffic worked, and isp.cingular connected devices started behaving normally.

Keep in mind that above all this, some mobile operator APNs may not work simply because the operator has blocked certain ports (such as UDP 500/4500). Please coordinate with your preferred MO(s) to ensure that you are using an APN that is appropriate for use with MDM, and that all of your SIMs are provisioned accordingly.

Chris Saint-Amant
Mobile LOB Architect