Enterprise Mobile

When installing System Center Mobile Device Manager (aka MDM) in a customer environment recently, I encountered a scenario where VPN connectivity worked on all but one of the mobile networks we tested. Here’s a recap of the challenges I encountered, and the eventual solution.
Summary
After installing all MDM components, we were able to successfully enroll and connect two T-mobile devices without any problems. Next, we tried connecting AT&T devices (using the recommended isp.cingular APN), and the tunnel would come up but the device could not access any internal systems. Policies would get pushed down to T-mobile (or even WiFi) connected devices flawlessly, yet isp.cingular would always fail. This happened consistently across a variety of devices, SIMs and regions of the country. We were also able to take the same device, SIM and APN and connect fine to our MDM lab here at Enterprise Mobile.
Troubleshooting
To troubleshoot this issue, we tried quite a few different tests. For each test, we would open up IE Mobile on the device and try to access both the FQDN and IP of the Device Management (DM) server. Keep in mind that the DM service typically run on port 8443, so to properly test connectivity, you need to access a URL in the format of https://<IP or FQDN>:8443/. When connectivity is working properly, you will get prompted to select a client certificate for authentication with the DM server.

The tests we attempted were:

• Enroll and connect a device using a T-Mobile SIM, then swapping in the AT&T SIM (failed).
• Run network captures on devices connected to both isp.cingular and wap.voicestream and compare the results. Behavior looked identical: the tunnel came up fine, and DNS requests were sent out over the internal network. However, the isp.cingular device never received the query response, while the wap.voicestream device did.
• Run network captures on the firewall between the Gateway (GW) and DC/DM servers. DNS requests from devices made it in/out, but never made it back to isp.cingular devices.
• Verified that the SIMs were using were properly provisioned for isp.cingular.
• Through all this, wap.cingular worked fine (although that is not recommended) while isp.cingular continued to fail.

Solution
After speaking with a number of people at EM and MS regarding this issue, we discovered that the MDM mobile VPN operates over slightly different ports depending on whether the client network uses NAT. When there is no NAT in use, Protocol 50 is used for encapsulation of data in the VPN (IPsec) tunnel. However, when NAT is in use, UDP 4500 is used for this encapsulation instead.

The fact that data was flowing over the tunnel fine for some connections but not other likely indicated a problem with connectivity over either Protocol 50 or UDP 4500 from the Internet to the GW. As it turns out, all of the networks were testing over used NAT, with the exception of isp.cingular. That narrowed our focus down to Protocol 50.

Now that we knew the likely issue was with Protocol 50, we focused on the firewall (a Sidewinder 1150) that was between the GW server and the internet. We eventually found the following entries in the FW logs:

Apr 4 14:35:46 192.168.122.10 auditd: date="Apr 4 18:35:46 2008 GMT",
fac=f_kern_ipfilt,area=a_general_area,type=t_info,pri=p_minor,pid=0,ruid=0,euid=0,pgid=0,logid=0,cmd=kernel,domain=Abot,edomain=Abot,hostname=<fw>,
rule_name=<ESP-rule-name>,srcip=<client-IP>,dstip=<GW-IP>,protocolname=esp,information=”=IP Filter: 1 packets successfully forwarded by rule”
Apr 4 14:35:46 192.168.122.10 auditd: date="Apr 4 18:35:46 2008 GMT"
,fac=f_kernel,area=a_nil_area,type=t_netprobe,pri=p_minor,hostname=<FW>,event=IPv4 netprobe,srcip=<GW-IP>,dstip=<client-IP>,
protocol=50,srcburb=<srcnet>,interface=vlan2,reason=”An IPv4 packet was received for a protocol not supported over IPv4.”

It turns out that the FW was allowing Protocol 50 inbound, but blocking it on the way back out. The rule that was defined for this traffic specified bi-directional, but the FW still blocked the outbound.

The final solution was to change the existing rule to inbound, and create a separate IP filter rule for outbound Protocol 50. With that change, outbound traffic worked, and isp.cingular connected devices started behaving normally.

Keep in mind that above all this, some mobile operator APNs may not work simply because the operator has blocked certain ports (such as UDP 500/4500). Please coordinate with your preferred MO(s) to ensure that you are using an APN that is appropriate for use with MDM, and that all of your SIMs are provisioned accordingly.

Chris Saint-Amant
Mobile LOB Architect

A recurring question I get is how to test and demo the software distribution capabilities of MDM.

People generally run into errors with importing test CAB files because the DM does not trust the signature the CAB was signed with - or the CAB is simply unsigned. First thing to note is the software distribution server can only import signed CAB files. You cannot disable this feature (as of this writing anyway.) The root certs of the certificate that signed the cab file must be in the Trusted Publisher store on the DM server. In most cases you will have to manually put it there.

If you have several unsigned apps, create a cert from your MDM CA and use that for signing all the CABs. The steps for how to do that are in a CAB Signing document on the Connect site or came with your MDM product documentation.

If you would like to do a quick test without going through the self signing cert process, you can deploy a Microsoft signed CAB like live search. You can download the cab here:

http://mobile.search.live.com/client/download_manual.aspx

You will need to prep your DM server to trust the certs and CA used to sign the live search cab. Here’s the steps to do that on your DM server.

  1.)   Download and import the Verisign / Microsoft M2M root certs from here:          

http://blogs.msdn.com/hegenderfer/attachment/3272332.ashx

2.)   Extract the contents to a temporary folder.

3.)   Start->Run-> MMC

4.)   Add / Remove Snap In - Choose Certificates MMC (Not certificate templates or certificate authority)

5.)   It will prompt for User, Service or Computer - choose computer.

6.)   Right click Trusted Root Certs ->All Tasks -> Import and import all of the .cer files extracted in step 2.

7.)   Right click on the LiveSearchWM5.cab file and choose properties.

8.)   Choose Digital Signatures

9.)   Click on Microsoft Corporation

10.)  Click Details Button

11.)  Click Details TAB

12.)  Click Copy to File… button.

13.)  Click Next twice in the Certificate Export Wizard

14.)  Enter c:\Microsoft.cer as the file name and click next

15.)  Click Finish

16.)  Back to the MMC Right click Trusted Publishers ->All Tasks -> Import and import the file c:\Microsoft.cer

17.)  Run the SW package import wizard and you should see success.

Once the CAB is imported you can simply follow the steps in the operations guide for how to deploy the application to your devices using the MDM software deployment MMC.

The Windows Mobile device management platform supports two different Open Mobile Alliance (OMA) standards:  OMA Client Provisioning (OMA CP) and OMA Device Management (OMA DM).  By the say, OMA CP is the new name for WAP Provisioning.  So, when you see Windows Mobile configuration XML with the root node of <wap-provisioningdoc>, you know you are using OMA CP.

Because Windows Mobile supports both OMA CP and OMA DM, you’ll find that MSDN documentation for most Windows Mobile Configuration Service Providers will include information on configuration XML for both standards.  OMA DM is suppose to be the new, improved standard (and it is in many ways).  So, you may wonder why OMA CP support is still included. 

As with most wireless-related standards and protocols, ubiquitous deployment and use of OMA DM has been slow.  OMA CP is still widely used.  Of course, the improvements inherent to OMA DM will drive eventual OMA DM ubiquity, but for now, OMA CP still rules the roost.

But, from a Windows Mobile developer point of view, there is another reason why OMA DM is slow to take hold.  The Windows Mobile device management platform does not come with  an OMA DM API.  The only way to take advantage of Windows Mobile OMA DM functionality is to bootstrap an OMA DM server to the device and interact with the platform through a client/server connection.

In comparison, the Windows Mobile device management platform provides much richer interfaces to configure the device using OMA CP.  There is nice API, DMProcessConfigXML (this is the native API, managed .NET CF API is called ProcessConfigXML in Microsoft.WindowsMobile.Configuration) .  You can also configure the device via Remote API (RAPI) that is exposed on desktop PCs running ActiveSync or Vista Windows Mobile Device Center (WMDC).  Of course, OMA CP can be configured via a WAP provisioning server for the client/server connection functionality.  But, for device developers, we need a client-side API which does not exist for OMA DM on Windows Mobile.

Why do we need a client-side API for OMA DM?  The main reason is to build a value-added device management solution for customers that want more then what you can get out-of-the-box with the Windows Mobile device management platform. 

For instance, a client application can save bandwidth and improve performance by adding smart query logic to device management transactions.  Otherwise, you’d need to offload all the logic to the server.  An example might be that a client can detect a new Bluetooth profile registration and allow or disallow its use based on a pre-configured policy.  Without this, the information must be uploaded to the server, processed by an admin or some server-side application and a new policy is provisioned to the device’s networkpolicy CSP.  That’s just one example though.

Another reason to build a client application for device management is to add new device management features to those provided out-of-the-box.  If you use a device with built in GPS, RFID reader, etc., you may want to manage these devices and their requisite applications as well.  And, if you want to support device management across multiple hardware and operating system platforms, you’ll probably end up opting to build a device management client.

You may be thinking that this is all a lot of whining though because the Windows Mobile device management platform still makes OMA CP configuration accessible to applications.  But, consider that in Windows Mobile 6.1, most all of the new software management Configuration Service Providers are OMA DM only with no ability to manage them via OMA CP.  This means that a Windows Mobile Device Management developer will need to use an OMA DM server or a proprietary client application to leverage the new, cool, built-in Configuration Service Providers that manage software and ROM update installations. 

In conclusion, the call to action is for Microsoft to provide and public, documented API for applications to interface with Windows Mobile OMA DM components just like OMA CP.  Perhaps DMProcessConfigXML() can be extended to support both OMA DM and OMA CP XML input.

Dave Field, CISSP, MCP

Device Managment and Security Architect

Enterprise Mobile, Inc.

Hello,

Here is a little issue that I researched this week and I thought I’d share it on the blog.

PROBLEM:  When accessing a website that is secured to use an NTLM authenticated password, the “Save Password” option does not work on Windows Mobile Pocket PCs and Smartphones.

Here are detailed steps that outline how to reproduce this problem using a Motorola/Symbol MC70 Pocket PC.  Note that the test device was running Windows Mobile 5.0 AKU3.

1.       Perform a “clean” reboot of the device which is a cold boot that resets all storage volumes on the device to factory defaults.

2.       Configure a website connected to an Active Directory domain to use password authentication.  During testing, you’ll need access to IIS Admin. 

a.       Start out using basic authentication.  In properties for the website, open the “Directory Securty” tab, click the “Edit” button for “Authentication and access control” and check the box for “Basic authentication”

b.      Use of HTTPS/SSL is irrelevant to the test.  However, basic should not be used without SSL because the basic password is sent in clear text.

3.       From the device, open the website in the Mobile IE browser.  You will be prompted for the password.  Note:  “Realm” is included in the Basic authentication dialog box.  For NTLM authentication, “domain” input is required.

4.       Enter your domain\username and your password.  Make sure to check the “Save Password” box.  Then tap “OK”.

5.       Confirm that your password-secured site renders in the Mobile IE browser.

6.       At this point, if you were to refresh the page or even browse to a couple sites before revisting the site, you would not be asked for the password again.  This is because the website page that was opened after authentication is cached.  By exiting out of the browser session and re-opening the page, authentication occurs again.  You can exist the browser session by warm resetting.  But, it is easier to stop and restart the “internet explorer” process as follows: 

a.       On the device, go to Settings/System tab/Memory/Running Programs tab.  S

b.      elect “Internet Explorer” from the “Running Programs List:”

c.       Tap the “Stop” button.  This stops the browser process.

7.       Re-open the Mobile IE browser on the device and browse to the website again.  You should find that you do not need to enter the credentials this time for the web site because they are cached. 

CONCLUSION:  Save Password works with “Basic Auth”

8.       Close the browser session again as described above in step #6

9.       In IIS admin, change the authentication for your website from basic to NTLM:

a.       Uncheck basic and check the box for “Integrated Windows Authentication” which is NTLM.

b.      Stop and restart the website to ensure the authentication change is implemented

10.   On the device, confirm that the “internet explorer” process is not running.  If it is, stop the process.

11.   Open Mobile IE and browse the website again.  This time, you’ll see the following authentication dialog for NTLM (Windows Integrated) authentication. You’ll see that username and domain are cached from the previous use of your credentials for basic authentication.  However, the previously cached password is not re-used.

12.   Enter your password and be sure to check the “save password” box prior to tapping the “OK” button.

13.   Confirm that your password-secured site renders in the Mobile IE browser.

14.   Close the browser session again as described above in step #6

15.   Now re-open the website .

16.   You will see the same authentication dialog that you saw in step# 11 above.

CONCLUSION:  By default, the “Save Password” option exposed in the Mobile IE authentication dialog does not save the password for NTLM authentication cases.

WHY DOES THIS HAPPEN?:  Underlying Mobile IE is the use of the Windows Internet API (wininet) which is a set of core code inherited from desktop windows.  You may have noticed that in order to cache NTLM with desktop IE, you need to configure the website to be an “intranet” zone website.  Furthermore, you have to set your security settings to use user authentication logon settings to use automatic logon only for intranet zone” or “Automatic Logon for user name and password”.  Once the user of desktop IE has entered and saved their password for the “intranet” website one time, they no longer need to enter it on subsequent use.  Mobile IE encounters the same problem as desktop IE, but in Windows Mobile IE, there is no configuration UI to allow the user to make these changes.

SOLUTION:  You can add the website to the list of intranet sites by hacking the registry on the device.  To configure http://www.yoursite.com/ and https://www.yoursite.com/ as “intranet” websites and therefore eligible for NTLM password caching:

1. Add the registry key:  HKLM\Software\Microsoft\Windows\CurrentVersion\Internet settings\ZoneMap\domains\yoursite.com\www
2. Under this new key add 2 values: “http” and “https”
3. Set each of the 2 values to “1″

NOTES:

• The website domain name prefix (eg, “www”) is configured as a sub-key of the domain name (eg, “yoursite.com”).
• You need to configure each URI scheme (http, https, ftp, etc.) that may be used for the website. Each different URI scheme will require a separate instance of the cached password
• By setting value=”1″ for the URI scheme, you are configuring that full URI as a site in the “intranet” zone. Here is the value associated with all zones: 0 My Computer, 1 Local Intranet Zone, 2 Trusted sites Zone, 3 Internet Zone, 4 Restricted Sites Zone.

• For a full discussion of all security zone registry settings see: http://support.microsoft.com/kb/182569

• There are a number of registry editors for Windows Mobile devices that you can download and use. However, you can use WAP Provisioning XML to query and write to the registry via the registry configuration service provider. You can use a tool such as rapiconfig to implement WAP provisioning XML on Windows Mobile devices. The rapiconfig tool is available in the Windows Mobile Software Development Kits.

Dave Field, CISSP, MCP

Device Managment and Security Architect

Enterprise Mobile, Inc.