I’ve run into a couple challenges after an upgrade to Expressway/VCS 8.5.2 where MRA for phones quit working.
I found a bug that broke MRA in 8.5.2 (the recommendation has been to downgrade back to 8.5.1). That bug is shows that it is now fixed in 8.6, so I upgraded to 8.6.1 recently. MRA started working again, but only on one out of every three login attempts. It was really weird. In looking at the logs it showed a bunch of errors:
Home CUCM not available – Unknown CUCM cluster for node sub02
Home CUCM not available – Unknown CUCM cluster for node sub03
The deployment I was working on is a three node (pub and two subs), running split DNS (different internal domain than the external domain name).
After a lot of digging it turns out that there is a change in the way MRA handles CUCM lookup. When I installed the system added my pub and subs to Expressway-C by IP address. But it looks like Expressway now attempts to communicate with them via hostname, and not IP as they were defined by me.
Since Expressway is using the domain suffix assigned to MRA (extdomain.com), it is attempting to lookup sub02.extdomain.com and sub03.extdomain.com. I didn’t have A records for these on my internal DNS server extdomain zone since I’ve never needed to resolve them by the external domain name.
Adding these two records fixed the login issue and it now logins on first attempt like it used to.
An internal name is a domain or IP address that is part of a private network. Common examples of internal names are:
Any server name with a non-public domain name suffix. For example, http://www.contoso.local or server1.contoso.internal.
NetBIOS names or short hostnames, anything without a public domain. For example, Web1, ExchCAS1, or Frodo.
Any IPv4 address in the RFC 1918 range.
Any IPv6 address in the RFC 4193 range.
Why do we care?
Jabber authenticates TLS encryption using certificates for services from CUCM, CUC, IM&P, etc. Historically these have typically been deployed with IP addresses only, or internal domains (e.g. domain.local, etc.). Because of this you can no longer get a certificate for the Expressway-C box that has SANs with IPs or internal names. Jabber requires valid certificates for login now.
Without a certificate with proper SANs, Jabber will either throw an invalid cert error, or will completely deny login to UC services.
Using Collab Edge MRA, Jabber authenticates to the Expressway-E server and uses it’s certifciate. Internally Jabber communicates directly with each component.
Dealing with the issue for Collab Edge MRA
Basically we have two options to work around the SAN issue:
1) Change the domain name of the UC components to a valid public domain name that the public CA will sign for. This doesn’t mean the server has to be accessible from the internet by any means or that it is an existing domain name your company is using.
Option 1a: Deploy a new public domain name for UC services internally. For example if your domain name was domain.com you might see if domain.info or domain.net or something similar is available to register and use as the internal UC domain name. The domain wouldn’t need to resolve externally at all.
If you do this, then you need to take in to consideration that the MRA deployment becomes a multi-domain (or split-domain) deployment which requires some special treatment like the VoiceServicesDomain option. (See my previous post about multi-domain deployments.)
Options 1b: The seemingly easier deployment would be to just match your public domain name that you use for email (e.g. domain.com) for your UC components (not suggesting all internal servers — file, print or otherwise, need to be in this domain). This makes services discovery nice and clean.
The challenge to this method is usually the need to deploy a split DNS for internal and external name resolution. (The internal DNS server also serving the domain.com zone and having the A records for internal services, where the external DNS server have A records for external services.)
2) Create certs using your own internal CA, like Microsoft AD Certificate Services, or OpenSSL, etc. There are no restrictions on SANs with your own certificate server. I detail how to use OpenSSL to sign certs in an earlier post.
The major constraint to this deployment option is the need to get the trusted cert from your CA server on to all devices that will use MRA. AD does it for your Windows machines automatically, but mobile devices will need to have this certificate installed. Using an MDM like Meraki MDM (freemium service) or others to push the certificates would be the way I’d attempt to deploy the certificates
The Implications of changing the domain name of CUCM/CUC/IM&P
Anyone who’s attempted to change the hostname of a CallManager knows the trainwreck and ensuing TAC calls that will ensue.
I’ve personally not tried to change the domain name of a CallManager or CUC in recent memory, but doing so for IM&P/CUP is relatively straightforward.
Collab Edge is now supported. The official Mobile-Remote-Access-via-Expressway-Deployment-Guide is located here.
I’m updating this document to reflect changes made in Expressway-C/E 8.1 that make importing the certificates MUCH easier.
This document explains how to deploy Collaboration Edge with on-prem presence (IM&P/CUP) on a non-redundant set of Expressway-E and C VMs. Deploying with WebEx Messenger is not covered here, but the bulk of the configuration is the same as far as the Expressway piece.
The biggest challenge in the initial deployment was finding all of the necessary documentation! Things you need to know like certificate chaining, or OpenSSL are in various docs. I’ve linked all of the documents that I used and tried to summarize things to make it quicker to deploy.
What you’ll need to deploy Collaboration Edge today:
IM&P (CUP) 9.1+
VCS or “Expressway” X8.1.1+
A Collaboration Edge enabled Jabber client: Cisco Jabber for Windows 9.7+, Cisco Jabber for iOS 9.6, Jabber for Android 9.6+ or Cisco Jabber for MAC 9.6+
Updated jabber-config.xml on CUCM with RemoteAccess turned on (This is no longer required for Jabber 9.6+)
Two certificates (one for VCSe another for VCSc) – either signed by your own CA (OpenSSL or similar) or publically signed certs like GoDaddy, Verisign, etc.
A few notes about nomenclature:
Collaboration Edgeis the architecture umbrella term for the VCS/Expressway edge proxy for CUCM-registered clients (Jabber and TC7.0 TP units). It’s commonly used to refer to the Jabber piece of it, but will support endpoints too. (The DX650 will support Collaboration Edge in a future release of firmware. Traditional IP phones will not be supported, they will use VPN Phone or CUBE lineside proxy.)
Mobile and Remote Access (MRA) is the term used in VCS/Expressway documentation for the VPN-less Jabber (and CUCM-registered TC7.0 endpoint) proxy feature.
Cisco Expressway-Edge is the same software as VCS-Expressway, just packaged for CUCM registered endpoints. Expressway-Edge is a VCS-Expressway that is deployed as a Mobile and Remote Access proxy or for traversal calls for CUCM registered endpoints. There is a license file actually changes the title to say “Expressway-E” when it is loaded. In the rest of the document, I will refer to VCS-Expressway, VCS-E, Expressway-Edge, Expressway-E as Expressway-E since we are primarily talking about MRA.
Expressway-Core is the same story. It is VCS-Control software deployed as an MRA proxy only with the Expressway-C license loaded. We call it VCS-Control when it is licensed for device registration, non-traversal calls, FindMe, and other features like Lync interop. For purposes of this document I’ll call it the Expressway-C below.
Customers with a valid UCSS contract for UCL-Enhanced, CUWL-STD, or CUWL-PRO are entitled to Expressway-Edge and Expressway-Core for free (for MRA) and their license will reflect the Expressway names. Licenses are charged for the other VCS features mentioned above. Licenses are required and have a cost for B2B/B2C (Jabber Guest) calls through Expressway-C/E Each box requires one media session license to get a session through.
VCS-C and VCS-E can have the MRA features turned on and run on a pair and do both functions. We are still awaiting clarification as to when you must break these apart and run a separate set of VCS (for B2B, interop) and Expressway (for MRA) servers. (Update: VCS is supported for limited sized deployments.)
Expressway licenses should be orderable via PUT, or you can use your existing VCS severs by upgrading to X8.1. (Update: I posted a later post that discusses what to order.)
2) Decide if you want to deploy valid security certificates on CUCM, IM&P and CUC. You will likely want to do this independent of Collaboration Edge as all of the Jabber clients are no longer trusting self-signed certificates. By providing a publicly trusted cert, Jabber won’t throw Invalid Certificate errors as you log in. Granted they are only shown once during the very first login if the user accepts them on each client. If you do put certificates on those components I’d suggest getting them for a 5-year term so you aren’t dealing with it in a year when the certificates expire.
Directory Lookup Considerations
MRA only supports UDS as the directory lookup service. If you are inside you can use LDAP (EDI/BDI), outside UDS.
Jabber-config.xml Update for 9.6 clients
Update: This section is no longer required as current versions of the clients (Win 9.7, iOS 9.6.1, Android 9.6, OS X 9.6) to do MRA by default, negating the requirement to pull the jabber-config.xml file first (expect in the case of split internal/external domains).
CUCM UC Service Profiles
Make sure that you’ve configure CUCM UC Service Profiles (this should have been done as part of your initial IM&P/CUP deployment and won’t be covered here) and assigned them to the end users.
Recall there is a single OVA that does both Expressway-E/C and VCS-E/C that you need to deploy — it’s just a matter of how you configure and license (request via PUT as mentioned above) it as to what it is called. Download the OVA from Cisco here
You’ll deploy the Expressway-C on your internal network (presumably on the same VLAN as CUCM and other UC components) e.g. 10.10.1.30
You’ll deploy Expressway-E one of two ways. Either on a stick in your DMZ (perhaps 10.99.99.30), or two-legged with the external interface in the DMZ network (e.g. 10.99.99.30 – or on your public address space), and the internal interface on the internal network (presumably on the same VLAN as Expressway-C and other UC components – e.g. 10.10.1.31).
You’ll need to trunk your DMZ to your ESXi host if you haven’t, or figure out how to deal with getting the Expressway-E external (LAN2) interface in the DMZ network.
Once the VM’s are deployed edit the Expressway-E VM settings to put LAN2 in the DMZ network (alternately you can put LAN2 on the inside and LAN1 in the DMZ). If you don’t see options for the second NIC, or options for NAT, you are missing the Advanced Networking license. You need this in order to have it two legged, or do NAT.
I don’t know how long I wasted on my first install because I forgot to modify the ACL to include the additonal ports that MRA requires. I just assumed that beause VCS was working for B2B that it would work for MRA. Not the case!
You’ll need to configure NAT on your firewall from a public IP address outside your network to the DMZ address of Expressway-E, or do 1:1 public to your DMZ if you’ve deployed it with a public address.
You will need two DNS servers for MRA to function properly. Jabber decides if it is inside the network or outside the network depending on what SRV records it can resolve. Depending on what records it resolves it will either try to use MRA or it will directly connect to CUCM/IM&P.
Internal DNS Server
Create two A records:
sjc-expressway-edge-01.domain.com A – (make this name whatever you want) Pointing to the INSIDE interface of Expressway-E for two-legged deployments, or pointing to the DMZ address if it’s on a stick. The record is used by Expressway-C to lookup and validate the certificate against. You will use this hostname anywhere you are asked for the expressway server’s name whe configuring the C server.
sjc-expressway-core-01.domain.com A – (any name you want) Pointing to Expressway-C.
Create two SRV records:
_cisco-uds._tcp.domain.com SRV 0 0 port 8443 – Pointing to CUCM. (NOT IM&P!)
_cuplogin._tcp.domain.com SRV 0 0 port 8443 – Pointing to IM&P (TBD if this is really required for Jabber 9.6 with IM&P 9.1 – I don’t believe it actually is)
When you launch Jabber, if it can resolve these DNS records, it knows it’s inside and pulls the service profile directly from CUCM and logs in to IM&P and CUCM.
External DNS Server
Create one A record:
sjc-expressway-edge-01.domain.com A – (any name you want) Pointing to the public address assigned (or NATted) to your Expressway-E.
Create one SRV record:
_collab-edge._tls.domain.com SRV 0 0 8443 – Pointing to Expressway-E (in our case sjc-expressway-edge-01.domain.com)
Configure Expressway-Edge and Expressway-Core
Follow this chapter of the Expressway Admin Guide – Mobile and Remote Access (feature preview) beginning at p.52 but stop half-way down p.56 (before the beginning of the Certificates section).
A couple notes: I did not enable TLS verify mode on my CUCM and IM&P server definitions because just wanted to get it up and running. I’m suggesting putting real certs on CUCM, IM&P, and CUC, and turning TLS verify on, but this can be done later.
Valid CA-signed certificates are required to setup the traversal zone for MRA. You can either get public ones, or sign your own with your own CA. I’ve done it both ways. The major reason for a valid trusted CA-signed certificate is to stop Jabber from throwing a certificate warning on the initial MRA login to Expressway-E itself. I highly recommend deploying a publicly trusted CA signed certificate.
Update: This is fixed in Expressway 8.1.1 Ignore this section below:
Add UC Domain (domain.com) and XMPP server information
Download the CSR
Upoad the CSR file to the CA to get the certificate signed
Get the signed server PEM and the root/intermediate chain PEM back from the CA.
Upload the signed server cert to Expressway-E under Maintenance | Security Certificates | Server Certificate
Break apart the CA-intermediate-root certificates into individual PEMs for import – See the WebEX instruction for VCS 8.1 to learn how to do this.
Import into the Trusted CA certificate list: the top-level cert (“CA”), then the root cert, then the intermediate cert found under Maintenance | Security Certificates | Trusted CA Certificates
Reboot to make them active.
Follow the Webex instructions to break apart the CA-intermediate and root PEM into individual certs using Windows so that you can import them into the CA trusted cert list properly.
Repeat this procedure for Expressway-C.
For the customers that I’ve worked with using GoDaddy certificates. I’ve worked with four certificates – Go Daddy Class 2 CA; Go Daddy Root Certificate Authority – G2; Go Daddy Secure Certificate Authority – G2; the server certificate itself.
I used Chrome on Windows to export the three Go Daddy certificates individually to Base 64 .PEM and then loaded them into the Expressway-E/C trusted CA list. This worked perfectly for me after loading and rebooting the servers. The UC traversal zones came right up.
Sign your own using OpenSSL if you’d like
If you want to use OpenSSL to create your own CA cert and sign your CSR, it is actually easier than you’d think.
Start at the bottom of p.13 of this document –
You’ll follow the procedure twice. Once for Expressway-E and once for Expressway-C. Take the CA root cert that you generated and import it into the trusted list on both boxes, and then import your signed server cert on the appropriate box.
Traversal Zone Configuration
Resume the configuration tasks in the Admin guide on p.58 making sure to put the proper settings for both Expressway-E and Expressway-C.
If your certificates are good, you will see the traversal zone go active on both servers under Status | Unified Communications. If not, double-check your configuration settings, and double-check your certificates.
Troubleshooting Zone Configuration
If the zone won’t go active and you think it looks good, check the logs to see what is happening. My initial attempt where the certificates were not chained properly showed a continuous loop of TLS failures. When I had my Expressway-C pointing to the external public address instead of the inside interface of Expressway-E, TLS looked good and even the SSH tunnel showed “up” but traffic wasn’t actually flowing.
The best place I found to troubleshoot this stuff was by putting the Expressway-C and E in “Devel mode” to enable the Experimental menu. (Instructions for this are found on p.207 of the admin guide.) The reason for this is because the CollabEdge/MRA feature is still considered experimental. You need to look at the Developer Logs. You can enable them for debug level as well as collect a tcpdump.
Make sure to add your Unity Connection, and any other servers that Jabber needs access to. Unity Connection requires it for Visual Voicemail to work.
Launch Jabber 9.6 internally
Update: No longer required unless you are doing separate internal and external domains. (I’ll detail this in a later post.)
Launch Jabber on your client device on the inside network (so that it has direct access to CUCM/IM&P). When you enter your email address Jabber should automatically discover your servers (using the before setup internal DNS SRV records). If Jabber does not auto-discover, troubleshoot your SRV records. The easiest method is to use dig or nslookup.
The quick nslookup method is to:
Launch the program,
Make sure it shows your internal DNS servers (that your device should be pulling via DHCP scope options)
Enter set type=SRV, then type _cisco-uds._tcp.domain.com. This should resolve to the hostname of CUCM, or the IP address of it.
If using the hostname, exit nslookup and try to ping the hostname.
Once you enter your credentials you will likely be presented with several invalid certs to accept, and your client should connect and have IM, Presence, CUCM, CUC, and be able to IM and do voice/video calls.
Sign out and close Jabber
Launch Jabber 9.6 externally
Disconnect from your internal network and make sure your device is outside your network where a) it cannot resolve the internal SRV records, and b) it can resolve the external _collab-edge SRV record and access your Expressway-E from the outside.
Launch Jabber on your device. Jabber will attempt to resolve _cisco-uds._tcp.domain.com and will fail to do so. It will also attempt to resolve _cuplogin._tcp.domain.com and will fail. It will then attempt to resolve _collab-edge._tls.domain.com and get pointed to the public IP address of Expressway-E.
It will then connect to Expressway-E, and a if everything is configured properly it will login and you’ll show connected to IM, CUCM and CUC!
Notes about iOS
On iOS the timeout for attempts to login is MINUTES long. Be very patient for it to either succeed or fail. It can take a significant amount of time to login successfully on the 9.6 build. 9.6.1 is supposed to be much faster.
If your login fails, click the Send Error Report and email it to yourself. Open the ZIP file and look through (going from bottom to top) to see where the errors are. The logs will include more than just the current login attempt, so note the time when you are attempting to login and look at the timestamps in the log. This is critical so that you aren’t troubleshooting an old login that isn’t relevant to your current problem.
From my experience:
When I had firewall issues, I was seeing CONNECTION_TIMEOUT errors when trying to login via MRA, but not when I was inside.
When I had neglected to enable RemoteAccess in jabber-config.xml I was seeing RemoteAccess Policy errors.
I’m impressed with the ability to finally be able to do voice/video calls from anywhere! It’s about time. Collab Edge is still considered a Feature Preview by Cisco and isn’t TAC supported yet. Please send me questions that you have as you attempt to deploy it.