DNS entries: Do you have forward and reverse DNS lookups for all infrastructure
systems that the Expressway interacts with? If the Expressway cannot resolve hostnames and IP addresses of systems,your complex deployments (eg.MRA) could stop working as expected after you upgrade.
Oddly enough, the two systems that I’ve been involved in the upgrade to 8.8.1 with, both had the Unified Communications traversal zone with show Active, and hard phones (8800 and DX 650) will register and work properly, but Jabber clients will be unable to login and Jabber will throw an error when trying to login through MRA:
"Unable to Communicate with Server."
Running the debugging logs on Expressway-C you see the following error:
"Certificate verification failed for host=x.x.x.x, additional info:
Invalid Hostname expressway-e.domain.com"
The fix is to make sure that Expressway-C can do a reverse DNS lookup on the IP address of Expressway-E. Then flush the DNS cache of C to make sure it re-queries DNS properly.
The debugging log will give you the address and hostname it is trying to do the lookup on.
In a dual-NIC Expressway-E deployment the PTR recrod should point to the private IP address that C talks to. In a single-NIC NAT hairpin deployment, I’ve seen it talk on the private and public IP. So check that debug log.
I began the deployment by registering the MX-300 I use directly to CUCM to make sure I had it successfully working before attempting registration through MRA. I’d previously had it registered to a VCS-C.
MRA requires TC 7.3 code, so I decided to deploy 7.3.4 since it is the latest bugfix version. I downloaded and deployed the COP file to CUCM since CUCM will be in control of what version of software the TC endpoint will use once it’s registered to CUCM (like typical phones get firmware).
Note that TC 7.3.3 or greater firmware have different functionality for remote screen monitoring of systems! TC 7.3.3 introduces the requirement to have an option key for remote system/screen monitoring of the TC endpoint. You’ll need to work with TAC/Cisco to get option keys cut if you are doing remote screen monitoring before going to 7.3.3+.
Registration to CUCM was straightforward. I defined the device on CUCM as you would normally define a phone, picking the appropriate SIP profiles. I didn’t do secure registration as this is optional. (The documentation above does mention that secure registration is optional, but the example works through a secure registration.)
I made sure to set the device association my end user in CUCM. This is important for MRA later.
Once the MX-300 was registered and making calls successfully through CUCM, I moved it out to a general internet connection to work on MRA.
On the touchpanel I launched the Provisioning wizard and selected Cisco UCM via Expressway. After putting in my credentials I was greeted with this error:
After doing quite a bit of research and looking at the detailed error logs from the MX-300, it turns out that your Expressway-E certificate must also include a SAN for the domain name itself (e.g. yourdomain.com). The error actually indicates that it wants a collab-edge.yourdomain.com SAN:
Edge TLS verification failed: Edge domain ‘yourdomain.com’ and corresponding SRVName ‘collab-edge._tls.yourdomain.com’ not found in certificate SAN list.
The challenge I had is that the certificate I’d bought from GoDaddy for Expressway-E (that was working with 8800 MRA) wasn’t a UCC or multi-SAN certificate and you need to have at least the Expressway-E as the CN and the domain as a SAN.
At this point I decided that it was time to move from GoDaddy to DigiCert since they have unlimited resigning of certificates without having to revoke any of them. This essentially allows you to create as many certificates as you want without having to keep buying more like GoDaddy. I bought the Wildcard Plus certificate and used it to create a multi-SAN certificate for my Expressway-E. The CN is always *.yourdomain.com, but you can add a bunch of SANs (like 20 or so.)
I generated the certificate with the following SANs – edge.yourdomain.com, yourdomain.com, collab-edge._tls.yourdomain.com and _collab-edge._tls.yourdomain.com. One of the partner engineers that I talked to said that he got it working without having to add either collab-edge SAN. (I’ve not looked into why/when we’ll need the collab-edge SAN and if it is actually be collab-edge as the error indicates, or if it needs the preceding underscore on collab-edge like the SRV records have.)
After applying the certificate to Expressway-E and rebooting it I tried the provisioning wizard on the touchpanel again and was greeted with the SAME ERROR.
After this the MX-300 registered like a champ and is able to do calls. I followed this same process to get an SX-10 registered as well.
The partner engineer I talked to said he had to work through a couple issues on his test: 1) Endpoint rejecting the user credentials when running through the provisioning wizard. Make sure the endpoint is associated with the End User and that the end user has CTI Enabled and CCM End user. 2) Getting an http download error after getting through the initial expressway authentication and that it was caused by the endpoint needing to do a _cisco-uds.yourdomain.com lookup internally to find out where CUCM is to download it’s configuration. He was in a split domain situation and didn’t have a _cisco-uds record for the external domain on the inside.
I’ll detail the adventure for the DX-series on another post.
I’ve run into a couple challenges after an upgrade to Expressway/VCS 8.5.2 where MRA for phones quit working.
I found a bug that broke MRA in 8.5.2 (the recommendation has been to downgrade back to 8.5.1). That bug is shows that it is now fixed in 8.6, so I upgraded to 8.6.1 recently. MRA started working again, but only on one out of every three login attempts. It was really weird. In looking at the logs it showed a bunch of errors:
Home CUCM not available – Unknown CUCM cluster for node sub02
Home CUCM not available – Unknown CUCM cluster for node sub03
The deployment I was working on is a three node (pub and two subs), running split DNS (different internal domain than the external domain name).
After a lot of digging it turns out that there is a change in the way MRA handles CUCM lookup. When I installed the system added my pub and subs to Expressway-C by IP address. But it looks like Expressway now attempts to communicate with them via hostname, and not IP as they were defined by me.
Since Expressway is using the domain suffix assigned to MRA (extdomain.com), it is attempting to lookup sub02.extdomain.com and sub03.extdomain.com. I didn’t have A records for these on my internal DNS server extdomain zone since I’ve never needed to resolve them by the external domain name.
Adding these two records fixed the login issue and it now logins on first attempt like it used to.
Cisco recently posted Expressway (and VCS) X8.5.2 and 10.3.1 firmware for the 8800 and 7800 series phones. The combination of these products allows these phones to register remotely to CUCM utilizing Collaboration Edge MRA. (The DX-series (650/70/80) is expected to support MRA in the next release of code due out shortly.)
This functionality isn’t TAC supported yet, and has been released in a “feature preview” form. I’ve set it up and tested it and it works well for the most part. However, there is not full feature parity for a phone registered via MRA vs directly registered to CUCM, but for testing and basic calls, it works well.
In order to set it up, make sure your Expressway MRA deployment for Jabber is working properly. MRA for the 7800/800 series phones uses the same service discovery process that Jabber uses, so if you have Jabber working, you’ll have 95% of the work done.
One important piece of information to know is that the phone firmware trusts 100+ public root CA certificates. If your Expressway-E server does not have a certificate signed by one of these CA’s it’s not going to work for the phones.
Here’s the basic procedure I followed to make it work:
Installed the 8800 10.3.1 COP file via OSadmin and restarted the TFTP service.
Logged in to Jabber via MRA to ensure the correct functionality of my MRA system and my login credentials.
Defined the phone in CUCM and then connected the phone directly to CUCM so that it would pull the version of firmware that supports MRA. (My 8851 phone shipped with an older version of code that did not have MRA support.)
Took the phone off of the corporate network to an internet-access only network.
I had initial problems with the phone not attempting MRA lookup after being connected to the internet-only network, so I followed the troubleshooting process of resetting the Network settings on the phone. It then started to try the MRA process.
Steps the phone follows in MRA registration:
1) Phone attempts normal TFTP registration/_cisco-uds._tcp.domain.com lookup process:
This fails because the phone has no direct access to CUCM.
2) Firmware now prompts for MRA credentials (These would be the same credentials you use for Jabber MRA login — in my case it is set to use LDAP/AD for authentication):
Phone now attemps _collab-edge._tls.domain.com service record lookup (like Jabber does) to discover the Expressway-E/VCS-E host.
3) Phone completes MRA login process
The phone is now registered and usable.
I’ve read conflicting information about the number of calls supported, and number of lines supported via MRA. In my experience I have two lines on the phone registered and am able to make two calls per line. (I’ve not tested more than two calls per line.) The list of features that may work or not is extensive, so be careful as things like Barge or Intercom may not work yet.
The phone also upgraded code via MRA successfully which is good to know.
I’ve noticed some oddities with on-hook vs. off-hook dialing. I know there are some limitations around KPML currently. In my experience it seems to off-hook dial fine on the primary line, but on a secondary line or when attempting a second call on the primary line you MUST on-hook dial.
Phone registration isn’t supported via TAC yet so feel free to post here and we can collectively attempt to assist. Remember the most basic step to troubleshoot is to see if your Jabber can successfully login.
An internal name is a domain or IP address that is part of a private network. Common examples of internal names are:
Any server name with a non-public domain name suffix. For example, http://www.contoso.local or server1.contoso.internal.
NetBIOS names or short hostnames, anything without a public domain. For example, Web1, ExchCAS1, or Frodo.
Any IPv4 address in the RFC 1918 range.
Any IPv6 address in the RFC 4193 range.
Why do we care?
Jabber authenticates TLS encryption using certificates for services from CUCM, CUC, IM&P, etc. Historically these have typically been deployed with IP addresses only, or internal domains (e.g. domain.local, etc.). Because of this you can no longer get a certificate for the Expressway-C box that has SANs with IPs or internal names. Jabber requires valid certificates for login now.
Without a certificate with proper SANs, Jabber will either throw an invalid cert error, or will completely deny login to UC services.
Using Collab Edge MRA, Jabber authenticates to the Expressway-E server and uses it’s certifciate. Internally Jabber communicates directly with each component.
Dealing with the issue for Collab Edge MRA
Basically we have two options to work around the SAN issue:
1) Change the domain name of the UC components to a valid public domain name that the public CA will sign for. This doesn’t mean the server has to be accessible from the internet by any means or that it is an existing domain name your company is using.
Option 1a: Deploy a new public domain name for UC services internally. For example if your domain name was domain.com you might see if domain.info or domain.net or something similar is available to register and use as the internal UC domain name. The domain wouldn’t need to resolve externally at all.
If you do this, then you need to take in to consideration that the MRA deployment becomes a multi-domain (or split-domain) deployment which requires some special treatment like the VoiceServicesDomain option. (See my previous post about multi-domain deployments.)
Options 1b: The seemingly easier deployment would be to just match your public domain name that you use for email (e.g. domain.com) for your UC components (not suggesting all internal servers — file, print or otherwise, need to be in this domain). This makes services discovery nice and clean.
The challenge to this method is usually the need to deploy a split DNS for internal and external name resolution. (The internal DNS server also serving the domain.com zone and having the A records for internal services, where the external DNS server have A records for external services.)
2) Create certs using your own internal CA, like Microsoft AD Certificate Services, or OpenSSL, etc. There are no restrictions on SANs with your own certificate server. I detail how to use OpenSSL to sign certs in an earlier post.
The major constraint to this deployment option is the need to get the trusted cert from your CA server on to all devices that will use MRA. AD does it for your Windows machines automatically, but mobile devices will need to have this certificate installed. Using an MDM like Meraki MDM (freemium service) or others to push the certificates would be the way I’d attempt to deploy the certificates
The Implications of changing the domain name of CUCM/CUC/IM&P
Anyone who’s attempted to change the hostname of a CallManager knows the trainwreck and ensuing TAC calls that will ensue.
I’ve personally not tried to change the domain name of a CallManager or CUC in recent memory, but doing so for IM&P/CUP is relatively straightforward.
Collab Edge is now supported. The official Mobile-Remote-Access-via-Expressway-Deployment-Guide is located here.
I’m updating this document to reflect changes made in Expressway-C/E 8.1 that make importing the certificates MUCH easier.
This document explains how to deploy Collaboration Edge with on-prem presence (IM&P/CUP) on a non-redundant set of Expressway-E and C VMs. Deploying with WebEx Messenger is not covered here, but the bulk of the configuration is the same as far as the Expressway piece.
The biggest challenge in the initial deployment was finding all of the necessary documentation! Things you need to know like certificate chaining, or OpenSSL are in various docs. I’ve linked all of the documents that I used and tried to summarize things to make it quicker to deploy.
What you’ll need to deploy Collaboration Edge today:
IM&P (CUP) 9.1+
VCS or “Expressway” X8.1.1+
A Collaboration Edge enabled Jabber client: Cisco Jabber for Windows 9.7+, Cisco Jabber for iOS 9.6, Jabber for Android 9.6+ or Cisco Jabber for MAC 9.6+
Updated jabber-config.xml on CUCM with RemoteAccess turned on (This is no longer required for Jabber 9.6+)
Two certificates (one for VCSe another for VCSc) – either signed by your own CA (OpenSSL or similar) or publically signed certs like GoDaddy, Verisign, etc.
A few notes about nomenclature:
Collaboration Edgeis the architecture umbrella term for the VCS/Expressway edge proxy for CUCM-registered clients (Jabber and TC7.0 TP units). It’s commonly used to refer to the Jabber piece of it, but will support endpoints too. (The DX650 will support Collaboration Edge in a future release of firmware. Traditional IP phones will not be supported, they will use VPN Phone or CUBE lineside proxy.)
Mobile and Remote Access (MRA) is the term used in VCS/Expressway documentation for the VPN-less Jabber (and CUCM-registered TC7.0 endpoint) proxy feature.
Cisco Expressway-Edge is the same software as VCS-Expressway, just packaged for CUCM registered endpoints. Expressway-Edge is a VCS-Expressway that is deployed as a Mobile and Remote Access proxy or for traversal calls for CUCM registered endpoints. There is a license file actually changes the title to say “Expressway-E” when it is loaded. In the rest of the document, I will refer to VCS-Expressway, VCS-E, Expressway-Edge, Expressway-E as Expressway-E since we are primarily talking about MRA.
Expressway-Core is the same story. It is VCS-Control software deployed as an MRA proxy only with the Expressway-C license loaded. We call it VCS-Control when it is licensed for device registration, non-traversal calls, FindMe, and other features like Lync interop. For purposes of this document I’ll call it the Expressway-C below.
Customers with a valid UCSS contract for UCL-Enhanced, CUWL-STD, or CUWL-PRO are entitled to Expressway-Edge and Expressway-Core for free (for MRA) and their license will reflect the Expressway names. Licenses are charged for the other VCS features mentioned above. Licenses are required and have a cost for B2B/B2C (Jabber Guest) calls through Expressway-C/E Each box requires one media session license to get a session through.
VCS-C and VCS-E can have the MRA features turned on and run on a pair and do both functions. We are still awaiting clarification as to when you must break these apart and run a separate set of VCS (for B2B, interop) and Expressway (for MRA) servers. (Update: VCS is supported for limited sized deployments.)
Expressway licenses should be orderable via PUT, or you can use your existing VCS severs by upgrading to X8.1. (Update: I posted a later post that discusses what to order.)
2) Decide if you want to deploy valid security certificates on CUCM, IM&P and CUC. You will likely want to do this independent of Collaboration Edge as all of the Jabber clients are no longer trusting self-signed certificates. By providing a publicly trusted cert, Jabber won’t throw Invalid Certificate errors as you log in. Granted they are only shown once during the very first login if the user accepts them on each client. If you do put certificates on those components I’d suggest getting them for a 5-year term so you aren’t dealing with it in a year when the certificates expire.
Directory Lookup Considerations
MRA only supports UDS as the directory lookup service. If you are inside you can use LDAP (EDI/BDI), outside UDS.
Jabber-config.xml Update for 9.6 clients
Update: This section is no longer required as current versions of the clients (Win 9.7, iOS 9.6.1, Android 9.6, OS X 9.6) to do MRA by default, negating the requirement to pull the jabber-config.xml file first (expect in the case of split internal/external domains).
CUCM UC Service Profiles
Make sure that you’ve configure CUCM UC Service Profiles (this should have been done as part of your initial IM&P/CUP deployment and won’t be covered here) and assigned them to the end users.
Recall there is a single OVA that does both Expressway-E/C and VCS-E/C that you need to deploy — it’s just a matter of how you configure and license (request via PUT as mentioned above) it as to what it is called. Download the OVA from Cisco here
You’ll deploy the Expressway-C on your internal network (presumably on the same VLAN as CUCM and other UC components) e.g. 10.10.1.30
You’ll deploy Expressway-E one of two ways. Either on a stick in your DMZ (perhaps 10.99.99.30), or two-legged with the external interface in the DMZ network (e.g. 10.99.99.30 – or on your public address space), and the internal interface on the internal network (presumably on the same VLAN as Expressway-C and other UC components – e.g. 10.10.1.31).
You’ll need to trunk your DMZ to your ESXi host if you haven’t, or figure out how to deal with getting the Expressway-E external (LAN2) interface in the DMZ network.
Once the VM’s are deployed edit the Expressway-E VM settings to put LAN2 in the DMZ network (alternately you can put LAN2 on the inside and LAN1 in the DMZ). If you don’t see options for the second NIC, or options for NAT, you are missing the Advanced Networking license. You need this in order to have it two legged, or do NAT.
I don’t know how long I wasted on my first install because I forgot to modify the ACL to include the additonal ports that MRA requires. I just assumed that beause VCS was working for B2B that it would work for MRA. Not the case!
You’ll need to configure NAT on your firewall from a public IP address outside your network to the DMZ address of Expressway-E, or do 1:1 public to your DMZ if you’ve deployed it with a public address.
You will need two DNS servers for MRA to function properly. Jabber decides if it is inside the network or outside the network depending on what SRV records it can resolve. Depending on what records it resolves it will either try to use MRA or it will directly connect to CUCM/IM&P.
Internal DNS Server
Create two A records:
sjc-expressway-edge-01.domain.com A – (make this name whatever you want) Pointing to the INSIDE interface of Expressway-E for two-legged deployments, or pointing to the DMZ address if it’s on a stick. The record is used by Expressway-C to lookup and validate the certificate against. You will use this hostname anywhere you are asked for the expressway server’s name whe configuring the C server.
sjc-expressway-core-01.domain.com A – (any name you want) Pointing to Expressway-C.
Create two SRV records:
_cisco-uds._tcp.domain.com SRV 0 0 port 8443 – Pointing to CUCM. (NOT IM&P!)
_cuplogin._tcp.domain.com SRV 0 0 port 8443 – Pointing to IM&P (TBD if this is really required for Jabber 9.6 with IM&P 9.1 – I don’t believe it actually is)
When you launch Jabber, if it can resolve these DNS records, it knows it’s inside and pulls the service profile directly from CUCM and logs in to IM&P and CUCM.
External DNS Server
Create one A record:
sjc-expressway-edge-01.domain.com A – (any name you want) Pointing to the public address assigned (or NATted) to your Expressway-E.
Create one SRV record:
_collab-edge._tls.domain.com SRV 0 0 8443 – Pointing to Expressway-E (in our case sjc-expressway-edge-01.domain.com)
Configure Expressway-Edge and Expressway-Core
Follow this chapter of the Expressway Admin Guide – Mobile and Remote Access (feature preview) beginning at p.52 but stop half-way down p.56 (before the beginning of the Certificates section).
A couple notes: I did not enable TLS verify mode on my CUCM and IM&P server definitions because just wanted to get it up and running. I’m suggesting putting real certs on CUCM, IM&P, and CUC, and turning TLS verify on, but this can be done later.
Valid CA-signed certificates are required to setup the traversal zone for MRA. You can either get public ones, or sign your own with your own CA. I’ve done it both ways. The major reason for a valid trusted CA-signed certificate is to stop Jabber from throwing a certificate warning on the initial MRA login to Expressway-E itself. I highly recommend deploying a publicly trusted CA signed certificate.
Update: This is fixed in Expressway 8.1.1 Ignore this section below:
Add UC Domain (domain.com) and XMPP server information
Download the CSR
Upoad the CSR file to the CA to get the certificate signed
Get the signed server PEM and the root/intermediate chain PEM back from the CA.
Upload the signed server cert to Expressway-E under Maintenance | Security Certificates | Server Certificate
Break apart the CA-intermediate-root certificates into individual PEMs for import – See the WebEX instruction for VCS 8.1 to learn how to do this.
Import into the Trusted CA certificate list: the top-level cert (“CA”), then the root cert, then the intermediate cert found under Maintenance | Security Certificates | Trusted CA Certificates
Reboot to make them active.
Follow the Webex instructions to break apart the CA-intermediate and root PEM into individual certs using Windows so that you can import them into the CA trusted cert list properly.
Repeat this procedure for Expressway-C.
For the customers that I’ve worked with using GoDaddy certificates. I’ve worked with four certificates – Go Daddy Class 2 CA; Go Daddy Root Certificate Authority – G2; Go Daddy Secure Certificate Authority – G2; the server certificate itself.
I used Chrome on Windows to export the three Go Daddy certificates individually to Base 64 .PEM and then loaded them into the Expressway-E/C trusted CA list. This worked perfectly for me after loading and rebooting the servers. The UC traversal zones came right up.
Sign your own using OpenSSL if you’d like
If you want to use OpenSSL to create your own CA cert and sign your CSR, it is actually easier than you’d think.
Start at the bottom of p.13 of this document –
You’ll follow the procedure twice. Once for Expressway-E and once for Expressway-C. Take the CA root cert that you generated and import it into the trusted list on both boxes, and then import your signed server cert on the appropriate box.
Traversal Zone Configuration
Resume the configuration tasks in the Admin guide on p.58 making sure to put the proper settings for both Expressway-E and Expressway-C.
If your certificates are good, you will see the traversal zone go active on both servers under Status | Unified Communications. If not, double-check your configuration settings, and double-check your certificates.
Troubleshooting Zone Configuration
If the zone won’t go active and you think it looks good, check the logs to see what is happening. My initial attempt where the certificates were not chained properly showed a continuous loop of TLS failures. When I had my Expressway-C pointing to the external public address instead of the inside interface of Expressway-E, TLS looked good and even the SSH tunnel showed “up” but traffic wasn’t actually flowing.
The best place I found to troubleshoot this stuff was by putting the Expressway-C and E in “Devel mode” to enable the Experimental menu. (Instructions for this are found on p.207 of the admin guide.) The reason for this is because the CollabEdge/MRA feature is still considered experimental. You need to look at the Developer Logs. You can enable them for debug level as well as collect a tcpdump.
Make sure to add your Unity Connection, and any other servers that Jabber needs access to. Unity Connection requires it for Visual Voicemail to work.
Launch Jabber 9.6 internally
Update: No longer required unless you are doing separate internal and external domains. (I’ll detail this in a later post.)
Launch Jabber on your client device on the inside network (so that it has direct access to CUCM/IM&P). When you enter your email address Jabber should automatically discover your servers (using the before setup internal DNS SRV records). If Jabber does not auto-discover, troubleshoot your SRV records. The easiest method is to use dig or nslookup.
The quick nslookup method is to:
Launch the program,
Make sure it shows your internal DNS servers (that your device should be pulling via DHCP scope options)
Enter set type=SRV, then type _cisco-uds._tcp.domain.com. This should resolve to the hostname of CUCM, or the IP address of it.
If using the hostname, exit nslookup and try to ping the hostname.
Once you enter your credentials you will likely be presented with several invalid certs to accept, and your client should connect and have IM, Presence, CUCM, CUC, and be able to IM and do voice/video calls.
Sign out and close Jabber
Launch Jabber 9.6 externally
Disconnect from your internal network and make sure your device is outside your network where a) it cannot resolve the internal SRV records, and b) it can resolve the external _collab-edge SRV record and access your Expressway-E from the outside.
Launch Jabber on your device. Jabber will attempt to resolve _cisco-uds._tcp.domain.com and will fail to do so. It will also attempt to resolve _cuplogin._tcp.domain.com and will fail. It will then attempt to resolve _collab-edge._tls.domain.com and get pointed to the public IP address of Expressway-E.
It will then connect to Expressway-E, and a if everything is configured properly it will login and you’ll show connected to IM, CUCM and CUC!
Notes about iOS
On iOS the timeout for attempts to login is MINUTES long. Be very patient for it to either succeed or fail. It can take a significant amount of time to login successfully on the 9.6 build. 9.6.1 is supposed to be much faster.
If your login fails, click the Send Error Report and email it to yourself. Open the ZIP file and look through (going from bottom to top) to see where the errors are. The logs will include more than just the current login attempt, so note the time when you are attempting to login and look at the timestamps in the log. This is critical so that you aren’t troubleshooting an old login that isn’t relevant to your current problem.
From my experience:
When I had firewall issues, I was seeing CONNECTION_TIMEOUT errors when trying to login via MRA, but not when I was inside.
When I had neglected to enable RemoteAccess in jabber-config.xml I was seeing RemoteAccess Policy errors.
I’m impressed with the ability to finally be able to do voice/video calls from anywhere! It’s about time. Collab Edge is still considered a Feature Preview by Cisco and isn’t TAC supported yet. Please send me questions that you have as you attempt to deploy it.