Helped a customer upgrade from 11.0 to CSR 11.5, CUCM 11.5(1)SU1; IM&P 11.5(1)SU1; CUC 11.5(1)SU1.
Unity Connection 11.5
You must apply ciscocm.cuc_11.5SU1_pre_upgrade.cop.sgn before you upgrade to 11.5 because of bugid CSCvb02774. The install of the patch is straightfoward and does not require a reboot. I also ran a utils iothrottle disable to make the upgrade run faster (since it was being done after hours.)
If you’re upgrading from 10.x or earlier it is CRITICAL to increase your VM RAM to 6GB. (This was something I ran into when going to 11.0. If you leave it at 4GB it will not function properly at all.)
The upgrade ran normally and took a quite a while for the switch-version to complete.
On a site note, I noticed that the new Unity Connection (CUC) 11.5 .ova files define a 200GB HDD for the bigger VM. I investigated increasing my HDD from 160 to 200GB, but found out that CUC does NOT support dynamic resize of the HDD. This will cause the partition to be unaligned and you’ll get to rebuild CUC from scratch. So leave it at it’s current size.
To save time during the upgrade window, the day before I preloaded the 11.5 ISO on my remote ESXi datastores so that it wouldn’t take forever for the ISO to SFTP over to the remote offices (they have limited bandwidth) , then I attached those ISOs as virtual DVDs to the CUCM servers via vShpere and then launched those upgrades as though they were coming from DVD instead of a remote file server.
The first attempt to launch the upgrade on the Pub failed with the old “common parition doesn’t have enough space” business. I used RTMT to decrease the Low and High logging watermark to 45 and 40 respectively (and restarted the log partition monitoring service) to create room.
Purge Log Files by Changing the Log Partition Watermarks
Another way to create additional disk space is by changing the high and low watermarks on the system. This informs Unified CM of the numbers of log files to purge once the watermark is reached. Use RTMT as follows:
Launch RTMT and log in to the desired cluster.
From the left pane, select Alert Central.
On the right pane, double-click LogPartitionHighWaterMarkExceeded. Change the threshold value to 40.
On the right pane, double-click LogPartitionLowWaterMarkExceeded. Change the threshold value to 45.
This data is polled every five minutes. Allow five to 10 minutes and then check the drive partitions for additional disk space by using one of the methods described above.
DNS entries: Do you have forward and reverse DNS lookups for all infrastructure
systems that the Expressway interacts with? If the Expressway cannot resolve hostnames and IP addresses of systems,your complex deployments (eg.MRA) could stop working as expected after you upgrade.
Oddly enough, the two systems that I’ve been involved in the upgrade to 8.8.1 with, both had the Unified Communications traversal zone with show Active, and hard phones (8800 and DX 650) will register and work properly, but Jabber clients will be unable to login and Jabber will throw an error when trying to login through MRA:
"Unable to Communicate with Server."
Running the debugging logs on Expressway-C you see the following error:
"Certificate verification failed for host=x.x.x.x, additional info:
Invalid Hostname expressway-e.domain.com"
The fix is to make sure that Expressway-C can do a reverse DNS lookup on the IP address of Expressway-E. Then flush the DNS cache of C to make sure it re-queries DNS properly.
The debugging log will give you the address and hostname it is trying to do the lookup on.
In a dual-NIC Expressway-E deployment the PTR recrod should point to the private IP address that C talks to. In a single-NIC NAT hairpin deployment, I’ve seen it talk on the private and public IP. So check that debug log.
Now that all of the core CSR 11 components have had a service release under their collective belts, it’s go-time. I helped a customer upgrade CUCM, IM&P and CUC from 10.5 to release 11.0.
CUCM IM&P 11.0(1)SU1
Jabber for Mac and Windows 11.5
Latest DX-series/8800 firmware
Expressway C/E X8.7.1
CWMS 2.6 MR1 Patch 1
Security COP to address CSCuy07473 for CUCM 11.0(1)
Permanent Licensing Surprises
Holding UCCX at 10.6 for now…. (Agent/Supervisor issues)
This is a four node system (pub and 3 subs) running the latest 10.5 SU. Upgraded the pub during evening hours and told it not to reboot the night before. Once it had completed the upgrade, I ran the upgrade on the three subs and told them not to reboot. The maintenance window was the next evening, so we didn’t make any changes during that window.
When trying to reboot the pub to the new version from the GUI it got into an ugly loop. Switch version reported that an upgrade was still in progress. Went to the upgrade menu option and it indicated that I had to assume control over the upgrade. Did so and the log file showed that the upgrade had completed successfully and that the lock files were released. Went back to switch version and it still indicated that it was in an upgrade…
Bailed on the GUI and issued the version switch from the CLI. It didn’t complain at all and did the version switch from 10.5 to 11. It was a faster process than I imagined, taking less than 10 minutes to reboot on 11. I was a bit concerned if it was going to work given the GUI seeming to be in a loop, but it rebooted just fine.
I rebooted the subs all from the CLI since I didn’t (and now perhaps don’t trust) the GUI switch version. They all rebooted quickly too and were up and running on v11.0
This was a textbook upgrade that evening. I’d prestaged 11.0 like CUCM and the reboot took about 15 minutes. All was well until I applied the permanent licensing. Which I’ll cover later
I wasn’t able to pre-stage this upgrade so ran it the evening of the maintenance window. It’s a fairly small system so it took about an hour to upgrade and reboot. The reboot seemed excessively long and I was worried, but it came back and workstation Jabber clients automatically connected.
In conjunction with this upgrade I updated my jabber-update.xml file and push out the latest 11.5(2) version of the Mac and Windows clients. We also updated all of the user photos on the webserver that houses them to current pictures. Jabber was hit and miss about actually pulling the new picture. It seemed that you had to manually view the profile on some users to get it to pull the new picture.
Latest DX/8800 series Firmware
The DX-series firmware has been a bumpy bumpy bumpy road. It’s finally pretty stable as of 10.2(5)154. A newer 10.2(5)195 is out so I pushed that out as it has a number of bugfixes. I also updated the photo location for the DX-series phones and they all now pull the photos correctly from the webserver that houses them. The super secret URL to put in the Company Photo Directory is this: http://<webserver ip address>/%%uid%%.jpg
I migrated the DX-es from Anyconnect VPN over to MRA through Expressway that night since this latest ASA Sev 10 Bugfix upgrade has caused an odd cert issue for the DX (not not normal Anyconnect software clients on other platforms). Remote phone control does work properly from Jabber (that is VPNed in) to the phone that is connected via MRA.
CUCM 11.0 default firmware also had older firmware for the 8800 series phones so I pushed the latest 11.0 version and am anxiously awaiting 11.5 for some really cool upcoming features for the 8800 series.
Expressway C/E X8.7.1
Textbook upgrade. I love the software that came from Tandberg.
CWMS 2.6MR1 Patch 1
This is still my favorite app to upgrade by miles. Attach the ISO to the Admin VM in vCenter and press go from the GUI. An hour or so later after a couple reboots of all the various VMs (Admin, Media, IRP) you kick it back out of maintenance mode and you’re done.
Security COP to address CSCuy07473 for CUCM 11.0(1)
This patch JUST released with the latest security fixes for CiscoSSL (a ciscoized variant of OpenSSL). Install on each CUCM node and you’re done. No reboot required.
After upgrading everything to 11.0 everything kicked into 60-day temp license mode as expected. (Upgrading to CSR 10.5 was bad news when it didn’t do what it was supposed to and CCX ate all of it’s licenses resulting in a P1 case.)
The TAC case for licensing was pretty straightforward. Had permanent licenses in about a day after providing the contract number that showed SWSS.
I held of installing the permanent licenses until after hours in the event that something would go wrong and take the system down (still nervous after the CCX incident). Installation went fine with one side issue.
I had complaints about SpeechConnect / voice enabled directory handlers on Unity Connection not working right. Turns out CUC didn’t like the permanent licenses as far as SpeechConnect. It had pulled the licenses from ELM/PLM properly and was in compliance, but it took a restart of the Conversation service for it to start doing the voice recognition stuff again. Rather odd.
Holding at CCX 10.6
Since 10.6 is the last version of CCX to support CAD/CSD and Finesse, I’m working to migrate the contact center over to Finesse. There are some usability complaints we’re working through. The users love the idea of a dedicated app that pops when a call comes in as well as the agent-to-agent chat inside CAD. Getting them to use a web-browser for Finesse has been a challenge. Once I have those details ironed out we’ll force them into Finesse when we upgrade to CSR 11.5 in the summer.
I recently helped one of our customers work through a split DNS domain Collab Edge MRA deployment that was quite educational. Here are a few tidbits that I picked up.
Customer’s external (internet facing) DNS domain is: collabdomain.com Customer’s internal domain is: internaldomain.local (not resolvable externally, only served by the internal DNS server)
Expressway-E lives on the internet so it’s hostname would be vcse.collabdomain.com
Expressway-C, CUCM, IM&P (CUP) and Unity Connection live on the internal domain, so they would be called ucm1.internaldomain.local, cup1.internaldomain.local, ucxn.internaldomain.local.
The users accounts would live in internaldomain.local (presumably the LDAP user would be firstname.lastname@example.org, etc.)
The external DNS server would serve the public collabdomain.com zone.
The internal DNS server would serve internaldomain.local, as well as an internal version of the collabdomain.com zone.
The challenge with these separate domain names is that when a user goes to login to Jabber via MRA, the service discovery process is going to fail because they are going to type in their IM address as email@example.com, and there is no _collab-edge._tls.internaldomain.local SRV record availble on the itnernet (nor could there be).
To get around this, the jabber-config.xml file can be modified to include the external domain name that Jabber should try to do service discovery against.
Jabber must first login internally (directly on your corporate network) to pull down the jabber-config.xml that includes this policy.
With this configuration now loaded in Jabber, when the user attempts to login from outside via MRA they will still use their firstname.lastname@example.org credentials, but Jabber will use the specified VoiceServicesDomain — in this case — collabdomain.com. It will do the SRV record lookup on _collab-edge._tls.collabdomain.com and now be able to resolve Expressway-E and login.
In addition to the the normal _cisco-uds._tcp.internaldomain.local SRV record, Jabber needs another _cisco-uds._tcp.collabdomain.com SRV record (on the internal DNS server) pointing to ucm1.internaldomain.local for MRA to work with the split domains.The one criticism to this method is requiring the Jabber client to login internally first, which isn’t an ideal scenario for large distributed organizations. One method that is supposed to work around this is to prime Jabber by installing it generically and launching it from an URL:ciscojabber://provision?ServicesDomain=internaldomain.local&VoiceServicesDomain=collabdomain.com
This could be provided in an email and will supposedly launch Jabber so that it tries the external domain without having to ever login internally first.
I recently had a customer upgrade to their cluster to CUCM 10.5(1) as well as their IM&P servers to 10.5(1). They ran into an situation with the IM&P subscriber server that ended up requiring intervention from TAC to get running. (I also noticed a number of other customers having this exact same issue.)
The symptom is that after the IM&P subscriber upgrade completes, the administrator screen goes blank when trying to login and the sync service won’t start.
The issues is caused by a CUCM LDAP sync occurring while the IM&P servers are being upgraded. This causes all of the users assigned to subscriber IM&P servers to be unable to login.
If your server ends up in this condition, a call to TAC would be the first thing to do so that they can do a remote support session to fix the issue.
Avoiding the Problem in the First Place
Before performing a system upgrade of CUCM and IM&P, change the LDAP sync interval to a sufficiently long period to avoid having a sync occur during your upgrade window. I’d push it out o a week or so, but make sure it isn’t going to happen during your upgrade. This is the cause of the issues.
Upgrade CUCM and IM&P as normal
Change your LDAP sync interval back to previous values (many engineers set it to 24 hours).
One important task in a Jabber (and/or Collab Edge MRA deployment), or a UCCX deployment is to make sure that the clients trust the certificates that are presented by the services. The current versions of Jabber force validation of certificates, as well as (obviously) web browsers used with Finesse.
I recently worked through applying trusted CA-signed certificates for a UCCX 10.5 Finesse desktop deployment. The customer had deployed UCCX Finesse without installing valid certificates on the server which was casing a number of problems.
CUIC report errors (either direct CUIC web interface or gadgets in Finesse)
I’d highly recommend installing trusted certificates before going production with a system to avoid these errors.
Basic flow to deploy trusted SSL certificates
Begin by going to OS Admin and generating a certificate server request (CSR) for the server. The CSR is then signed by a certificate authority (CA).
To have Jabber or the web browser trust the certificate it must be signed by a trusted public CA like Verisign, GoDaddy, Starfield, Thawte, etc., or by a private CA that’s been added to the trusted list of the computer/mobile device. (e.g. Private CAs include MS Active Directory Certificate Service for a client that’s part of an AD domain, or an OpenSSL (ahem, if you still trust it… ;-)) that’s been added to the client’s trust list.) The most compatible, but more expensive option is to use a public CA to sign the server certificate request (CSR) as it is trusted inherently by the most clients.
One of the challenges we’re seeing from public CA’s is the CSR being signed with an intermediate certificate. Usually the certificate chain involves the server certificate, an intermediate certificate and a root certificate. In the past it was much more popular to have the server certificate linked directly to a root certificate. This has been a problem historically for some servers that don’t know how to deal with intermediate or chained certificates. Most VOS applications 9.0 and higher support chained certificates, but documentation is non-existent.
After having the CSR signed, you’ll want to request the certificates in Base64 encoded format (usually a .pem or .cer extension which is an ASCII File that has the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– markers). Normally you’ll receive a ZIP file with the server certificate, as well as a root certificate, or in the case of a chained certificate that has an intermediate certificate you may receive a bundle .pem/cer with the intermediate and root certificates.
The server certificate is easy to deal with, as it will typically have the name of the server in the filename and only include that certificate.
Figuring out how to split up the intermediate and root certificates from the bundle .pem/cer can be a challenge. The file will normally include three certificates, delimited by the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– for each certificate.
Typically they are chained in the bundle .pem/cer with the Root certificate first, the Intermediate second. They may have another certificate at the bottom which is a Class 2 Certificate Authority identifier. This is not needed for the import of the certificates into UCCX.
Windows makes it easy to open the bundle and export just the certificates you want by double-clicking the .cer and looking at the chain and exporting the certificates. On the Mac, I just take the text file and cut and paste it into separate files for each certificate and then manually open them in Keychain access to figure out which one is which. Then I name each file appropriately (like root.cer and intermediate.cer). Now I have all three certificates that I need — root, intermediate and server.
At this point to into OS Admin on the VOS box (UCCX in this example) and import them according to the documentation below.
UCCX 10.5 allows importing of chained certificates but the instructions are incorrect.
They indicate the proper order of importing the certificates (1. Root; 2. Intermediate; 3. Server), but step 7b and 8b are incorrect. It has outdated information indicating that the Root Certificate must be specified as in the screenshot below:
This field does not exist in UCCX 10.5, so ignore steps 7b and 8b. It will automatically chain the certificates.
Once the certificates are uploaded, I bogged down in trying to restart individual services like tomcat and finesse which fixed the client cert errors, but the gadget and CUIC errors persisted. The eventual fix was a complete server reboot, so I would suggest doing that immediately.
Please take a look at the release notes. While Collab Edge MRA is supported in this client it is not TAC supported until Expressway 8.1.1 releases. There are some notable feature exceptions to be aware of.
From the release notes:
When using Expressway Mobile and Remote Access to connect to services from outside the corporate firewall, the client does not support:
LDAP for contact resolution. Instead, the client must use UDS for contact resolution.
File transfer, including screen capture, is not supported with on-premise deployments. File transfer using Expressway Mobile and Remote Access is only supported using WebEx Cloud deployments.
Desk phone control mode (CTI), including extension mobility.
Extend and Connect. You cannot use the Jabber client to make and receive calls on a non-Cisco IP Phone in the office; to control a non-Cisco IP Phone in the office, such as hold/resume; or control a home or hotel phone when connecting with Expressway Mobile and Remote Access.
Dial via Office – Reverse calls.
Session persistency. The client cannot recover from disruptions caused by network transitions. For example, if a users start a Cisco Jabber call inside their office and then they walk outside their building and lose Wi-Fi connectivity, the call drops as the client switches to use Expressway Mobile and Remote Access.
Cisco WebEx Meetings Server. The client cannot access the Cisco WebEx Meetings Server server, or join or start on-premises Cisco WebEx meetings.
Sending problem reports. To work around this issue, users can save the report locally and send the report in another manner.
Early Media. Early Media allows the client to exchange data between endpoints before a connection is established. For example, if a user makes a call to a party that is not part of the same organization, and the other party declines or does not answer the call, Early Media ensures that the user hears the busy tone or is sent to voicemail. When using Expressway Mobile and Remote Access, the user does not hear a busy tone if the other party declines or does not answer the call. Instead, the user hears approximately one minute of silence before the call is terminated.
Self Care Portal.
End-to-end media encryption. Media is not encrypted on the call path between the Cisco VCS Control or Cisco Expressway-C and devices that are registered locally to Cisco Unified Communications Manager. The media path outside of the enterprise is encrypted.
Head over to the Collaboration Community EAP Space and get the latest build (or sign up to join if you haven’t). It is reported to have MRA enabled by default.
To clarify on the version number change: Prior to this build the product was going to be called 10.0, but has been rebranded to 9.6. The reasoning behind the change is to have version numbers aligned with the capabilities of each client across the board (iOS, Windows, Mac, Droid).
Collab Edge is now supported. The official Mobile-Remote-Access-via-Expressway-Deployment-Guide is located here.
I’m updating this document to reflect changes made in Expressway-C/E 8.1 that make importing the certificates MUCH easier.
This document explains how to deploy Collaboration Edge with on-prem presence (IM&P/CUP) on a non-redundant set of Expressway-E and C VMs. Deploying with WebEx Messenger is not covered here, but the bulk of the configuration is the same as far as the Expressway piece.
The biggest challenge in the initial deployment was finding all of the necessary documentation! Things you need to know like certificate chaining, or OpenSSL are in various docs. I’ve linked all of the documents that I used and tried to summarize things to make it quicker to deploy.
What you’ll need to deploy Collaboration Edge today:
IM&P (CUP) 9.1+
VCS or “Expressway” X8.1.1+
A Collaboration Edge enabled Jabber client: Cisco Jabber for Windows 9.7+, Cisco Jabber for iOS 9.6, Jabber for Android 9.6+ or Cisco Jabber for MAC 9.6+
Updated jabber-config.xml on CUCM with RemoteAccess turned on (This is no longer required for Jabber 9.6+)
Two certificates (one for VCSe another for VCSc) – either signed by your own CA (OpenSSL or similar) or publically signed certs like GoDaddy, Verisign, etc.
A few notes about nomenclature:
Collaboration Edgeis the architecture umbrella term for the VCS/Expressway edge proxy for CUCM-registered clients (Jabber and TC7.0 TP units). It’s commonly used to refer to the Jabber piece of it, but will support endpoints too. (The DX650 will support Collaboration Edge in a future release of firmware. Traditional IP phones will not be supported, they will use VPN Phone or CUBE lineside proxy.)
Mobile and Remote Access (MRA) is the term used in VCS/Expressway documentation for the VPN-less Jabber (and CUCM-registered TC7.0 endpoint) proxy feature.
Cisco Expressway-Edge is the same software as VCS-Expressway, just packaged for CUCM registered endpoints. Expressway-Edge is a VCS-Expressway that is deployed as a Mobile and Remote Access proxy or for traversal calls for CUCM registered endpoints. There is a license file actually changes the title to say “Expressway-E” when it is loaded. In the rest of the document, I will refer to VCS-Expressway, VCS-E, Expressway-Edge, Expressway-E as Expressway-E since we are primarily talking about MRA.
Expressway-Core is the same story. It is VCS-Control software deployed as an MRA proxy only with the Expressway-C license loaded. We call it VCS-Control when it is licensed for device registration, non-traversal calls, FindMe, and other features like Lync interop. For purposes of this document I’ll call it the Expressway-C below.
Customers with a valid UCSS contract for UCL-Enhanced, CUWL-STD, or CUWL-PRO are entitled to Expressway-Edge and Expressway-Core for free (for MRA) and their license will reflect the Expressway names. Licenses are charged for the other VCS features mentioned above. Licenses are required and have a cost for B2B/B2C (Jabber Guest) calls through Expressway-C/E Each box requires one media session license to get a session through.
VCS-C and VCS-E can have the MRA features turned on and run on a pair and do both functions. We are still awaiting clarification as to when you must break these apart and run a separate set of VCS (for B2B, interop) and Expressway (for MRA) servers. (Update: VCS is supported for limited sized deployments.)
Expressway licenses should be orderable via PUT, or you can use your existing VCS severs by upgrading to X8.1. (Update: I posted a later post that discusses what to order.)
2) Decide if you want to deploy valid security certificates on CUCM, IM&P and CUC. You will likely want to do this independent of Collaboration Edge as all of the Jabber clients are no longer trusting self-signed certificates. By providing a publicly trusted cert, Jabber won’t throw Invalid Certificate errors as you log in. Granted they are only shown once during the very first login if the user accepts them on each client. If you do put certificates on those components I’d suggest getting them for a 5-year term so you aren’t dealing with it in a year when the certificates expire.
Directory Lookup Considerations
MRA only supports UDS as the directory lookup service. If you are inside you can use LDAP (EDI/BDI), outside UDS.
Jabber-config.xml Update for 9.6 clients
Update: This section is no longer required as current versions of the clients (Win 9.7, iOS 9.6.1, Android 9.6, OS X 9.6) to do MRA by default, negating the requirement to pull the jabber-config.xml file first (expect in the case of split internal/external domains).
CUCM UC Service Profiles
Make sure that you’ve configure CUCM UC Service Profiles (this should have been done as part of your initial IM&P/CUP deployment and won’t be covered here) and assigned them to the end users.
Recall there is a single OVA that does both Expressway-E/C and VCS-E/C that you need to deploy — it’s just a matter of how you configure and license (request via PUT as mentioned above) it as to what it is called. Download the OVA from Cisco here
You’ll deploy the Expressway-C on your internal network (presumably on the same VLAN as CUCM and other UC components) e.g. 10.10.1.30
You’ll deploy Expressway-E one of two ways. Either on a stick in your DMZ (perhaps 10.99.99.30), or two-legged with the external interface in the DMZ network (e.g. 10.99.99.30 – or on your public address space), and the internal interface on the internal network (presumably on the same VLAN as Expressway-C and other UC components – e.g. 10.10.1.31).
You’ll need to trunk your DMZ to your ESXi host if you haven’t, or figure out how to deal with getting the Expressway-E external (LAN2) interface in the DMZ network.
Once the VM’s are deployed edit the Expressway-E VM settings to put LAN2 in the DMZ network (alternately you can put LAN2 on the inside and LAN1 in the DMZ). If you don’t see options for the second NIC, or options for NAT, you are missing the Advanced Networking license. You need this in order to have it two legged, or do NAT.
I don’t know how long I wasted on my first install because I forgot to modify the ACL to include the additonal ports that MRA requires. I just assumed that beause VCS was working for B2B that it would work for MRA. Not the case!
You’ll need to configure NAT on your firewall from a public IP address outside your network to the DMZ address of Expressway-E, or do 1:1 public to your DMZ if you’ve deployed it with a public address.
You will need two DNS servers for MRA to function properly. Jabber decides if it is inside the network or outside the network depending on what SRV records it can resolve. Depending on what records it resolves it will either try to use MRA or it will directly connect to CUCM/IM&P.
Internal DNS Server
Create two A records:
sjc-expressway-edge-01.domain.com A – (make this name whatever you want) Pointing to the INSIDE interface of Expressway-E for two-legged deployments, or pointing to the DMZ address if it’s on a stick. The record is used by Expressway-C to lookup and validate the certificate against. You will use this hostname anywhere you are asked for the expressway server’s name whe configuring the C server.
sjc-expressway-core-01.domain.com A – (any name you want) Pointing to Expressway-C.
Create two SRV records:
_cisco-uds._tcp.domain.com SRV 0 0 port 8443 – Pointing to CUCM. (NOT IM&P!)
_cuplogin._tcp.domain.com SRV 0 0 port 8443 – Pointing to IM&P (TBD if this is really required for Jabber 9.6 with IM&P 9.1 – I don’t believe it actually is)
When you launch Jabber, if it can resolve these DNS records, it knows it’s inside and pulls the service profile directly from CUCM and logs in to IM&P and CUCM.
External DNS Server
Create one A record:
sjc-expressway-edge-01.domain.com A – (any name you want) Pointing to the public address assigned (or NATted) to your Expressway-E.
Create one SRV record:
_collab-edge._tls.domain.com SRV 0 0 8443 – Pointing to Expressway-E (in our case sjc-expressway-edge-01.domain.com)
Configure Expressway-Edge and Expressway-Core
Follow this chapter of the Expressway Admin Guide – Mobile and Remote Access (feature preview) beginning at p.52 but stop half-way down p.56 (before the beginning of the Certificates section).
A couple notes: I did not enable TLS verify mode on my CUCM and IM&P server definitions because just wanted to get it up and running. I’m suggesting putting real certs on CUCM, IM&P, and CUC, and turning TLS verify on, but this can be done later.
Valid CA-signed certificates are required to setup the traversal zone for MRA. You can either get public ones, or sign your own with your own CA. I’ve done it both ways. The major reason for a valid trusted CA-signed certificate is to stop Jabber from throwing a certificate warning on the initial MRA login to Expressway-E itself. I highly recommend deploying a publicly trusted CA signed certificate.
Update: This is fixed in Expressway 8.1.1 Ignore this section below:
Add UC Domain (domain.com) and XMPP server information
Download the CSR
Upoad the CSR file to the CA to get the certificate signed
Get the signed server PEM and the root/intermediate chain PEM back from the CA.
Upload the signed server cert to Expressway-E under Maintenance | Security Certificates | Server Certificate
Break apart the CA-intermediate-root certificates into individual PEMs for import – See the WebEX instruction for VCS 8.1 to learn how to do this.
Import into the Trusted CA certificate list: the top-level cert (“CA”), then the root cert, then the intermediate cert found under Maintenance | Security Certificates | Trusted CA Certificates
Reboot to make them active.
Follow the Webex instructions to break apart the CA-intermediate and root PEM into individual certs using Windows so that you can import them into the CA trusted cert list properly.
Repeat this procedure for Expressway-C.
For the customers that I’ve worked with using GoDaddy certificates. I’ve worked with four certificates – Go Daddy Class 2 CA; Go Daddy Root Certificate Authority – G2; Go Daddy Secure Certificate Authority – G2; the server certificate itself.
I used Chrome on Windows to export the three Go Daddy certificates individually to Base 64 .PEM and then loaded them into the Expressway-E/C trusted CA list. This worked perfectly for me after loading and rebooting the servers. The UC traversal zones came right up.
Sign your own using OpenSSL if you’d like
If you want to use OpenSSL to create your own CA cert and sign your CSR, it is actually easier than you’d think.
Start at the bottom of p.13 of this document –
You’ll follow the procedure twice. Once for Expressway-E and once for Expressway-C. Take the CA root cert that you generated and import it into the trusted list on both boxes, and then import your signed server cert on the appropriate box.
Traversal Zone Configuration
Resume the configuration tasks in the Admin guide on p.58 making sure to put the proper settings for both Expressway-E and Expressway-C.
If your certificates are good, you will see the traversal zone go active on both servers under Status | Unified Communications. If not, double-check your configuration settings, and double-check your certificates.
Troubleshooting Zone Configuration
If the zone won’t go active and you think it looks good, check the logs to see what is happening. My initial attempt where the certificates were not chained properly showed a continuous loop of TLS failures. When I had my Expressway-C pointing to the external public address instead of the inside interface of Expressway-E, TLS looked good and even the SSH tunnel showed “up” but traffic wasn’t actually flowing.
The best place I found to troubleshoot this stuff was by putting the Expressway-C and E in “Devel mode” to enable the Experimental menu. (Instructions for this are found on p.207 of the admin guide.) The reason for this is because the CollabEdge/MRA feature is still considered experimental. You need to look at the Developer Logs. You can enable them for debug level as well as collect a tcpdump.
Make sure to add your Unity Connection, and any other servers that Jabber needs access to. Unity Connection requires it for Visual Voicemail to work.
Launch Jabber 9.6 internally
Update: No longer required unless you are doing separate internal and external domains. (I’ll detail this in a later post.)
Launch Jabber on your client device on the inside network (so that it has direct access to CUCM/IM&P). When you enter your email address Jabber should automatically discover your servers (using the before setup internal DNS SRV records). If Jabber does not auto-discover, troubleshoot your SRV records. The easiest method is to use dig or nslookup.
The quick nslookup method is to:
Launch the program,
Make sure it shows your internal DNS servers (that your device should be pulling via DHCP scope options)
Enter set type=SRV, then type _cisco-uds._tcp.domain.com. This should resolve to the hostname of CUCM, or the IP address of it.
If using the hostname, exit nslookup and try to ping the hostname.
Once you enter your credentials you will likely be presented with several invalid certs to accept, and your client should connect and have IM, Presence, CUCM, CUC, and be able to IM and do voice/video calls.
Sign out and close Jabber
Launch Jabber 9.6 externally
Disconnect from your internal network and make sure your device is outside your network where a) it cannot resolve the internal SRV records, and b) it can resolve the external _collab-edge SRV record and access your Expressway-E from the outside.
Launch Jabber on your device. Jabber will attempt to resolve _cisco-uds._tcp.domain.com and will fail to do so. It will also attempt to resolve _cuplogin._tcp.domain.com and will fail. It will then attempt to resolve _collab-edge._tls.domain.com and get pointed to the public IP address of Expressway-E.
It will then connect to Expressway-E, and a if everything is configured properly it will login and you’ll show connected to IM, CUCM and CUC!
Notes about iOS
On iOS the timeout for attempts to login is MINUTES long. Be very patient for it to either succeed or fail. It can take a significant amount of time to login successfully on the 9.6 build. 9.6.1 is supposed to be much faster.
If your login fails, click the Send Error Report and email it to yourself. Open the ZIP file and look through (going from bottom to top) to see where the errors are. The logs will include more than just the current login attempt, so note the time when you are attempting to login and look at the timestamps in the log. This is critical so that you aren’t troubleshooting an old login that isn’t relevant to your current problem.
From my experience:
When I had firewall issues, I was seeing CONNECTION_TIMEOUT errors when trying to login via MRA, but not when I was inside.
When I had neglected to enable RemoteAccess in jabber-config.xml I was seeing RemoteAccess Policy errors.
I’m impressed with the ability to finally be able to do voice/video calls from anywhere! It’s about time. Collab Edge is still considered a Feature Preview by Cisco and isn’t TAC supported yet. Please send me questions that you have as you attempt to deploy it.
Collaboration Edge is an umbrella term for an architecture. It allows Jabber clients (Win/Mac/iOS/Droid) to be proxied though a server (Expressway E) in the DMZ back to Expressway C and then CUCM. Collaboration Edge also includes the ability to proxy remote CUCM registered video endpoints (SX20, EX90, etc.).
Currently Collaboration Edge components are/will be:
VCS X8.1 software (called VCSc, VCSe, Expressway C, or Expressway E depending on deployment model. It’s one actual piece of software.)
Jabber 9.6/9.7 software (depending on platfrom)
TC 7.0 video endpoint firmware
Expressway C and Expressway E is analogous of the VCS Control/VCS Express model, except that it is only for CUCM registered endpoints. It is actually VCS software. VCS Control is called “Expressway C” and VCS Expressway called “Expressway E” when deployed as Collaboration Edge (Jabber proxy, and endpoint proxy to CUCM) only. Depending on size, scale and deployment model it may run co-resident if you already have VCS deployed, or you may need to stand up two new VMs to run it.