With the release of CE8 code for Cisco video endpoints (like the SX10 (8.1), SX20 (also MX200-G2 and MX300-G2) and SX80-based endpoints like the SX80, MX700 and MX800), and the appropriate infrastructure components, multistream video is a possibility. Multistream video allows an endpoint to send multiple resolution video streams and have the bridge pass the most appropriate streams to the far-end video units. The far end video unit would receive a full resolution stream of the active speakers, and then low quality streams of the other participants. The most useful feature of multistream video is the ability to use both screens of a dual-screen video unit to see remote participants (when doing single-stream transcoded mode, you can only do single screen video, and secondary screen content.) Multistream also allows for ActiveControl layout, which allows the endpoint to choose the video layout vs. the video bridge determining the layout of the participants (which has rudimentary DTMF layout control).
Components used in my lab configuration:
(2) 8845 video phones (used to inject more video streams — these endpoints do not support multistream, they do single stream and receive their layout from the bridge)
CUCM 11.0(1)21900-11 (Latest and greatest version is a requirement) -or- VCS X8.7.1
This guide assumes you’ve already setup a Rendezvous (aka MeetMe) number/URI that is routed to Conductor/vTS and you’re able to to normal conference calls. We’ll modify settings to enable multistream.
The relevant portion of this configuration is to make sure your SIP trunk to conductor is in a Location that supports full quality video. I sent the inter-region bandwidth to UNLIMITED in my test system. Cisco recommends a minimum of 1mpbs per screen, otherwise the vTS bridge may kick that video unit down to single-stream transcoded mode.
Configure the endpoint to support multistream
In CUCM the setting is in the device specific settings, Multistream Mode needs to be set to Auto. Despite some of the documentation reading otherwise, Auto will attempt to do multistream, there is not actually an On setting.
Configure the SIP Profile used by the SIP trunk to Conductor to include the following settings:
Allow iX Application Media and Allow multiple codecs in answer SDP are checked on.
SDP Transparency Profile is set to Pass all unknown SDP attributes
In System > Service Parameters > Call Manager Service > click advanced > set SIP Maximum Incoming Message Size to 18000.
On the Conductor server, under the Conference Template you’re using for your conference, select advanced template parameters and add:
Enable iX protocol – True and the box checked
Multiscreen layout – ActivePresence and the box checked
No settings on vTS need to be changed, it will automatically do multistream if the endpoints meet the requirements, and CUCM (or VCS) and Conductor are properly configured.
When you join with a multi-stream endpoint you will see the following on vTS Conferences page:
You’ll notice the endpoints that support multistream show Multistream, and the 8845 phone named “Mike White” is Standard because it only supports a single stream.
If we look at the statistics for 5580 (the SX80) you’ll see multiple video streams being sent and received:
Lastly if we look at the call statistics from the video endpoint itself, we see the same information:
The touchpanel now shows more details in the layout. You can see each participant in the conference and the active speaker.
While you can select from several canned layout modes (same typical layouts are you’re used to), this version doesn’t yet support complete drag and drop layout of individual participants where you want them. If you select a particular participant, you can see information about any of the participants and boot them if you are meeting organizer:
Overall its very cool, and sets the groundwork for much more flexibility in the future with layout control.
I’ve run into a couple challenges after an upgrade to Expressway/VCS 8.5.2 where MRA for phones quit working.
I found a bug that broke MRA in 8.5.2 (the recommendation has been to downgrade back to 8.5.1). That bug is shows that it is now fixed in 8.6, so I upgraded to 8.6.1 recently. MRA started working again, but only on one out of every three login attempts. It was really weird. In looking at the logs it showed a bunch of errors:
Home CUCM not available – Unknown CUCM cluster for node sub02
Home CUCM not available – Unknown CUCM cluster for node sub03
The deployment I was working on is a three node (pub and two subs), running split DNS (different internal domain than the external domain name).
After a lot of digging it turns out that there is a change in the way MRA handles CUCM lookup. When I installed the system added my pub and subs to Expressway-C by IP address. But it looks like Expressway now attempts to communicate with them via hostname, and not IP as they were defined by me.
Since Expressway is using the domain suffix assigned to MRA (extdomain.com), it is attempting to lookup sub02.extdomain.com and sub03.extdomain.com. I didn’t have A records for these on my internal DNS server extdomain zone since I’ve never needed to resolve them by the external domain name.
Adding these two records fixed the login issue and it now logins on first attempt like it used to.
I took some time to help a customer upgrade their system from CSR (Cisco System Release) 9.1 to 10.5 recently. As any good upgrade goes, it wasn’t without some significant drama….
[In particular, they were upgrading from CUCM 9.1(2) to CUCM 10.5(2); CUCM IM&P 9.1 to 10.5(2); CUC 9.1(2) to 10.5(2) and UCCX 9.0(2) to 10.6(1); Expressway x8.1 to x8.5]
Expressway-C and E was a textbook upgrade using the .gz upgrade files.
Unity Connection (aka CUC) was the first core component that we chose to bite off because it doesn’t have any version dependencies with the other components. It was a textbook upgrade without issues.
The minor snag was CUC going unlicensed immediately because it was pointing to ELM on CUCM 9.1 and CUCM didn’t have 10.x licenses installed. So watch out for that. It was rather odd that CUC didn’t give us a 60-day grace period. I moved it to it’s own PLM with appropriate licensing installed there.
We next chose to upgrade UCCX as 10.6(1) is compatible with CUCM 9.1(2). [http://docwiki.cisco.com/wiki/Unified_CCX_Software_Compatibility_Matrix_for_10.6%281%29]
This is a refresh upgrade so you must install a refresh upgrade COP file on UCCX before installing 10.6(1). Because it is a refresh upgrade the system upgrades the underlying VoS (RHEL-based Voice Operating System) and CCX is down while the system reboots and upgrades the OS. I selected to have CCX stay on 9.0(2) after the upgrade, so that 10.6(1) is the inactive version and will just need a version switch reboot to go live.
The switch version reboot turned into a bit of a mess. I issued it but the server didn’t seem to actually do anything. I issued a CLI utils system reboot command about 2 minutes later and it screamed back that I shouldn’t do that as the system was in a version switch and the database could be corrupted. I let it sit about 30 minutes and tried again. This time it rebooted without complaining and came up on 10.6(1).
I had to run the typical process of updating the CAD client (this customer will move to Finesse in the next phase of the upgrade), using the Client Configuration tools you download and install from CCX.
Next was the CUCM publisher. This ended up being a multi-hour affair. I’d already heard about the very common problem of the Common partition being full, so I’d taken the liberty to use RTMT to clear out old logfiles. You basically go to Trace and Log Central in RTMT, select Collect Files, choose a time period (I chose the previous 5 month period) did not ZIP the files (for time sake) and most importantly checked the box to Delete the files from the server.
I had the Common partition down to 50% utilized before the upgrade.
It failed citing the good old bugid CSCuc63312:
There is not enough disk space in the common partition to perform the upgrade. For steps to resolve this condition please refer to the Cisco Unified Communications Manager 9.1(1) Release Notes or view defect CSCuc63312 in Bug Toolkit on cisco.com.
So second-guessing myself, I decided to use the sledge hammer known as ciscocm.free_common_space_v1.1.cop.sgn COP file to clean out the common partition (this COP file script nukes the currently installed inactive version). I gave that a run and rebooted the server.
The next attempt at install also failed with the same CSCuc63312 error!
Knowing that the Common partition wasn’t the issue (which you can see in the install logs that you can copy and paste from the GUI), I started doing some digging around. It turns out that if the main partition doesn’t have enough room (which you can see in the log files by searching for the word “needed”), it will throw the CSCuc63312 error erroneously.
I found a couple of TAC cases where the next step to clean up room on the main partition is to clean up the TFTP directory. This customers TFTP directory was over 5GB in size from multiple versions of large firmware for endpoints like the DX650 and Telepresence codec firmware for endpoints like the C40, SX20, SX10, etc.
Even after cleaning up the TFTP folder I was still hit with the same stupid error message: Not enough space in the Common partition. Even though I knew Common had plenty of room and I was fighting a main partition space issue.
I remembered that from CUCM 10.0 on, the OVA template had increased the disk size from 80GB to 110GB. It turns out that if you increase the size of the VM’s disk in 10.0 or greater, CUCM will automatically see the space and take advantage of it. CUCM 9.x doesn’t do this automatically.
The special sauce to get 9.1 to expand the disk is to install the ciscocm.vmware-disk-size-reallocation-1.0.cop.sgnCOP file, shut the VM down, resize the VM HDD in ESXi to 110GB and then boot it back up. CUCM will reboot a couple of times during the process and then come all the way up.
After doing this CUCM 10.5(2) was able to install successfully. Keep in mind that this is a refresh upgrade so the system will be down for an hour or so while VoS is updated, and the server will go through a couple reboots.
CUCM IM&P (CUP)
I initially tried to install the upgrade ISO and received an invalid checksum error. Thinking it was the version of IM&P I’d downloaded from CCO (it’s been updated twice in the past week) I re-downloaded and hit the same error. Note the current version of CUP is 10.5(2a) as of this writing. If I’d paid attention to the documentation I’d have realized that you need to install the ciscocm.version3-keys.cop.sgn COP File which has the new keys that the 10.5(x) software images are signed with. After installing this, the upgrade would recognize the ISO and upgrade.
Collab Edge is now supported. The official Mobile-Remote-Access-via-Expressway-Deployment-Guide is located here.
I’m updating this document to reflect changes made in Expressway-C/E 8.1 that make importing the certificates MUCH easier.
This document explains how to deploy Collaboration Edge with on-prem presence (IM&P/CUP) on a non-redundant set of Expressway-E and C VMs. Deploying with WebEx Messenger is not covered here, but the bulk of the configuration is the same as far as the Expressway piece.
The biggest challenge in the initial deployment was finding all of the necessary documentation! Things you need to know like certificate chaining, or OpenSSL are in various docs. I’ve linked all of the documents that I used and tried to summarize things to make it quicker to deploy.
What you’ll need to deploy Collaboration Edge today:
IM&P (CUP) 9.1+
VCS or “Expressway” X8.1.1+
A Collaboration Edge enabled Jabber client: Cisco Jabber for Windows 9.7+, Cisco Jabber for iOS 9.6, Jabber for Android 9.6+ or Cisco Jabber for MAC 9.6+
Updated jabber-config.xml on CUCM with RemoteAccess turned on (This is no longer required for Jabber 9.6+)
Two certificates (one for VCSe another for VCSc) – either signed by your own CA (OpenSSL or similar) or publically signed certs like GoDaddy, Verisign, etc.
A few notes about nomenclature:
Collaboration Edgeis the architecture umbrella term for the VCS/Expressway edge proxy for CUCM-registered clients (Jabber and TC7.0 TP units). It’s commonly used to refer to the Jabber piece of it, but will support endpoints too. (The DX650 will support Collaboration Edge in a future release of firmware. Traditional IP phones will not be supported, they will use VPN Phone or CUBE lineside proxy.)
Mobile and Remote Access (MRA) is the term used in VCS/Expressway documentation for the VPN-less Jabber (and CUCM-registered TC7.0 endpoint) proxy feature.
Cisco Expressway-Edge is the same software as VCS-Expressway, just packaged for CUCM registered endpoints. Expressway-Edge is a VCS-Expressway that is deployed as a Mobile and Remote Access proxy or for traversal calls for CUCM registered endpoints. There is a license file actually changes the title to say “Expressway-E” when it is loaded. In the rest of the document, I will refer to VCS-Expressway, VCS-E, Expressway-Edge, Expressway-E as Expressway-E since we are primarily talking about MRA.
Expressway-Core is the same story. It is VCS-Control software deployed as an MRA proxy only with the Expressway-C license loaded. We call it VCS-Control when it is licensed for device registration, non-traversal calls, FindMe, and other features like Lync interop. For purposes of this document I’ll call it the Expressway-C below.
Customers with a valid UCSS contract for UCL-Enhanced, CUWL-STD, or CUWL-PRO are entitled to Expressway-Edge and Expressway-Core for free (for MRA) and their license will reflect the Expressway names. Licenses are charged for the other VCS features mentioned above. Licenses are required and have a cost for B2B/B2C (Jabber Guest) calls through Expressway-C/E Each box requires one media session license to get a session through.
VCS-C and VCS-E can have the MRA features turned on and run on a pair and do both functions. We are still awaiting clarification as to when you must break these apart and run a separate set of VCS (for B2B, interop) and Expressway (for MRA) servers. (Update: VCS is supported for limited sized deployments.)
Expressway licenses should be orderable via PUT, or you can use your existing VCS severs by upgrading to X8.1. (Update: I posted a later post that discusses what to order.)
2) Decide if you want to deploy valid security certificates on CUCM, IM&P and CUC. You will likely want to do this independent of Collaboration Edge as all of the Jabber clients are no longer trusting self-signed certificates. By providing a publicly trusted cert, Jabber won’t throw Invalid Certificate errors as you log in. Granted they are only shown once during the very first login if the user accepts them on each client. If you do put certificates on those components I’d suggest getting them for a 5-year term so you aren’t dealing with it in a year when the certificates expire.
Directory Lookup Considerations
MRA only supports UDS as the directory lookup service. If you are inside you can use LDAP (EDI/BDI), outside UDS.
Jabber-config.xml Update for 9.6 clients
Update: This section is no longer required as current versions of the clients (Win 9.7, iOS 9.6.1, Android 9.6, OS X 9.6) to do MRA by default, negating the requirement to pull the jabber-config.xml file first (expect in the case of split internal/external domains).
CUCM UC Service Profiles
Make sure that you’ve configure CUCM UC Service Profiles (this should have been done as part of your initial IM&P/CUP deployment and won’t be covered here) and assigned them to the end users.
Recall there is a single OVA that does both Expressway-E/C and VCS-E/C that you need to deploy — it’s just a matter of how you configure and license (request via PUT as mentioned above) it as to what it is called. Download the OVA from Cisco here
You’ll deploy the Expressway-C on your internal network (presumably on the same VLAN as CUCM and other UC components) e.g. 10.10.1.30
You’ll deploy Expressway-E one of two ways. Either on a stick in your DMZ (perhaps 10.99.99.30), or two-legged with the external interface in the DMZ network (e.g. 10.99.99.30 – or on your public address space), and the internal interface on the internal network (presumably on the same VLAN as Expressway-C and other UC components – e.g. 10.10.1.31).
You’ll need to trunk your DMZ to your ESXi host if you haven’t, or figure out how to deal with getting the Expressway-E external (LAN2) interface in the DMZ network.
Once the VM’s are deployed edit the Expressway-E VM settings to put LAN2 in the DMZ network (alternately you can put LAN2 on the inside and LAN1 in the DMZ). If you don’t see options for the second NIC, or options for NAT, you are missing the Advanced Networking license. You need this in order to have it two legged, or do NAT.
I don’t know how long I wasted on my first install because I forgot to modify the ACL to include the additonal ports that MRA requires. I just assumed that beause VCS was working for B2B that it would work for MRA. Not the case!
You’ll need to configure NAT on your firewall from a public IP address outside your network to the DMZ address of Expressway-E, or do 1:1 public to your DMZ if you’ve deployed it with a public address.
You will need two DNS servers for MRA to function properly. Jabber decides if it is inside the network or outside the network depending on what SRV records it can resolve. Depending on what records it resolves it will either try to use MRA or it will directly connect to CUCM/IM&P.
Internal DNS Server
Create two A records:
sjc-expressway-edge-01.domain.com A – (make this name whatever you want) Pointing to the INSIDE interface of Expressway-E for two-legged deployments, or pointing to the DMZ address if it’s on a stick. The record is used by Expressway-C to lookup and validate the certificate against. You will use this hostname anywhere you are asked for the expressway server’s name whe configuring the C server.
sjc-expressway-core-01.domain.com A – (any name you want) Pointing to Expressway-C.
Create two SRV records:
_cisco-uds._tcp.domain.com SRV 0 0 port 8443 – Pointing to CUCM. (NOT IM&P!)
_cuplogin._tcp.domain.com SRV 0 0 port 8443 – Pointing to IM&P (TBD if this is really required for Jabber 9.6 with IM&P 9.1 – I don’t believe it actually is)
When you launch Jabber, if it can resolve these DNS records, it knows it’s inside and pulls the service profile directly from CUCM and logs in to IM&P and CUCM.
External DNS Server
Create one A record:
sjc-expressway-edge-01.domain.com A – (any name you want) Pointing to the public address assigned (or NATted) to your Expressway-E.
Create one SRV record:
_collab-edge._tls.domain.com SRV 0 0 8443 – Pointing to Expressway-E (in our case sjc-expressway-edge-01.domain.com)
Configure Expressway-Edge and Expressway-Core
Follow this chapter of the Expressway Admin Guide – Mobile and Remote Access (feature preview) beginning at p.52 but stop half-way down p.56 (before the beginning of the Certificates section).
A couple notes: I did not enable TLS verify mode on my CUCM and IM&P server definitions because just wanted to get it up and running. I’m suggesting putting real certs on CUCM, IM&P, and CUC, and turning TLS verify on, but this can be done later.
Valid CA-signed certificates are required to setup the traversal zone for MRA. You can either get public ones, or sign your own with your own CA. I’ve done it both ways. The major reason for a valid trusted CA-signed certificate is to stop Jabber from throwing a certificate warning on the initial MRA login to Expressway-E itself. I highly recommend deploying a publicly trusted CA signed certificate.
Update: This is fixed in Expressway 8.1.1 Ignore this section below:
Add UC Domain (domain.com) and XMPP server information
Download the CSR
Upoad the CSR file to the CA to get the certificate signed
Get the signed server PEM and the root/intermediate chain PEM back from the CA.
Upload the signed server cert to Expressway-E under Maintenance | Security Certificates | Server Certificate
Break apart the CA-intermediate-root certificates into individual PEMs for import – See the WebEX instruction for VCS 8.1 to learn how to do this.
Import into the Trusted CA certificate list: the top-level cert (“CA”), then the root cert, then the intermediate cert found under Maintenance | Security Certificates | Trusted CA Certificates
Reboot to make them active.
Follow the Webex instructions to break apart the CA-intermediate and root PEM into individual certs using Windows so that you can import them into the CA trusted cert list properly.
Repeat this procedure for Expressway-C.
For the customers that I’ve worked with using GoDaddy certificates. I’ve worked with four certificates – Go Daddy Class 2 CA; Go Daddy Root Certificate Authority – G2; Go Daddy Secure Certificate Authority – G2; the server certificate itself.
I used Chrome on Windows to export the three Go Daddy certificates individually to Base 64 .PEM and then loaded them into the Expressway-E/C trusted CA list. This worked perfectly for me after loading and rebooting the servers. The UC traversal zones came right up.
Sign your own using OpenSSL if you’d like
If you want to use OpenSSL to create your own CA cert and sign your CSR, it is actually easier than you’d think.
Start at the bottom of p.13 of this document –
You’ll follow the procedure twice. Once for Expressway-E and once for Expressway-C. Take the CA root cert that you generated and import it into the trusted list on both boxes, and then import your signed server cert on the appropriate box.
Traversal Zone Configuration
Resume the configuration tasks in the Admin guide on p.58 making sure to put the proper settings for both Expressway-E and Expressway-C.
If your certificates are good, you will see the traversal zone go active on both servers under Status | Unified Communications. If not, double-check your configuration settings, and double-check your certificates.
Troubleshooting Zone Configuration
If the zone won’t go active and you think it looks good, check the logs to see what is happening. My initial attempt where the certificates were not chained properly showed a continuous loop of TLS failures. When I had my Expressway-C pointing to the external public address instead of the inside interface of Expressway-E, TLS looked good and even the SSH tunnel showed “up” but traffic wasn’t actually flowing.
The best place I found to troubleshoot this stuff was by putting the Expressway-C and E in “Devel mode” to enable the Experimental menu. (Instructions for this are found on p.207 of the admin guide.) The reason for this is because the CollabEdge/MRA feature is still considered experimental. You need to look at the Developer Logs. You can enable them for debug level as well as collect a tcpdump.
Make sure to add your Unity Connection, and any other servers that Jabber needs access to. Unity Connection requires it for Visual Voicemail to work.
Launch Jabber 9.6 internally
Update: No longer required unless you are doing separate internal and external domains. (I’ll detail this in a later post.)
Launch Jabber on your client device on the inside network (so that it has direct access to CUCM/IM&P). When you enter your email address Jabber should automatically discover your servers (using the before setup internal DNS SRV records). If Jabber does not auto-discover, troubleshoot your SRV records. The easiest method is to use dig or nslookup.
The quick nslookup method is to:
Launch the program,
Make sure it shows your internal DNS servers (that your device should be pulling via DHCP scope options)
Enter set type=SRV, then type _cisco-uds._tcp.domain.com. This should resolve to the hostname of CUCM, or the IP address of it.
If using the hostname, exit nslookup and try to ping the hostname.
Once you enter your credentials you will likely be presented with several invalid certs to accept, and your client should connect and have IM, Presence, CUCM, CUC, and be able to IM and do voice/video calls.
Sign out and close Jabber
Launch Jabber 9.6 externally
Disconnect from your internal network and make sure your device is outside your network where a) it cannot resolve the internal SRV records, and b) it can resolve the external _collab-edge SRV record and access your Expressway-E from the outside.
Launch Jabber on your device. Jabber will attempt to resolve _cisco-uds._tcp.domain.com and will fail to do so. It will also attempt to resolve _cuplogin._tcp.domain.com and will fail. It will then attempt to resolve _collab-edge._tls.domain.com and get pointed to the public IP address of Expressway-E.
It will then connect to Expressway-E, and a if everything is configured properly it will login and you’ll show connected to IM, CUCM and CUC!
Notes about iOS
On iOS the timeout for attempts to login is MINUTES long. Be very patient for it to either succeed or fail. It can take a significant amount of time to login successfully on the 9.6 build. 9.6.1 is supposed to be much faster.
If your login fails, click the Send Error Report and email it to yourself. Open the ZIP file and look through (going from bottom to top) to see where the errors are. The logs will include more than just the current login attempt, so note the time when you are attempting to login and look at the timestamps in the log. This is critical so that you aren’t troubleshooting an old login that isn’t relevant to your current problem.
From my experience:
When I had firewall issues, I was seeing CONNECTION_TIMEOUT errors when trying to login via MRA, but not when I was inside.
When I had neglected to enable RemoteAccess in jabber-config.xml I was seeing RemoteAccess Policy errors.
I’m impressed with the ability to finally be able to do voice/video calls from anywhere! It’s about time. Collab Edge is still considered a Feature Preview by Cisco and isn’t TAC supported yet. Please send me questions that you have as you attempt to deploy it.
If you’re upgrading an existing system, you’ll need:
VCS X8.1 upgrade tar file (as opposed to the ova file for a fresh deplyoment) – Found Here
New VCS 8.x release keys from Cisco – request via PUT
Once you’ve baked up your systems (VCSc and VCSe), and you’re in a maintenance window (this will disrupt all system registrations and calls), SSH in to each machine and put the system in Maintenance Mode: xConfiguration SystemUnit Maintenance Mode: On
Initiate the upgrade in the GUI, and add the new release keys.
My VCSc upgrade failed with an error:
System error: tar zxf /tmp/tandberg-image.tar.gz: Execution failed: tar (child): /tmp/tandberg-image.tar.gz: Cannot open: No such file or directory
I rebooted VCSc and attempted the upgrade a second time and it succeeded.
Collaboration Edge is an umbrella term for an architecture. It allows Jabber clients (Win/Mac/iOS/Droid) to be proxied though a server (Expressway E) in the DMZ back to Expressway C and then CUCM. Collaboration Edge also includes the ability to proxy remote CUCM registered video endpoints (SX20, EX90, etc.).
Currently Collaboration Edge components are/will be:
VCS X8.1 software (called VCSc, VCSe, Expressway C, or Expressway E depending on deployment model. It’s one actual piece of software.)
Jabber 9.6/9.7 software (depending on platfrom)
TC 7.0 video endpoint firmware
Expressway C and Expressway E is analogous of the VCS Control/VCS Express model, except that it is only for CUCM registered endpoints. It is actually VCS software. VCS Control is called “Expressway C” and VCS Expressway called “Expressway E” when deployed as Collaboration Edge (Jabber proxy, and endpoint proxy to CUCM) only. Depending on size, scale and deployment model it may run co-resident if you already have VCS deployed, or you may need to stand up two new VMs to run it.