Lync 2013 and Skype for Business 2015 on Server 2012R2/2016 – *Updated*

 

Once upon a time not so long ago whilst on leave my largest lync deployment ‘acted’ up and practically fell over for no apparent reason whatsoever.

As luck has it I was no where near an internet connection and the current preview build of Windows 10 at the time did not support Cisco VPN so I was stuck until I returned from my trip.

By the time I returned home almost 12 hours later the friendly staff from premier was ready to begin troubleshooting my issue, which at the time I did not even have a full view of what was wrong exactly, I just knew it was broken and that no one in my team was able to even find anything on the internet or TechNet which could point them in any direction.

In short, my two pools were no longer communicating with one another, meaning users were fine inside the same pool but could not see anyone from the other pool.

About 14 hours and 4 Lync Premier support engineers later this was resolved by adding to my existing list of 2 Server 2012R2 SSL related registry settings another 2 settings and from there it all worked perfectly as if nothing was ever wrong.

Again this week I found myself upgrading another Lync 2013 environment to Skype for Business and everything worked perfectly, but for some reason I started struggeling to move users to my other pool as I went through the motions of the upgrade and moving things around so my users could continue working whilst I did the upgrades.

After what felt like hours of analyzing traces and testing replication and settings, digging into the SQL backends, looking into the mirroring on my sql databases, analyzing the local sql express instances etc I finally thought to just check the state of these keys in these servers only to find they were not there at all. Now keeping in mind these servers have been running for almost two years now with no issues whatsoever I can see how these would have been missed, the symptoms was just not there.

So imported the keys, rebooted and problem solved.

Today, whilst upgrading my final edge pool again ‘issues’ one of the two edge servers upgraded perfectly whilst the other one found every possible excuse to not be upgraded, down to the point where even powershell stopped working giving me either a blank screen or allowing me to enter a single command and then hanging on me indefinitely.

Again I checked the keys since I had imported them just recently, and as luck would have it I only imported these keys into my Frontend pool servers.

Again I imported keys and rebooted and just like magic, again everything started working.

Now since I personally have never seen all these keys in 1 place I thought I must share them, they apply to Lync 2013 as well as Skype for Business 2015 servers running on Server 2012R2 , any lync/skype server rolls, not the SQL backend, any server with certificates that forms part of the lync environment benefits from this, there may be other uses also I don’t know.

*Update – We found these keys to be needed for CMS replication related issues on Server 2016 Edge Servers*

Here are they keys:
( Make a .reg and merge )

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL]
“EventLogging”=dword:00000001
“EnableSessionTicket”=dword:00000002
“ClientAuthTrustMode”=dword:00000002
“SendTrustedIssuerList”=dword:00000000

Exchange 2013 and Lync 2013 IM Integration in an Exchange PA aligned environment

Let me firstly say that I have configured Lync and Exchange IM Integration numerous times and every time I perform this configuration in an Exchange Preferred Architecture Deployment I end up confusing myself and then spend a day or so fixing my configuration. (* TIP – Now that I have blogged this naturally I can reference my own blog in future J )

Scenario:

1 Exchange Stretched DAG which resides in 2 different data centers on the same subnet using Cisco OTV Stretched Vlan networking technology.

2 Lync Pools configured in the same network subnet.

The confusing part:

Typically I confuse myself with where I should point my “IMPROVIDER” in my exchange server web.config files since technically I have 1 Exchange, however I have two different lync pools which could potentially serve my IM configuration.

Quick and Easy Guide:

I have resorted to pointing my Exchange nodes to the Lync Pool which resides in the same physical data center.

My logic in this is that in the event of a datacenter going down, my traffic from both internal and external networks will not be routed to that datacenter which also means my exchange naturally would not be able to speak to the lync pool in that datacenter either, makes sense.

Step 1: Ensure all Exchange nodes have a separate internally signed UM certificate with the actual node FQDN in the certificate.

Step 2: Using the script provided on the technet script centre by Michel de Rooij here I configure each Exchange node to point to the local lync pool. – Note you typically have to rerun this after each CU or SP you install on Exchange.

Step 3: Enable the IM on the Exchange nodes by running the following command:

Get-OwaVirtualDirectory | Set-OwaVirtualDirectory –InstantMessagingEnabled $True –InstantMesssagingType OCS

Step 3: Utilising get-cstrustedapplication, get-cstrustedapplicationpool I ensure that I have no reference to my Exchange 2013 nodes of load balanced names.

Step 4: Configure the Exchange Nodes as TrustedApplicationPools using the actual node FQDN’s and the local lync frontend pools:

New-CsTrustedApplicationPool -Identity Exchangenode.FQDN -Registrar LocalLyncPool -Site “SitenameofLyncPool” -RequiresReplication $False

Repeat this for all Exchange Nodes remembering to populate the registrar with the lync pool which resides in the physical datacenter where the Exchange node is located.

Step 5: Configure the OutlookWebapp Trusted Application:

New-CsTrustedApplication -ApplicationId OutlookWebApp1 -TrustedApplicationPoolFqdn Exchangenode.FQDN -Port
5199

New-CsTrustedApplication -ApplicationId OutlookWebApp2 -TrustedApplicationPoolFqdn Exchangenode.FQDN -Port
5199

I resorted to adding a numerical number starting at 1 to the ApplicationId, in my case I have 8 of these configured.

This will add the Exchange Nodes associated with the Lync Pool in the local data centre in the same site as the Lync Pool with which I am communicating.

It is not possible to create a single application pool with all the nodes configured in the pool since you can not configure the shared name in both sites, assuming that your lync pools are configured in separate sites.

Step 6: Configure the Exchange Partner Application

New-CsPartnerApplication –Identity Exchange –ApplicationTrustLevel Full –MetadataUrl https://yourinternalautodiscover.FQDN /autodiscover/metadata/json/1

Step 7: Enable OATH and Commit:

Set-CsOAuthConfiguration –Real yourlocaldomain.local

Enable-CsTopology

Assuming all the certificates in your enviroment was deployed correctly and there are no communications issues the integration should work at this point:

Good luck and note this guide also works for single server deployments or scenarios without a stretched dag or multiple lync pools.

Powershell and Automation

I will have to start this article with a real world example, by posing a question:

“What if you build something so magnificent that you no longer spend all your time ensuring the uptime of your environment?”

To explain, I just recently completed a fairly large upgrade from 2007 Exchange and OCS 2007R2 to a fully site resilient PA aligned solution that thanks to careful and proper planning by myself and the other individuals involved resulted something that no longer requires much TLC.

So, my environment is now extremely manageable, with very little complexity and fully HA, allowing for all the fun things in life such as fully automated patching cycles and no more late night change controls.

In my scenario this left me with 4 basic requirements:

1. I want to be able to a look at a real-time view of the environment at any point in time and be able to spot potential problems immediately.

2. I don’t want to have to memorize every single PowerShell command in the world to always be able to perform any required task without having to reverence my notes, get-help or even a web search for the answer.

3. I want to empower the people who are tasked with performing actions in my environments consistently and correctly.

4. I don’t want to be dependent on a product rollout or configuration or implementation to meet my needs.

Without going into the details of all the various methods I have tried to achieve what I wanted whilst being able to satisfy all 4 needs, I will jump straight into the solution.

THE SOLUTION: PowerShell within ASP.NET

My final choice is the result of months of thinking, wondering, wanting and also testing various methods available to me whilst remaining within the boundaries of my 4 core requirements.

To go into a slight level of depth, I played with various options, from writing out HTML code via powershell and running it on schedules and putting all my output in a central location hosted within a iframe on a website, to attempting to revive poshboard and various other fun things such as exporting data to csv and importing to excel and visio but yet somehow nothing ever produced the results I wanted.

So let us call this PART 1: How to turn an ordinary website into a powershell interface?

STEP 1: Find a copy of Visual Studio – We want to create an ASP.NET Web Form Page with some packages installed using NUGET.

STEP 2: Once installed we literally need to create ASP.NET web application using forms, it can be a standalone website or a web application that portion does not matter.

STEP 3: We need to host this application somewhere, and for purposes of simplicity for initial testing we can use a service or testing account to host the application pool running the website to reduce initial configuration requirements.

WHAT HAPPENS NEXT?

We create an empty page using the designer view on visual studio and we place 3 items on the page.
1 x Textbox labelled “input”
1 x Textbox labelled “output”
1 x Button labelled “Execute”

This basically looks like this:

Next we need to install our required packages, I used NUGET since it was easy, but ultimately all we really need to proceed is the “system.management.automation” package.

Once this is installed, we merely double click on our execute button above which takes us to the “CODE BEHIND” portion of our page, which is basically C#, which looks a little like powershell.

For anyone not knowing what goes on here, much like myself, this is what I have gathered so far:
Top of the .cs file defines the modules and the page
Next we have a section of actions to be performed when the page is loaded
In the remaining section we have actions such as button click execution code as well as anything else relevant to the page to be defined and or executed.

For our basic example we will import the system.management.automation module at the top and produce some code for allow our button to execute our input field command and write it into the output box:

Note the } highlighted above, as with any other code for every opening bracket “{” there must also be a matching closing bracket “}”.
NB: My screenshot above does not show that.

Our end result is simple, a web site from where to run powershell commands using the elevation we chose to assign to the application pool, or alternatively, running on our local machine using the preview on visual studio, a sample of our basic text results:

This concludes the basic introduction to making this idea turn into a reality.
There will be some time spent on creating pages and interfaces and many hours stuck trying to figure out why something does not work, however the frustration for me personally thus far has been worth every second. 

At this point my project has resulted in a 13 function page with dropdown selections where applicable and utilizing the ‘.visible = true;” or “false” statements to show and hide buttons and boxes and text labels on a single page, driven by a dropdown to allow operators to perform tasks based on an email input.

I look forward to writing the next part of this article, in which I will show the code to manipulate the powershell output data.

Feel free to contact me for a sample project file or portions of the code, I found manipulating a working piece of code in these early stages to be a lot less effort than attempting to construct something from scratch.

 

 

 

 

 

 

 

 

 

 

 

 

 

Exchange 2013 Signatures HTTP Issue Workaround

After being a little disappointed at the lack of image integration support in the signature/disclaimer configuration on my exchange 2013 environment I investigated other options such as base64 image integration.

Sadly there is 5000 character limit which pretty much leaves very little space for a decent image/logo on the signature which is applied.

Naturally placing a http URL in the email signature does not work since Outlook and many other email clients wont download the file as it may not be trusted.

Solution:

This came as a slight surprise but it works:

Use an HTTPS Url for your signature image.

Below is an example of the signature I created:

Whilst still a work in progress at this point the results look good.

Below is a piece of it taken from my Hotmail account, after stripping out all the company info there is not much left, but I am you can get the idea:

 

* UPDATE *

Sadly this will not work as a signature.  REASON: The disclaimer function stamps every single email for the users included in this rule, resulting in multiple’s of these being stamped at the bottom of the emails.

Exchange 2013 Troubleshooting with LOGPARSER STUDIO

Whilst looking for an answer to a very strange issue I had with a particular application receiving random logon failures to my Exchange environment on POP3 via my hardware load balanced hostname, I decided it was time to go to the logs.

Step 1 and 2 is only the commands, I have 8 servers so I did not execute them on every single server one by one but in the interest of time I have resorted to the documentation to bring this portion to you:

Step 1 – Enable protocol logging on POP3:

Set-PopSettings -Server “CAS01” -ProtocolLogEnabled $true

Step 2: – Stop and start the pop3 and pop3backend services:

Stop-service MSExchangePOP3
Stop-service MSExchangePOP3BE
Start-service MSExchangePOP3
Start-service MSExchangePOP3BE

Step 3: Used the pop3 folders under the logging folder in Exchange 2013 as my source data, then I chose CSV as my logfile type and configured the source with a headers file contained all the headers I found in the pop3 source, then I devised the following query and within a few seconds I have my answer.
(SAMPLE BELOW I PUT HEALTH MAILBOXES BACK – TO MUCH COMPANY INFORMATION IN THE REAL OUTPUT )

I wanted connection information did not respond as OK, I did not want system mailboxes and I also wanted all the other data where certain fields were incompete to not be included.

This is what my query looked like in the end:

SELECT top 10000 DISTINCT
dateTime AS dateTime,
user AS user,
cIp AS cIp,
sIp AS sIp,
context AS context,
command as command,
duration as duration
FROM ‘[LOGFILEPATH]’
where user <> NULL and command <> NULL and CONTEXT NOT LIKE ‘%R=ok%’ and CONTEXT <> NULL and user not like ‘%health%’
order by DateTime desc

Very powerful data. Whilst it would be entirely possible to pull those logs into a central place and then process it using excel, this only took about 5 minutes to compile and yielded instant results and my issue was escalated to the relevant area and resolved within the hour.

Head over to the Exchange Team Blog Post or more information and download information

 

 

 

 

Exchange 2013 SP1 Unified Messaging Lync 2013 Integration SNAG

I recently stepped in this little snag myself and thought I would share.

Despite numerous TechNet articles explaining how to load balance your Exchange 2013 UM Roles, how to integrate your Exchange Deployment with Lync etc, somehow there is not many articles explaining the actual certificate requirements.

In my case I assigned my CAS name as my SN, servers names and other services as SAN’s and assigned the certificate to all uses.

Everything worked great except my UM stopped functioning and my Lync OWA Integration also failed.

In my Exchange Server Application Logs the following Started showing up:

Event ID: 1113
MSExchange Unified Messaging UMService N/A
The Client Access server failed to exchange the required certificates with an IP gateway to enable Transport Layer Security (TLS). Please check that the gateway is configured to operate in the correct security mode. If the gateway is required to operate in TLS mode, check that the certificates being used are correct. More information: ‘A TLS failure occurred because the remote server disconnected while TLS negotiation was in progress. The error code = 0x80131500 and the message = Unknown error (0x80131500).’. Remote certificate: (). Remote end point: [::1]:35702. Local end point: [::1]:5063.
3/24/2014 8:23:27 AM Warning

Event ID: 1649
MSExchange Unified Messaging UMCallRouter N/A
The Microsoft Exchange Unified Messaging Call Router service failed to exchange the required certificates with an IP gateway to enable Transport Layer Security (TLS). Please check that the gateway is configured to operate in the correct security mode. If the gateway is required to operate in TLS mode, check that the certificates being used are correct. More information: ‘A TLS failure occurred because the remote server disconnected while TLS negotiation was in progress. The error code = 0x80131500 and the message = Unknown error (0x80131500).’. Remote certificate: (). Remote end point: 127.0.0.1:37877. Local end point: 127.0.0.1:5061.

According to the articles Lync and Exchange in this case use MTLS and the servers are in fact expecting a certificate with the FQDN of the UM Server on the Subject Name.

The same also applies to the Lync OWA IM Integration component.

This is documented however am unable to even find the article to insert the URL here for you to review it.

So a quick summary will have to suffice:

1. Exchange 2013 UM Roles when integrating requires a certificate that is trusted by both the Lync Servers and the Exchange servers in the environment.

2. The certificate can not be self signed – So a certificate from your internal root CA would be required.

3. The FQDN of the UM server must be in the subject name of the certificate.

4. This same certificate can be used to populate the thumbprint field of the web.config when configuring the Lync 2013 OWA IM integration.

Once all this is done, everything naturally starts working again.

Hope this helps someone!

Exchange 2013 SP1 installation on Server 2012R2

Not the most fun I have had in a long time but here is a quick rundown.

I have a 8 Node Multirole stretched DAG with a witness server in a third datacenter.

To perform the SP1 installations I followed the following steps:

1. Fail over all servers in Datacentre B and ensure all services are fully operational from Datacentre A.

2. Uninstall the Exchange UM language Packs from the previous CU3 release, followed by a reboot.

3. Initiate installation of SP1 on all passive servers followed by another reboot.

4. Install the SP1 UM language packs followed by another reboot.

5. Fail over all databases and services to Datacentre B and repeat the process.

FINDINGS:

Whilst everything went rather smooth with no service interruption the SP1 installation took about 8 hours in my environment, which could potentially be reduced.

Performing the installation using the unattended option from the command line does greatly reduce execution times.

2014 – Update – Large Lync and Exchange 2013 Deployments in progress

Firstly let me start by saying sorry for neglecting my blog a little this year.

I have been busy deploying Lync 2013 and Exchange 2013 for a large enterprise customer and with all the excitement and naturally all the work involved have not been able to share all my experiences and issues as well as fixes with everyone as I normally do.

As things are cooling off I will be posting my findings and experiences to hopefully assist anyone else performing similar work and uncovering those weird and wonderfull things that climb out of the wood work as you progress on your project or migration.

Worth a Mention:

1. Exchange 2013 on premises deployment with DNS Load Balancing from an internet breakout point of view – Wonderfull work done Exchange Team – real world DMZ failover tests yielded >10ms reconnect times.

2. Exchange 2013 on premises deployment stretched VLAN Deployments – Again brilliant work – Dropping an entire Datacentre in my pre-production test environment had my Databases remounted and users reconnected in less than 15 seconds.

3. No More sleepness or late nights – My personal favourite outcome – I no longer patch my enviroments at night, instead I merely drain them, fail them over and patch my passive and now idle servers, works great, keeps costs down and gives me more time to focus on other things in life, like updating this blog 🙂

4. Skype Lync Integration – This works wonderfully, I even enabled the GMAIL XMPP integration and I am truly impressed thus far.

 

Exchange 2013 – Health Monitoring Mailboxes Missing

I experienced this issue on one of my Exchange 2013 deployments running CU2.
AD PREP, Forest prep and installations completed successfully without a single error, however the “Health Monitoring” Mailboxes are missing from the “Monitoring Mailboxes” container under “Microsoft Exchange System Objects” in AD.

Typical Event Log Entry:

Error 2014-01-30 01:15:39 AM ManagedAvailability 4 Monitoring

Error:

The mail delivery smtp probe failed 6 times over 42 minutes.
Active Directory operation failed on servername.com . This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031A1256, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

WORKAROUND:
The only real way to fix this is running “setup /r” again which would fix the permissions, which is not always something that can be done at a whim, so you could wait for the next update to be available and schedule your maintenance or you could perform this workaround

STEP 1:

Change that startup type on the “Microsoft Exchange Health Manager” to the credentials of the Exchange Administrator.

Step 2:

Restart the service and wait for the Health Service to initiate a probe which requires the existance of the accounts.

Step 3:

Review the containers in AD and you will see the mailboxes have now been created:

Whilst this is a temporary workaround I have also used this when setting up the OATH between my Exchange 2013 and Lync 2013 Deployments when the accounts for oath cannot be created due to AD permissions.

Once you are done just remember to change the startup account back to the local system account.

 

 

 

Asus P5N7A-VM uATX HTPC Motherboard – HDMI HD Audio Guidance

I donated this board to a friend after he experienced problems achieving DTS Audio on his Sony AMP using a Western Digital Media Player Unfortunately the HDMI Audio would only ever show 2 Channel Stereo.

Through numerous reinstalls and late night fiddling I managed to assist him in finally getting all the AMP’s sound properties to become available and from some of the searches I did I noticed quite a lot of people are still struggling with this.

PROBLEM:
HDMI Audio only showed 2 Channel Stereo despite numerous ‘fiddles’ on his SONY Amplifier, this had worked perfectly on my Yamaha Amplifier.

My HDMI Audio Solutions Guide – This should work on other systems as well.

I believe the below steps should help just about anyone experiencing issues with the HDMI Audio on this board.
* This has been tested on Windows 7 and 8 x64. *

1. Ensure the amplifier Audio Settings are configured correctly:

It appears that when the Audio setup is configured for “AMP + TV” often only the stereo output from the Display are passed via the HDMI detection which results in the AMP’s capabilities not being detected. This should be set to just “AMP”

2. Ensure SPDIF Audio in the BIOS is set to SPDIF and not HDMI

This is audio redirection of the onboard REALTEK sound, this is not required for HDMI from the NVIDIA HDMI interface, Nvidia produces its own sound via HDMI passthrough, this effectively means that the amplifier becomes the sound card.

3. Drivers

The below detail is specific to the NVIDIA GFORCE Range, however the ATI display cards also have a separate Audio Driver to be installed.
The below screenshot is from the NVIDIA PUREHD website GUIDE:( http://www.nvidia.com/content/purevideoHD/pdf/NVIDIA_GeForce_HTPC_Guide_2_8.pdf )

We did however manage this with only the GEFORCE Driver and the HDAUDIO driver from the NVIDIA Website.

GFORCE: http://www.geforce.co.uk/drivers

NVIDIA HDAUDIO KB: http://nvidia.custhelp.com/app/answers/detail/a_id/2384 ( Driver links are on the page )

4. Audio Setup in Windows

By now everything should be ready to roll.
Note there will be 2 audio devices, one for the SPDIF interface and one for the HDMI Audio Output ( On my Yamaha Amplifier it showed the amplifier model as my Audio device – results may vary )

Selecting properties on the NVIDIA HDMI Interface should now yield the AMP’s capabilities:

There is some additional guidance from the NVIDIA PUREVIEW HD Guide with regards to 7.1 ( http://www.nvidia.com/content/purevideoHD/pdf/NVIDIA_GeForce_HTPC_Guide_2_8.pdf  )

I believe this resolution would also be applicable to other HDMI Audio related issues on GFX Cards which support HDMI Audio.

Good Luck!