Exchange 2013 and Lync 2013 IM Integration in an Exchange PA aligned environment

Let me firstly say that I have configured Lync and Exchange IM Integration numerous times and every time I perform this configuration in an Exchange Preferred Architecture Deployment I end up confusing myself and then spend a day or so fixing my configuration. (* TIP – Now that I have blogged this naturally I can reference my own blog in future J )

Scenario:

1 Exchange Stretched DAG which resides in 2 different data centers on the same subnet using Cisco OTV Stretched Vlan networking technology.

2 Lync Pools configured in the same network subnet.

The confusing part:

Typically I confuse myself with where I should point my “IMPROVIDER” in my exchange server web.config files since technically I have 1 Exchange, however I have two different lync pools which could potentially serve my IM configuration.

Quick and Easy Guide:

I have resorted to pointing my Exchange nodes to the Lync Pool which resides in the same physical data center.

My logic in this is that in the event of a datacenter going down, my traffic from both internal and external networks will not be routed to that datacenter which also means my exchange naturally would not be able to speak to the lync pool in that datacenter either, makes sense.

Step 1: Ensure all Exchange nodes have a separate internally signed UM certificate with the actual node FQDN in the certificate.

Step 2: Using the script provided on the technet script centre by Michel de Rooij here I configure each Exchange node to point to the local lync pool. – Note you typically have to rerun this after each CU or SP you install on Exchange.

Step 3: Enable the IM on the Exchange nodes by running the following command:

Get-OwaVirtualDirectory | Set-OwaVirtualDirectory –InstantMessagingEnabled $True –InstantMesssagingType OCS

Step 3: Utilising get-cstrustedapplication, get-cstrustedapplicationpool I ensure that I have no reference to my Exchange 2013 nodes of load balanced names.

Step 4: Configure the Exchange Nodes as TrustedApplicationPools using the actual node FQDN’s and the local lync frontend pools:

New-CsTrustedApplicationPool -Identity Exchangenode.FQDN -Registrar LocalLyncPool -Site “SitenameofLyncPool” -RequiresReplication $False

Repeat this for all Exchange Nodes remembering to populate the registrar with the lync pool which resides in the physical datacenter where the Exchange node is located.

Step 5: Configure the OutlookWebapp Trusted Application:

New-CsTrustedApplication -ApplicationId OutlookWebApp1 -TrustedApplicationPoolFqdn Exchangenode.FQDN -Port
5199

New-CsTrustedApplication -ApplicationId OutlookWebApp2 -TrustedApplicationPoolFqdn Exchangenode.FQDN -Port
5199

I resorted to adding a numerical number starting at 1 to the ApplicationId, in my case I have 8 of these configured.

This will add the Exchange Nodes associated with the Lync Pool in the local data centre in the same site as the Lync Pool with which I am communicating.

It is not possible to create a single application pool with all the nodes configured in the pool since you can not configure the shared name in both sites, assuming that your lync pools are configured in separate sites.

Step 6: Configure the Exchange Partner Application

New-CsPartnerApplication –Identity Exchange –ApplicationTrustLevel Full –MetadataUrl https://yourinternalautodiscover.FQDN /autodiscover/metadata/json/1

Step 7: Enable OATH and Commit:

Set-CsOAuthConfiguration –Real yourlocaldomain.local

Enable-CsTopology

Assuming all the certificates in your enviroment was deployed correctly and there are no communications issues the integration should work at this point:

Good luck and note this guide also works for single server deployments or scenarios without a stretched dag or multiple lync pools.

Exchange 2013 Signatures HTTP Issue Workaround

After being a little disappointed at the lack of image integration support in the signature/disclaimer configuration on my exchange 2013 environment I investigated other options such as base64 image integration.

Sadly there is 5000 character limit which pretty much leaves very little space for a decent image/logo on the signature which is applied.

Naturally placing a http URL in the email signature does not work since Outlook and many other email clients wont download the file as it may not be trusted.

Solution:

This came as a slight surprise but it works:

Use an HTTPS Url for your signature image.

Below is an example of the signature I created:

Whilst still a work in progress at this point the results look good.

Below is a piece of it taken from my Hotmail account, after stripping out all the company info there is not much left, but I am you can get the idea:

 

* UPDATE *

Sadly this will not work as a signature.  REASON: The disclaimer function stamps every single email for the users included in this rule, resulting in multiple’s of these being stamped at the bottom of the emails.

Exchange 2013 Troubleshooting with LOGPARSER STUDIO

Whilst looking for an answer to a very strange issue I had with a particular application receiving random logon failures to my Exchange environment on POP3 via my hardware load balanced hostname, I decided it was time to go to the logs.

Step 1 and 2 is only the commands, I have 8 servers so I did not execute them on every single server one by one but in the interest of time I have resorted to the documentation to bring this portion to you:

Step 1 – Enable protocol logging on POP3:

Set-PopSettings -Server “CAS01” -ProtocolLogEnabled $true

Step 2: – Stop and start the pop3 and pop3backend services:

Stop-service MSExchangePOP3
Stop-service MSExchangePOP3BE
Start-service MSExchangePOP3
Start-service MSExchangePOP3BE

Step 3: Used the pop3 folders under the logging folder in Exchange 2013 as my source data, then I chose CSV as my logfile type and configured the source with a headers file contained all the headers I found in the pop3 source, then I devised the following query and within a few seconds I have my answer.
(SAMPLE BELOW I PUT HEALTH MAILBOXES BACK – TO MUCH COMPANY INFORMATION IN THE REAL OUTPUT )

I wanted connection information did not respond as OK, I did not want system mailboxes and I also wanted all the other data where certain fields were incompete to not be included.

This is what my query looked like in the end:

SELECT top 10000 DISTINCT
dateTime AS dateTime,
user AS user,
cIp AS cIp,
sIp AS sIp,
context AS context,
command as command,
duration as duration
FROM ‘[LOGFILEPATH]’
where user <> NULL and command <> NULL and CONTEXT NOT LIKE ‘%R=ok%’ and CONTEXT <> NULL and user not like ‘%health%’
order by DateTime desc

Very powerful data. Whilst it would be entirely possible to pull those logs into a central place and then process it using excel, this only took about 5 minutes to compile and yielded instant results and my issue was escalated to the relevant area and resolved within the hour.

Head over to the Exchange Team Blog Post or more information and download information

 

 

 

 

Exchange 2013 SP1 Unified Messaging Lync 2013 Integration SNAG

I recently stepped in this little snag myself and thought I would share.

Despite numerous TechNet articles explaining how to load balance your Exchange 2013 UM Roles, how to integrate your Exchange Deployment with Lync etc, somehow there is not many articles explaining the actual certificate requirements.

In my case I assigned my CAS name as my SN, servers names and other services as SAN’s and assigned the certificate to all uses.

Everything worked great except my UM stopped functioning and my Lync OWA Integration also failed.

In my Exchange Server Application Logs the following Started showing up:

Event ID: 1113
MSExchange Unified Messaging UMService N/A
The Client Access server failed to exchange the required certificates with an IP gateway to enable Transport Layer Security (TLS). Please check that the gateway is configured to operate in the correct security mode. If the gateway is required to operate in TLS mode, check that the certificates being used are correct. More information: ‘A TLS failure occurred because the remote server disconnected while TLS negotiation was in progress. The error code = 0x80131500 and the message = Unknown error (0x80131500).’. Remote certificate: (). Remote end point: [::1]:35702. Local end point: [::1]:5063.
3/24/2014 8:23:27 AM Warning

Event ID: 1649
MSExchange Unified Messaging UMCallRouter N/A
The Microsoft Exchange Unified Messaging Call Router service failed to exchange the required certificates with an IP gateway to enable Transport Layer Security (TLS). Please check that the gateway is configured to operate in the correct security mode. If the gateway is required to operate in TLS mode, check that the certificates being used are correct. More information: ‘A TLS failure occurred because the remote server disconnected while TLS negotiation was in progress. The error code = 0x80131500 and the message = Unknown error (0x80131500).’. Remote certificate: (). Remote end point: 127.0.0.1:37877. Local end point: 127.0.0.1:5061.

According to the articles Lync and Exchange in this case use MTLS and the servers are in fact expecting a certificate with the FQDN of the UM Server on the Subject Name.

The same also applies to the Lync OWA IM Integration component.

This is documented however am unable to even find the article to insert the URL here for you to review it.

So a quick summary will have to suffice:

1. Exchange 2013 UM Roles when integrating requires a certificate that is trusted by both the Lync Servers and the Exchange servers in the environment.

2. The certificate can not be self signed – So a certificate from your internal root CA would be required.

3. The FQDN of the UM server must be in the subject name of the certificate.

4. This same certificate can be used to populate the thumbprint field of the web.config when configuring the Lync 2013 OWA IM integration.

Once all this is done, everything naturally starts working again.

Hope this helps someone!

Exchange 2013 SP1 installation on Server 2012R2

Not the most fun I have had in a long time but here is a quick rundown.

I have a 8 Node Multirole stretched DAG with a witness server in a third datacenter.

To perform the SP1 installations I followed the following steps:

1. Fail over all servers in Datacentre B and ensure all services are fully operational from Datacentre A.

2. Uninstall the Exchange UM language Packs from the previous CU3 release, followed by a reboot.

3. Initiate installation of SP1 on all passive servers followed by another reboot.

4. Install the SP1 UM language packs followed by another reboot.

5. Fail over all databases and services to Datacentre B and repeat the process.

FINDINGS:

Whilst everything went rather smooth with no service interruption the SP1 installation took about 8 hours in my environment, which could potentially be reduced.

Performing the installation using the unattended option from the command line does greatly reduce execution times.

2014 – Update – Large Lync and Exchange 2013 Deployments in progress

Firstly let me start by saying sorry for neglecting my blog a little this year.

I have been busy deploying Lync 2013 and Exchange 2013 for a large enterprise customer and with all the excitement and naturally all the work involved have not been able to share all my experiences and issues as well as fixes with everyone as I normally do.

As things are cooling off I will be posting my findings and experiences to hopefully assist anyone else performing similar work and uncovering those weird and wonderfull things that climb out of the wood work as you progress on your project or migration.

Worth a Mention:

1. Exchange 2013 on premises deployment with DNS Load Balancing from an internet breakout point of view – Wonderfull work done Exchange Team – real world DMZ failover tests yielded >10ms reconnect times.

2. Exchange 2013 on premises deployment stretched VLAN Deployments – Again brilliant work – Dropping an entire Datacentre in my pre-production test environment had my Databases remounted and users reconnected in less than 15 seconds.

3. No More sleepness or late nights – My personal favourite outcome – I no longer patch my enviroments at night, instead I merely drain them, fail them over and patch my passive and now idle servers, works great, keeps costs down and gives me more time to focus on other things in life, like updating this blog 🙂

4. Skype Lync Integration – This works wonderfully, I even enabled the GMAIL XMPP integration and I am truly impressed thus far.

 

Exchange 2013 – Health Monitoring Mailboxes Missing

I experienced this issue on one of my Exchange 2013 deployments running CU2.
AD PREP, Forest prep and installations completed successfully without a single error, however the “Health Monitoring” Mailboxes are missing from the “Monitoring Mailboxes” container under “Microsoft Exchange System Objects” in AD.

Typical Event Log Entry:

Error 2014-01-30 01:15:39 AM ManagedAvailability 4 Monitoring

Error:

The mail delivery smtp probe failed 6 times over 42 minutes.
Active Directory operation failed on servername.com . This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031A1256, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

WORKAROUND:
The only real way to fix this is running “setup /r” again which would fix the permissions, which is not always something that can be done at a whim, so you could wait for the next update to be available and schedule your maintenance or you could perform this workaround

STEP 1:

Change that startup type on the “Microsoft Exchange Health Manager” to the credentials of the Exchange Administrator.

Step 2:

Restart the service and wait for the Health Service to initiate a probe which requires the existance of the accounts.

Step 3:

Review the containers in AD and you will see the mailboxes have now been created:

Whilst this is a temporary workaround I have also used this when setting up the OATH between my Exchange 2013 and Lync 2013 Deployments when the accounts for oath cannot be created due to AD permissions.

Once you are done just remember to change the startup account back to the local system account.

 

 

 

Exchange 2007 CCR Cluster Strange Replication Issue

Last night whilst performing regular monthly maintenance at my current client, I found that my passive node no longer could replicate and had in fact placed the storage groups in ‘suspended’ state:

A look at the event logs indicated some sort of file lock, which I attempted to resolve with another reboot which did not change anything:

EVENTS LOGGED:

Event 2070 MSExchangeRepl

The Microsoft Exchange Replication Service encountered an error while inspecting the logs and database for EXCHANGESERVER\SG05 on startup. The specific error code returned is: Microsoft.Exchange.Cluster.Replay.FileCheckAccessDeniedException: File check failed : File ‘Z:\Program Files\Microsoft\Exchange Server\SG05\MDB05.edb’ could not be opened.  The file is locked or in use. —> Microsoft.Exchange.Isam.IsamFileAccessDeniedException: Cannot access file, the file is locked or in use (-1032)

at Microsoft.Exchange.Isam.?A0x3fee5ce8.HandleError(Int32 err)

at Microsoft.Exchange.Isam.Interop.MJetGetDatabaseFileInfo(String database)

at Microsoft.Exchange.Cluster.Replay.FileChecker.GetDatabaseFileInfo(String database)

— End of inner exception stack trace —

at Microsoft.Exchange.Cluster.Replay.FileChecker.GetDatabaseFileInfo(String database)

at Microsoft.Exchange.Cluster.Replay.FileChecker.CheckDatabase()

at Microsoft.Exchange.Cluster.Replay.FileChecker.RunChecks()

at Microsoft.Exchange.Cluster.Replay.ReplicaInstance.ConfigurationChecker(Object stateIgnored).

EVENT 489 ESE

Microsoft.Exchange.Cluster.ReplayService (5176) An attempt to open the file “Z:\Program Files\Microsoft\Exchange Server\SG05\MDB05.edb” for read only access failed with system error 32 (0x00000020): “The process cannot access the file because it is being used by another process. “.  The open file operation will fail with error -1032 (0xfffffbf8).

For more information, click http://www.microsoft.com/contentredirect.asp.

FURTHER INVESTIGATION:

I ran Process Monitor and had a look at the path and found only exchange accessing the files, thus ruling out an actual ‘lock’ on the files:

I searched tirelessly for an answer and found mostly suggestions to make new databases and restart services.
With my user count and maintenance window hours left none of these were a possibility.

At a very early hour of the morning I decided to look at the folder permissions, purely out of interest.

A Database folder which would not replicate:

And a view of a healthy database folder:

Rather Strange, a controlled failover and a reboot of a server caused the “DOMAIN\Exchange Servers” READ permission to disappear from 4 of my 6 Database Folders.

I checked the log and system paths and found the same result.

Once I added the permissions onto the folders, my databases and storage groups once again were able to replicate:

Here is hoping if this happens or happened to you that you found this article!

Exchange Mailbox Moves When Custom Size Limits go wrong

Just a quick script I hammered out last night whilst pulling out my hair during a mailbox database ‘evacuation’.

Of the 650 users I was moving to a new database I had about 100 users with Custom Limits and all of them were over their limit and refused to move.

Previously I would just do a quick CSV, do some magic with excel and paste the commands but I had some time and finally produced a script to do all of this ‘automatically’

It is not pretty, but it does work.

—– SCRIPT START —–
$strSOURCEDB = Read-Host “SOURCE DATABASE ( FORMAT – SERVERNAME\STORE\DTB )”
$strTARGETEDB = Read-Host “TARGETDATABASE ( FORMAT – SERVERNAME\STORE\DTB )”
Get-MailboxStatistics -Database “$StrSOURCEDB”| Where {$_.StorageLimitStatus -eq “MailboxDisabled”} | Add-Member -MemberType ScriptProperty -Name TotalItemSizenew -Value {$this.totalitemsize + 10000 } -PassThru | Select DisplayName,TotalItem* | export-csv -path USERLIST1.csv
import-csv userlist1.csv | ForEach-Object -Process { Set-Mailbox -identity $_.DisplayName -ProhibitSendReceiveQuota $_.TotalItemSizenew}
import-csv userlist1.csv | ForEach-Object -Process { move-mailbox -identity $_.DisplayName -targetdatabase $strTARGETEDB -Confirm:$False –PreserveMailboxSizeLimit:$True}

—– SCRIPT END —–

Save as a .ps1 and ensure you allow unsigned scripts to run.

The script prompts for the source and destination database in “SERVERNAME\STORE\DTB” format.

Happy Mailbox Moves!

Microsoft Certified Solutions Expert: Messaging (MCSE) Charter Member

I recently found out purely by coincidence about the CHARTER member status awarded to individuals who successfully complete “new” Microsoft Certifications.

From the website:

Charter Members are the pioneering group of individuals who achieve a certification within six months following the retail release date of the certification. (People who pass the beta exams will receive the Charter certificate after the certification is commercially released.) Charter Members are recognized by being given the Charter version of the certificate acknowledging their early adoption of the technology solution. The Charter version of the certificate includes the word “Charter.”

I passed my 70-342 Exam yesterday and received this status, below my certificate:

Not as flashy as the golden certificates I received for my Lync 2010 and Exchange 2010 MCITP certifications, but I still feel very proud to have it.

For those wondering about preparation for these exams at this point in time, perhaps to grab a Charter status for yourself or for other reasons, below my preparation materials:

Teched 2013 – Attended mostly Exchange sessions – Very valuable for * new * features and concepts

Exchange, Office & Office 365 Ignite – Presented by Nicholas Blank (MCM ) in South Africa and *free* , more information and labs available here

Exchange 2013 Compiled Helpfile – Updated April2013

Teched Exchange 2013 Pages – Lots of documentation and guides available from here

MSEXCHANGETEAM.COM – All happenings direct from the Exchange Team, very valuable resource, worth visiting daily

Last and most definately not the least, my HP Microserver home lab, Running Server 2012 and bumped up to 16GB RAM, fitted with SSD and Sata discs.  This inexpensive Gem has been instrumental in my preparation for this exam, if finances allow I would recommend 2 of these to do some interesting clustering and additional role tests.  I lost out on some of the DAG lab experiments, but since I work in a Enterprise Enviroment I deal with HA in exchange Daily.  A note on this server, with all my labs for Exchange 2013 still running, ‘production’ – accessable from the internet, outlook anywhere, owa, oac, activesync etc, I am currently deploying my Lync Labs to hopefully achieve Lync 2013 Charter Status aswell.