Kerberos is Easy – Part 2

kids coding

Yes my friends, this post has been long overdue. Life, work and all the other good excuses got in the way. However, there is nothing like a friend calling you out with a “WTF I need part 2” to get the motivation and kerberos mana flowing again. So where did we leave off? In part 1 we discussed some of the most common issues with Kerberos authentication and the necessity to break the problem down to client-side vs server-side authentication.

In part 2 we will look at some of the first troubleshooting steps I take to determine why nothing is working – yes, this happens to me too.

ADTest is Your New BF4L

You know those people that open a web browser to see if their Internet works… yea, don’t be that person. Open a terminal, check if you can ping your gateway, public DNS server, etc. and then, only then, open a browser.

ADTest is your equivalent of ping. Don’t assume because you configured an Active Directory AAA object that authentication is just going to work. Please open a console and verify that Kerberos authentication against the AD server is working with ADTest. Check out my APM Troubleshooting with ADTest for more information.

Time is not on Your Side

If you have worked with Kerberos before you know it is supper picky about time drift. If ADTest just won’t work and you can’t figure out why ensure the BIG-IP’s time matches the KDC. If you need to adjust the BIG-IP follow the F5 SOL3381.

These Are Not the KDCs You’re Looking For

Ever been in a multi-domain environment and the AD admin swears the KDC you’re talking to is the correct one; never mind the “Kerberos Principal Unknown” error you keep getting. So if ADTest doesn’t work then we need to ensure the AD server we’re talking to is a KDC for the expected realm. Now, if someone can RDP into the server this can be ruled out pretty quickly, but when in life is anything that easy.

So it’s helpful to use nslookup to find all KDCs for the intended domain and ensure the IP you were give is in this list – example below:

C:\Users\user>nslookup -type=SRV


Address: SRV service location:

priority = 0

weight = 100

port = 88

svr hostname = internet address = internet address =

Playing Go Fish with SPNs

You have to admire how Microsoft took something as complicated as Kerberos and made it trivial to deploy and manage inside of Active Directory. Having managed MIT and MS version of Kerberos myself I felt a little guilty after setting up my first AD server without hours of troubleshooting issues. However, with that ease of use Microsoft also made it easy to shoot your own foot off if you have no basic understanding of Kerberos.

Think of the KDC as a key, value pair database. The KDC will let you store multiple keys of the same value even though you shouldn’t do that. So if you have multiple SPN entries in AD you are not guarantied that a request for a ticket will return the value you’re looking for. This typically presents itself in APM as authentication works onetime and not the other.

An easy way to check this is to log into a domain machine and issue:

setspn -X

This will print out any duplicate SPNs in your KDC. If the SPN you are working with appears in this list then you need to correct this issue. The easiest way I find is to delete the service account you have created for APM and just use the service account the web server application pool is using.

Cached Tickets

APM caches Kerberos tickets for both client side Kerberos authentication and server side Kerberos SSO. If you’re troubleshooting Kerberos be sure to clear these caches after you’ve made modifications.

For Kerberos AAA:

bigstart restart apd rba

For Kerberos SSO:

bigstart restart websso

I’ll Just Do It Myself

If you are working with Kerberos SSO then you have to sometimes determine if the issue is with APM obtaining the token or the web server not accepting the token. If you have your WebSSO logging set to debug and Kerberos SSO is working then you should see


If not, then there are a few CLI commands you can use to simulate the request APM makes to the KDC.

First, remove all kerberos tickets.


Second, obtain a kerberos ticket as the AD delegation account (if this works you won’t get a response)

kinit -f <SPN of AD delegation account>

Finally, test if the ticket you obtain has delegation capabilities

knvo -C -U <username> <SPN of AD delegation account>

If you receive a key version number on the kvno command then everything is working and it proves that ASREQ and RSREQ work. So the issue more than likely is on the web server side and not the F5 – more on this in part 3.

F5 APM and Okta Integration

I’m happy to announce the F5 APM and Okta integration guide has been published on Okta’s website.  I’ve been playing with this solution for the past 4 months and I have to say it’s pretty cool.  F5 Access Policy Manager and Okta complement each other well and provide customers a solution to address identity, access and single sign-on for cloud and on-premises applications regardless of their authentication requirements.

In this integration guide F5 and Okta focus on single sign-on capabilities for on-premises legacy applications that cannot consume a SAML or Claim assertion.  For these legacy applications you can leverage F5’s Access Policy Manger to perform Kerberos Constrained Delegation or Header authentication.

I deviated from the deployment guide and used APM’s per-request policy engine to insert the header versus the iRule.  I prefer this method as it is easier for people new to F5 and it will survive future upgrades.

I’ve provided a video demo below:

SAML – Client versus Server Authentication with F5 APM

As organizations start to utilize Software as a Service (SaaS) the concern on how to authenticate users becomes a critical security issue. Many organizations look to federated authentication mechanisms, such as SAML, to help address this security risk. The benefits of using SAML are that user credentials are not replicated across each vendor cloud instance and that it greatly simplifies user management. However, not all applications will be migrated to the cloud due to limitations or security concerns. So when you start adopting SAML what happens to the applications that are left in your data centers.

Many of the customers I work with adopt SAML as an authentication platform for both internal and cloud base solutions. The road block I typically run into here is that they try to implement SAML on all service – this is a “square peg in a round hole” approach.  While many commercial applications and programming frameworks are started to support SAML the need to implement SAML on the backend servers is not always necessary.

In my view, SAML works great as a client-side authentication mechanism for self hosted applications but does not always “fit the bill” on a server-side authentication. When you use the F5 BIG-IP to provide Single Sign-on service to your internal applications you have several server-side authentication options to choose from: HTTP basic, form, NTLM and Kerberos to name a few.  In certain situations these authentication options are much easier to maintain and implement across a large number of applications.

When using SAML for authentication the IdP provides an assertion to prove the user’s identity.  In this situation the F5 does not have the user’s password so HTTP Basic, HTTP form and NTLM authenticate are not possible – this is ideal in security conscious environments because sensitive credentials are not being passed around. However, the F5 can convert the SAML assertion into a token and use Kerberos Constrained Delegation to authenticate the user to the backend web server.  A huge benefit to this model is that it scales very well with a large number of applications because most web servers supports Kerberos authentication and you do not have to create one-off APM SSO profiles for each application.

In conclusion, F5 BIG-IP provides organizations the ability to authenticate users securely to services whether in the cloud or in your data center.  The important thing to consider when designing your authentication solution is to examine all authentication option available to you and stop trying to squeeze the square peg into the round hole.