APM Cookbook – Okta MFA Integration

2016-09-12_21-41-09Since the launch of the Okta and F5 Integration Guide I’ve seen interest in leveraging this partnership take off.  One aspect I’ve enjoyed is watching how customers address pain points they were not able to address previously.  For example, providing multi-factor authentication (MFA) for Microsoft Exchange Outlook Web Access (OWA).

This particular customer standardized on Okta’s MFA solution but
OWA was behind Microsoft Threat Management Gateway (TMG) and could not easily integrate with Okta.  For this solution F5’s Access Policy Manager (APM) will replace the TMG servers  and leverage Okta’s on-premises RADIUS agent for MFA via Okta Verify, which supports push notification – by far my favorite feature.

I’ve included a video below that walks through the process of configuring Okta for RADIUS based multifactor as well as configuring APM to leverage Okta’s RADIUS agent.

Okta Configuration

On the Okta administrator portal you’ll need to create a new Okta Sign-on policy: Security -> Policies.  Once you name the new policy you’ll need to add a rule:


The crucial part here is to select RADIUS for the And Authenticates via option.

F5 Configuration

The F5 APM configuration is pretty straight forward since you can use the built-in VPE macro template for RADIUS authentication but we’ll need to create a RADIUS AAA object first.


Once the RADIUS AAA object is created go ahead and create a new Access Profile and customize your VPE as shown below – for detailed steps please watch the attached video.


Pretty easy solution and we’re just scratching the surface on what is possible.  Can’t wait to start playing with Okta’s API via iRules LX!

Kerberos is Easy – Part 2

kids coding

Yes my friends, this post has been long overdue. Life, work and all the other good excuses got in the way. However, there is nothing like a friend calling you out with a “WTF I need part 2” to get the motivation and kerberos mana flowing again. So where did we leave off? In part 1 we discussed some of the most common issues with Kerberos authentication and the necessity to break the problem down to client-side vs server-side authentication.

In part 2 we will look at some of the first troubleshooting steps I take to determine why nothing is working – yes, this happens to me too.

ADTest is Your New BF4L

You know those people that open a web browser to see if their Internet works… yea, don’t be that person. Open a terminal, check if you can ping your gateway, public DNS server, etc. and then, only then, open a browser.

ADTest is your equivalent of ping. Don’t assume because you configured an Active Directory AAA object that authentication is just going to work. Please open a console and verify that Kerberos authentication against the AD server is working with ADTest. Check out my APM Troubleshooting with ADTest for more information.

Time is not on Your Side

If you have worked with Kerberos before you know it is supper picky about time drift. If ADTest just won’t work and you can’t figure out why ensure the BIG-IP’s time matches the KDC. If you need to adjust the BIG-IP follow the F5 SOL3381.

These Are Not the KDCs You’re Looking For

Ever been in a multi-domain environment and the AD admin swears the KDC you’re talking to is the correct one; never mind the “Kerberos Principal Unknown” error you keep getting. So if ADTest doesn’t work then we need to ensure the AD server we’re talking to is a KDC for the expected realm. Now, if someone can RDP into the server this can be ruled out pretty quickly, but when in life is anything that easy.

So it’s helpful to use nslookup to find all KDCs for the intended domain and ensure the IP you were give is in this list – example below:

C:\Users\user>nslookup -type=SRV _kerberos._tcp.dc._msdcs.f5demo.com

Server: ad1.f5demo.com


_kerberos._tcp.dc._msdcs.f5demo.com SRV service location:

priority = 0

weight = 100

port = 88

svr hostname = ad1.f5demo.com

ad1.f5demo.com internet address =

ad1.f5demo.com internet address =

Playing Go Fish with SPNs

You have to admire how Microsoft took something as complicated as Kerberos and made it trivial to deploy and manage inside of Active Directory. Having managed MIT and MS version of Kerberos myself I felt a little guilty after setting up my first AD server without hours of troubleshooting issues. However, with that ease of use Microsoft also made it easy to shoot your own foot off if you have no basic understanding of Kerberos.

Think of the KDC as a key, value pair database. The KDC will let you store multiple keys of the same value even though you shouldn’t do that. So if you have multiple SPN entries in AD you are not guarantied that a request for a ticket will return the value you’re looking for. This typically presents itself in APM as authentication works onetime and not the other.

An easy way to check this is to log into a domain machine and issue:

setspn -X

This will print out any duplicate SPNs in your KDC. If the SPN you are working with appears in this list then you need to correct this issue. The easiest way I find is to delete the service account you have created for APM and just use the service account the web server application pool is using.

Cached Tickets

APM caches Kerberos tickets for both client side Kerberos authentication and server side Kerberos SSO. If you’re troubleshooting Kerberos be sure to clear these caches after you’ve made modifications.

For Kerberos AAA:

bigstart restart apd rba

For Kerberos SSO:

bigstart restart websso

I’ll Just Do It Myself

If you are working with Kerberos SSO then you have to sometimes determine if the issue is with APM obtaining the token or the web server not accepting the token. If you have your WebSSO logging set to debug and Kerberos SSO is working then you should see


If not, then there are a few CLI commands you can use to simulate the request APM makes to the KDC.

First, remove all kerberos tickets.


Second, obtain a kerberos ticket as the AD delegation account (if this works you won’t get a response)

kinit -f <SPN of AD delegation account>

Finally, test if the ticket you obtain has delegation capabilities

knvo -C -U <username> <SPN of AD delegation account>

If you receive a key version number on the kvno command then everything is working and it proves that ASREQ and RSREQ work. So the issue more than likely is on the web server side and not the F5 – more on this in part 3.

F5 APM and Okta Integration

I’m happy to announce the F5 APM and Okta integration guide has been published on Okta’s website.  I’ve been playing with this solution for the past 4 months and I have to say it’s pretty cool.  F5 Access Policy Manager and Okta complement each other well and provide customers a solution to address identity, access and single sign-on for cloud and on-premises applications regardless of their authentication requirements.

In this integration guide F5 and Okta focus on single sign-on capabilities for on-premises legacy applications that cannot consume a SAML or Claim assertion.  For these legacy applications you can leverage F5’s Access Policy Manger to perform Kerberos Constrained Delegation or Header authentication.

I deviated from the deployment guide and used APM’s per-request policy engine to insert the header versus the iRule.  I prefer this method as it is easier for people new to F5 and it will survive future upgrades.

I’ve provided a video demo below:

Yubikey One-Time-Password Authentication with APM

yubikey_4Well my Yubikey 4 arrived today so I had a chance to play around with their one-time-password capabilities – read about their U2F and APM capabilities here. The primary benefit about OTP over U2F is it’s supported across almost every major browser and OS.  This makes the Yubikey 4 a little more palatable for enterprises – note the Yubikey 4 supports both OTP and U2F.

Jason Rahm posted an article on DevCentral regarding 2FA using Yubikey, YubiCloud and BIG-IP LTM  back in 2013.  I’ve adapted this iRules to use APM Agent Events so we can leverage Yubikeys for 2FA in APM.  For more information on Yubikey OTP clients check out the Getting Started Writing Clients page.


  1. To configure this you’ll need to add the iRule below to your BIG-IP and XXXXXX with your YubiCloud client ID and Secret Key.
  2. Add a data group (yubikey_users) and populate it with username:serial pairs
  3. add an iRule event to your APM VPE
    1. set the name to OTP Valid
    2. set the ID to “otp_verify”
    3. add a branch rule
      1. name it Yes
      2. add an advanced expression of:

expr { [mcget {session.custom.otp_valid} ] == 1}


No too difficult.  Some ways that we could extend this code would be to try multiple cloud instances (api1.yubico.com-api5.yubico.com) and provide a self enrollment page if the user’s serial number is not in our data group – I’m writing an example of this with Google Authenticator and iRules LX so stay tuned.

U2F Authentication with F5 APM and Duo Security

Security-Key-by-Yubico-1000-2016-444x444I’ve been working on Universal 2nd Factor (U2F) authentication today and it’s a very interesting concept.  There is no requirement to enter a 6-digit code for 2nd factor authentication.  The website I’m logging into detects my Yubikey and the key button flashes a blue light.  Press the button and you’re automatically authenticated.

I’ve configured it with some of the websites I use and I’m going to give it a try for the next few weeks to see what I think.  Now, how to make sure I don’t lose it since I don’t normally carry my key chain around…

See the demo below with F5 APM and Duo Security:

iRules LX – AFM/APM Dynamic Firewall Rules

One of the iRules LX projects I’ve been working on is the ability to dynamically open firewall rules based upon a user’s APM Access Session.  In this example the user is external trying to access internal server resources but is blocked by AFM’s default firewall policies.  The user then logs into an APM policy and upon successful login AFM allows that external user access to internal resources.  See the video below for a demo:

iRules do not have direct access to control plane settings.  So prior to TMOS 12.1 this wasn’t easily doable without some serious iRule foo around sideband connections. However, with the introduction of iRules LX we can leverage Node.js to help us interact with the control plane via API calls.

So how does it work?

You still leverate iRules in iRules LX but a new set of commands allow you to pass data from TMM (data plane) to Node.js (control plane) and process that data via javascript – more information in the Getting Started With iRules LX series on DevCentral.

So what does this look like?

Your iRules do not change much except for the addition of the ILX commands:

  • ILX::init – invokes the specified node methods
  • ILX::call – establishes a communication path from an iRules to the node process
  • ILX::notify  – sends a message to the specified node method but does not wait for a response

As an example I’ve included the  iRule used in the AFM/APM dynamic firewall rule project below:

Notice the ILX::init on line 7 – this is configured with the iRules LX plugin name and the iRules LX extension inside our workspace.


What about Node.js

Next, you provide the Node.js methods your iRule calls in the index.js file

Not too hard and it offers a world of possibilities with the inclusion of NPM modules.

Where’s the rest of the code?

For a complete view of the iRules LX code please visit the Github repository.

iRules LX – 5 Tips to Get You Started

iRules LX is now GA with the release of TMOS 12.1.  I’ve been using iRules LX over the past two weeks working on some cool projects I hope to share in the near future.  In the meantime I wanted to provide some tips I’ve picked up while learning Node.JS and iRules LX.

1 – NPM makes life easier

Wow… I was speechless when I first heard that iRules LX would have native access to NPM.  I’ve seen some pretty cool internal projects that leverage publically available NPM tool sets and now you have the power of NPM inside TMOS.  So what does this mean… well for all you APM fans out that you can now read/write to SQL!  NPM allows you to quickly create applications and features without having to write everything from scratch.  This opens up a new world of possibilities for F5’s advance modules like APM and AFM.  More to come on this in the near future.

2 – Learn Node.js or make friends with a developer

There are probably more Node.js developers in your city than there are TCL developers in your state.. and probably your surrounding states.  So I was happy to hear F5 was opening up our development capabilities to a wider audience.  If you’re new to Node.JS I highly recommend Code School or if your organization has a membership to Safari Books  they have some great resources as well.

I’m also a big fan of test driven development. Mocha and Chai are great tools and supper easy to learn.  The benefit of TDD is you always know your code works.  If any update breaks something, the TDD process will find it.I also like developing the Node.js code on my laptop so TDD helps me verify that my java libraries work before I upload them to the BIG-IP for integration with iRules.

3 – DevCentral

Eric Flores over on DevCentral publish a great 5 part Getting Started with iRules LX series.  Between this and the Node.js video tutorial in the Safari Books online library I was ready to go.  DevCentral is also a great place to ask question and get assistance.  With over 250,000+ members it’s definitely a site you should familiarize yourself with.

4 – {}; or {}

Okay, so I’ve written some Node.js code and I’m finishing my iRules code to glue the processes together and when I hit the “Reload from Workspace” button I get a slew of iRule error…. guess what I did wrong 🙂  Can’t write JavaScript in iRules

Syntax issues bit me several times as I moved between iRules and JavaScript so be sure you keep an eye on this.  Some of the most common issues:

Accessing a variable in iRules requires a $, in JavaScript it does not
JavaScript uses a semicolon (;) at the end of a statement, iRules does not

5 – Callback hell

One of the hardest things for me to wrap my head around while learning Node.js is the concept of synchronous vs asynchronous processes.  I’ve always been use to writing code in a synchronous way so it stumped me why my functions would not return a value or my debug statements were always empty.

An easy way to think about it:

if you’re reading/writing to a filesystem or a web server then use an asynchronous process
if you’re reading/writing to a local function, say generating a secret key for Google Authenticator, then use a synchronous process
Also, in your Mocha test framework you’ll need to use the break() function for any process that are synchronous.

I hope these tips help you has your dive into iRules LX.  I’ve definitely enjoyed working with iRules LX over the past two weeks and I’ll share my projects with you once they’re publically available.