BIG-IP Troubleshooting 101

When you work with any technology there reaches a point where the “it’s a black box” approach is no longer valid and you have to dig in a little deeper and understand how the product works. With F5 BIG-IP this means understanding how traffic flows through the appliance and how to monitor and watch it.

TMOS – Client and Server Traffic
The F5 BIG-IP Traffic Management Operating System (TMOS) is a dual-stack full proxy which means the client terminates their TCP connection with the BIG-IP and the BIG-IP then makes a new TCP connection to the backend server. So as far as the client is concerned the F5 is the server and as far as the backend server is concerned the F5 is the client. So when you are troubleshooting it is important to understand there will always be client-side traffic and server-side traffic. The names are pretty self explaining but can be misleading in the troubleshooting process. What I mean by this is you’ll more than likely need to look at both client and server side traffic to gain a better understanding in how the application behaves/operated.

LTM Monitors
LTM monitors are used to evaluate the health of a pool member or a node. They typically run at a set interval (15 seconds by default) and will mark a pool member or node down after 3 failed intervals (this setting is configurable). If you are unsure why a monitor is failing the first place to look is the Local Traffic Manager logs. These logs are accessible via the GUI (System -> Logging) or the CLI (less /var/log/ltm) and will give you some basic information such as:
– when the resource was makes offline
– if the resource is flapping

The LTM log will not however tell you why the monitor failed. To determine this you typically need to run a synthetic request using a CLI based tool such as curl or the TMSH monitor test command. Please note: if you can not access the application using these steps the F5 is probably not at fault – no matter how much the application owner swears everything works on the server ūüôā

If this is a new application deployment I typically see monitor failures resulting from:
– Networking/firewall issues
– does the BIG-IP have a Self-IP on the sames network as the server?
– If not does the BIG-IP have a route to that network?
– Application issues
– Is the web server using name bases virtual directories? If so, what HTTP host header is it expecting?
– does the host OS have a firewall installed/configured?

If this is an existing application you need to answer the tried and true “what changed” question. In these scenarios I typically work my way down this checklist to see where the problem lies:
– can I ping the server?
– can I telnet to the port?
– can I run a synthetic request using a CLI tool like curl?
– does the web server respond with the correct website (if you’re using customized HTTP monitors – which I highly recommend)

These steps will usually lead me to the underlying issue or point me to the team who managed the device/server with the issue.

I will follow up with an additional post that provide examples of synthetic transactions as well as a 201 post regarding APM troubleshooting and ASM troubleshooting.

Clustered Multi-processing (CMP) versus Traditional Shared Memory Architecture

Over on DevCentral Robert Haynes has posted a great article outlining the advantages of F5’s clustered multiprocessing (CMP) architecture versus traditional shared memory architecture.

F5 clustered multiprocessing architecture
F5 clustered multiprocessing architecture

So why does this matter?  Because attacks today are designed to stress the performance of security devices and bring them to their knees.  If your architecture is designed correctly then it can utilize all of the available CPU and memory for your critical applications and not waste it on antiquated OS design issues.

SAML – Client versus Server Authentication with F5 APM

As organizations start to utilize Software as a Service (SaaS) the concern on how to authenticate users becomes a critical security issue. Many organizations look to federated authentication mechanisms, such as SAML, to help address this security risk. The benefits of using SAML are that user credentials are not replicated across each vendor cloud instance and that it greatly simplifies user management. However, not all applications will be migrated to the cloud due to limitations or security concerns. So when you start adopting SAML what happens to the applications that are left in your data centers.

Many of the customers I work with adopt SAML as an¬†authentication platform for both internal and cloud base solutions. The road block I typically run into here is that they try to implement SAML on all service – this is a “square¬†peg in a round¬†hole” approach. ¬†While many commercial applications and programming frameworks are started to support SAML the need to implement SAML on the backend servers is not always necessary.

In my view, SAML works great as a client-side authentication mechanism for self hosted applications but does not always “fit the bill” on a server-side authentication. When you use the F5 BIG-IP to provide Single Sign-on service to your internal applications¬†you have several server-side authentication options to choose from: HTTP basic, form, NTLM and Kerberos¬†to name¬†a few. ¬†In certain situations these authentication options¬†are much easier to maintain and implement across a large number of applications.

When using SAML for authentication the IdP provides an assertion to prove the user’s identity. ¬†In this situation the F5 does not have the user’s password so HTTP Basic, HTTP form and NTLM authenticate are not possible – this is ideal in security conscious environments because sensitive credentials¬†are not being passed around. However, the F5 can convert the SAML assertion into a token and use Kerberos Constrained Delegation to authenticate the user to the backend web server. ¬†A huge benefit to this model is that it scales very well with a large number of applications because¬†most¬†web servers supports Kerberos authentication and you do not have to create one-off APM SSO profiles for each application.

In conclusion, F5 BIG-IP provides organizations the ability to authenticate users securely to services whether in the cloud or in your data center.  The important thing to consider when designing your authentication solution is to examine all authentication option available to you and stop trying to squeeze the square peg into the round hole.