How to Monitor a TLS 1.0 Application

Overview

With the slew of SSL and TLS based vulnerabilities over the last two years F5 administrators have been forced to become more cognizant of the encryption standards used in their environment.  While disabling SSLv3 and TLSv1 is a critical step in securing your infrastructure you may find yourself stuck with applications servers that only support TLSv1 or weaker protocols.

HTTPS monitors in TMOS always default to the latest protocol version supported by OpenSSL but when you upgrade to 11.5.0 and higher the HTTPS monitors will not utilize SSLv3 or TLSv1.  If you’re stuck with application servers that require TLSv1 this puts you in a sticky situation.  Now I don’t know the dynamics of your organization and while upgrading the application server to support a more secure protocol is the ideal way to solve this issue it might not be feasible you for.  For those customers the process below outlines the process of creating an external monitor that uses TLSv1 to perform health checks.

Note: this article is based upon TMOS 11.5 and higher.  If you’re running another version this process would still work but the step by step instruction may differ for your configuration.

Create an Monitor Script

Grab the HTTP monitor script from CodeShare on DevCentral and modify the curl statement on line 48 from:

curl -fNs http://${IP}:${PORT}${URI} | grep -i "${RECV}" 2>&1 > /dev/null

to

curl -NksSf --tlsv1 https://${IP}:${PORT}${URI} | grep -i "${RECV}" 2>&1 > /dev/null

Save the modified script to your desktop and upload the script to your BIG-IP through the File Management options under the System menu.

Note: I named my monitor tlsv1_monitor which will be referenced throughout this document.

Create an External Monitor

Now we can create an external monitor based upon our monitor script.

  1. In the left hand menu click Local Traffic -> Monitors
  2. Click the create button in the top right corner
  3. Name your monitor (remember you can’t use spaces in the name)
  4. Select External for the type
  5. Select your external monitor for the External Program
  6. Click Finished

external_monitor

Test Your External Monitor

Now that you’ve created an external monitor based upon your monitor script we need to test it. You could go for broke and assign it to a pool but I prefer to know things are working as intended and not because i goofed up somewhere!

When you upload your monitor script TMOS stores it in the filestore.  So to test this script we’ll need to SSH into the BIG-IP and access the BASH console.

Initial Test

Note: TMOS adds a unique identifier to the script name. So your script name will be different than the example below.  You’ll also need to enter your own IP address and port for the two script arguments.

  1. cd /config/filestore/files_d/Common_d/external_monitor_d
  2. ./\:Common\:tlsv1_monitor_386911_1 10.128.20.11 443

If everything works the script should return UP.

TCPDump

Now that our script is working we need to verify that it’s actually using TLSv1.  To determine this we’ll take a tcpdump while issuing the command above and then verify the protocol with the ssldump command.

Note: You’ll need to modify the IP address and TCP port to match your environment.

Type the following commands on the CLI:

  1. tcpdump -vvv -s0 -nni External -w /var/tmp/tlsv1.cap host 10.128.20.11 and port 443
  2. ctrl+z
  3. bg
  4. ./\:Common\:tlsv1_monitor_386911_1 10.128.20.11 443
  5. fg
  6. ctrl+c

This will start a TCP dump and then send the process to the background (ctrl+z pauses and bg sends it to the background). Once the monitor command is executed the fg command will bring the tcpdump process back to the forground and ctrl+c will terminate the tcpdump.

Note: The tcpdump command will still display information to the CLI so you may have a hard time seeing what you’re typing.  My recommendation is to paste the monitor script command.

SSLDump

Once you have the tcpdump we can use ssldump to view the protocol used between the F5 and your server.  Issue the following commands at your CLI:

  1. ssldump -H -nr /var/tmp/tlsv1.cap | grep Version

This will display the SSL record messages and search for the Version used.  In our case we’re looking for Version 3.1:

ssldump_tlsv1

Assign your External Monitor

Now that everything is working as intended you can assign the new external monitor to your application pool.  If you don’t know how to do this I highly recommend you checkout the DevCentral Whiteboard Wednesday session on monitors.

BIG-IP Troubleshooting 101

When you work with any technology there reaches a point where the “it’s a black box” approach is no longer valid and you have to dig in a little deeper and understand how the product works. With F5 BIG-IP this means understanding how traffic flows through the appliance and how to monitor and watch it.

TMOS – Client and Server Traffic
The F5 BIG-IP Traffic Management Operating System (TMOS) is a dual-stack full proxy which means the client terminates their TCP connection with the BIG-IP and the BIG-IP then makes a new TCP connection to the backend server. So as far as the client is concerned the F5 is the server and as far as the backend server is concerned the F5 is the client. So when you are troubleshooting it is important to understand there will always be client-side traffic and server-side traffic. The names are pretty self explaining but can be misleading in the troubleshooting process. What I mean by this is you’ll more than likely need to look at both client and server side traffic to gain a better understanding in how the application behaves/operated.

LTM Monitors
LTM monitors are used to evaluate the health of a pool member or a node. They typically run at a set interval (15 seconds by default) and will mark a pool member or node down after 3 failed intervals (this setting is configurable). If you are unsure why a monitor is failing the first place to look is the Local Traffic Manager logs. These logs are accessible via the GUI (System -> Logging) or the CLI (less /var/log/ltm) and will give you some basic information such as:
– when the resource was makes offline
– if the resource is flapping

The LTM log will not however tell you why the monitor failed. To determine this you typically need to run a synthetic request using a CLI based tool such as curl or the TMSH monitor test command. Please note: if you can not access the application using these steps the F5 is probably not at fault – no matter how much the application owner swears everything works on the server 🙂

If this is a new application deployment I typically see monitor failures resulting from:
– Networking/firewall issues
– does the BIG-IP have a Self-IP on the sames network as the server?
– If not does the BIG-IP have a route to that network?
– Application issues
– Is the web server using name bases virtual directories? If so, what HTTP host header is it expecting?
– does the host OS have a firewall installed/configured?

If this is an existing application you need to answer the tried and true “what changed” question. In these scenarios I typically work my way down this checklist to see where the problem lies:
– can I ping the server?
– can I telnet to the port?
– can I run a synthetic request using a CLI tool like curl?
– does the web server respond with the correct website (if you’re using customized HTTP monitors – which I highly recommend)

These steps will usually lead me to the underlying issue or point me to the team who managed the device/server with the issue.

I will follow up with an additional post that provide examples of synthetic transactions as well as a 201 post regarding APM troubleshooting and ASM troubleshooting.

Configuring and Testing an External Monitor

In follow-up to my post on Monitoring Access Policy Manager this post will walk you through how to assign the monitor to a GTM pool and simulate a down status to ensure GTM is configured correctly. If you do not know how to import the external monitor into GTM please reference SOL13423.

Someone asked what my external monitor configuration looked like (which is pretty basic) so I’ve included a screenshot below.

Example monitor for APM
Example monitor for APM

To add the external monitor to an existing GTM pool navigate to DNS -> GSLB -> Pools and select the application pool you’re looking to monitor. Under the Configuration tab move the apm_monitor from Available to Active. If you refresh the page after a few seconds you should see the Availability go green.

Adding monitor to GTM pool.
Adding monitor to GTM pool.

To simulate a down status you can update the external monitor to look for F6 versus F5.

  1. Navigate to System -> File Management -> External Monitor Program File List and select your APM monitor.
  2. Scroll towards the bottom and change the string “This product is licensed from F5 Networks” to “This product is licensed from F6 Networks”.
  3. Navigate to DNS -> GSLB -> Pools
  4. Verify that your APM pool is now red
Shows a GTM Pool as down
Shows a GTM Pool as down

Be sure to run through steps 1 through 4 again and set the monitor back to F5 otherwise the application will not work 🙂

Monitoring Access Policy Manager

I have a customer who recently needed to monitor the APM logon page but noticed the default HTTP monitor was eating up APM access sessions. To address this issue we need the HTTP monitor to retain the MRHSession cookie as it follows the redirect from / to /my.policy. After the monitor determines the health of the APM logon page it needs to also access the logout page to delete it’s current access session.

To accomplish this we can use the cookie jar option in curl:

  • -c create the cookie jar
  • -b uses an existing cookie jar

We also tell curl to follow redirects with the -L argument.

In the example code below the monitor is looking for the F5 licensing statement at the bottom of the APM logon page. If you’ve customized the APM logon page this monitor can easily be modified to fit your specific requirements. You can also download a PDF version of the file here.

Update: I’ve added a new post on Configuring and Testing an External Monitor if you need help adding this monitor to GTM.

#!/bin/sh
#
# (c) Copyright 1996-2014 F5 Networks, Inc.
#
# This software is confidential and may contain trade secrets that are the
# property of F5 Networks, Inc.  No part of the software may be disclosed
# to other parties without the express written consent of F5 Networks, Inc.
# It is against the law to copy the software.  No part of the software may
# be reproduced, transmitted, or distributed in any form or by any means,
# electronic or mechanical, including photocopying, recording, or information
# storage and retrieval systems, for any purpose without the express written
# permission of F5 Networks, Inc.  Our services are only available for legal
# users of the program, for instance in the event that we extend our services
# by offering the updating of files via the Internet.
#
# @(#) $Id: http_monitor_cURL+GET,v 1.0 2007/06/28 16:10:15 deb Exp $
# (based on sample_monitor,v 1.3 2005/02/04 18:47:17 saxon)
#
# these arguments supplied automatically for all external monitors:
# $1 = IP (IPv6 notation. IPv4 addresses are passed in the form 
#                         ::ffff:w.x.y.z
#                         where "w.x.y.z" is the IPv4 address)
# $2 = port (decimal, host byte order)
#

# remove IPv6/IPv4 compatibility prefix (LTM passes addresses in IPv6 format)
IP=`echo ${1} | sed 's/::ffff://'`
PORT=${2}

PIDFILE="/var/run/`basename ${0}`.${IP}_${PORT}.pid"
# kill of the last instance of this monitor if hung and log current pid
if [ -f $PIDFILE ]
then
   echo "EAV exceeded runtime needed to kill ${IP}:${PORT}" | logger -p local0.error
   kill -9 `cat $PIDFILE` > /dev/null 2>&1
fi
echo "$$" > $PIDFILE

# send request & check for expected response
curl -Lsk https://${IP}:${PORT} -c cookiejar.txt | grep "This product is licensed from F5 Networks" > /dev/null

# mark node UP if expected response was received
if [ $? -eq 0 ]
then
    rm -f $PIDFILE
    echo "UP"
else
    rm -f $PIDFILE
fi

# delete APM session
curl -Lsk https://${IP}:${PORT}/my.logout.php3 -b cookiejar.txt > /dev/null
exit

Continue reading