Integrate new firewalls into Panorama via CLI

I recently switched jobs and I am excited to announce that I am working with Palo Alto firewalls again. I am not gonna lie, I am beyond excited seeing PANs again. There is just something about them that I like.

As much as I love the UI it is somewhat annoying to integrate a new Palo Alto in Panorama. The reason is that PANs come with a standard configuration which has a standard security policy, virtual-wires and security zones. Now when you want to add it to Panorama this will cause some problems because it interferes with the config you want to push from Panorama. And after all, you want to get rid of any configuration that you may not use. Now you must click through the WebUI and delete the standard configuration in the correct order to avoid dependency errors. Just to hit “commit” and get another error thrown at you.

Hence I referred to the CLI and looked up all commands to wipe and stage a new device for our environment. I basically wanted to have a copy/paste deployment procedure to save time and drive standardization. The Palo Alto CLI is very capable and I was pleasantly surprised about the awesome readability of the commands. Below you will find my staging scripts for the local device and Panorama.

 

Example deployment “script” for PA firewall

 

For easier deployment, I build a small excel file that will build all the commands to

• wipe the standard configuration
• setup the management interface
• set the panorama server
• setup the interfaces so that interface configurations can be pushed via Panorama templates
• setup HA
• update the license
• check and install dynamic updates
• check and install software updates

Basic setup of 3 interfaces as layer 3 and port 7 and 8 for HA1/2:

 

# Setup MGMT interface

set deviceconfig system ip-address <MGMT-IP> netmask <netmask in X.X.X.X> default-gateway <default-gw-IP> dns-setting servers primary <primary-DNS>

# Setup Panorama server

set deviceconfig system panorama-server <IP/DNS of your Panorama Server>

# Delete standard config

delete network virtual-wire default-vwire

delete rulebase security rules rule1

delete zone trust

delete zone untrust

delete network interface ethernet ethernet1/1

delete network interface ethernet ethernet1/2

delete network interface ethernet ethernet1/3

delete network interface ethernet ethernet1/4

delete network interface ethernet ethernet1/5

delete network interface ethernet ethernet1/6

delete network interface ethernet ethernet1/7

delete network interface ethernet ethernet1/8

# Assign minimum configuration to network interfaces to be able to push Panorama templates

set network interface ethernet ethernet1/7 ha

set network interface ethernet ethernet1/7 comment HA1

set network interface ethernet ethernet1/8 ha

set network interface ethernet ethernet1/8 comment HA2

set network interface ethernet ethernet1/1 layer3 ip X.X.X.X/XX

set network interface ethernet ethernet1/1 comment <interface1comment>

set network interface ethernet ethernet1/2 layer3 ip X.X.X.X/XX

set network interface ethernet ethernet1/2 comment <interface2comment>

set network interface ethernet ethernet1/3 layer3 ip X.X.X.X/XX

set network interface ethernet ethernet1/3 comment <interface3comment>

set network virtual-router default interface ethernet1/1

set network virtual-router default interface ethernet1/2

set network virtual-router default interface ethernet1/3

# Setup hostname

set deviceconfig system hostname <hostname>

# Setup HA

set deviceconfig high-availability enabled yes

set deviceconfig high-availability group 1

set deviceconfig high-availability group 1 peer-ip X.X.X.X peer-ip-backup X.X.X.X

set deviceconfig high-availability interface ha1 port ethernet1/7

set deviceconfig high-availability interface ha1 ip-address X.X.X.X netmask X.X.X.X

set deviceconfig high-availability interface ha1-backup port management

set deviceconfig high-availability interface ha2 port ethernet1/8

set deviceconfig high-availability interface ha2 ip-address X.X.X.X netmask X.X.X.X

commit

# Update license on local firewall

request license fetch

# Check for App updates and install latest

request content upgrade check

request content upgrade download latest

request content upgrade install sync-to-peer yes version latest

# Check for AV updates and install latest

request anti-virus upgrade check

request anti-virus upgrade download latest

request anti-virus upgrade install sync-to-peer yes version latest

 

Example deployment “script” for Panorama

 

This will do the following on the Panorama Server:

• Add serial number to Managed Devices
• Clone a template
• Create a template stack
• Add templates to template stack
• Add new device to template stack
• Add new device to log-collector

 

set mgt-config devices <serial-nr>

# copy a template

copy template <template1> to <template2>

# Create template stack, add templates and add new device

set template-stack <template-stack>

set template-stack <template-stack> templates <template1>

set template-stack <template-stack> templates <template2>

set template-stack <template-stack> devices <serial-nr>

# Add new device to log collector

set log-collector-group default logfwd-setting devices <serial-nr> collectors <serial-nr-of-collector>

 

Hit commit and push the template to device (don’t forget to force values on first push).

No rocket science here, it is all very simple CLI commands, but maybe I will save someone some time to look up all these commands and it will for sure save you some time clicking through the WebUI and hitting dependency errors.

I used the commands above together with an excel file which will automatically put in IPs and serial numbers collected from a form. All commands are tested with version PAN-OS 8.0.4

 

How to replace blank/space with line break (Alt+Enter) in Excel

Ever wondered how to replace blank or spaces with a line break in Microsoft Excel? it is a lifesaver when dealing with firewall policy sheets 🙂

Push Ctrl + H (for replace) and find what:” “(just a space).
In the replace with field you enter 0010 while holding Alt key (so ALT + 0010). You will now notice the cursor jumping into a second line (hence, there is only a small piece visible – it is looking like a flashing dot). Now hit Replace or Replace all. Et voila!

Of course this also works the other way around.

before:

after:

Short n quick: DIY BayTech RPC serial adapter to use with standard ethernet cables (RJ45)

How to build a BachTech RPC serial console adapters which can be used with standard RJ45 (Cat5/Cat6/..) ethernet cables:

Serial RS-232 Pinout (DE-9 female):
______________
\ 5  4  3  2  1 /
\ 9  8  7  6 /

RS-232 to RJ45 (T-568A) Pins

RS-232 Pin#	RS-232 Color	RJ45 PIN#	RJ45 Color
5	Orange	7	White Brown
4	White	1	White green
3	Green	4	Blue
2	Red	5	White blue
1	Black	6	Orange
9	None	None	None
8	Blue	8	Brown
7	Yellow	3	White orange
6	Brown	2	green

Accessing your modem from OpenWRT Router

Very useful if you want to check for errors on your Cable modem Accessing your modem from OpenWRT Router.

Unfortunately, you don’t have fulll access to your modem on AT&T, TWC or Comcast. Hence, you are not able to do SNMP monitoring 😦

Short n quick: Reset Alteon AD3 load balancer to factory default.

Case: Old load balancer, unknown IP, no serial/console access

Don’t even try to get the serial port up – it’s impossible.

• Connect a computer with crossover cable to one of the ports.
• Check ARP requests with Wireshark
• Telnet into IP
• Enter “boot” (boot menu)
• Enter “conf” (config block to use next boot)
• Enter “factory” (in order to boot factory defaults)
• Power off
• Power on

Good Luck!

Problem solved: Monitoring Kemp Loadbalancers with Check_MK (kemplb_real_servers, kemplb_rsvs and kemp_virtual_server)

About two weeks ago I found a plug-in for monitoring Kemp loadbalancers on Check_MK Exchange (go here: http://exchange.check-mk.org/index.php?option=com_remository&Itemid=59&func=fileinfo&id=135).  Until today it was working absolutely flawless on two loadbalancers and was a really big help on monitoring all the services (including connection and pool data).

Unfortunately, it would not recognize the services on a third Kemp loadbalancer. That´s when I went CLI and checked the snmp_scan_function. (Note: The plug-in consist of three different checks: kemplb_real_servers, kemplb_rsvs and kemp_virtual_server)

Turns out the script is checking for OID .1.3.6.1.4.1.12196.12.8.1.2.1 in order to inventory the servers. So I decided to compare the output from the working and faulty Kemp LBs when doing a snmpget for this OID:

Turns out that the working LB is giving back a value, while the other Loadbalancer does not. This is quite interesting since both LBs are on same firmware version and platform (virtual machines).

Then I used a MIB-Browser to investigate which values in this OID tree would be available.

As we can see from the screenshot there is no OID .1.3.6.1.4.1.12196.12.8.1.2.1. Instead there are a couple other OIDS like .1.3.6.1.4.1.12196.12.8.1.2.4. and .18 etc in the subtree.

While checking the Kemp LB MIBs (accessible here: http://kemptechnologies.com/files/downloads/documentation/7.0/LM_mibs.zip) I found out that OID .1.3.6.1.4.1.12196.12.8 is “a table containing Totals for Real Server (RS) specific information.”

So it seems that there is no Real Server “1” on the faulty LB. Thereforet Check_MK is not doing an inventory (even though it would find services on the LB).

So here is a solution that worked for me: I changed the last lines of the three check scripts as following:

So basically it is checking OID .1.3.6.1.2.1.1.2.0 (“The vendor’s authoritative identification of the network management subsystem contained in the entity”)  and validates if the returned value starts with .1.3.6.1.4.1.12196 (because in that case it is a Kemp LB and we want the plug-in to start the inventory).

Feedback appreciated.

Update: Check PAN Firewall´s Sessions Counters with SNMP

About a year ago I published my first ever developed plug-in for check_mk (https://sitweak.wordpress.com/2012/08/30/snmp-based-check_mk-plug-in-for-palo-alto-firewalls/). For all who missed out on that milestone in my programming carrer: It´s doing not more than fetching some sessions counters with the help of SNMP.

Even though I realized that there is not a high demand for such an plug-in (until now it has one rating on the check_mk Exchange – haha) I am still very proud that there is a piece of self-made code which is actually working. J That’s why I decided to give a small update.

Some hours ago I uploaded and version 1.0. Here´s a Changelog:

Plug-In

o   Two more SNMP values are being fetched
– 1.3.6.1.4.1.25461.2.1.2.3.7.0 Total number of active SSL proxy sessions
– 1.3.6.1.4.1.25461.2.1.2.3.8.0 Total number of active SSL proxy sessions.

o   Code updated
– Added a snmp_scan_function
– Added declaration according to new Check_MK API (https://mathias-kettner.de/checkmk_devel_newapi.html)

o   Added warning and critical levels (Service State is based on SNMP value of sessions utilization)
– Factory default: 80% warning level and 90% critical level. You can define your own values by editing ~/local/share/check_mk/checks/paloalto_sessions.

Perfdata
o Added value ‘proxied ssl sessions’ to perfdata

PNP-Template
o   Added graph for value ‘proxied ssl sessions’
o   Cleaned PNP-Template

Plug-In Output
o   Added value for proxied SSL sessions
o   Added value for utilization of SSL proxy

 

This is what it looks like:

Tested on Check_MK 1.2.4

 

I´m looking forward to any feedback here, on twitter or check_mk Exchange!

 

Importing Palo Alto Firewall´s Policies into Microsoft Excel

 

Last week I started my second approach on importing the Policies (Ruleset) from our PaloAlto firewall into Microsoft Excel. I really love the way PaloAlto designed the web-interface on their Pan-OS: It has a cross-browser compatibility and works flawless on any screen resolution. Unfortunately, it is missing an option to export the policies into any standardized form, which would give me the possibility to keep a record of the ruleset outside the web-interface.

However, this was something I wanted to do for quite a while. More and more often we have the problem that system administrators complain about a missing transparency of the ruleset. This problem usually comes up when they are deploying a new service, and there is exactly one policy missing in order to complete the deployment.

One solution for that problem is granting a read-only access on the firewall. Downside: They have to be taught on how to use the web-interface and how to correctly read the policies. More important the will have access to other areas of the configuration, which may be considered sensitive or conflict with the goal of protecting sensitive data.

Presenting the ruleset in an Excel sheet is a good solution in order to address these problems: Everyone is able to open and read an Excel-sheet (Freeware viewers are also available). As a plus you can hide or exclude any information that is not necessary and likely people will know how to find the information they are looking for. On the other hand you have endless options on how to add additional information with the help of texts or conditional formatting.

Anyhow, enough for now – let´s start with the solution.

A while ago I already found two articles in the PaloAlto forums on how to import the Policies into Microsoft Excel. One is here: https://live.paloaltonetworks.com/docs/DOC-1617, the other one you will find here: https://live.paloaltonetworks.com/docs/DOC-5754. Unfortunately, both solutions didn´t work for me.

In fact, I got the following result using the tutorials:

As you can see, there is a new column for every source and target address. The reason is, that the every address-object is enclosed by <member> tags, which leads to a misinterpretation in the structure of the XML-file.

The advice from my co-worker was to use formulas in order to fix the layout. This didn´t seem to be the perfect solution for me, since I wanted a procedure which could be eas

ily reproduced by any other firewall-administrator.

After checking a few of the comments under the above stated links I found a comment mentioning a solution based on a linux bash command. After analyzing the command line I could adopt the procedure to a windows environment. Here is what you have to do (tested on Pan-OS 5.X with PAN-5050 and Google Chrome browser):

1)      Export your current running configuration:  In the web-interface you go to Device -> Setup -> Operations -> Export named configuration snapshot.

2)      Open the configuration snapshot with a compatible text-editor (as Notepad++ for example).

3)      Search for string <security> (press Strg+H in most ext-editors) and deleted everything before the tag.

4)      Search for string </security> (press Strg+H in most ext-editors) and deleted everything after the tag. You now should have everything between <security> and </security>. Save the file (for security ;))

5)      Delete all tags <member> and </member>: Press Strg+H again for search and replace. Search for string <member> and replace it with nothing (delete it!). Do the same for the string </member>.

6)      Save the file as an XML document.

7)      Open Excel and import the XML file by clicking: Data -> Import –> other Sources –> XML-Dataimport and choose the XML file.

8)      As a result you should see your complete ruleset, where every rule is in exactly one row. However, there were slight layout problems caused by blanks in front of the address objects:

You can easily fix that by using the replace-function again, and replace the blanks with nothing:

As a side-note:  You can use that procedure also for importing the address-objects of your PAN-Firewall. In that case you have to import everything between <address> and </address> tags.

Side-Note 2: For additional XML settings you may want to activate the developers tab in Excel. It offers you additional features when working with XML data.

Palo Alto Firewall and ESX Session Time-Outs (Management)

Did you ever wonder how PAN firewalls are calculating session time-outs? I did, and it took me a while to find out about it. As you can see from my last post, it wasn´t the first time I was having problems with session timeouts and I´m pretty sure that there´s a ton of people having similar problems out there – so I want to share our results.

Scenario

There is an ESX-server located in the Inside Zone of our firewall (let´s call it ESX-Management). It is opening SSL management connections to remote ESX-hosts located in the Outside zone (ESX-remote). The session timeout value was set to 4 hours. Unfortunately these sessions were running into timeouts because the PAN firewall was dropping them (we could verify that by checking the monitor tab and seeing the timeout counter running from 14400 to 0). On the other hand we could track a constant flow of heartbeat packets between the hosts. The ESX-management host kept sending heartbeat packages every 5-10 minutes in average.

Result and Issue

The result of that behavior was that the ESX-remote host ran out of sockets (the limit of ESX management connections is limited to a maximum of 321 connections), and therefore wasn´t manageable anymore. (HINT: restart esx management agent to reset all connections ‘/etc/init.d/mgmt-vmware restart’ )

We were able to identify the core problem very easily by entering the ‘netstat –an’ command in linux. The outpot showed us hundreds of established connections. However, when doing the same command on the ESX-management server it only showed us 3-4 active connections to the remote host. After that we checked the session in the PAN traffic monitor and found out that they were marked ‘ended’.

Conclusion: The ESX-management host was recognizing ended sessions, while the ESX-remote host kept the connections in established state-

Why is the PAN dropping sessions because of a timeout which have a constant flow of (heartbeat) packages?

The problem is related to the fact that PAN firewalls are doing session offloading, and data is processed by the data plane (see additional Info in PAN community: https://live.paloaltonetworks.com/docs/DOC-3950 or Wikipedia: http://en.wikipedia.org/wiki/SSL_acceleration). In case of session offloading  the PAN firewall needs a flow of 16 packets (unidirectional, so on one direction of the data-flow) in order to refresh the timeout timer. These values are for TCP sessions – as far as my information goes, UDP sessions need half the amount of packages (so 8 packages in one direction).

Conclusion and Fix

The conclusion you can make from the described the behavior is the following:

• The application heartbeat value has to be at least 16 times shorter than the session timeout value set on the PAN firewall.

Or the other way around:

• The application timeout value configured on the PAN firewall has to be changed to a 16 times higher value the heartbeat setting of the application. (You can change timeout values for every application in the objects tab).

For example: In our case the average heartbeat interval was 10 minutes. 16 multiplied by 10 = 160 Minutes. So we would have to set the session timeout value for the application ‘SSL’ to 9600 seconds (160 minutes) – or for a predefined application timeout of 1200 seconds, we would have to configure a heartbeat interval  of at least 75 seconds (1200 divided by 16 = 75).

As mentioned, the problem is related to the session offloading done by PAN firewalls. Taking this into concern, there is a second way to avoid session timeouts: Turn off session offloading. This can only be done via command-line.

To turn off hardware offload temporarily you can use the following commands (in PAN configure mode):
#set session offload no
or permanently with
#set deviceconfig setting session offload no (followed by commit).

As a result every heartbeat package will refresh the TTL timer, since the packages are not directly processed by the data plane. There´s no more need to send 16 packages in order to refresh the timer. In this case you only have to make sure, that the heartbeat interval from the application is smaller than the configured value on the PAN firewall. However there´s a big downside: The utilization of the data plane increases. Somewhere in the forums I read that disabling session offloading can decrease the total throughput by 15%.

PAN (Palo Alto) firewalls and flapping Cisco WLC data/control paths

Just yesterday we migrated to a (Palo Alto Networks) PAN-5050 firewall in active-passive configuration. After that we experienced problems with flapping control and data paths between our foreign and anchor WLCs (Wireless LAN Controller). In this article I will describe a possible solution for flapping data/control paths in a Palo Alto / Checkpoint firewall environment. Scroll down to ‘Solution’ to skip preamble.

Our set up: We have four Cisco 4402 Series WLCs which are located in remote affiliates. These ‘foreign WLCs’ set up etherchannels to an ‘anchor WLC’, which is located in one of our DMZs.

After changing our firewall from Cisco to PAN we experienced flapping data and control paths. Investigation led to assumption that the direction of initialization was somehow related to the flapping. Cisco´s documentation says that the WLC with the lower MAC-Address is initializing the etherchannel. This is leading to the first conclusion:

• It´s important to know which side is initiating the etherchannel. As a result you may want to implement bidirectional rules for easier handling. Otherwise make sure that your firewall rules match the traffic flow.

The goal of this first step was to make sure that no more packages are dropped. As with PAN-OS 5.0.3 and AppVer 365-1733 (03/26/13) the Application were detected correctly (etherip and cisco-wlc-mobility). Only one thing seems a little bit weird: The traffic log says that etherip is using Port 0 (I´m not sure about that one). However, we were still experiencing flapping paths.

In the second step we changed the values for the timeouts on application level (on the PAN you can set customized values for TCP/UDP timeouts under Objects > Applications tab). Unfortunately the paths still didn´t stop flapping.

SOLUTION: After that we changed the default values for the session timeouts (you´ll find them under Device > Setup > Session Tab in PAN-OS) and rebooted the foreign as well as the Anchor WLCs and all Data and Control Paths seem to work fine. In the picture you will find the values which worked for me.

As to my surprise this behavior/problem doesn´t seem to be Palo Alto specific. In fact I found a topic on the Cisco Support forums where someone is having the same problems with a Checkpoint firewall. So this fix might be a general fix for problems with WLC data or control paths.