I ran into a fun error today while trying to commit a configuration on our Palo Alto Firewall core.  No matter what I tried, I could not get a commit to take.  Keep in mind, we’ve run into the “Management server failed to send ID request to client device” error about 5 or 6 times this week.  Every time, however, a delete of the suspect policy has allowed a commit.  We could then drop the exact policy back in and commit the configuration no problem.  This time, however, I had no such luck.  After a support call and some additional troubleshooting, I discovered that this is a bug in the current 4.0.2 release.  However, two things did help this commit to go through and get us back up and running:

debug software restart management-server

debug software restart device-server

After restarting both servers, the commit went through without issue.  I’ll definitely be adding this one to my short list of things to try the next time this pops up.

Palo Alto support engineer highly recommends moving to 4.0.3 when it is release mid June.  So far, with my experience on 4.0.2, I would have to agree.

 

My team has been very busy since implementing the new Palo Alto Networks Firewalls in our core.  The short summary is that the firewalls are in and we have had no show-stopping production traffic outages.  We also took the opportunity to remove some legacy policies related to traffic between sites and that has caused a significant number of policy additions that were necessary in order to have a truly locked down WAN infrastructure.  That has been the bulk of our time over the last week.

In addition, if you think your developers really know what “applications” their apps use, you may want to think again.  We had case after case of little applications pop up as denied in the Palo Alto that they were not aware needed to be defined…

I’ll have a full recap this week along with what we learned…

 

Palo Alto Network’s support team generously offered to remote in and examine our pending go-live configuration.  Additionally, they walked us through each section and went over what we had in there.  We looked at a sample of the translation and security policies, as well as applications, content filtering, vulnerability scanning, file blocking, data filtering, and other major areas in the firewalls.  Everything looked great, which we expected, but it was great to have a vendor offer a quick look through.  They also opened a support ticket in preparation for our go-live and passed along the information to their support team.  In all they spent over an hour going through everything and giving us a thumbs up.  They did all of this without us paying an extra dime for any of it.  Keep in mind we didn’t pay for any services either… I must say that I am impressed.

The customer service experience is definitely different with Palo Alto Networks so far.  They are still going through some growing pains with support, but I am genuinely impressed by their willingness to keep us informed and ensure that we are successful.

 

 

Policies… Policies… Policies…

That has been my life the last few weeks.  I have been working with one of my Sr. Engineers on migrating all of our policy base over.  Crunch time is here and we are prepping for go-live this weekend.  We’ve reviewed every policy individually.  On each, we have identified the application if available, the purpose, and tried to restrict things more wherever possible.  We are loving the new Palo Alto firewalls.  Let’s hope we’ll be saying the same thing on Monday…

The reality is that my team has come a long way over the last 10 years.  We may be an oddity in the technology world in that I have managed to keep the same team together for a decade.  We lost one of our team members to cancer last year, but other than that the team has been static and has grown together.  This will be our third core firewall replacement in a decade.  When my team took over security for the company, it was being outsourced to an individual in Florida for $3k a month.  We were on Checkpoint on AIX at the time.  There is nothing like having your firewall several software releases behind with no updates in site due to the operating system.  We never felt like Checkpoint did a good job supporting the AIX platform, so we decided to migrate away from them.  At the time, we were a huge Cisco shop – everything was Cisco.  If Cisco had a solution for it, you can bet we were using it in our world.  So, the Pix line was a natural fit for us.  We went through a similar migration, only we had a lot less information about our environment than we did now.  Because of that, it was a disaster.  We spent the next two weeks cleaning up problems.

The second replacement was a transition from Cisco Pix to Juniper Netscreen.  We were having a host of VPN problems at the time that were due to Pix issues holding the VPN tunnels open after a hiccup in the network connection at the remote site.  Cisco could not provide a reasonable fix, so we evaluated the Netscreen platform.  We really liked the interface and the ease of getting things done, so we began an orderly migration to Juniper.  At the same time, we were replacing branch office routers so we felt it was an opportune time to firewall our branch offices.  This was unusual at the time, but we wanted to improve our security on our inside network as well.  After the Juniper core firewall replacement, it took us a couple of days to get everything straight.  This was largely due to the fact that our Pix firewalls had the standard any any any outbound policy in effect because of the security levels of our interfaces.  So, we took a big step forward in building up our rule-set from a deny all policy.

Now we approach our third and likely most significant core firewall replacement in my 15 year technology career.  It is significant largely because we are moving to an application aware firewall.  We have decided rather than just migrating our policy base over and stick with the traditional source ip / destination ip / port model, we would dive in and embrace application awareness.  Additionally, we would utilize A/D groups and user accounts where possible to further isolate the policy base and present a more secure rule set.  As I reflect on the past decade I realize that this migration has been more time-consuming and significantly more complex than ever before.  As we increase the awareness of our firewall, we also assume more responsibility for the traffic.  Can I be sure that our firewalls are going to be up to the task?  Am I certain that the traffic that traverses the firewall will be properly identified and thus hit the correct policy?  Do we have unusual traffic patterns that are going to trigger my drop rules in my scanning engine?

At any rate, we are as prepared as we can be at this point.  We have thought and re-thought.  We have scoured through our policy base line by line.  We have cleaned up our rules and identified more secure ways of doing things.  I have no doubt that we have created a policy base that is as secure as we can make it within our business requirements.  The question is, will these Palo Alto firewalls really do the job we have come to expect?  We will find out the answer on Sunday as we begin our transition into the world of Next Generation Firewalls.  We fully believe in the platform we are migrating to.  We have proven it in the field in smaller deployments.  However, there is always some apprehension in my mind when we begin to pass our core data center traffic through a new box…  In the end, it is a new world for me and my team.  We will have visibility like we have never had before.  We will have significantly increased our security posture and ability to report on individual user traffic rather than hunting an IP address down.  It will help us every day in helping to protect our organization.  And it is to that new world that we travel to this weekend.  Here’s hoping the journey will be pleasant…

 

It occurred to me over the last few days how set I have been on the whole source IP / destination IP / port configuration.  My team is in the process of migrating our rule base from a Juniper ISG firewall to a Palo Alto Network firewall.  The PA-4020 HA pair we are migrating to is considered a “Next Generation Firewall.”  What that basically means is there is significantly more features built into the firewall that would normally be separated out into separate boxes.  It also means that this firewall is fully application aware.  As of the last update, these firewalls are fully aware of 1,249 separate applications.

That creates some interesting problems for a team migrating policies from the old source / destination / port rule base.  For example, if I were to permit port 80 in my Juniper firewall, applications like Facebook for our marketing team would go through without problem.  I might use a web filtering engine to block that site in order to prevent other users from going there, but essentially the port was opened outbound.  Now, with an application aware firewall, opening up the “Web Browsing” application isn’t good enough.  In fact, other applications that utilize web browsing would not be permitted through the policy.  That has created an interesting delve into what applications we really want to allow over common ports like 80.  This is both a very good and a very bad thing.  It is great that we can now tie into an Active Directory group and tell our firewall that our Marketing team can post to Facebook, but cannot play games.  It is bad in that it has created a ton of work for us in migrating our policies from a typical firewall.  Based on our existing work so far, this has increased our workload in migration by at least twice.  We have had to review each policy and determine what we are really saying.  Does an application definition exist for this?  If so, what other application dependencies exist and how does this affect our policy?  What extra rules need to be added now that applications that were permitted through the standard port 80 will be blocked in our shiny new firewall?

We have answered these questions in large part by identifying traffic patterns while running a PA-4020 in monitor mode.  We have had to isolate application traffic and create new policies for our Go-live.  We will soon see if all of our preparation to move to these “Next Generation Firewalls” will pay off or if we still have some work to do.  In fact, we identified over 90 applications that we want to allow that would have been blocked, even though they are “Web Browsing” type applications.  The transition from the old world hasn’t been easy.  It has opened up our eyes to a host of new problems that we now have to deal with.  It would be easier if we had a clearer mandate from our businesses other than “Secure our world without blocking anything we need.  Oh, by the way, we aren’t really sure what we need to conduct business.”  However, that is the reality that many companies deal with.  Our core firewall replacement will occur this Sunday, so we will have a chance to see just how well we have prognosticated for our businesses…

 

Cisco Corner was created to provide networking professionals a place to get information, tips, tricks, and a wealth of up to date knowledge about networking topics.  My plan is to include the vendors and products that I work with on a day-to-day basis.  These include Cisco, Juniper, Palo Alto Networks, Aerohive, A10 Networks, and many others.  Please visit back regularly or subscribe.  I will be providing regular updates which will provide all of the little tips that I pick up as I manage a global network.

You can also visit the forum to ask questions or discuss networking topics with professionals around the world.

© 2012 Cisco Corner Suffusion theme by Sayontan Sinha