PDF -Diagnosing and Resolving Latch Contention on SQL Server - How to Identify and Troubleshoot Memory Leaks on IPSO
Wait Loading...

PDF :1 PDF :2 PDF :3 PDF :4 PDF :5 PDF :6 PDF :7

Like and share and download

How to Identify and Troubleshoot Memory Leaks on IPSO

Diagnosing and Resolving Latch Contention on SQL Server

How to Identify and Troubleshoot Memory Leaks on IPSO downloads checkpoint fileserver SOURCE This document explains how to identify and troubleshoot memory leaks on IPSO systems Supported Versions This document was developed for Check Point R70 However, it applies to all versions of the Check Point firewall Supported OS IPSO 6 2 MR1 (GA029a02)

Related PDF

How to Identify and Troubleshoot Memory Leaks on IPSO

downloads checkpoint fileserver SOURCE This document explains how to identify and troubleshoot memory leaks on IPSO systems Supported Versions This document was developed for Check Point R70 However, it applies to all versions of the Check Point firewall Supported OS IPSO 6 2 MR1 (GA029a02) IPSO 6 2 MR2 (GA039)

Detect & Troubleshoot Performance Issues Faster

launch solarwinds rs 564 VFR 008 images 1705 SAM Detect & Troubleshoot Performance Issues Faster page 1 SolarWinds ® Server & Application Monitor (SAM) monitors your applications and their supporting infrastructure, whether they are running on premises, in the cloud, or in a hybrid environment Do not let slow applications and downtime impact your end users and business services


cdn swcdn 1211 WP NetworkTroubleshooting pdf Memory is a limited resource on all network devices and must be controlled and monitored to help ensure that utilization is kept in check A memory allocation failure means either the network device has used all available memory, or the memory has fragmented such that the device cannot find a usable available block

SQL Server Performance Tuning Accelerator - SQLMaestros

sqlmaestros resources Performance tuning Accelerator pdf • Identify, troubleshoot & diagnose workloads that are consuming high memory • Understand Query Workspace Memory • Identify, troubleshoot & diagnose queries that demand high memory grant Module 5 Tempdb Troubleshooting • How to identify Tempdb usage • Identify, troubleshoot & diagnose workloads that are causing maximum tempdb usage

Identify, Monitor & Test LAN Issues User Manual

platinumtools upload doc 1519337019TNP The Net Prowler has the capability to identify, monitor and troubleshoot LAN issues associated with physical layer and link conditions in the network It also has the functions of a high end cable tester; measuring length, identifying link status, link capability and load testing for PoE voltage, all in

Solving the Five Most Common VMware Virtual Machine Issues

virtualizationsoftware wp content uploads 2014 Solving the Five Most Common VMware Virtual Machine Issues Page 11 you’ll be able to eliminate the finger pointing that all too commonly occurs when there is poor network performance in the virtual infrastructure #3 Virtual Machine Memory Limits When a virtual machine is moved out of a resource pool that had a memory limit, the stand

Diagnosing and Resolving Latch Contention on SQL Server


How to improve the necessity test of the European Court of Human Rights

Code of practice on confidential personal information - CQC

PDF Necessity Toolkit European Data Protection Supervisor europa eu edps europa eu 17 06 01 necessity toolkit final en 0 pdf PDF Salmond, Necessity, and the State Victoria University of Wellington victoria

mind uk media 715750 how to increase How to increase your self esteem Cycle of low self esteem and mental health problems Low self esteem and mental health problems can reinforce each other, creating an unhelpful cycle Low self esteem contributed to my developing depression – and the

If your cluster doesn't have the requisite software you will need to install it For example on Ubuntu Linux $ sudo apt get install ssh $ sudo apt get install rsync This setup and configuration document is a guide to setup a Single Node Apache

  1. Single Node Setup
  2. Apache Hadoop
  3. Installation and Configuration Documentation
  4. 15-Minute Guide to Install a Hadoop Cluster
  5. Installing Hadoop-2.6.x on Windows 10
  6. How to Install Hadoop
  7. Hadoop 1.0.3 Installation
  8. Cloudera Installation
  9. Apache Hadoop Tutorial
  10. RDMA for Apache Hadoop 3.x 0.9.1 User Guide

HOW TO KEEP A SHINING FACE. 2 Corinthians 3:4 18. Dr. George O. Wood

The Symbolism of the Veil - York Rite of California

cdn1 ag 05 How To Keep A Shining Face pdf HOW TO KEEP A SHINING FACE 2 Corinthians 3 4–18 Dr George O Wood Our Scripture today continues to be from Paul‘s second letter to the Corinthians We begin today with chapter 3,

How to Manage Your Money

Managing Your Money in Retirement - AARP

nelnet pdf Tips to Manage Your Money pdf Managing Your Money Nine Tips to Achieving Financial Wellness 1 w Where Your Money Goes Kno Be aware of how you are spending your money A $5 cup of coffee five days

How to Measure for Your New Countertops


PDF Measuring guidelines handbook American Forests americanforests AF Tree Measuring Guidelines LR pdf PDF how to measure for your new kitchen Kitchen & Bath Unlimitedkitchenandbathunlimited files 2016 06 KB measure pdf PDF How

  1. measuring tree caliper
  2. isa measuring dbh
  3. measuring dbh of multi stemmed trees
  4. how to measure for a garage door
  5. tree height measurement pdf
  6. tree measurements
  7. how to measure garage door for opener
  8. how to measure for a overhead garage door



ww2 nycourts gov sites default files document files 2018 When the tenant has finished testifying, the landlord has the right to cross examine the tenant Sometimes a Judge may ask some questions to clarify matters Other witnesses can be presented in support of the tenant’s claims, and they,


Andrew Devereau SixSix---BetBetBet Baccarat Baccarat

bettorsolutions files Unused Files paying 2 way parlays, each completed parlay will pay more if we hit it Naturally, longer parlays (ways) require more winners so they are harder to hit, but the good news is that when you do, you will make a lot more

PDF Producing to plan Klüber Lubrication klueber ecomaXL get blob php?name en PDF BARRIERTA® K Reihe Klüber Lubrication klueber ecomaXL get blob php? Klueber PDF Sustainability at Klüber Lubrication

Home back Next

y and Troubleshoot Memory Leaks on IPSO 5 July 20102 2010 Check Point Software Technologies Ltd.

All rig...


How to Identify and Troubleshoot Memory Leaks on IPSO

© 2010 Check Point Software Technologies Ltd.

All rights reserved.

This product and related documentation are protected by copyright and distributed under licensing restricting their use,

No part of this product or related documentation may be reproduced in any form or by any means without prior written authorization of Check Point.

While every precaution has been taken in the preparation of this book,

Check Point assumes no responsibility for errors or omissions.

This publication and features described herein are subject to change without notice.


or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013 and FAR 52.227-19.

TRADEMARKS: See the Copyright page (http://www.checkpoint.com/copyright.html) for a list of our trademarks.

See the Third Party copyright notices (http://www.checkpoint.com/3rd_party_copyright.html) for a list of relevant copyrights.

Important Information Latest Version The latest version of this document is at: http://supportcontent.checkpoint.com/documentation_download

?ID=10906 For additional technical information visit Check Point Support Center (http://supportcenter.checkpoint.com).

Revision History Date


Initial version

Feedback Check Point is engaged in a continuous effort to improve its documentation.

Please help us by sending your comments (mailto:[email protected]

?subject=Feedback on How to Identify and Troubleshoot Memory Leaks on IPSO ).

Contents Important Information .............................................................................................3 Objective .................................................................................................................5 Supported Versions ................................................................................................5 Supported OS..........................................................................................................5 Supported Appliances ............................................................................................5 Before You Start .....................................................................................................6 Related Documentation and Assumed Knowledge ..............................................

Related Documentation and Assumed Knowledge

Objective This document explains how to identify and troubleshoot memory leaks on IPSO systems.

Supported Versions This document was developed for Check Point R70.


it applies to all versions of the Check Point firewall.

Supported OS 

IPSO 6.2 MR1 (GA029a02)

IPSO 6.2 MR2 (GA039)

The same concepts directly translate to any POSIX-compliant system

Supported Appliances All IP Series appliances running IPSO 6.2 and Check Point firewall.


Page 5

Related Documentation and Assumed Knowledge

Before You Start Related Documentation and Assumed Knowledge 

A fundamental knowledge of Unix memory management is recommended.

Check Point Technical Assistance Center (TAC) assistance is not required to interpret the output of the ps,

These are POSIX compliant commands present on all Unix systems,

therefore the customer administrator is expected to have sufficient knowledge to run these tools and understand the output.

This base knowledge is not strictly required to perform the steps outlined in this document,

however the results will be confusing to people without the base level of knowledge.

Therefore the actual analysis of the generated data should be performed by Check Point support.

only help identify or prove that an issue exists.

Impact on the Environment and Warnings Important

but in ANY procedure to trace memory leaks,

there will be a significant impact on CPU resources.

None of the steps outlined should be performed during core business hours on mission-critical high performance systems.

Before You Start

Page 6

Impact on the Environment and Warnings

Background on Unix Memory management Zones The IPSO kernel manages several pools of memory that can be allocated to either kernel functions or processes.

The memory pools can be viewed with vmstat –z,

which displays all of the Universal Memory Allocator zones.

Each zone is constructed with a distinctive name,

and statistics about the usage: 

Zone references by slab children,

Memory usage of the zone,

Number of total requests in the zone.

slabs There are children memory pools which are directly connected the zones,

either by the kernel or running processes.

They are called slabs.

Every slab is a child of a UMA memory zone

the zone sets certain size restrictions on the type of memory pages that may be requested.

The list of memory slabs may be viewed with vmstat –m.

The reason to use memory slabs,

It is a much faster operation to create a slab of memory as a child of a pre-initialized UMA zone,

than to create a memory zone from scratch.

The UMA memory zones are children of a "primitive" structure called a Keg (outside the scope of this document,

This hierarchy of memory allocation is relevant because it can help you track down when a kernel virtual memory zone’s used pages count is increasing and never going down.

This is the definition of a memory leak within the kernel.

Memory Pages Memory within the children slabs or UMA zones is allocated on a page basis by the virtual memory manager.

The size of these memory pages is fixed within the system,

The IPSO memory allocator works by maintaining a set of lists that are ordered by increasing powers of two.

Each list contains a set of memory blocks of its corresponding size.

To fulfill a memory request,

the size of the request is rounded up to the next power of two.

A piece of memory is then removed from the list corresponding to the specified power of two and returned to the requester.


a request for a block of memory of size 53 returns a block from the 64-sized list.

This sizing less than 4kb occurs within the 4kb slices that are allocated to a page

consider that a memory operation is happening within a particular slab,

and the requesting function needs 3 slots of memory,

Since all of this can fit within a single virtual memory (VM) page,

a single page could be used to store the data.

The single VM page is allocated to a single requesting memory slab.

Allocation Chain It is a critical requirement that memory allocation occur quickly.


there are 3 structures that are defined which manage individual memory pages.

Chaining together the structures allows a program to use predefined memory types,

This speeds up the allocation substantially and increases performance.

Keg – Provides structural information about the memory allocation 2.

UMA Zone – Inherits information from a certain Keg structure,

and further defines sizing and key attributes 3.

Memory slab – Inherits information from one or more UMA zones,

and is linked to directly by the requesting task

Virtual Memory Any modern operating system uses a Virtual Memory system to allocate memory space to the kernel and applications.

This virtual system maps memory requests to contiguous address space,

which maps to discontiguous physical page locations.

The virtual memory handler is used to prevent resource exhaustion of physical memory.

By tracking memory accesses and allocation/free operations,

it is able to dynamically free up physical memory as needed by remapping a physical memory page to the disk media – this is called Background on Unix Memory management

Page 7

Impact on the Environment and Warnings

Swapping of memory is normal and happens as part of memory management.

However during standard operating conditions,

a Check Point firewall should NOT swap as this would lead to severely degraded performance.

Wired Memory A portion of the physical memory can be reserved as WIRED memory.

Wired memory refers to memory pages that may not be swapped out of physical memory to disk.

Often the main usage of Wired memory is for core kernel memory structures.

Programs running on the system may request a certain percentage of their memory space to be reserved as Wired memory.

The total memory allocations and usage may be viewed using the "top" command.

Background on Unix Memory management

Page 8

The Memory Leak Detection Script

How to Use This Document Most often this document will be used because there is a suspected memory leak on a system.

The reason for assuming there is a memory leak,

is typically that the system crashed.

A system could crash because there was insufficient Free memory,

and no other type of memory could be Free’d (possibly too much was Wired,

or the Virtual Memory system was exhausted),

and a new memory allocation for a core system function failed.

In this instance it is normal for the system to crash or lock up.

After the system has been recovered with a hard power cycle,

there are very few clues about what originally caused the problem.

The logfiles most often could not be written to,

because there was no memory available to initialize the function that would write the log.

All crucial IPSO kernel counters available via ipsctl would be cleared on a system reboot.


the "top" "vmstat" and "ps" output would also be cleared.

In the case where there is no available core file to analyze,

an analysis must be done on the newly-booted system to determine if the problem is persistent and likely to occur again.

Memory leaks are notoriously difficult to track down.

The Memory Leak Detection Script The script accompanying this document is intended to help trace memory leaks and determine if there is indeed a memory leak present,

so that the Check Point development team may be engaged to try and help narrow down the exact cause.

The script by itself will only serve as instrumentation.

If the script is aborted or the system crashes before the script is done,

the raw stats will still be present but the script must be hacked (by removing the data collection portion and all sections above it) and rerun to generate the final output.

The script is included in a package that also includes a sample output showing the script running on IPSO 6.2 MR2 (GA039).

How to Use This Document

Page 9

The Memory Leak Detection Script

Running the Memory Leak Detection Script 1.

Download the memory leak detection script mem-html.sh (http://supportcontent.checkpoint.com/file_download

and copy the script to the target system The script is included in a package that also includes a sample output showing the script running on IPSO 6.2 MR2 (GA039).

You may wish to read the script before executing it.


For typical operations it is recommended to run the script with the following syntax: sh mem-html.sh 3600 This will execute the script,

and run for approximately one hour – 3600 seconds.

If you are concerned about the CPU utilization of the script,

you may run it with the following syntax: nice +20 sh mem-html.sh 3600 The script will be executed with the lowest possible CPU priority on the system

however the results may be less accurate.

The script works by writing an HTML page and several subdirectories containing raw statistics.

The raw stats are compiled and written in an abbreviated format to the HTML page as graphs and tables.

Script output,

collected before the intensive data collection The script will collect several important data and include them at the top of the document.

Running the Memory Leak Detection Script

Page 10

The Memory Leak Detection Script

The basic information which is collected includes the duration of the script run,

and date when the script was executed.

The "top" output is also collected for later reference.

The vmstat outputs listed are collected before,

Intensive data collection Until the end timer is reached,

the script will iterate through data collection using ps and top.

This output is collected in the appropriate subdirectory.

Wrapping up the data Once the script’s main loop is complete,

the data is graphed within the HTML page using CSS.

This is to assist the analyst by providing a historical plot of the memory utilization in megabytes.

Running the Memory Leak Detection Script

Page 11

The Memory Leak Detection Script

Tables are also created containing the start and end values of both kernel memory stats,

This allows the tracking of memory leaks both within the kernel space,

A leak in the kernel should be easily detectable by seeing the Free memory consistently shrink,

and Wired memory growing consistently.

There should be a corresponding growing Virtual Memory slab corresponding to the system function with the leak.


the RSS for every process would drop as the virtual memory system tries to reclaim the least frequently used memory pages from the running processes.

A leak in a userland process would show similar behavior,

except there would be a large increase in RSS and VSZ on one specific process.

The system *should* normally panic in these circumstances either when the process reserved Wired memory increases beyond the available Free space,

or the Virtual Memory address space is exhausted.

It has been observed that a system may simply hang rather than panic and generate a core file.

As a convenience to the analyst,

the file descriptor allocations are also tracked via the script,

since file descriptor leaks could also be interpreted as a memory leak.

All of the raw data that is used by the script is stored in the subdirectories so that the analyst may drop the results into Excel and plot them.

Script output

The resulting index.html and subdirectories are put into a script-output.tgz tarball,

and the original files and directories are then deleted.

The script-output.tgz must be collected and provided to Check Point TAC for analysis.

Check Point will extract the contents of the tarball.

Running the Memory Leak Detection Script

Page 12

The Memory Leak Detection Script

Running the Memory Leak Detection Script

Page 13

The Memory Leak Detection Script

Analysis of the Memory Leak Detection script The analyst of the script output must consider several items: 

How busy is the system

The system may only leak memory when there is a lot of traffic

Running the script during a maintenance window may not yield the desired output

How long it has been up

Has the memory gone up or down normally,

A system will during a normal day request and release large amounts of memory

"Abnormal" growth could indicate the problem,

but it must be considered with everything else that is happening

Where is the memory going

Depending on the uptime and how busy the system is,

memory can be allocated to Wired,

Inactive and process memory space in a way that looks wrong

This is dependent on too many factors to worry about any one indicator

Has the condition that triggered the memory leak occurred

A system could leak memory over very long timeframes – potentially weeks or months

A memory leak is usually triggered by a specific function that requests memory and never releases it.

If this function doesn’t execute,

It’s important to run the script on the actual system that is leaking memory,

When running the script,

ensure that an appropriate duration is selected.

Unless the memory leak is extreme,

days or weeks may be required.

Analysis of the Memory Leak Detection script

Page 14

The Memory Leak Detection Script

Additional Tools Commands The following commands may prove useful when trying to track down memory leaks: top ps –auxwwlSHm vmstat –z vmstat –m vmstat 1 5 vmstat

Kernel Flags IPSO is compiled with an additional debug kernel.

Since IPSO is FreeBSD-based,

you can look up additional information about what the kernel flags are for.

The additional flags that the kernel is compiled with,

are Witness Witness_KDB Witness_Skipspin Invariants Invariants_Support Diagnostic KDB_Unattended Choosing to run this kernel.debug_g is extremely unwise unless directed to do so by Check Point engineering.

One purpose of this debug kernel is to crash/panic when an error is detected,

much more frequently than on a non-debug kernel.

Therefore the kernel is very rarely provided to a customer and would rather be implemented in a lab where the error occurred.

References The following resources are good starting places to learn about memory management: 

MIT 6.823 Computer System Architecture Lectures Fall 2009 (http://csg.csail.mit.edu/6.823/lecnotes.html)

A Study of Initialization in Linux and OpenBSD",

Operating System Review Vol 39,

No 2,

April 2005 (http://cisr.nps.edu/downloads/05paper_linux.pdf)

The Slab Allocator: An Object-Caching Kernel Memory Allocator (http://citeseerx.ist.psu.edu/viewdoc/download


Introduction to Debugging the FreeBSD Kernel (http://www.bsdcan.org/2008/schedule/attachments/45_article.pdf)

Week 3 of Berkeley CS61C Spring 2010 Lectures (http://inst.eecs.berkeley.edu/~cs61c/sp10/)

cse522: Advanced Operating Systems Lecture Notes,

Spring 2008,

Washington State University in St Louis (http://www.arl.wustl.edu/~fredk/courses/cs523/notes.html)

Computer Science 152: Computer Architecture and Engineering Lectures,

Berkeley University (http://inst.eecs.berkeley.edu/~cs152/sp10/)

Network Buffer Allocation in the FreeBSD Operating System,

Bosko Milekic (http://bmilekic.unixdaemons.com/netbuf_bmilekic.pdf)

Understanding the Linux Virtual Memory Manager,

Mel Gorman (http://ptgmedia.pearsoncmg.com/images/0131453483/downloads/gorman_book.pdf)

Additional Tools

Page 15