Intermediate SQL Color Coded SQL, UNIX and Database Essays

How AIX Paging Space works. Part 1: Why your program memory footprint gets bloated sometimes

If you recall AIX memory saga from my previous “AIX memory” posts (Process memory, SGA Memory), one of the points was that during the lifetime of the process (or shared memory segment) at any given moment each allocated memory page could reside in one of the two places:

• Either the page is located in a physical RAM
• OR the page is located in paging space

In memory location is, obviously, preferred. If I was a memory page I would WANT to be there In paging space ? – not so much, but, of course I could be forced there if I’m not popular enough and AIX needs memory for other things.

As you can see, logically, “page location” is an exclusive EITHER/OR deal – either the page is in memory OR it is in paging space – it can never be “in between”. In a way, virtual memory “flows” between physical memory and the paging space and because of that, the following approximate formula should work:

This formula is very logical and very easy to understand … and would even work in a perfect world (that does not require memory optimizations).

Unfortunately in our world we may get stuck with a memory allocations that look like this:

Here, we obviously have a problem with our calculation as 7,796 + 6,550 = 14,346, which is much larger than the virtual size of: 7,914.

So, what happened ? Where is the memory ? And how much of it are we really using ?

The answer to the “what happened” question sends us again to the AIX bag of tricks.

In short:

In this post, to show how AIX paging space works in details, I’m going to create a few small C programs that will allocate AIX shared memory segment (mimicking ORACLE SGA), but the same logic should apply to any AIX computational memory that is regularly managed and is eligible for page out (*). So, let’s begin, shall we ?

(*) And here is an example of not regularly managed AIX memory

To see how paging space behaves, we clearly need to use it. That means that we have to create system conditions that will encourage paging … or, in other words, our programs should request (a lot) more memory than available in the system …

First of all, so that not to make our job too difficult, let’s artificially limit the amount of available memory to ~ 3 Gb

Next, to the test programs. I’m going to need 3 of them:

1. shmalloc: A program to allocate shared memory. We will use it to mimic ORACLE SGA as well as (in a separate instance) to create “external” workload and push “SGA” into paging space
2. shmwrite: A ‘writer’ program to change SGA memory, to represent UPDATE operations
3. shmread: A ‘reader’ program to read SGA memory and emulate SELECT operations

You can find the source code for these programs in the TOOLS section.

Ok, now all the preparations are made, so let’s start the test.

This is a first snapshot of our “SGA” right after it started up. Notice that right now it uses ~ 180 Mb of physical RAM and not the full 1.2 Gb that was requested. This is because, as you might recall, AIX does not allocate memory until it is actually used (and at the moment, we are only using 180 Mb).

Current SGA memory snapshot looks like this:

Let’s load more of the “SGA” into memory … We are going to use shmwrite program for that (but with real ORACLE instance, you could use i.e. a FULL TABLE SCAN for the large table … at least if you are still in ORACLE 10g).

After updating “SGA” memory it is now fully allocated. Moreover, as our “SGA” is still the only game in town, it fits entirely in the available physical RAM:

And the overall “SGA” memory picture looks like this:

Notice a subtle thing here: No paging space has been allocated as of yet !

You might wonder why it is “subtle”, after all, obviously, there is still plenty of “real” memory available …

But what seems obvious right now is, in fact, a fairly recent improvement in AIX paging space behavior. Only since AIX 5.1 did IBM introduce a deferred paging space allocation policy that does not ALLOCATE paging space until system needs to PAGE OUT.

In older versions, with then default late paging space allocation policy AIX would not USE paging space until PAGE OUT, but would ALLOCATE it anyway. So, the picture that you’d see before AIX 5.1 would look like this:

Ok, moving on … Now, let’s simulate some workload and create a huge memory pressure for the system. This will hopefully make AIX page out some of “SGA” pages…

To simulate “external memory pressure” let’s request lots of of RAM by another process.

First of all, how much memory do we have right now ?

That would be: 276343 * 4k or ~ 1.3 Gb of available RAM. To be sure, let’s request 2 Gb now. Hopefully, this will force some of our “SGA” out to the paging space.

Since our “SGA” was idle and thus represented older workload, it must give way … and it does:

A significant portion of the “SGA” memory was moved to the paging space, and, the overall “SGA” memory picture looks like this now:

Note that our simple memory estimation formula still works as Virtual (that is: overall requested) memory size is equal to “in memory” + “in paging space”. In other words, memory more or less cleanly “flows” from “in memory” to “in paging space” location.

We’ve seen how memory and paging space behave on PAGE OUTs … Now, let’s see how they behave on PAGE INs …

To do that, let’s kill the “outside workload” (so that the system once again has lots of available memory) and request “SGA” pages back in RAM by requesting another update on them:

Ok, the memory “flowed” again. 100% of our RAM is still accounted for by our simple memory allocation formula as i.e.: 60362+5174 = 65536. Our “SGA” memory is still roughly a zero sum game …

It looks like paging space is deallocated once associated pages are read into memory during UPDATE operation…

This paging space behavior, by the way, is actually another major feature that became available only with AIX 5.3. Paging space deallocation is controlled by a garbage collection mechanism, which, by default, works only during PAGE IN operations …

We’ve seen that paging space is deallocated when memory pages are brought back in memory for UPDATE … Would the same hold true if we simply READ ? Let’s find out …

First, let’s repeat the steps to spill “SGA” out to paging space again:

And now, let’s kill the “external workload” and read “SGA” memory …

Wow, this is weird ! Take the 2nd line for example: 65536 + 37293 = 102829 … Yet only 65536 (4k) blocks are really requested …

As you can see, during READ operations paging space is NOT released, instead, many memory pages now appear to be “double allocated” (they “exist” in both “real memory” and paging space) and the overall memory picture looks like this:

This is where our simple formula finally breaks down !

And the question naturally arises – why AIX is not releasing paging space on reads ? What good are the “leftover” paging space copies if our memory is “truly” in RAM ?

Well, this is actually pretty simple to figure out once you realize that:

• Memory that has already been paged out once is likely to do so again …
• After PAGE IN caused by READ, leftover “in paging space” page copy remains identical to its “in memory” twin

Thus, on a likely subsequent PAGE OUT (and assuming memory did not change in between), AIX can simply discard the memory page and will NOT need to copy its contents back to paging space … which does represent some serious savings for AIX …

So, essentially, by keeping “associated” memory copies in paging space AIX is positioning itself to potentially (and likely) skip some serious work in the future and a bit more disk space that is used for that is only a small price to pay for this gain in efficiency.

You might wonder – what would happen if memory pages are PAGED IN “for read” initially, but then changed afterwards ? Are they going to be “released” from the paging space at the time of change ?

Let’s find out …

Looks like the answer is: NO. “SGA” memory is still double allocated even though memory has really changed and no longer matches its “in paging space” copy.

This is because the default garbage collection process only works during PAGE IN operations (at least, up until 6.1 ML5) and if there is no PAGE IN (and remember, after the READ, all “SGA” pages were already “in memory”) then there cannot be a paging space release.

The bottom line here is that with the way AIX paging space works our simple memory estimation formula may no longer yield correct results.

So, is there a way to fix it ?

The answer is a qualified YES and, curiously, the formula (and the results) that you get will depend on the exact question that you ask.

Let’s say that you want to know how much of your program’s memory is “really” in the paging space ? (this is a performance tuning question).

The key insight here is that during the time when memory pages are double allocated, the primary page copy is ALWAYS in memory. Thus, svmon “inuse” field returns “true” data, while “pgsp” value is artificially “bloated” (double counting paging space pages that are “really” in memory).

Thus, “real” paging space use will be:

Alternatively, you might want to know, how much paging space is being used by your program ? (this is paging space sizing question).

In this case, svmon “pgsp” values is not going to be “bloated” any longer – yes, the pages are double counted here, but they are nevertheless physically allocated.

Hence the resource usage formula would look like our old simple memory allocation formula:

It’s just that “what is used” may NOT equal “what is requested” (Virtual) anymore …