Widely Vision

Memory overcommit in Linux Memory management

Posted by aboelnour on September 16, 2011

What’s happening when a program request more memory on machine running OS with Linux kernel?

last days I get the answer.When you use malloc() to reserve more memory it has a return value.

from malloc() man page:

 malloc() returns a pointer to the allocated memory,

which is suitably aligned for any kind of variable.

On error,these functions return NULL.

NULL is returned when error occur in allocating the block of memory that include the system didn’t find enough space to allocate it to your program.

With Linux kernel by default malloc() always succeed. Linux deal with memory allocating like airline when sell tickets, it sells more tickets than they have actual seats, in the hopes that some of the passengers don’t show up.

Linux assuming you’re not “really” going to use all of the memory you just asked for. The malloc()‘s will continue to succeed, but not until you actually try to use the memory you allocated will the kernel “really” allocate it.

As an example, consider the fork() system call, which copies all of a process’s memory for the new child process. In fact, all it does is to mark the memory as “copy on write” and allow parent and child to share it. Should either change a page shared in this way, a true copy is made.

This called “Memory overcommit

“Memory overcommit is a Linux kernel feature that lets applications allocate more memory than is actually available. The idea behind this feature is that some applications allocate large amounts of memory “just in case”, but never actually use it. Thus, memory overcommit allows you to run more applications than actually fit in your memory, provided the applications don’t actually use the memory they’ve allocated. If they do, then the kernel terminates the application.”

To make things more clearer you can try this examples:

[WARNING] “This examples may cause harm to your computer if you don’t know how to rescue your system from process eating all of your memory don’t run it”

example 1: allocate memory without using

#include <stdio.h>
#include <stdlib.h>

int main (void) {
int n = 0;

while (1) {
if (malloc(1<<20) == NULL) {
printf(“malloc failure after %d MiB\n”, n);
return 0;
}
printf (“got %d MiB\n”, ++n);
}
}

example 2: allocate memory and actually touch it all.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main (void) {
int n = 0;
char *p;

while (1) {
if ((p = malloc(1<<20)) == NULL) {
printf(“malloc failure after %d MiB\n”, n);
return 0;
}
memset (p, 0, (1<<20));
printf (“got %d MiB\n”, ++n);
}
}

example 3: first allocate, and use later.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define N       10000

int main (void) {
int i, n = 0;
char *pp[N];

for (n = 0; n < N; n++) {
pp[n] = malloc(1<<20);
if (pp[n] == NULL)
break;
}
printf(“malloc failure after %d MiB\n”, n);

for (i = 0; i < n; i++) {
memset (pp[i], 0, (1<<20));
printf(“%d\n”, i+1);
}

return 0;
}

In example 1 you will notice that the program will print smoothly without harming your computer and it will stop when the program reach your ram boundary and exit successfully.

Explanation:

the program doesn’t use the memory he allocate it,so Linux doesn’t care to allocate it.

——–

In example 2 you will notice that program make the computer print slower.

Explanation:

the program after allocating each block he use it, so Linux allocate it after each memset() so it takes alot of time to reach ram boundary.

——–

In example 3 you will notice that the program print quickly and when he reach the ram boundary the computer will be slower.

Explanation:

the program will allocate the whole block of memory after malloc() finishes.program will print smoothly and after printing “malloc failure after MiB” the computer will become slower.

———————————

So what’s happing when process request for more memory and there no free memory to allocate it for the process?

Linux start a job called OOM killer “Out Of Memory” killer, Its job is to kill processes and free up some memory. Getting it to kill the right processes has been an ongoing challenge, however. One person’s useless memory hog is another’s crucial application.

This challenge is like:

“An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused.”

———–

Refrences:

http://lwn.net/Articles/104179/

http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html

http://www.win.tue.nl/~aeb/linux/lk/lk-9.html

http://www.gnu.org/software/gnusound/Documentation/ar01s05.html#id2572059

———–

I hope that any one find this post useful. any questions or feedbacks are welcomed.

regards,

aboelnour

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: