Tracking High Memory Usage on Windows 10 (Non-Paged pool) and Event Tracing Buffers

If you’re wondering why your memory usage is very high but you don’t see any process using so much memory in the task manager, you’ve found the right tutorial.

The problem is that the non-paged pool doesn’t appear in the process list, since it’s memory used by the system kernel and 3rd party drivers. Most likely, if the non-paged pool is using the majority of your memory, one of your installed drivers has a memory leak. So you should start by updating your drivers.

If you have already updated your drivers, you can continue reading. Since the task manager is useless for this case, we’re going to use a tool called poolmon. Poolmon is part of the Windows Driver Kit, which you can download here.

Using Poolmon

Once it’s installed, if it’s in the default folder you’ll find poolmon here:

 C:\Program Files (x86)\Windows Kits\10\Tools\{arm, arm64, x86, x64}\poolmon.exe

You need to choose the right folder depending on your processor. Mine is x64, so I’ll open …\x64\poolmon.exe

You can also add the folder to the PATH of the system if you want, so you can open poolmon by typing it in the command prompt directly

Once it’s open, you’re going to see this:

Let’s start by pressing ‘b’ to sort by size. Now, you have the list of pool allocation tags ordered by size. In the «Bytes» column you’ll see the amount of memory each one is using in bytes (obviously). Most likely, if you have a excessive memory usage, you’re already seeing the problem: it’s probably the first one. You can also filter by paged or non-paged by pressing ‘p’ if you need it.

In my case it’s obvious because I have 981 MB in the non-paged pool (as shown in the task manager) and EtwB is using 860 MB alone. At this point you’ll need to search for the tag in pooltag.txt, which is in Debuggers\x64\triage , inside the folder of the Windows Driver Kit. You can also read it here if it’s easier for you.

The file has the following format:
<PoolTag> - <binary-name> - <Description>

If it doesn’t appear in there, that’s because it’s not a known tag. In that case you should search for it on Google. If it’s part of a driver, you should report the issue to the company responsible of the drivers so they can update it to fix the issue. There are many possibilities here. In my case, I searched for the tag (EtwB) and I discovered the following information:

EtwB - nt!etw       - Etw Buffer

Turns out this is part of Windows Event Tracing facility.

Analyzing Event Tracing Sessions Sizes

If your problem is related to Windows Event Tracing, you should be able to know what is causing high memory usage with logman.

logman creates and manages Event Trace Session and Performance logs. Let’s start by listing them:

logman -ets

You can read more about each one by typing logman [name] -ets. This is going to list some information about it, such as name, status, type and buffer size. If the name has spaces, you should wrap it in quotation marks. In my case, the problem was in the last one: RivetActivity. How did I know? I typed:

logman RivetActivity -ets

And the following info appeared:

Name:                 RivetActivity
 Status:               Running
 Root Path:            C:\ProgramData\RivetNetworks\Killer\ActivityLog
 Segment:              Off
 Schedules:            On
 Segment Max Size:     1000 MB
 Name:                 RivetActivity\RivetActivity
 Type:                 Trace
 Output Location:      C:\ProgramData\RivetNetworks\Killer\ActivityLog\ActivityLog.etl
 Append:               Off
 Circular:             On
 Overwrite:            Off
 Buffer Size:          1000
 Buffers Lost:         0
 Buffers Written:      1
 Buffer Flush Timer:   1
 Clock Type:           Performance
 File Mode:            File

With all this information, I know that it has a maximum segment size of 1000MB, and a Buffer Size of 1000. That made me suspect from it. As you can see, it’s related to Killer Networking, which is the manufacturer of my Ethernet adapter. To finally check if that was the problem, I stopped the data collector set by typing the following command:

logman stop RivetActivity -ets

And just like that, my non-paged pool was reduced from almost 1GB to 149MB. This is not the first time I have a problem with Killer Ethernet in this computer. The last one was effectively a memory leak in the Killer Suite, which was using all my memory. And I discovered it by doing the same steps here, using poolmon and searching for it. After that issue, I started to download the driver-only installation without the Killer Suite, but turns out they keep screwing everything up.

I’ll contact to Killer to see if they can fix this, and I’ll update this post if necessary.

UPDATE [05-06-2019]

Turns out I had traces of an old driver installation. I had updated the driver to the latest version before, but I don’t know why it didn’t uninstalled some old bugged services. As they said: «If you install the driver only version, it removes the Killer software, thus removes this feature (and its related bug, which as since been fixed). «

Anyways, I used the Killer Uninstaller to delete everything, I made a fresh installation and the problem is now gone. Killer also made an explanation for the problem:

«Rivet Activity/Activity log are for internal tracking of events. This is only used to display the ‘Events’ in the Killer Control Center’s Overview/Notifications panel. The version you were running had a known issue where it does not properly ask for just the memory it needs for these events (very small, in the 10MB range), but instead allocates a full buffer which was not intended. «

– Killer Support

Thank you for reading; I sincerely hope it has been useful for you. You can leave a comment if you have any questions. I would also like to thank Killer Support for the -really- quick answer.

Bye!

Deja un comentario