Advanced search

Message boards : Frequently Asked Questions (FAQ) : FAQ - Recommended pagefile sizes for Windows users for Python on GPU tasks

Author Message
Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1335
Credit: 7,373,467,459
RAC: 13,434,696
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59136 - Posted: 19 Aug 2022 | 23:02:44 UTC

For Windows users attempting to run Python on GPU tasks so that they do not fail it is recommended to change the default pagefile settings.

Change from default or system managed settings to Custom with a minimum starting pagefile size of 35000 MB and the upper limit to be 512000 MB.

See this post for location images in Windows.
https://www.gpugrid.net/forum_thread.php?id=5336&nowrap=true#59029

[CSF] Aleksey Belkov
Send message
Joined: 26 Dec 13
Posts: 86
Credit: 1,263,181,270
RAC: 225,076
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 59139 - Posted: 20 Aug 2022 | 3:57:57 UTC

Useful notice.
But in a good way, this information should be in project preferences when choosing applications to crunch(how it's done in WCG project).
Or at least in the top post of announcing new application's thread.
A lot of time and effort could be saved(by many people) if such important information was placed in the right place : /

djo5296
Send message
Joined: 31 Jul 18
Posts: 1
Credit: 3,374,175
RAC: 0
Level
Ala
Scientific publications
wat
Message 59219 - Posted: 11 Sep 2022 | 0:24:32 UTC - in response to Message 59136.

Is it safe to allocate such a huge virtual memory on my SSD? Will it not affect the lifespan of my SSD? If yes, can I divide the recommended virtual memory size to multiple drives and maybe even put the bulk of the virtual memory on my HDD instead?

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1335
Credit: 7,373,467,459
RAC: 13,434,696
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59220 - Posted: 11 Sep 2022 | 3:57:38 UTC

There is no danger to your SSD's. That trope is a long disproven fallacy about too many writes to an SSD killing your drive.

Many articles have tested the longevity of modern SSD drives and the none of them have any failures in literally petabytes of data written to them.

The figure in SSD specifications to pay attention to is Total Drive Writes per day or DWPD and TBW or Terabytes Written. The higher the number the better.

For typical consumer drives in the 256GB-2TB range drive life expectancy is on the order of tens of years.

A Samsung 850 Pro 1TB drive will last before failure in 342 years for example.

You would kill the HDD much faster than you ever would using the SSD.

https://www.enterprisestorageforum.com/hardware/ssd-lifespan-how-long-will-your-ssd-work/

WPrion
Send message
Joined: 30 Apr 13
Posts: 96
Credit: 2,243,834,111
RAC: 21,138,713
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 59242 - Posted: 12 Sep 2022 | 16:33:35 UTC - in response to Message 59136.

Since we're talking about pagefile, I assume this storage won't be used until most/all of RAM is consumed.

What size RAM are those pagefile requirements based upon?

For example, if a computer has 128GB of RAM is a max pagefile of 512GB still needed or could that be less?

In other words, what's the max combined RAM + pagefile needed?

Thanks,

W

KAMasud
Send message
Joined: 27 Jul 11
Posts: 137
Credit: 523,901,354
RAC: 0
Level
Lys
Scientific publications
watwat
Message 59243 - Posted: 12 Sep 2022 | 17:46:56 UTC

50GB is recommended. Where did the 512GB come from?

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1335
Credit: 7,373,467,459
RAC: 13,434,696
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59244 - Posted: 12 Sep 2022 | 19:25:04 UTC - in response to Message 59242.

I don't have an answer for you since I don't run Windows.

Linux handles the issue much differently than Windows.

On my Linux hosts with 128GB of system RAM, I have not used any of the default 2GB pagefile.

On my Linux hosts with 32GB of system RAM, I see less than a 1GB of pagefile in use with the host using 17-18GB of the system RAM.

On this daily driver with 32GB of system RAM I see around 24-26GB of RAM in use and around 2.2-2.4 GB of the 6GB pagefile in use.

This host also supports two Gridcoin wallets along with BOINC usage and is why it is using more memory and pagefile than just my simple BOINC crunchers.

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1335
Credit: 7,373,467,459
RAC: 13,434,696
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59245 - Posted: 12 Sep 2022 | 19:27:58 UTC - in response to Message 59243.

50GB is recommended. Where did the 512GB come from?

Yes, it looks like a max 50GB pagefile is being sufficient for Windows.

No need for a huge 512GB pagefile size. Assume typo here in play.

WPrion
Send message
Joined: 30 Apr 13
Posts: 96
Credit: 2,243,834,111
RAC: 21,138,713
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 59246 - Posted: 12 Sep 2022 | 23:48:57 UTC - in response to Message 59243.

50GB is recommended. Where did the 512GB come from?


The original post in this FAQ stated 512000 MB

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1335
Credit: 7,373,467,459
RAC: 13,434,696
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59247 - Posted: 13 Sep 2022 | 1:07:40 UTC - in response to Message 59246.
Last modified: 13 Sep 2022 | 1:09:40 UTC

50GB is recommended. Where did the 512GB come from?


The original post in this FAQ stated 512000 MB


That's an upper limit for the pagefile to dynamically grow. Not that it would ever need one that large but it certainly would be a safe value.

I was just saying if you set it that large for the upper limit you would never have the issue.

And since I don't do Windows, I was just making a safe guess. With further comments from actual Windows users the upper safe limit is 50GB or 50000 MB.

WPrion
Send message
Joined: 30 Apr 13
Posts: 96
Credit: 2,243,834,111
RAC: 21,138,713
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 59249 - Posted: 13 Sep 2022 | 12:40:41 UTC - in response to Message 59247.
Last modified: 13 Sep 2022 | 13:02:39 UTC

Thanks!

Now I'll circle back to my original question, which I guess is re: Windows only.


Is that 50GB upper limit for pagefile PLUS RAM, or just pagefile?

If a computer has 64GB or 128GB of RAM is any pagefile needed??

Keith Myers
Send message
Joined: 13 Dec 17
Posts: 1335
Credit: 7,373,467,459
RAC: 13,434,696
Level
Tyr
Scientific publications
watwatwatwatwat
Message 59250 - Posted: 13 Sep 2022 | 18:19:16 UTC - in response to Message 59249.

Thanks!

Now I'll circle back to my original question, which I guess is re: Windows only.


Is that 50GB upper limit for pagefile PLUS RAM, or just pagefile?

If a computer has 64GB or 128GB of RAM is any pagefile needed??


Again, I state I don't run Windows so don't know all the particulars for everyone running Windows that has been successful or what their hosts have for hardware.

I am just stating that 50GB of pagefile seems to work. I don't know how much RAM they are using.

The reason why this issue only occurs for Windows is here:

https://www.gpugrid.net/forum_thread.php?id=5322&nowrap=true#58908

Has to do with how Windows enforces memory allocation.

Post to thread

Message boards : Frequently Asked Questions (FAQ) : FAQ - Recommended pagefile sizes for Windows users for Python on GPU tasks

//