Add direct access to rss without iterating pages#34291
Add direct access to rss without iterating pages#34291Aschen wants to merge 1 commit intonodejs:masterfrom kuzzleio:add-direct-access-to-rss
Conversation
bnoordhuis
left a comment
There was a problem hiding this comment.
For your (and everyone else's) consideration, perhaps:
-
the
osmodule is a more appropriate place for a new method? -
instead of a new method, make
process.memoryUsage()take an optional{ rssOnly: true }options object?
test/parallel/test-memory-usage.js
Outdated
There was a problem hiding this comment.
This is not reliable because the RSS can change between the two calls.
src/node_process_methods.cc
Outdated
There was a problem hiding this comment.
| // Get the double array pointer from the Float64Array argument. | |
| CHECK(args[0]->IsFloat64Array()); | |
| Local<Float64Array> array = args[0].As<Float64Array>(); | |
| CHECK_EQ(array->Length(), 1); | |
| Local<ArrayBuffer> ab = array->Buffer(); | |
| double* fields = static_cast<double*>(ab->GetBackingStore()->Data()); | |
| fields[0] = rss; | |
| args.GetReturnValue().Set(static_cast<double>(rss)); |
Float64Array is overkill if you're returning a single floating-point value. You can expose the method directly, without going through a JS method first.
|
IMHO the But I like your idea of a |
It would be possible to use |
I thought about this one but I couldn't figure an easy way to document it since it will involve a property on a function which is a function |
|
@bnoordhuis I updated the PR with the |
test/parallel/test-memory-usage.js
Outdated
There was a problem hiding this comment.
| const result = process.memoryUsage({ rssOnly: true }); | |
| process.memoryUsage(); | |
| const result = process.memoryUsage({ rssOnly: true }); |
Then re-run the test. :-)
(You need to clear the other fields when rssOnly=true or they'll contain stale values.)
test/parallel/test-worker-memory.js
Outdated
There was a problem hiding this comment.
Leftover from the previous iteration of the PR? There's no process.rss() anymore.
|
@Aschen :
Take a look at the documentation for |
|
@bnoordhuis Are you also ok with the |
|
@Aschen Sure, there's precedent, e.g., |
This comment has been minimized.
This comment has been minimized.
|
I got a strange error on the file https://github.com/nodejs/node/pull/34291/checks?check_run_id=1371318709 |
doc/api/process.md
Outdated
There was a problem hiding this comment.
I find can be slow quite vague - in special as a caller can't do anything to avoid this to test if it will be slow.
Are we talking about µs, ms, or seconds here?
There was a problem hiding this comment.
It depends on the program size in memory. Access the RSS take a fixed time but compute heap statistics is quite costly since you have to iterate over every memory page.
The difference can be in seconds. (see the benchmark in the PR description)
There was a problem hiding this comment.
The sample in PR description prints milli seconds not seconds so not that bad.
I tried it local on my machine and even for 4GB heaps it was below 0.2ms. But for sure the rss only API is much faster.
There was a problem hiding this comment.
At the beginning I have done this PR because I found very strange results while benchmarking promises (#33384).
So even if the difference is not that huge it can be misleading in some cases
I think you have to add |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@nodejs/process |
PR-URL: #36768 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michael Dawson <[email protected]>
PR-URL: #36769 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36766 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: James M Snell <[email protected]>
Accessing the rss value through memoryUsage() can be expensive because this method will also generate memory usage statistics by iterating on each page. This commit intend to offer a more direct access to rss value. Refs: #33384 PR-URL: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: James M Snell <[email protected]>
Refs: #34291 PR-URL: #36781 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Rich Trott <[email protected]>
PR-URL: #36768 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michael Dawson <[email protected]>
PR-URL: #36769 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36766 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) [#29412](#29412) * support AbortSignal in fork (Benjamin Gruenbaum) [#36603](#36603) * crypto: * implement basic secure heap support (James M Snell) [#36779](#36779) * fixup bug in keygen error handling (James M Snell) [#36779](#36779) * introduce X509Certificate API (James M Snell) [#36804](#36804) * implement randomuuid (James M Snell) [#36729](#36729) * doc: * update release key for Danielle Adams (Danielle Adams) [#36793](#36793) * add dnlup to collaborators (Daniele Belardi) [#36849](#36849) * add panva to collaborators (Filip Skokan) [#36802](#36802) * add yashLadha to collaborator (Yash Ladha) [#36666](#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) [#36685](#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) [#36623](#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) [#34291](#34291) * v8: * fix native constructors (ExE Boss) [#36549](#36549)
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) [#29412](#29412) * support AbortSignal in fork (Benjamin Gruenbaum) [#36603](#36603) * crypto: * implement basic secure heap support (James M Snell) [#36779](#36779) * fixup bug in keygen error handling (James M Snell) [#36779](#36779) * introduce X509Certificate API (James M Snell) [#36804](#36804) * implement randomuuid (James M Snell) [#36729](#36729) * doc: * update release key for Danielle Adams (Danielle Adams) [#36793](#36793) * add dnlup to collaborators (Daniele Belardi) [#36849](#36849) * add panva to collaborators (Filip Skokan) [#36802](#36802) * add yashLadha to collaborator (Yash Ladha) [#36666](#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) [#36685](#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) [#36623](#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) [#34291](#34291) * v8: * fix native constructors (ExE Boss) [#36549](#36549)
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) [#29412](#29412) * support AbortSignal in fork (Benjamin Gruenbaum) [#36603](#36603) * crypto: * implement basic secure heap support (James M Snell) [#36779](#36779) * fixup bug in keygen error handling (James M Snell) [#36779](#36779) * introduce X509Certificate API (James M Snell) [#36804](#36804) * implement randomuuid (James M Snell) [#36729](#36729) * doc: * update release key for Danielle Adams (Danielle Adams) [#36793](#36793) * add dnlup to collaborators (Daniele Belardi) [#36849](#36849) * add panva to collaborators (Filip Skokan) [#36802](#36802) * add yashLadha to collaborator (Yash Ladha) [#36666](#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) [#36685](#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) [#36623](#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) [#34291](#34291) * v8: * fix native constructors (ExE Boss) [#36549](#36549)
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) [#29412](#29412) * support AbortSignal in fork (Benjamin Gruenbaum) [#36603](#36603) * crypto: * implement basic secure heap support (James M Snell) [#36779](#36779) * fixup bug in keygen error handling (James M Snell) [#36779](#36779) * introduce X509Certificate API (James M Snell) [#36804](#36804) * implement randomuuid (James M Snell) [#36729](#36729) * doc: * update release key for Danielle Adams (Danielle Adams) [#36793](#36793) * add dnlup to collaborators (Daniele Belardi) [#36849](#36849) * add panva to collaborators (Filip Skokan) [#36802](#36802) * add yashLadha to collaborator (Yash Ladha) [#36666](#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) [#36685](#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) [#36623](#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) [#34291](#34291) * v8: * fix native constructors (ExE Boss) [#36549](#36549)
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) [#29412](#29412) * support AbortSignal in fork (Benjamin Gruenbaum) [#36603](#36603) * crypto: * implement basic secure heap support (James M Snell) [#36779](#36779) * fixup bug in keygen error handling (James M Snell) [#36779](#36779) * introduce X509Certificate API (James M Snell) [#36804](#36804) * implement randomuuid (James M Snell) [#36729](#36729) * doc: * update release key for Danielle Adams (Danielle Adams) [#36793](#36793) * add dnlup to collaborators (Daniele Belardi) [#36849](#36849) * add panva to collaborators (Filip Skokan) [#36802](#36802) * add yashLadha to collaborator (Yash Ladha) [#36666](#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) [#36685](#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) [#36623](#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) [#34291](#34291) * v8: * fix native constructors (ExE Boss) [#36549](#36549)
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) [#29412](#29412) * support AbortSignal in fork (Benjamin Gruenbaum) [#36603](#36603) * crypto: * implement basic secure heap support (James M Snell) [#36779](#36779) * fixup bug in keygen error handling (James M Snell) [#36779](#36779) * introduce X509Certificate API (James M Snell) [#36804](#36804) * implement randomuuid (James M Snell) [#36729](#36729) * doc: * update release key for Danielle Adams (Danielle Adams) [#36793](#36793) * add dnlup to collaborators (Daniele Belardi) [#36849](#36849) * add panva to collaborators (Filip Skokan) [#36802](#36802) * add yashLadha to collaborator (Yash Ladha) [#36666](#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) [#36685](#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) [#36623](#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) [#34291](#34291) * v8: * fix native constructors (ExE Boss) [#36549](#36549)
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) (#29412) * support AbortSignal in fork (Benjamin Gruenbaum) (#36603) * crypto: * implement basic secure heap support (James M Snell) (#36779) * fixup bug in keygen error handling (James M Snell) (#36779) * introduce X509Certificate API (James M Snell) (#36804) * implement randomuuid (James M Snell) (#36729) * doc: * update release key for Danielle Adams (Danielle Adams) (#36793) * add dnlup to collaborators (Daniele Belardi) (#36849) * add panva to collaborators (Filip Skokan) (#36802) * add yashLadha to collaborator (Yash Ladha) (#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) (#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) (#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) (#34291) * v8: * fix native constructors (ExE Boss) (#36549)
PR-URL: #36889 Notable changes: * child_process: * add 'overlapped' stdio flag (Thiago Padilha) (#29412) * support AbortSignal in fork (Benjamin Gruenbaum) (#36603) * crypto: * implement basic secure heap support (James M Snell) (#36779) * fixup bug in keygen error handling (James M Snell) (#36779) * introduce X509Certificate API (James M Snell) (#36804) * implement randomuuid (James M Snell) (#36729) * doc: * update release key for Danielle Adams (Danielle Adams) (#36793) * add dnlup to collaborators (Daniele Belardi) (#36849) * add panva to collaborators (Filip Skokan) (#36802) * add yashLadha to collaborator (Yash Ladha) (#36666) * http: * set lifo as the default scheduling strategy in Agent (Matteo Collina) (#36685) * net: * support abortSignal in server.listen (Nitzan Uziely) (#36623) * process: * add direct access to rss without iterating pages (Adrien Maret) (#34291) * v8: * fix native constructors (ExE Boss) (#36549)
PR-URL: nodejs#36839 Refs: nodejs#34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36839 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36839 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: James M Snell <[email protected]>
Accessing the rss value through memoryUsage() can be expensive because this method will also generate memory usage statistics by iterating on each page. This commit intend to offer a more direct access to rss value. Refs: #33384 PR-URL: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: James M Snell <[email protected]>
Refs: #34291 PR-URL: #36781 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Rich Trott <[email protected]>
PR-URL: #36768 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michael Dawson <[email protected]>
PR-URL: #36769 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36766 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36839 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: James M Snell <[email protected]>
PR-URL: #36768 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michael Dawson <[email protected]>
PR-URL: #36769 Refs: #34291 Reviewed-By: Gerhard Stöbich <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: James M Snell <[email protected]>
Accessing the rss value through
process.memoryUsage().rsscan be expensive because this method will also generate memory usage statistics by iterating on each page.The overhead can be quite huge when a lot of memory has been allocated, consider this snippet:
rss-access.js
This PR aim to offer an API to access directly the rss value in a (almost) constant time.
EDIT: 2020-07-13
Following @bnoordhuis proposal I added a
rssOnlyoption toprocess.memoryUsageinstead of adding the newprocess.rssmethod:EDIT: 2020-08-11
Following @jasnell and @bnoordhuis advices the rss value is now accessible through
process.memoryUsage.rssSee #33384
Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passes (Unix is ok but unfortunately I don't have a windows computer)