fs: optimize `fs.cpSync` js calls by anonrig · Pull Request #53614 · nodejs/node

anonrig · 2024-06-27T18:38:31Z

#53612 is required to do a benchmark

A different approach to #53541

cc @nodejs/performance @nodejs/cpp-reviewers @billywhizz @jasnell @nodejs/fs

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1573/

Small win on happy path, and probably high win on error path:

                           confidence improvement accuracy (*)   (**)   (***)
fs/bench-cpSync.js n=1            ***     21.96 %       ±6.85% ±9.14% ±11.97%
fs/bench-cpSync.js n=100          ***     14.91 %       ±2.86% ±3.81%  ±4.97%
fs/bench-cpSync.js n=10000        ***      9.74 %       ±1.28% ±1.71%  ±2.23%

anonrig · 2024-06-27T21:32:11Z

cc @nodejs/tsc @nodejs/fs appreciate your input on which path to take.

previous pr:

is a sem-ver major
introduces 2 different implementations (js only and cpp)
had limitations due to std::filesystem::copy

this pr:

is not a sem-ver major
optimizes all flags passed to cpSync
optimizes error cases as same as the previous pr
has a lot more optimization to do

mcollina

lgtm

nodejs-github-bot · 2024-07-08T18:29:44Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60175/

nodejs-github-bot · 2024-07-09T06:01:34Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60191/

nodejs-github-bot · 2024-07-09T06:01:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60190/

nodejs-github-bot · 2024-07-11T19:04:07Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60257/

nodejs-github-bot · 2024-07-12T10:48:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60274/

nodejs-github-bot · 2024-07-12T15:29:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60277/

nodejs-github-bot · 2024-07-16T19:58:32Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60366/

aduh95 · 2024-07-16T21:58:04Z

Windows compilation is failing:

C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\mksnapshot.vcxproj]
  embedded-file-writer.cc
  mksnapshot.cc
  embedded-empty.cc
  platform-embedded-file-writer-base.cc
  platform-embedded-file-writer-generic.cc
  platform-embedded-file-writer-win.cc
  platform-embedded-file-writer-aix.cc
  platform-embedded-file-writer-mac.cc
  snapshot-empty.cc
  static-roots-gen.cc
     Creating library ..\..\out\Debug\mksnapshot.lib and object ..\..\out\Debug\mksnapshot.exp
  mksnapshot.vcxproj -> ..\..\out\Debug\\mksnapshot.exe
  generating: "..\..\out\Debug\obj\v8_snapshot\/snapshot.cc" "..\..\out\Debug\obj\v8_snapshot\/embedded.S"
  Assembling ..\..\out\Debug\obj\v8_snapshot\\embedded.S...
  setup-isolate-deserialize.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
  snapshot.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
  v8_snapshot.vcxproj -> ..\..\out\Debug\lib\v8_snapshot.lib

anonrig · 2024-07-17T00:35:24Z

@nodejs/platform-windows any idea why this build is failing?

H4ad · 2024-07-17T01:43:14Z

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
 'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

StefanStojanovic · 2024-07-17T06:31:13Z

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
 'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

anonrig · 2024-07-21T21:19:19Z

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

It seems I forgot a src_path.c_str() there. Thank you! @StefanStojanovic @H4ad

nodejs-github-bot · 2024-07-21T23:41:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60519/

nodejs-github-bot · 2024-07-22T06:15:53Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60528/

jasnell

Still LGTM

nodejs-github-bot · 2024-07-22T16:00:41Z

Landed in 88027e8

PR-URL: #53614 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

joyeecheung · 2024-08-22T09:10:08Z

src/node_file.cc

+                        ? std::filesystem::symlink_status(src_path, error_code)
+                        : std::filesystem::status(src_path, error_code);
+  if (error_code) {
+    return env->ThrowUVException(EEXIST, "lstat", nullptr, src.out());


This looks wrong, the error code should be properly translated instead of being directly mapped to EEXIST, there are many other reasons why this could fail.

joyeecheung · 2024-08-22T09:20:52Z

src/node_file.cc

+  ToNamespacedPath(env, &src);
+  THROW_IF_INSUFFICIENT_PERMISSIONS(
+      env, permission::PermissionScope::kFileSystemRead, src.ToStringView());
+  auto src_path = std::filesystem::path(src.ToStringView());


BufferValue converts the string to UTF8, on Windows this requires automatic conversion for char8_t to work which may not be supported by the version of MSVC we use, I think that may be the source of bugs like #54476

Actually no, this is simply broken on Windows, because this just gets returned in char. It needs to be returned as std::u8string/std::u8string_view instead.

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jun 27, 2024

anonrig mentioned this pull request Jun 27, 2024

fs: improve cpSync performance #53541

Closed

anonrig force-pushed the optimize-cp-sync branch from 02e0883 to d7d4ae8 Compare June 27, 2024 18:44

anonrig requested a review from lemire June 27, 2024 18:44

anonrig force-pushed the optimize-cp-sync branch 2 times, most recently from bebf131 to 2d5fb50 Compare June 27, 2024 21:23

anonrig force-pushed the optimize-cp-sync branch from 2d5fb50 to 6a216bd Compare June 27, 2024 21:33

This comment was marked as outdated.

Sign in to view

mcollina approved these changes Jun 28, 2024

View reviewed changes

benjamingr approved these changes Jun 28, 2024

View reviewed changes

anonrig force-pushed the optimize-cp-sync branch from 6a216bd to 371d7a7 Compare July 11, 2024 18:54

anonrig force-pushed the optimize-cp-sync branch from 371d7a7 to e2c27ae Compare July 21, 2024 21:16

anonrig requested a review from jasnell July 21, 2024 21:17

This comment was marked as outdated.

Sign in to view

jasnell approved these changes Jul 21, 2024

View reviewed changes

anonrig added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jul 21, 2024

fs: optimize fs.cpSync js calls

a590e84

anonrig force-pushed the optimize-cp-sync branch from e2c27ae to a590e84 Compare July 21, 2024 23:40

jasnell approved these changes Jul 22, 2024

View reviewed changes

anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024

nodejs-github-bot merged commit 88027e8 into nodejs:main Jul 22, 2024

targos pushed a commit that referenced this pull request Jul 30, 2024

fs: optimize fs.cpSync js calls

1b0d5d4

PR-URL: #53614 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

RafaelGSS mentioned this pull request Jul 30, 2024

v22.6.0 proposal #54123

Merged

RafaelGSS pushed a commit that referenced this pull request Aug 5, 2024

fs: optimize fs.cpSync js calls

5e03c17

PR-URL: #53614 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: James M Snell <[email protected]>

joyeecheung reviewed Aug 22, 2024

View reviewed changes

joyeecheung mentioned this pull request Aug 22, 2024

v22.6.0 fs.cpSync crash #54476

Closed

markandrus mentioned this pull request Sep 11, 2024

[Bug?]: fsBinding.cpSyncCheckPaths error in Node 22 yarnpkg/berry#6488

Open

1 task

jazelly mentioned this pull request Sep 12, 2024

fs: translate error code properly in cpSync #54906

Merged

joyeecheung mentioned this pull request Sep 20, 2024

src: revert filesystem::path changes #55015

Merged

targos added the dont-land-on-v20.x PRs that should not land on the v20.x-staging branch and should not be released in v20.x. label Sep 21, 2024

Uh oh!

Conversation

anonrig commented Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anonrig commented Jun 27, 2024

Uh oh!

This comment was marked as outdated.

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 8, 2024

Uh oh!

nodejs-github-bot commented Jul 9, 2024

Uh oh!

nodejs-github-bot commented Jul 9, 2024

Uh oh!

nodejs-github-bot commented Jul 11, 2024

Uh oh!

nodejs-github-bot commented Jul 12, 2024

Uh oh!

nodejs-github-bot commented Jul 12, 2024

Uh oh!

nodejs-github-bot commented Jul 16, 2024

Uh oh!

aduh95 commented Jul 16, 2024

Uh oh!

anonrig commented Jul 17, 2024

Uh oh!

H4ad commented Jul 17, 2024

Uh oh!

StefanStojanovic commented Jul 17, 2024

Uh oh!

This comment was marked as outdated.

anonrig commented Jul 21, 2024

Uh oh!

nodejs-github-bot commented Jul 21, 2024

Uh oh!

nodejs-github-bot commented Jul 22, 2024

Uh oh!

jasnell left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 22, 2024

Uh oh!

joyeecheung Aug 22, 2024

Choose a reason for hiding this comment

Uh oh!

joyeecheung Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joyeecheung Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

anonrig commented Jun 27, 2024 •

edited

Loading

joyeecheung Aug 22, 2024 •

edited

Loading

joyeecheung Aug 22, 2024 •

edited

Loading