Fix includes normalize win32

[digit-google]

Remove the use of fixed-size buffers in src/includes_normalize-win32.cc in order to support long file paths.

This also requires using GetFullPathNameW() as, surprisingly, GetFullPathNameA() would fail with long paths even when long paths are enabled on the host system running the tests.

To achieve this, introduce ConvertWin32UnicodeToUTF8() and ConvertUTF8ToWin32Unicode() functions to perform conversions properly in src/util.cc.

[jhasse]

Have you created a bug report for Microsoft? I think we should try to avoid working around bugs in Windows as much as possible, so that the underlying issue can be fixed and everyone benefits.

[digit-google]

So I actually tried to file a bug, which requires registering a Microsoft account (don't have one, don't want one), and from what I can gather on various Internet comments, essentially goes into a /dev/null bucket most of the time. Even if they fixed this, existing Vista / Windows 10 / whatever installs would never get the fix, so I think we'll have to just work-around this here.

[jhasse]

I think someone who wants to keep this moving probably has an account and could do it for us. Until then let's merge the #else version so that we are doing the correct thing and would immediately benefit after a Windows update.

[maxhmv]

@jhasse After reading the Maximum File Path Limitation documentation again from my understanding this section states that only the *W functions don't have any MAX_PATH limitations, which is in line with @digit-google observation that GetFullPathNameA() fails for long paths even when the system option is set in the registry and during build.

Therefore I don't understand how working with GetFullPathNameA() can fix the problem.

Thanks to the both of you for looking into this.

[jhasse]

The fact that only the W versions don't have the MAX_PATH limitation is a bug on Microsoft's side which I don't want to work around.

[maxhmv]

@jhasse While I agree with you about this being a bug on Microsoft's side (I would say the whole existence of a MAX_PATH in 2025 is a bug), i strongly doubt that this is something they will likely fix. The implications of such a change could be quite big from Microsoft's perspective, because potentially people will depend on the current (documented) behavior.

Just to clarify:

• You will not merge the PR as is but only the #else part with the *A version, and ninja wont support long path names under Windows for the time being, right? If this is the case, as a compromise would you accept a cmake option to switch the behavior?
• Would you merge the *W part if there is a bug report at Microsoft?

Thank you.

[digit-google]

It's even crazier than that:

• The W functions are the only documented APIs that support long paths, when the feature is enabled and the application has the right manifest attribute. See for yourself: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

• The fact that most A functions support them too when the mode is enabled is an implementation detail, that just happens to work most of the time. There is no guarantee that this won't break in the future (though Microsoft is very good at maintaining backwards compatibility). Maybe GetFullPathNameA() does not work because it returns an output path (while most other functions take an input path). Who knows?

• Ninja relying on A functions for long paths support could be considered a bug in Ninja itself. The correct fix is to only use W functions for all file path-related operations.

I would not be surprised if a bug filed for this "issue" would be closed as "working as intended". We can try, but I don't have high hopes it will work.

[jhasse]

You will not merge the PR as is but only the #else part with the *A version, and ninja wont support long path names under Windows for the time being, right?

Well, not for the normalization of long paths at least. There is a subset of features where we support long paths already.

If this is the case, as a compromise would you accept a cmake option to switch the behavior?

No, if someone builds from source anyway, they can checkout one of the PRs that use the *W function.

Would you merge the *W part if there is a bug report at Microsoft?

That depends on Microsoft's answer.

There is no guarantee that this won't break in the future (though Microsoft is very good at maintaining backwards compatibility).

The best thing to make them not break it, is to rely on it ;)

Ninja relying on A functions for long paths support could be considered a bug in Ninja itself.

We're using A functions for UTF-8, not for long path support.

[sztomi]

As a data point, I managed to build a large Qt codebase that previously failed to compile because of moc-generated long relative include paths; it works with the ninja from this branch.

[sztomi]

@jhasse To also reflect on the discussion above: from its inception, long path support was documented to only be supported with the *W APIs. The fact that it's not ergonomic/surprising that the *A versions don't support it does not make it a bug. It's unlikely that Microsoft will act on (or respond to) this. Converting paths to wide chars is the correct thing to do on Windows and not a workaround.

[jhasse]

The fact that Microsoft documented their bug does not make it not a bug.

[sztomi]

@jhasse On the other hand (and I say this with no malice), your judgement does not make it a bug. It is not documented as a bug, there is simply a list of functions on MSDN that support long paths if the setting is enabled in the registry and the application manifest, and they happen to be the *W family of functions only.

Whether or not it is a bug, boils down to what the intention of Microsoft was while introducing long paths. It is a valid choice to only add a new feature to the *W APIs (even if we, not being aware of the entire implementation constraints feel like we would have chosen differently).

[jhasse]

Well, I disagree.