Cannot clone Linux kernel on Windows

[ader1990]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options
git version 2.20.1.windows.1
cpu: x86_64
built from commit: 7c9fbc07db0e2939b36095df45864b8cda19b64f
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.15063]

• What options did you set as part of the installation? Or did you choose the
  defaults?

Editor Option: VIM
Custom Editor Path:
Path Option: CmdTools
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: ConHost
Performance Tweaks FSCache: Enabled
Use Credential Manager: Disabled
Enable Symlinks: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

No.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

PowerShell or CMD, same issue

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git clone https://github.com/torvalds/linux

• What did you expect to occur after running these commands?

git clone to finish correctly (exit code 0)

• What actually happened instead?

error: unable to create file drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c: No such file or directory
error: unable to create file drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.h: No such file or directory
error: unable to create file include/soc/arc/aux.h: No such file or directory

echo $LASTEXITCODE
128

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

https://github.com/torvalds/linux

• The thing is this issue can be fixed, as touch.exe and ls.exe which comes with the git environment do not fail when creating the aforementioned files.

PS C:\linux> New-Item -Type File drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c
New-Item : Could not find file '\\.\aux'.
At line:1 char:1
+ New-Item -Type File drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (C:\linux...ubdev\i2c\aux.c:String) [New-Item], FileNotFoundException
    + FullyQualifiedErrorId : NewItemIOError,Microsoft.PowerShell.Commands.NewItemCommand

PS C:\linux> touch.exe drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c
PS C:\linux> ls drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c
ls : Cannot find path 'C:\linux\drivers\gpu\drm\nouveau\nvkm\subdev\i2c\aux.c' because it does not exist.
At line:1 char:1
+ ls drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\linux...ubdev\i2c\aux.c:String) [Get-ChildItem], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand

PS C:\linux> ls.exe drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c
drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c

-- ls.exe version

PS C:\> ls.exe --version
ls (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

-- touch.exe version

PS C:\> touch.exe --version
touch (GNU coreutils) 8.30
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Rubin, Arnold Robbins, Jim Kingdon,
David MacKenzie, and Randy Smith.

[ader1990]

I am not looking for workarounds, what I am trying to say is that this issue might get fixed if the fileopen api is the same as the one used by gnu touch or ls implementation.

Can someone give more insight on what is the definition of the open method used here:
https://github.com/git-for-windows/git/blob/master/tempfile.c#L138
If we can replace the open method with the gnu ls/touch implementation, that might be a step in solving this age old problem.

Thank you,
Adrian Vladu

[dscho]

aux is an illegal file name on Windows (but not in NTFS per se). So while some hacks exist in some applications to generate those file names, most Windows tools will have serious problems to access those files.

Therefore it is an intentional design of Git for Windows to not allow such file names.

Granted, you might be able to browbeat Git to check out those files, but you will run into trouble later anyway.

If you still insist that it should be a good idea to generate those illegal file names, I dare you to patch (and open a PR) compat/mingw.c (look for mingw_open() and mingw_fopen()) to optionally use the \\?\ hack that is also used for core.longpaths and that would allow such illegal file names (yes, it must be an option, opt-in even, because there is no way that I will switch this on by default because I know exactly who would have to deal with the tons of bug reports about files that Git checks out without complaining but that cannot be read or even be deleted and what were we thinking anyway).

[PhilipOakley]

@ader1990 If you can manage to create a method that will detect such files and hold them in abeyance as if they are in the index, but not in the worktree then that may be of benefit.

There is a (potential) corresponding issue for partial clones where a promised file isn't being released to the user though the oid/sha1 is known. At the moment there is a bit of a blind spot for such cases.

Maybe it's an alternat version of a smudge/clean filter that detects some special prefix that locall is used to cope with these forbidden file names (and maybe Linux D/F conflic names?)

[t-b]

@ader1990 I have a pre-commit hook checking for legal filenames on windows -> https://github.com/t-b/git-pre-commit-hook-windows-filenames. In case that helps.

[dscho]

There's some development on that front: https://github.com/davidwin/git/commit/f21c7af658a7bfeb11ecde569336c8ba6d9a9354

Feel free to work with David to drive this forward.

[dscho]

Closing due to obvious lack of interest.

[fdk17]

Is it possible to workaround the reserved files names mentioned in #1393 using sparse-checkouts?
For myself I can't figure out how to do this as it still complains about these files even if I use "clone -n" and try to setup files that shouldn't be a problem. My apologies if this isn't the best place to ask but this is the exact problem I'm having with checkout of the Linux kernel on Windows to just inspect some files.

[dscho]

Is it possible to workaround the reserved files names mentioned in #1393 using sparse-checkouts?

Absolutely.

For myself I can't figure out how to do this as it still complains about these files even if I use "clone -n" and try to setup files that shouldn't be a problem.

Wait, what? You cannot clone -n? Mind pasting the verbatim output of a problematic run?

[fdk17]

$ git clone -n git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
Cloning into 'linux'...
remote: Enumerating objects: 8334952, done.
remote: Counting objects: 100% (8334952/8334952), done.
remote: Compressing objects: 100% (1253242/1253242), done.
remote: Total 8334952 (delta 7059432), reused 8307363 (delta 7031856)
Receiving objects: 100% (8334952/8334952), 1.41 GiB | 1.51 MiB/s, done.
Resolving deltas: 100% (7059432/7059432), done.

$ git sparse-checkout init --cone
error: invalid path 'drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c'
error: invalid path 'drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c'

$ ls -A
.git/

$ git sparse-checkout set arch/powerpc/boot/dts/fsl
error: invalid path 'drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c'
error: invalid path 'drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c'

$ ls -A
.git/

$ git status | head -n 20
On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        deleted:    .clang-format
        deleted:    .cocciconfig
        deleted:    .get_maintainer.ignore
        deleted:    .gitattributes
        deleted:    .gitignore
        deleted:    .mailmap
        deleted:    COPYING
        deleted:    CREDITS
        deleted:    Documentation/.gitignore
        deleted:    Documentation/ABI/README
        deleted:    Documentation/ABI/obsolete/sysfs-bus-usb
        deleted:    Documentation/ABI/obsolete/sysfs-class-dax
        deleted:    Documentation/ABI/obsolete/sysfs-class-net-batman-adv
        deleted:    Documentation/ABI/obsolete/sysfs-class-net-mesh
        deleted:    Documentation/ABI/obsolete/sysfs-class-typec

[dscho]

So the clone actually works, the checkout fails (because the offending files haven't been excluded, nothing has been excluded, actually).

[fdk17]

I didn't intend to make it sound like clone -n didn't work but that excluding files are not working.

According to the documentation git sparse-checkout init --cone performs these actions:

Enable the core.sparseCheckout setting. If the sparse-checkout file does not exist, then populate it with patterns that match every file in the root directory and no other directories, then will remove all directories tracked by Git.

If I perform the git clone -n and git sparse-checkout init --cone on Linux then this is what I see.

$ cat .git/info/sparse-checkout
/*
!/*/

$ ls
COPYING  CREDITS  Kbuild  Kconfig  MAINTAINERS  Makefile  README

On Linux the file 'drivers/gpu/drm/nouveau/nvkm/subdev/i2c/aux.c' seems to be excluded.

https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/#using-sparse-checkout-with-an-existing-repository

Are these examples then incorrect on how to use spare-checkout? Can you offer some guidance on how to perform the exclusion of the proper files?

[dscho]

$ cat .git/info/sparse-checkout
/*
!/*/

$ ls
COPYING  CREDITS  Kbuild  Kconfig  MAINTAINERS  Makefile  README

It is good to see that this file was created, I must have missed that fact earlier. This looks good, the problem is that we do not even allow such files with invalid names to enter the Git index, I think.

When that patch was developed, we were under embargo and wanted to err on the side of safety: there are too many ways to write out files from the index, so it looked like a sensible thing to be strict even with the index.

In your case, it might be too strict. As a quick workaround to get you unblocked, you can set core.protectNTFS = false in that repository before checking out the files, but that will of course make you vulnerable against a whole slew of attack vectors. Use with care.

[fdk17]

@dscho Thanks for the help. On windows 10 I decided it was easier to use the Linux for Windows and to checkout the Linux kernel from within Ubuntu-18.04. Then from within Ubuntu-18.04 you can run 'explorer.exe .' and explorer will open up as "\wsl$\Ubuntu-18.04\home\fdk17\linux_sparse".

[ader1990]

@dscho Thanks for the help. On windows 10 I decided it was easier to use the Linux for Windows and to checkout the Linux kernel from within Ubuntu-18.04. Then from within Ubuntu-18.04 you can run 'explorer.exe .' and explorer will open up as "\wsl$\Ubuntu-18.04\home\fdk17\linux_sparse".

Another option would be to use msys2 and clone from there, afterwards you can use the folder from Windows. Example here how to install msys2: https://lazyprogrammer.io/entry/lazy-build-qemu-img-exe-statically-from-source

[linquize]

Due to AUX files, git clone --no-checkout

[mfide]

is this problem solved? why is it closed?

[dscho]

@mfide you probably missed #2046 (comment) while reading this thread (it is long, I admit), which details a way forward. If anybody is interested enough in fixing this, the information is all right there to work on it.