Skip to content
This repository was archived by the owner on Nov 11, 2022. It is now read-only.

FileBasedSource does not work on Windows #2

Closed
jaroy opened this issue Dec 23, 2014 · 3 comments
Closed

FileBasedSource does not work on Windows #2

jaroy opened this issue Dec 23, 2014 · 3 comments
Assignees
Labels

Comments

@jaroy
Copy link

jaroy commented Dec 23, 2014

Apologies if this is not the right place to submit issues.

The following code

Path path = Files.createTempFile("test", ".txt");
Pipeline pipeline = Pipeline.create(PipelineOptionsFactory.create());
pipeline.apply(TextIO.Read.from(path.toString()));
pipeline.run();

throws an exception on a Windows client:

java.lang.RuntimeException: Failed to read from source: com.google.cloud.dataflow.sdk.runners.worker.TextSource@644baf4a
...
Caused by: java.io.IOException: No match for file pattern 'C:\Users\jroy\AppData\Local\Temp\test8262931969830113037.txt'
at com.google.cloud.dataflow.sdk.runners.worker.FileBasedSource.iterator(FileBasedSource.java:100)

The file exists, so this exception is unexpected.

The bug is in FileIOChannelFactory::match(), which improperly appends the value returned by File::getAbsolutePath() to a glob expression. On Windows, getAbsolutePath() returns the path with backslash path separators, which are interpreted as escape characters by FileSystem::getPathMatcher(). So no matches are returned.

@davorbonaci
Copy link
Contributor

Thanks for the bug report. Yes, this is the right place for these kinds of things.

The issue is understood. We'll try to address it quickly. Stay tuned!

@davorbonaci
Copy link
Contributor

I wanted to follow up that we have addressed several Windows-specific issues in the codebase, including this one and a few separate unit test failures.

We'd appreciate reports of any additional issues that may come up. Thanks!

@jaroy
Copy link
Author

jaroy commented Jan 15, 2015

Verified fixed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants