Skip to content

#336: fix hanging git subprocess in case of too much output on stdout/stderr. #396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 21, 2018

Conversation

andreas-eternach
Copy link
Contributor

@andreas-eternach andreas-eternach commented Nov 5, 2018

Context

#336: fix hanging git subprocess in case of too much output on stdout/stderr.

The previous implementation read the output after the subprocess finished. In case the buffer is full, the subprocess waits infinitly.

Therefore the new implementation flushes the subprocess buffer by reading from stdout/stderr on background threads.

Contributor Checklist

  • Added relevant integration or unit tests to verify the changes
  • Update the Readme or any other documentation (including relevant Javadoc)
  • Ensured that tests pass locally: mvn clean package
  • Ensured that the code meets the current checkstyle coding style definition: mvn clean verify -Pcheckstyle -Dmaven.test.skip=true -B

@andreas-eternach
Copy link
Contributor Author

i see that the conflict relates to a new timeout-setting for the native-git-provider.

@andreas-eternach
Copy link
Contributor Author

andreas-eternach commented Nov 6, 2018

I fixed the conflicts locally, there came up two more questions when doing that:

  • the timeout did not fix the hanging process, because even after the timeout the streams where completely flushed (depending on the stream which caused the overflow, because the implementation processes only one of them, not in parallel)
  • the new implementation is still not completely safe, there is a risk of an OutOfMemoryException in case the changes are really huge (larger than the heap).

Nevertheless i would recommend to merge the changes.

@TheSnoozer
Copy link
Collaborator

Thanks for creating a PR for that (greatly appreciated)!
I#ll try to review those changes on the weekend (sorry for the delay).

@TheSnoozer TheSnoozer added this to the 3.0 milestone Nov 8, 2018
@TheSnoozer
Copy link
Collaborator

Hi,
thanks again for the merge request and sorry for the delay on the review.
It honestly took me some time to review the changes, since it seems that you just moved some methods around without any actual changes.
I find it hard to read diffs where most of the stuff was just moved to some other place without any actual change (sorry, but its just super hard). For better readability I condensed the current changes down to this that IMHO creates a better visibility what has changed exactly.

The BigDiff Tests passes, but that passed manually on my side before any changes. From my perspective the only thing I currently would ask for is to condense down the current MR to either the version I created (hopefully i didn't miss anything), or achieve the same in any other shape or form. Regardless on how you want to proceed I think it should also be verified that the bug is gone (that's the entire goal here). Besides that the changes seem reasonable and thus i would be to happy to merge the condensed form (where one can see in the diff what changed).

@andreas-eternach
Copy link
Contributor Author

Indeed the methods have been rearranged by my IDE automatically, this is a setting i am using for other projects, which is somehow not suitable for multi-peer-projects. I am sorry for that.

About the BigDiffTest: its also not reliable reproducing the issue on my machine. In 1 of 4 cases it passes with the old implementation. Not sure why, i assume that the JDK dynamically allocates pipe-sizes.

I try to look into your condensed patch later on this week. BR

@TheSnoozer
Copy link
Collaborator

The auto-formatting changes happen for me all the time too....so nothing to be sorry about it happens :-)

Interesting, let me know how you would like to proceed and in particular if the final patch (whatever we conclude is the right thing) works as expected.

…shed. In case the buffer is full, the subprocess waits infinitly.

Therefore the prototype implementation flushes the subprocess buffer by reading from stdout/stderr on background threads.
@andreas-eternach
Copy link
Contributor Author

Hello again, i fixed the test case, you could probably not repdroduce because it used jgit instead of native git. This change probably happened when i merged HEAD into the PR.

I now reverted the PR via a force-push, actually using your cleaned patch, only additional with the BigDiffTest, which i enabled and which should pass now.

outGobbleThread.start();
errorGobbleThread.start();
if (!proc.waitFor(nativeGitTimeoutInMs, TimeUnit.MILLISECONDS)) {
proc.destroy();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!
I was somehow under the impression that proc.waitFor throws an exception when the timeout has been reached, but that doesn't seem to be the case (interesting)...

I'm wondering if we also would need to terminate the threads in weird conditions...I would assume once the process is killed the input / output stream would be empty and then the threads would automatically terminate...mhhh

@TheSnoozer TheSnoozer merged commit c618ceb into git-commit-id:master Nov 21, 2018
TheSnoozer pushed a commit to TheSnoozer/git-commit-id-maven-plugin that referenced this pull request Nov 21, 2018
TheSnoozer added a commit that referenced this pull request Nov 26, 2018
#336 / #396: apply some refactoring to simplify the entire process launching
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants