Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apply_command_response kills _autoTickThread #189

Closed
betanummeric opened this issue Feb 18, 2025 · 0 comments · Fixed by #190
Closed

apply_command_response kills _autoTickThread #189

betanummeric opened this issue Feb 18, 2025 · 0 comments · Fixed by #190

Comments

@betanummeric
Copy link
Contributor

I observed this exception with pysyncobj 0.3.13 in the auto tick thread, which killed it. The syncobj was unsable then because the tick thread was not running anymore.

Exception in thread Thread-8 (_autoTickThread):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/dist-packages/pysyncobj/syncobj.py", line 537, in _autoTickThread
    self._onTick(self.__conf.autoTickPeriod)
  File "/usr/local/lib/python3.11/dist-packages/pysyncobj/syncobj.py", line 646, in _onTick
    self._poller.poll(timeToWait)
  File "/usr/local/lib/python3.11/dist-packages/pysyncobj/poller.py", line 97, in poll
    self.__descrToCallbacks[descr](descr, eventMask)
  File "/usr/local/lib/python3.11/dist-packages/pysyncobj/tcp_connection.py", line 223, in __processConnection
    self.__onMessageReceived(message)
  File "/usr/local/lib/python3.11/dist-packages/pysyncobj/transport.py", line 108, in _onMessageReceived
    self._onMessageReceivedCallback(node, message)
  File "/usr/local/lib/python3.11/dist-packages/pysyncobj/syncobj.py", line 972, in __onMessageReceived
    assert idx > self.__raftLastApplied
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

This error seems to be a rare race condition. It happened directly after this syncobj transitioned from candidate to follower status (within the same second). Based on the origin of the exception it looks like this syncobj received a apply_command_response which it didn't like.
3 nodes participated in this raft in total, all pysyncobj 0.3.13.

I don't understand why this AssertionError appeared but I think it should not stop the auto tick thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant