Skip to content

Commit 6025433

Browse files
geertjgeertj
committedJun 27, 2015
Break out SkipList into separate module.
0 parents  commit 6025433

26 files changed

+2340
-0
lines changed
 

‎.gitignore

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
build/
2+
dist/
3+
*.egg-info
4+
*.pyc
5+
*.pyo
6+
.*.swp
7+
.*.swo
8+
*~
9+
__pycache__
10+
docs/_build
11+
docs/html
12+
.tox
13+
tests/memory.txt
14+
tests/performance.txt
15+
.coverage
16+
htmlcov
17+
README.html

‎.travis.yml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
language: python
2+
python: 3.3
3+
env:
4+
- TOX_ENV=py27
5+
- TOX_ENV=py33
6+
- TOX_ENV=py34
7+
- TOX_ENV=docs
8+
- TOX_ENV=flake8
9+
- TOX_ENV=coverage
10+
install:
11+
- pip install tox
12+
script:
13+
- tox -e $TOX_ENV
14+
- if [ $TOX_ENV = "coverage" ]; then pip install coveralls; coveralls; fi

‎AUTHORS

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* Geert Jansen <geertj@gmail.com>

‎LICENSE

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
Copyright (c) 2012-2015 the PySkipList authors.
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy
4+
of this software and associated documentation files (the "Software"), to deal
5+
in the Software without restriction, including without limitation the rights
6+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
copies of the Software, and to permit persons to whom the Software is
8+
furnished to do so, subject to the following conditions:
9+
10+
The above copyright notice and this permission notice shall be included in
11+
all copies or substantial portions of the Software.
12+
13+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
19+
THE SOFTWARE.

‎README.rst

+122
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
Welcome to PySkipList
2+
=====================
3+
4+
PySkipList is a fast, pure Python implementation of an indexable skiplist. It
5+
implements a ``SkipList`` data structure that provides an always sorted,
6+
list-like data structure for (key, value) pairs. It efficiently supports the
7+
following operations:
8+
9+
* Insert a pair in the list, maintaining sorted order.
10+
* Find the value of a given key.
11+
* Remove a given pair based on a key.
12+
* Iterate over all pairs in sorted order.
13+
* Find the position of a given key.
14+
* Access a pair at a certain position.
15+
* Delete a pair at a certain position.
16+
17+
Since PySkipList is a pure Python implementation, it should work well on
18+
alternative Python implementations such as PyPy and Jython.
19+
20+
21+
Example
22+
=======
23+
24+
The following provides a few examples on how to use the ``SkipList`` API::
25+
26+
>>> from pyskiplist import SkipList
27+
>>> sl = SkipList()
28+
>>> sl.insert('foo', 'bar')
29+
>>> sl.insert('baz', 'qux')
30+
>>> sl
31+
SkipList((('baz', 'qux'), ('foo', 'bar')))
32+
>>> sl.search('foo')
33+
'bar'
34+
>>> sl[0]
35+
('baz', 'qux')
36+
>>> sl.remove('foo') # remove by key
37+
>>> del sl[0] # remove by position
38+
39+
40+
Asymptotic Complexity
41+
=====================
42+
43+
Below are the Big-O complexities of the various operations implemented by
44+
pyskiplist:
45+
46+
================== ==========
47+
Operation Complexity
48+
================== ==========
49+
insertion O(log N)
50+
search by key O(log N)
51+
removal by key O(log N)
52+
forward iteration O(1)
53+
find by position O(log N)
54+
access by position O(log N)
55+
delete by position O(log N)
56+
================== ==========
57+
58+
59+
Performance
60+
===========
61+
62+
Below are the results of some performance tests. These are for Python 3.4.2 on
63+
my Linux laptop:
64+
65+
=================== ===================
66+
Test Operations / second
67+
=================== ===================
68+
Insert @ 1k nodes 45,056
69+
Insert @ 10k nodes 42,137
70+
Insert @ 100k nodes 28,086
71+
Remove @ 1k nodes 54,316
72+
Remove @ 10k nodes 46,240
73+
Remove @ 100k nodes 35,114
74+
Search @ 1k nodes 137,248
75+
Search @ 10k nodes 109,480
76+
Search @ 100k nodes 77,939
77+
=================== ===================
78+
79+
80+
Memory usage
81+
============
82+
83+
PySkipList tries to be efficient with regards to memory usage. The numbers
84+
below are for Python 3.4.2 on my Linux laptop. This specific test stores pairs
85+
of integer keys and an integer values in a skiplist. The total size of the two
86+
integers on this Python version is 56 bytes.
87+
88+
===== ============ ===============
89+
Nodes Bytes / node Overhead / node
90+
===== ============ ===============
91+
1k 164 108
92+
10k 162 106
93+
100k 162 106
94+
===== ============ ===============
95+
96+
97+
Implementation notes
98+
====================
99+
100+
Reference papers on skiplists:
101+
102+
* ftp://ftp.cs.umd.edu/pub/skipLists/skiplists.pdf (original paper)
103+
* http://drum.lib.umd.edu/bitstream/1903/544/2/CS-TR-2286.1.pdf (cookbook)
104+
105+
This implementation uses a novel (as far as I know) technique where it stores
106+
just a single link width per node, and only in nodes with level > 0. The link
107+
corresponds to the number of nodes skipped by the highest incoming link. Other
108+
implementations that I've seen all store a width for every link. The approach
109+
taken here saves a lot of memory. The overhead should just be 1/e (0.37)
110+
integer per node.
111+
112+
Duplicate keys are allowed in this implementation, and insertion order is
113+
maintained. That said, no special API support is provided for working with
114+
duplicate keys. For example, the `remove()` function removes the first pair it
115+
encounters with the given key, irrespective of the key's value.
116+
117+
Skiplist nodes are plain lists instead of objects. This saves memory. Kudos to
118+
http://pythonsweetness.tumblr.com/post/45227295342 for the idea.
119+
120+
The built-in Mersenne Twister is used as the random number source. This is
121+
preferable over SystemRandom since it doesn't require a system call and there
122+
is no need for cryptographically secure numbers.

‎dev-requirements.txt

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
-r requirements.txt
2+
sphinx
3+
wheel
4+
flake8
5+
tox
6+
detox
7+
invoke
8+
coverage
9+
coveralls

‎docs/Makefile

+192
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# Makefile for Sphinx documentation
2+
#
3+
4+
# You can set these variables from the command line.
5+
SPHINXOPTS =
6+
SPHINXBUILD = sphinx-build
7+
PAPER =
8+
BUILDDIR = _build
9+
10+
# User-friendly check for sphinx-build
11+
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
12+
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
13+
endif
14+
15+
# Internal variables.
16+
PAPEROPT_a4 = -D latex_paper_size=a4
17+
PAPEROPT_letter = -D latex_paper_size=letter
18+
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
19+
# the i18n builder cannot share the environment and doctrees with the others
20+
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
21+
22+
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
23+
24+
help:
25+
@echo "Please use \`make <target>' where <target> is one of"
26+
@echo " html to make standalone HTML files"
27+
@echo " dirhtml to make HTML files named index.html in directories"
28+
@echo " singlehtml to make a single large HTML file"
29+
@echo " pickle to make pickle files"
30+
@echo " json to make JSON files"
31+
@echo " htmlhelp to make HTML files and a HTML help project"
32+
@echo " qthelp to make HTML files and a qthelp project"
33+
@echo " applehelp to make an Apple Help Book"
34+
@echo " devhelp to make HTML files and a Devhelp project"
35+
@echo " epub to make an epub"
36+
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
37+
@echo " latexpdf to make LaTeX files and run them through pdflatex"
38+
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
39+
@echo " text to make text files"
40+
@echo " man to make manual pages"
41+
@echo " texinfo to make Texinfo files"
42+
@echo " info to make Texinfo files and run them through makeinfo"
43+
@echo " gettext to make PO message catalogs"
44+
@echo " changes to make an overview of all changed/added/deprecated items"
45+
@echo " xml to make Docutils-native XML files"
46+
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
47+
@echo " linkcheck to check all external links for integrity"
48+
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
49+
@echo " coverage to run coverage check of the documentation (if enabled)"
50+
51+
clean:
52+
rm -rf $(BUILDDIR)/*
53+
54+
html:
55+
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
56+
@echo
57+
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
58+
59+
dirhtml:
60+
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
61+
@echo
62+
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
63+
64+
singlehtml:
65+
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
66+
@echo
67+
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
68+
69+
pickle:
70+
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
71+
@echo
72+
@echo "Build finished; now you can process the pickle files."
73+
74+
json:
75+
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
76+
@echo
77+
@echo "Build finished; now you can process the JSON files."
78+
79+
htmlhelp:
80+
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
81+
@echo
82+
@echo "Build finished; now you can run HTML Help Workshop with the" \
83+
".hhp project file in $(BUILDDIR)/htmlhelp."
84+
85+
qthelp:
86+
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
87+
@echo
88+
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
89+
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
90+
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/PySkipList.qhcp"
91+
@echo "To view the help file:"
92+
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/PySkipList.qhc"
93+
94+
applehelp:
95+
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
96+
@echo
97+
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
98+
@echo "N.B. You won't be able to view it unless you put it in" \
99+
"~/Library/Documentation/Help or install it in your application" \
100+
"bundle."
101+
102+
devhelp:
103+
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
104+
@echo
105+
@echo "Build finished."
106+
@echo "To view the help file:"
107+
@echo "# mkdir -p $$HOME/.local/share/devhelp/PySkipList"
108+
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/PySkipList"
109+
@echo "# devhelp"
110+
111+
epub:
112+
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
113+
@echo
114+
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
115+
116+
latex:
117+
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
118+
@echo
119+
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
120+
@echo "Run \`make' in that directory to run these through (pdf)latex" \
121+
"(use \`make latexpdf' here to do that automatically)."
122+
123+
latexpdf:
124+
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
125+
@echo "Running LaTeX files through pdflatex..."
126+
$(MAKE) -C $(BUILDDIR)/latex all-pdf
127+
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
128+
129+
latexpdfja:
130+
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
131+
@echo "Running LaTeX files through platex and dvipdfmx..."
132+
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
133+
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
134+
135+
text:
136+
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
137+
@echo
138+
@echo "Build finished. The text files are in $(BUILDDIR)/text."
139+
140+
man:
141+
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
142+
@echo
143+
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
144+
145+
texinfo:
146+
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
147+
@echo
148+
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
149+
@echo "Run \`make' in that directory to run these through makeinfo" \
150+
"(use \`make info' here to do that automatically)."
151+
152+
info:
153+
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
154+
@echo "Running Texinfo files through makeinfo..."
155+
make -C $(BUILDDIR)/texinfo info
156+
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
157+
158+
gettext:
159+
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
160+
@echo
161+
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
162+
163+
changes:
164+
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
165+
@echo
166+
@echo "The overview file is in $(BUILDDIR)/changes."
167+
168+
linkcheck:
169+
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
170+
@echo
171+
@echo "Link check complete; look for any errors in the above output " \
172+
"or in $(BUILDDIR)/linkcheck/output.txt."
173+
174+
doctest:
175+
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
176+
@echo "Testing of doctests in the sources finished, look at the " \
177+
"results in $(BUILDDIR)/doctest/output.txt."
178+
179+
coverage:
180+
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
181+
@echo "Testing of coverage in the sources finished, look at the " \
182+
"results in $(BUILDDIR)/coverage/python.txt."
183+
184+
xml:
185+
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
186+
@echo
187+
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
188+
189+
pseudoxml:
190+
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
191+
@echo
192+
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

0 commit comments

Comments
 (0)
Please sign in to comment.