|
1 | | -# Whispers 2.0.0 release notes |
2 | | - |
3 | | -## :dizzy: Licensing changes :dizzy: |
4 | | - |
5 | | -Version 1 was developed and open sourced by [Artёm Tsvetkov](https://github.com/adeptex) at [Skyscanner](https://github.com/Skyscanner/whispers) under [Apache License 2.0](https://github.com/Skyscanner/whispers/blob/master/LICENSE), which states that `licensed works, modifications, and larger works may be distributed under different terms and without source code.` |
6 | | - |
7 | | -Version 2 is an independent continuation of my work, which is now released under [GNU General Public License v3.0](https://github.com/adeptex/whispers/blob/master/LICENSE) that is `intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users.` |
8 | | - |
| 1 | +# Whispers 2.1.0 release notes |
9 | 2 |
|
10 | 3 | ## :x: Breaking changes :x: |
11 | 4 |
|
12 | | -### :x: Integration :x: |
13 | | -In version 1, Python integration required multiple imports and a correctly-formatted list of values ([ref](https://github.com/Skyscanner/whispers#python)). |
14 | | - |
15 | | -In version 2, the integration is simplified to a single import and a string of CLI arguments. The following example illustrates current Python integration: |
16 | | - |
17 | | -```py |
18 | | -import whispers |
19 | | - |
20 | | -args = ( |
21 | | - "-c whispers/config.yml " |
22 | | - "-r apikey,aws-secret,password " |
23 | | - "-s BLOCKER,CRITICAL,MAJOR " |
24 | | - "tests/fixtures" |
25 | | -) |
26 | | - |
27 | | -for secret in whispers.secrets(args): |
28 | | - print(f"[{secret.file}:{secret.line}] {secret.key} = {secret.value}") |
29 | | -``` |
30 | | - |
31 | | -### :x: File exclusion globs are now regex :x: |
32 | | -In version 1, the configuration file expected file exclusion specification to be a list of globs. Whispers would then resolve included globs, resolve excluded globs, and finally subtract the two lists to get applicable scope. The entire target directory tree would be traversed twice to compute applicable files (highly resource-intensive operation!) |
33 | | - |
34 | | -In version 2, file exclusions are specified as regex. Instead of resolving globs, Whispers now uses the generator directly. Every file path received from the glob generator is now checked against the file exclusion regex to determine whether the file should be excluded on-the-fly. |
35 | | - |
36 | | -This highly improves performance for cases where the target directory contains a large number of files. In version 2 the tree is traversed file by file, individually checking if the file path matches a pre-compiled exclusion regex. This decreases CPU, RAM and time needed to scan directories of potentially unlimited trees and depths. |
37 | | - |
38 | | - |
39 | | -### :x: Rule names and severity levels :x: |
40 | | -Rule names and severity levels were adapted to make usage more intuitive, and results more useful. The following is the exhaustive list of default rules included in version 2: |
41 | | - |
42 | | -| Group | Rule ID | Severity | |
43 | | -|----------------------|----------------------|---------------------| |
44 | | -| files | file-known | MINOR | |
45 | | -| infra | dockercfg | CRITICAL | |
46 | | -| infra | htpasswd | MAJOR | |
47 | | -| infra | npmrc | CRITICAL | |
48 | | -| infra | pip | CRITICAL | |
49 | | -| infra | pypirc | CRITICAL | |
50 | | -| keys | apikey | MAJOR | |
51 | | -| keys | apikey-known | CRITICAL | |
52 | | -| keys | aws-id | BLOCKER | |
53 | | -| keys | aws-secret | BLOCKER | |
54 | | -| keys | aws-token | BLOCKER | |
55 | | -| keys | privatekey | CRITICAL | |
56 | | -| misc | comment | INFO | |
57 | | -| misc | creditcard | MINOR | |
58 | | -| misc | secret | MINOR | |
59 | | -| misc | webhook | MINOR | |
60 | | -| passwords | password | CRITICAL | |
61 | | -| passwords | uri | CRITICAL | |
62 | | -| python | cors | MINOR | |
63 | | -| python | system | MINOR | |
64 | | - |
65 | | -Make sure to review and adapt your integration accordingly. |
66 | | - |
67 | | - |
68 | | -### :x: Rule specification format changes :x: |
69 | | -In version 1 the rules were defined as a dictionary with rule ID as the key and rule config as the value. This created awkward parsing practices and unintuitive code. For example: |
70 | | -```yaml |
71 | | -npmrc: |
72 | | - description: Hardcoded .npmrc authToken |
73 | | - message: .npmrc authToken |
74 | | - severity: CRITICAL |
75 | | - key: |
76 | | - regex: ^npm authToken$ |
77 | | - ignorecase: False |
78 | | -``` |
79 | | -
|
80 | | -In version 2 the rules are defined as a list of dictionaries. The rule ID now has its own `id` key inside the rule config definition. For example: |
81 | | -```yaml |
82 | | -- id: npmrc |
83 | | - group: infra |
84 | | - description: Hardcoded .npmrc authToken |
85 | | - message: .npmrc authToken |
86 | | - severity: CRITICAL |
87 | | - key: |
88 | | - regex: ^npm authToken$ |
89 | | - ignorecase: False |
90 | | -``` |
91 | | - |
92 | | -If you have any custom rule definitions, you will have to adjust them for migrating to version 2. |
93 | | - |
94 | | -There is an additional new `groups` parameter that can be used to group rules. |
95 | | - |
96 | | - |
97 | | -### :x: Rule naming :x: |
| 5 | +### :x: Arguments :x: |
98 | 6 |
|
99 | | -Some rules IDs were changed to gain a consitent naming format, please [refer to rules](whispers/rules). |
| 7 | +Several arguments have been modified and/or adapted to improve usability. |
100 | 8 |
|
| 9 | +- Human readable output is shown in logs (2.1), `-H` and `--human` (2.0) are removed. |
101 | 10 |
|
102 | | -### :x: Output file format :x: |
103 | | -In version 1 the output file was written in YAML with awkward indexing, which made results unusable. |
| 11 | +- Version can be shown with `--version` (2.1), `-v` (2.0) is removed. |
104 | 12 |
|
105 | | -In version 2 the same JSON output as `stdout` is written to the output file, making it easier to parse. |
| 13 | +- Extended help can be shown with `--info` (2.1), `-i` (2.0) is removed. |
106 | 14 |
|
| 15 | +- Debug mode can be enabled with `--debug` (2.1), `-d` (2.0) is removed. |
107 | 16 |
|
108 | | -### :x: Log file :x: |
109 | | -In version 1, `whispers.log` is always created in the same directory from which Whispers was executed. The log file remains after execution. |
| 17 | +- Logs can be redirected to a file with `--log log.txt` (2.1), constant `/tmp/whispers.log` (2.0) is removed. |
110 | 18 |
|
111 | | -In version 2, the log file will not be created by default, unless explicitly enabled with an argument: `whispers --log src`. The log is only useful for reviewing exceptions and bugs, not for common usage. In addition, the log will now be written to `/tmp/whispers.log` (Posix) or `%TEMP%\whispers.log` (Windows) so that it does not interfere with analysis or permissions. |
| 19 | +- Configuration template can be created with `--init` (2.1), `--print_config` (2.0) is removed. |
112 | 20 |
|
113 | | -Together, `--log` and `--debug` can be used to investigate exceptions and bugs. Please [submit a bug report](issues/new) if you find something unexpected! |
114 | 21 |
|
| 22 | +### :x: Logging :x: |
115 | 23 |
|
116 | | -### :x: Removed support for dynamic languages :x: |
117 | | -In version 1 the following language files were parsed as text and checked for common variable declaration and assignment patterns: |
118 | | -* JavaScript |
119 | | -* Java |
120 | | -* Go |
121 | | -* PHP |
| 24 | +**Version 2.0:** Opt-in logging for tracing execution flow, useful only for debugging. Results printed to `stdout` using `print()` as a JSON dict, one result per line. Enabling logging required adding the `--log` argument. |
122 | 25 |
|
123 | | -It is not possible to parse these languages as Abstract Syntax Trees (ASTs) in Python. The initial attempt was to detect "low hanging fruit" by parsing the files as text instead. This lead to poor functional coverage, as well as a potentially false sense of security. |
124 | | - |
125 | | -In version 2 the support for these dynamic languages is dropped. This allowed bringing unit test coverage up to 100%, and in this way ensuring result reliability and true security coverage. It is recommended to rely on AST-based parsing for dynamic languages for getting reliable results. Check out [Semgrep](https://github.com/returntocorp/semgrep)! |
126 | | - |
127 | | -Python3 remains fully supported in Whispers 2. |
128 | | - |
129 | | - |
130 | | -### :x: Replace Levenshtein with Jaro-Winkler :x: |
131 | | -In version 1, [python-Levenshtein](https://github.com/ztane/python-Levenshtein) was used for key-value similarity comparisons (`similar` config parameter). This library is written in Cython and requires additional dependencies for installing. This made it not easily compatible with Windows systems, because additional Visual Studio dependencies needed to be present before installing Whispers. |
132 | | - |
133 | | -In version 2, [Jaro-Winkler](https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance) algorithm is used for similarity comparisons, using the [jellyfish](https://github.com/jamesturk/jellyfish) library, for improved approximate and phonetic string matching. As an additional effect, this change allows installing Whispers on Windows through `pip` without Visual Studio dependencies. |
134 | | - |
135 | | -This change should have no effect and behave in a consistent manner. If you have rules that specifically rely on `similar` for key-value comparisons, these may need to be manually tuned. |
136 | | - |
137 | | - |
138 | | -## :hammer_and_wrench: Improvements :hammer_and_wrench: |
139 | | - |
140 | | -### :hammer_and_wrench: Improved support for Windows and MacOS :hammer_and_wrench: |
141 | | - |
142 | | -Whispers now runs on Linux, MacOS, and Windows. Install it from PyPI like so: `pip3 install whispers`. |
143 | | - |
144 | | -### :hammer_and_wrench: Secrets detection :hammer_and_wrench: |
145 | | - |
146 | | -- Added support for Gradle and Maven credentials |
147 | | -- Improved private key detection |
148 | | -- Added known API key formats ([GitGuardian](https://docs.gitguardian.com/secrets-detection/detectors/)) |
149 | | -- Added knwon file extensions ([tell_me_your_secrets](https://github.com/valayDave/tell-me-your-secrets/blob/master/tell_me_your_secrets/config.yml)) |
150 | | - |
151 | | - |
152 | | -### :hammer_and_wrench: Include and Exclude by Rule and Severity :hammer_and_wrench: |
153 | | -Individual and grouped rules and severity levels to be included or excluded can now be directly specified with CLI args: |
154 | | - |
155 | | -Exclude known files from results: `whispers -R known-file` |
156 | | -Exclude miscellaneous results: `whispers -G misc` |
157 | | -Exclude MAJOR and MINOR severity level results: `whispers -S MAJOR,MINOR` |
158 | | - |
159 | | -It is also possible to specify included and excluded rules and severity levels via config.yml. Custom rules can be added directly to the list using the following format: |
160 | | -```yaml |
161 | | -exclude: |
162 | | - files: |
163 | | - - \.npmrc |
164 | | - - .*coded.* |
165 | | - - \.git/.* |
166 | | - keys: |
167 | | - - SECRET_VALUE_KEY |
168 | | - values: |
169 | | - - SECRET_VALUE_PLACEHOLDER |
170 | | - rules: |
171 | | - - password |
172 | | - - uri |
173 | | - - id: starks |
174 | | - message: Whispers from the North |
175 | | - severity: CRITICAL |
176 | | - value: |
177 | | - regex: (Aria|Ned) Stark |
178 | | - ignorecase: True |
179 | | - groups: |
180 | | - - misc |
181 | | -
|
182 | | -exclude: |
183 | | - severity: |
184 | | - - MINOR |
185 | | -
|
186 | | -``` |
187 | | - |
188 | | -If you don't specify any rules, all built-in rules will be used be default. If you do, only those that you specify will be applicable. For a full list of available rules check `whispers --info`. |
189 | | - |
190 | | -If you don't specify any severity, all built-in severity levels will be used be default - BLOCKER, CRITICAL, MAJOR, MINOR, INFO. |
191 | | -If you do, only those that you specify will be applicable. |
| 26 | +**Version 2.1:** Logging is used to alert identified secrets during execution with `WARNING` level. Results are written to `stdout` as a JSON list at the end. This improves results parseability as a JSON list, while maintaining live results display that was previously achieved by printing secrets as JSON one per line. |
192 | 27 |
|
193 | 28 |
|
194 | 29 | ## :white_check_mark: New features :white_check_mark: |
195 | 30 |
|
196 | | -**No new features** were introduced in this release. Whispers 2 still does the same thing as 1, only better and faster. |
197 | | - |
198 | | -The primary objective of the present release was to optimize version 1 logic in order to make it easier to read, understand, and work with in general. This refactoring, along with the aforementioned breaking changes, have shown to increase scanning speed up to 7-10 times (depending on conditions). In addition, it allowed achieving 100% unit test coverage. |
199 | | - |
200 | | -Other focus areas of version 2 were improving usability, like being able to easily filter results in and out from CLI; not writing a log file by default; dropping support for untestable dynamic code scanning; and making code more Pythonic by using built-in features and dataclass models. |
| 31 | +### :white_check_mark: Results as JSON list :white_check_mark: |
201 | 32 |
|
202 | | -Complete list of arguments, rules, and severity levels can be found in `whispers --info`, along with documentation in [README.md](https://github.com/adeptex/whispers/blob/master/README.md). |
| 33 | +To improve integration and downstream processing, Whispers now outputs results as a JSON list of dictionaries with all detected secrets together (2.1), instead of one JSON dictionary per line (2.0). This list is directly loadable and parsable as JSON. |
0 commit comments