Strange difference between passing proxy as None vs an empty string

[xjcl]

At work we have to deal with an annoying proxy server when making automated HTTP requests, e.g. in Python. To deal with this we usually add proxies={"http": None, "https": None} to any request, but recently this started breaking down when we started receiving HTTP redirects (301 Moved Permanently). After struggling with this issue by trying out different versions of dependencies, we finally figured out that passing an empty string vs None produced different behavior, with the behavior of None being strictly worse.

Reproduction Steps

To reproduce this you need some kind of proxy server setup, I cannot give instructions on how to achieve this.

Code snippet "empty string":

requests.get("https://pypi.org/pypi/pillow/9.0.0/json", proxies={"http": "", "https": ""})

Code snippet "None":

requests.get("https://pypi.org/pypi/pillow/9.0.0/json", proxies={"http": None, "https": None})

Code snippet "(no proxies)":

requests.get("https://pypi.org/pypi/pillow/9.0.0/json")

Expected Result

Using uppercase Pillow should result in a successful HTTP request (200).

Using lowercase pillow should result in a redirect to uppercase Pillow (301) which requests resolves with a second GET that is successful (200).

Actual Result

Unexpected result in italic:

	Pillow	pillow
empty string	OK	OK
None	OK	SSLError
(no proxies)	SSLError	SSLError

I will shorten the actual very long set of 3 tracebacks we receive to:

  File "...\python\lib\ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)

During handling of the above exception, another exception occurred:

  File: "...\site-packages\urllib3\util\retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(...): Max retries exceeded with url: ...

During handling of the above exception, another exception occurred:

  File "..\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exception.SSLError: HTTPSConnectionPool(...): Max retries exceeded with url: ...

It is not surprising that we get an SSLError when using no proxy parameter, but I find it odd that there is a semantic difference between passing None and "", (empty string), and could not find any documentation referencing that difference.

Further reading on StackOverflow: https://stackoverflow.com/a/72529216/2111778

System Information

• chardet 4.0.0
• cryptography absent
• idna 2.10
• implementation CPython 3.7.7
• platform Windows 10
• pyOpenSSL absent
• requests 2.25.1
• system_ssl 1010104f
• urllib3 1.26.3
• using_pyopenssl false

[xjcl]

Stepping through the code with breakpoints I noticed a difference in the function "rebuild_proxies" which seems to get called when a redirect occurs. In the empty string case we already have "" defined as http(s) proxy so nothing gets overwritten. In the None case the keys for http(s) proxies are absent so they get set using get_environ_proxies.

[xjcl]

Looks like the function merge_setting in the sessions.py module explicitly deletes any Nones (check the variable none_keys). This means http and https proxies end up missing from the dictionary which works fine for the first request, but rebuild_proxies tries to add them back in for the redirect.

[nateprewitt]

Hi @xjcl, this is the expected behavior and generally how the library operates. None values will be stripped from headers and most settings. I don't think we'd intend to change that behavior without a very strong reason.

What you're trying to accomplish here though is already supported without trying to override the proxies with null values. If you set trust_env to False on your Session, it won't pull proxy information from your env vars or system settings. Is that not working for your case? We'd also recommend making sure the issue you're seeing is still present in the latest version (2.27.1) as proxy behavior was recently reworked.

[nateprewitt]

Going to resolve this since we haven't heard back. Feel free to reopen if you have further questions or find the recommended approach isn't working.

[jtpereyda]

@nateprewitt The use case here is to disable the proxies without the other side effects of trust_env. What is the recommended approach in that case?

[nateprewitt]

@jtpereyda would you mind providing a concrete example of the other side effects you're trying to avoid?

[jtpereyda]

@nateprewitt Are all the effects documented anywhere? I have read that trust_env also ignores:

• Authentication information from .netrc
• CA bundles defined in REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE

Both of which are relevant for some of my use cases.

[nateprewitt]

.netrc info can be supplied via the auth parameter in this case and the ca_bundle via verify. If neither of those are satisfactory, the original posts details how you can disable the proxies with empty strings.

The original question was about the difference between an empty string and None. None has a semantic of "do not use this key" in the library, hence the difference in behavior.