Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Post requests with unicode chars in string as data sends incomplete/trimmed data (in >=2.30 with urllib3 2.0) #6658

Closed
viiand opened this issue Mar 7, 2024 · 1 comment

Comments

@viiand
Copy link

viiand commented Mar 7, 2024

In the requests>=2.30.0 (with urllib3 2+), when you send a post request with data=<a string containing unicode chars>, the server receives incomplete/trimmed data. This is due to the fact content-length set by requests will not match the actually sent data. It is set to len(data), yet it sends data.encoded() - which is longer.

If requests uses urllib3 1.x versions this throws an exception, for example:

UnicodeEncodeError: 'latin-1' codec can't encode character '\u0159' in position 10: Body ('ř') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

Expected Result

Either:
a) send complete data (where content-length header == len(data.encode()), not len(data) )
b) raise an exception as urlllib3 1.x versions did, suggesting the str should be utf-8 encoded
c) raise an exception telling the user he is not supposed to pass strings to the data argument

Actual Result

No error is reported on the client side, nor is the data sent correctly. Server reads incomplete data.

Reproduction Steps

import requests
d="Lidová tvořivost na sebe nenechala dlouho čekat."
requests.post('http://localhost:8083/echo', data=d)

Server receives Lidová tvořivost na sebe nenechala dlouho ček in the requests body, which is incomplete

System Information

$ python -m requests.help
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.8"
  },
  "platform": {
    "release": "22.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "300000d0"
  },
  "urllib3": {
    "version": "2.2.1"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
@nateprewitt
Copy link
Member

Hi @viiand, this is a duplicate of #6586 which is already fixed in #6589. Please check opened and closed issues before opening new ones. Thanks!

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 8, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants