request help: upstream sent too big header while reading response header from upstream

[eavesmy]

upstream 返回超大的 HTTP 头信息时报错

upstream-sent-too-big-header-while-reading-response-header-from-upstream
请问我需要更改什么设置？

[Firstsawyou]

You can try to set the configuration information of proxy_busy_buffers_size, proxy_buffers and · proxy_busy_buffers_size through customize-nginx-configuration method.
https://www.cyberciti.biz/faq/nginx-upstream-sent-too-big-header-while-reading-response-header-from-upstream/

[tokers]

The upstream response header will be saved to the first buffer, which size is controlled by proxy_buffer_size directive.

[eavesmy]

I edit apisix config file like this:

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "logs/error.log"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "logs/access.log"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Real-IP"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 'unix:'
    proxy_busy_buffers_size:   512k
    proxy_buffers:   4 512k
    proxy_buffer_size:   256k

    #lua_shared_dicts:              # add custom shared cache to nginx.conf
    #  ipc_shared_dict: 100m        # custom shared cache, format: `cache-key: cache-size`

But dose not work.

[spacewander]

You need to configure http_server_configuration_snippet, see https://github.com/apache/apisix/blob/master/docs/en/latest/customize-nginx-configuration.md

[eavesmy]

You need to configure http_server_configuration_snippet, see https://github.com/apache/apisix/blob/master/docs/en/latest/customize-nginx-configuration.md

Thx.

[ty793941]

I have solved this problem。Please look at here!

Error Info:

2022/05/08 14:24:31 [error] 47#47: *6887749 upstream sent too big header while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: _, request: "POST /signin-oidc HTTP/1.1", upstream: "http://xx.xx.xxx.xxxx:xxxx/signin-oidc", host: "xs.xxxxxxx.com:9080"

The problem I have is that the response size is 5kb,bigger than proxy_buffer_size size,which default is 4kb. This leads to API SIX reporting 502 error.

proxy_buffer_size:
Nginx uses proxy_ buffer_ Size request corresponding read_ BUF, which specifies the maximum length of the upstream header. If the response header exceeds this length, nginx will report the error of upstream sent too big header.

Solution:

change the nginx proxy_buffer_size size to the right size.
I got the nginx config from apisix doc.
https://github.com/apache/apisix/blob/master/docs/en/latest/customize-nginx-configuration.md
The Nginx configuration used by APISIX is generated via the template file apisix/cli/ngx_tpl.lua and the options from conf/config-default.yaml /conf/config.yaml.

so we modify the /conf/config.yaml,and add the some snippets,like this:

nginx_config:                      # config for render the template to generate nginx.conf
  http_configuration_snippet: |
      proxy_buffer_size 128k;
      proxy_buffers 32 128k;
      proxy_busy_buffers_size 128k;
  http_server_configuration_snippet: |
      proxy_buffer_size 128k;
      proxy_buffers 32 128k;
      proxy_busy_buffers_size 128k;

The http_configuration_snippet and http_server_configuration_snippet will attach to the nginx.conf,which the options from conf/config-default.yaml and /conf/config.yaml.

If you want to learn more about how the http_configuration_snippet and http_server_configuration_snippet works,look at
https://github.com/apache/apisix/blob/master/conf/config-default.yaml ,and find the snippets.

[deonash]

i am still facing problem with this.

upstream sent too big header while reading response header from upstream, client: 172.18.0.1,

tried below in docker compose and no luck

apisix:
image: apache/apisix:2.13.1-alpine
restart: always
volumes:
- ./apisix_log:/usr/local/apisix/logs
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
- ./nginx_custom/nginx.conf:/usr/local/apisix/conf/nginx.conf
depends_on:
- etcd
ports:
- "9080:9080/tcp"
- "9091:9091/tcp"
- "9443:9443/tcp"
- "9092:9092/tcp"
networks:
apisix:

cloning api docker and modifying conf/conf.yaml file sound tedious task.

appreciate help guys. :)