net.http: http client with dailContext send a https request different from client without dailContext

[liaogang]

What version of Go are you using (go version)?

$ go version
go version go1.20.5 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/xxx/Library/Caches/go-build"
GOENV="/Users/xxx/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/xxx/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/xxx/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/xxx/root/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/ph/czc9w09j34x7f67hzpvh5p5m0000gn/T/go-build4108418818=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

example one, Test_WithoutProxy, using http.client send a https request directly
example two Test_WithProxy, using http.client send a https request with dailConext to go socks proxy

see the two test here

package uu

import (
	"context"
	"encoding/hex"
	"fmt"
	"github.com/armon/go-socks5"
	"golang.org/x/net/proxy"
	"log"
	"net"
	"net/http"
	"testing"
)

func StartSimpleHttpsServer() string {
	l, err0 := net.Listen("tcp", "127.0.0.1:40030")
	if err0 != nil {
		log.Fatal(err0)
	}

	go func() {
		for true {
			conn, err := l.Accept()
			if err != nil {
				fmt.Println("accept fail -> ", err)
			} else {
				var buf = make([]byte, 1024*24*10)
				n, err1 := conn.Read(buf)
				if err1 != nil {
					fmt.Println("read fail -> ", err1)
				} else {
					fmt.Println("read bytes -> ", n)
					fmt.Println("hex dump -> \n", hex.Dump(buf[:n]))
				}
			}

		}

	}()

	return l.Addr().String()
}

func TestWithBuildInProxy(t *testing.T) {
	conf := &socks5.Config{}
	server, err := socks5.New(conf)
	if err != nil {
		panic(err)
	}

	var socksProxyServer = "127.0.0.1:40012"

	go func() {
		if err := server.ListenAndServe("tcp", socksProxyServer); err != nil {
			panic(err)
		}
	}()

	var httpsServerAddr = StartSimpleHttpsServer()
	workFlow(httpsServerAddr, socksProxyServer)
}

func Test_WithoutProxy(t *testing.T) {

	var httpsServerAddr = StartSimpleHttpsServer()
	workFlow(httpsServerAddr, "")
}

func workFlow(httpsServerAddr, socksServerAddr string) {

	fmt.Println("https server -> ", httpsServerAddr)
	fmt.Println("socks server -> ", socksServerAddr)

	var client = &http.Client{}

	if socksServerAddr != "" {
		dialer, err := proxy.SOCKS5("tcp", socksServerAddr, nil, &net.Dialer{})
		if err != nil {
			log.Fatal(err)
		}

		var transport = &http.Transport{
			DialContext: func(ctx context.Context, network, address string) (net.Conn, error) {
				return dialer.Dial(network, address)
			},
		}

		client.Transport = transport
	}

	//start a request
	resp, err := client.Get(fmt.Sprintf("https://%s/hello", httpsServerAddr))
	if err != nil {
		fmt.Println("http get fail -> ", err)
	} else {
		fmt.Println("http get ok -> ", resp)
	}

}

What did you expect to see?

http client with dailContext act the some like without dailContext

What did you see instead?

the tls server can see tls extension application_lay_protocol_negotiation if http client send sequest directly

the tls server can not see tls extension application_lay_protocol_negotiation if http client send sequest with dailContext

=== RUN   Test_WithoutProxy
https server ->  127.0.0.1:40030
socks server ->  
read bytes ->  257
hex dump -> 
 00000000  16 03 01 00 fc 01 00 00  f8 03 03 68 5b ef e0 03  |...........h[...|
00000010  76 5c 28 fb c2 d0 d5 a2  47 3a 79 f9 a4 e4 b3 e5  |v\(.....G:y.....|
00000020  c3 6b f5 da 06 f2 36 fe  f3 2d 4e 20 98 66 ce 20  |.k....6..-N .f. |
00000030  62 8e af 93 32 8b 97 7d  14 b7 b6 87 07 93 4b 23  |b...2..}......K#|
00000040  df 99 4c e9 2d 4d be f0  b0 fa 8e 41 00 26 c0 2b  |..L.-M.....A.&.+|
00000050  c0 2f c0 2c c0 30 cc a9  cc a8 c0 09 c0 13 c0 0a  |./.,.0..........|
00000060  c0 14 00 9c 00 9d 00 2f  00 35 c0 12 00 0a 13 01  |......./.5......|
00000070  13 02 13 03 01 00 00 89  00 05 00 05 01 00 00 00  |................|
00000080  00 00 0a 00 0a 00 08 00  1d 00 17 00 18 00 19 00  |................|
00000090  0b 00 02 01 00 00 0d 00  1a 00 18 08 04 04 03 08  |................|
000000a0  07 08 05 08 06 04 01 05  01 06 01 05 03 06 03 02  |................|
000000b0  01 02 03 ff 01 00 01 00  00 10 00 0e 00 0c 02 68  |...............h|
000000c0  32 08 68 74 74 70 2f 31  2e 31 00 12 00 00 00 2b  |2.http/1.1.....+|
000000d0  00 05 04 03 04 03 03 00  33 00 26 00 24 00 1d 00  |........3.&.$...|
000000e0  20 21 dd f5 7e 7c 02 ae  25 95 68 97 66 fc 04 86  | !..~|..%.h.f...|
000000f0  5e c0 cb 67 66 1b 11 9a  34 f4 76 cb 91 5a 38 2b  |^..gf...4.v..Z8+|
00000100  26                                                |&|

=== RUN   TestWithBuildInProxy
https server ->  127.0.0.1:40030
socks server ->  127.0.0.1:40012
read bytes ->  239
hex dump -> 
 00000000  16 03 01 00 ea 01 00 00  e6 03 03 96 b5 15 e6 b4  |................|
00000010  48 6b a6 26 bf 1a 11 85  f2 ad f3 05 0f cf 77 9f  |Hk.&..........w.|
00000020  b2 7b 6b 7a 19 77 a9 4a  f8 f4 f5 20 45 16 00 24  |.{kz.w.J... E..$|
00000030  3d 99 da 47 15 7e f2 c4  50 60 14 70 88 86 1f b3  |=..G.~..P`.p....|
00000040  09 cd 57 9d 19 10 f5 bb  d7 54 c4 80 00 26 c0 2b  |..W......T...&.+|
00000050  c0 2f c0 2c c0 30 cc a9  cc a8 c0 09 c0 13 c0 0a  |./.,.0..........|
00000060  c0 14 00 9c 00 9d 00 2f  00 35 c0 12 00 0a 13 01  |......./.5......|
00000070  13 02 13 03 01 00 00 77  00 05 00 05 01 00 00 00  |.......w........|
00000080  00 00 0a 00 0a 00 08 00  1d 00 17 00 18 00 19 00  |................|
00000090  0b 00 02 01 00 00 0d 00  1a 00 18 08 04 04 03 08  |................|
000000a0  07 08 05 08 06 04 01 05  01 06 01 05 03 06 03 02  |................|
000000b0  01 02 03 ff 01 00 01 00  00 12 00 00 00 2b 00 05  |.............+..|
000000c0  04 03 04 03 03 00 33 00  26 00 24 00 1d 00 20 6f  |......3.&.$... o|
000000d0  5a 5b 94 c2 13 0c fe db  fb c4 55 34 06 60 6c ca  |Z[........U4.`l.|
000000e0  ef a6 5e 9d 2d ee 89 99  e8 49 50 34 95 04 39     |..^.-....IP4..9|

the diffrence extension is here

000000c0  32 08 68 74 74 70 2f 31  2e 31 00 12 00 00 00 2b  |2.http/1.1.....+|

[seankhliao]

See https://pkg.go.dev/net/http@go1.20.5#Transport 's ForceAttemptHTTP2 field.