Is the HTTPS proxy support known-working under real-world conditions?

[ag-TJNII]

I've been debugging HTTPS proxy support all day and I'm coming to the conclusion the released implementation may be be broken.

Stepping through an example invocation:

n = Net::HTTP.new('google.com', 443, '127.0.0.1', 4433, nil, nil, nil, true)
n.use_ssl = true
n.get('/')

• connect is invoked
• Which opens a TCP socket to the proxy]
• And then opens a SSL socket over the TCP proxy socket
• It then sends the CONNECT command with the needed headers
• Then, after more setup, it attempts to open the endpoint SSL socket against the raw proxy socket, not the proxy SSL socket

I don't believe the last step works.

To test I banged out a little helper class that does the initial proxy connection setup:

require 'net/http'

class ProxySock
  attr_accessor :proxy_sock, :s

  def initialize
    @s = TCPSocket.open('127.0.0.1', 4433, nil, nil)
    @proxy_sock = OpenSSL::SSL::SSLSocket.new(@s)
    Net::Protocol.new.send(:ssl_socket_connect, @proxy_sock, 1.0)
  end

  def close
    @proxy_sock.close
  ensure
    @s.close
  end
end

Then, as a baseline, I checked that basic HTTP proxying was working:

ps = ProxySock.new
begin
  ps.proxy_sock.write("GET http://google.com/ HTTP/1.1\r\n\r\n")
  puts ps.proxy_sock.gets("\r\n\r\n")
ensure
  ps.close
end

This returns the expected HTTP/1.1 301 Moved Permanently Location: http://www.google.com/ response.

Then, I tried the flow Net::HTTP currently does:

ps = ProxySock.new
begin
  ps.proxy_sock.write("CONNECT google.com:443 HTTP/1.1\r\n\r\n")
  puts ps.proxy_sock.gets("\r\n\r\n")
  
  endpoint_sock = OpenSSL::SSL::SSLSocket.new(ps.s)
  Net::Protocol.new.send(:ssl_socket_connect, endpoint_sock, 1.0)
  endpoint_sock.write("GET http://google.com/ HTTP/1.1\r\n\r\n")
  puts endpoint_sock.gets("\r\n\r\n")
ensure
  ps.close
end

This throws the following error, which is the same error I get from Net::HTTP:

home/tom/.rbenv/versions/3.4.1/lib/ruby/3.4.0/net/protocol.rb:46:in 'OpenSSL::SSL::SSLSocket#connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:4433 state=error: invalid alert (OpenSSL::SSL::SSLError)
        from /home/tom/.rbenv/versions/3.4.1/lib/ruby/3.4.0/net/protocol.rb:46:in 'Net::Protocol#ssl_socket_connect'
        from tmp/logic_test.rb:59:in '<main>'

As a second test I tried performing another HTTP proxy test, this time using CONNECT:

ps = ProxySock.new
begin
  ps.proxy_sock.write("CONNECT google.com:80 HTTP/1.1\r\n\r\n")
  puts ps.proxy_sock.gets("\r\n\r\n")

  ps.proxy_sock.write("GET http://google.com/ HTTP/1.1\r\n\r\n")
  puts ps.proxy_sock.gets("\r\n\r\n")

  ps.s.write("GET http://google.com/ HTTP/1.1\r\n\r\n")
  puts ps.s.gets("\r\n\r\n")
ensure
  ps.close
end

This outputs two blocks. The first block uses the SSL socket and returns HTTP/1.1 301 Moved Permanently, as expected. The second block attempts to use the underlying TCP socket, same as we're trying to do for the endpoint SSL socket, and that returns �*o�Ń7�t��4��w4Q���k�9o� which appears to be encrypted data.

When using a HTTPS proxy the socket s IO will be encrypted, I don't believe this is the correct handle to use for the endpoint encryption.. I believe we need to initialize the endpoint ssl over the proxy_sock to nest the encryption.

To this end, I tried endpoint_sock = OpenSSL::SSL::SSLSocket.new(ps.proxy_sock) but that simply throws wrong argument type OpenSSL::SSL::SSLSocket (expected File) (TypeError). So I don't have a working HTTPS over HTTPS proxy example on hand.

I'm currently of the opinion the implementation here is broken. Am I mistaken? Is there a flaw in my analysis and test cases?

[ag-TJNII]

Proof-of-concept that shows the endpoint SSL does need to happen against the proxy_sock:

require 'net/http'

class ProxySock
  attr_accessor :proxy_sock, :s

  def initialize
    @s = TCPSocket.open('127.0.0.1', 4433, nil, nil)
    @proxy_sock = OpenSSL::SSL::SSLSocket.new(@s)
    Net::Protocol.new.send(:ssl_socket_connect, @proxy_sock, 1.0)
  end

  def close
    @proxy_sock.close
  ensure
    @s.close
  end
end

puts "HTTPS over HTTPS"
ps = ProxySock.new
begin
  ps.proxy_sock.write("CONNECT google.com:443 HTTP/1.1\r\n\r\n")
  puts ps.proxy_sock.gets("\r\n\r\n")

  server = TCPServer.new 2000
  Thread.new do
    server_client = server.accept

    Thread.new do
      loop do
        ps.proxy_sock.write(server_client.read(1)) # POC ONLY
      end
    end

    loop do
      server_client.write(ps.proxy_sock.read(1)) # POC ONLY
    end
  end

  client = TCPSocket.open('127.0.0.1', 2000, nil, nil)
  endpoint_sock = OpenSSL::SSL::SSLSocket.new(client)
  Net::Protocol.new.send(:ssl_socket_connect, endpoint_sock, 1.0)
  endpoint_sock.write("GET https://google.com/ HTTP/1.1\r\n\r\n")
  puts endpoint_sock.gets("\r\n\r\n")
ensure
  ps.close
end

HTTPS over HTTPS
HTTP/1.1 200 Connected

HTTP/1.1 301 Moved Permanently
Location: https://www.google.com/
Content-Type: text/html; charset=UTF-8
Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-hwWOBT4F4XDkfa9ufwjlJw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
Date: Sat, 01 Mar 2025 05:33:33 GMT
Expires: Mon, 31 Mar 2025 05:33:33 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 220
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

<snip thread/socket handling related exception... POC level code only...>

This is obviously not a suitable solution for this library. Unfortunately I rarely need to work with this low level of IO so I'm not sure the best way to solve the wrong argument type OpenSSL::SSL::SSLSocket (expected File) (TypeError) problem that prevents using OpenSSL::SSL::SSLSocket.new(ps.proxy_sock).

[liath]

Just to close the loop for anyone else looking for this, the fix is being worked on here:
ruby/openssl#736