HTTPProxy requestHeadersPolicy fails to deduplicate merged Content-Type headers for Spring Boot backends.

[amanas91]

Hi Team,

I am migrating from a standard Kubernetes Ingress controller to Contour v1.33.2. My upstream application is a Spring Boot service.
When a client sends duplicate Content-Type headers (e.g., two instances of application/json), Envoy merges them into a single comma-separated string: application/json,application/json.
This causes the Spring application to crash with an org.springframework.http.InvalidMediaTypeException because it cannot parse the comma-separated value as a valid MimeType.

The Problem:
Even when using requestHeadersPolicy to remove and then set the header, the backend still receives the merged/invalid string. It appears the header manipulation is not successfully overriding the merged Envoy string before it reaches the upstream service.

Config Used:
yaml
requestHeadersPolicy:
remove:
- Content-Type
set:
- name: Content-Type
value: application/json

I would like to how can we fix this issue in contour, as we did not face this issue in existing k8s ingress controller.

i have contour, envoy running as deployment and certgen job fyi in terms of env set up.

[github-actions]

Hey @amanas91! Thanks for opening your first issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on Contour. You can also join us on our mailing list and in our channel in the Kubernetes Slack Workspace

[tsaarni]

Hi @amanas91,

I just tested this with echoserver as the upstream.

First, apply the HTTPProxy without any header manipulation:

kubectl apply -f - <<'EOF'
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: echoserver
spec:
  virtualhost:
    fqdn: echoserver.127-0-0-101.nip.io
  routes:
    - services:
        - name: echoserver
          port: 80
EOF

I executed following command

curl -s -H "Content-Type: application/json" -H "Content-Type: application/json" http://echoserver.127-0-0-101.nip.io/

Response I received is

{
  "content_length": 0,
  "env": {
    "ENV_NAMESPACE": "default",
    "ENV_NODE_NAME": "contour-worker",
    "ENV_POD_IP": "10.244.1.3",
    "ENV_POD_NAME": "echoserver-6dcfbc7c44-6rrgb",
    "ENV_POD_UID": "391eb371-7e0f-4dcb-8ac3-ecca8e32571a"
  },
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Content-Type": [
      "application/json,application/json"
    ],
    "User-Agent": [
      "curl/8.5.0"
    ],
    "X-Envoy-Expected-Rq-Timeout-Ms": [
      "15000"
    ],
    "X-Envoy-External-Address": [
      "172.20.0.1"
    ],
    "X-Forwarded-For": [
      "172.20.0.1"
    ],
    "X-Forwarded-Proto": [
      "http"
    ],
    "X-Request-Id": [
      "92ee6bc2-1a51-4227-be75-b9c9f29d69f8"
    ],
    "X-Request-Start": [
      "t=1773089952.502"
    ]
  },
  "host": "echoserver.127-0-0-101.nip.io",
  "method": "GET",
  "proto": "HTTP/1.1",
  "remote": "10.244.1.6:46456",
  "url": "/"
}

Without header manipulation, the upstream receives "Content-Type": ["application/json,application/json"], the two headers merged into one by Envoy as you mentioned (which seems to be correct according to rfc7230 3.2.2).

Next, I apply with the requestHeadersPolicy workaround:

kubectl apply -f - <<'EOF'
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: echoserver
spec:
  virtualhost:
    fqdn: echoserver.127-0-0-101.nip.io
  routes:
    - services:
        - name: echoserver
          port: 80
      requestHeadersPolicy:
        remove:
          - Content-Type
        set:
          - name: Content-Type
            value: application/json
EOF

I executed the same curl command and received response

{
  "content_length": 0,
  "env": {
    "ENV_NAMESPACE": "default",
    "ENV_NODE_NAME": "contour-worker",
    "ENV_POD_IP": "10.244.1.3",
    "ENV_POD_NAME": "echoserver-6dcfbc7c44-6rrgb",
    "ENV_POD_UID": "391eb371-7e0f-4dcb-8ac3-ecca8e32571a"
  },
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Content-Type": [
      "application/json"
    ],
    "User-Agent": [
      "curl/8.5.0"
    ],
    "X-Envoy-Expected-Rq-Timeout-Ms": [
      "15000"
    ],
    "X-Envoy-External-Address": [
      "172.20.0.1"
    ],
    "X-Forwarded-For": [
      "172.20.0.1"
    ],
    "X-Forwarded-Proto": [
      "http"
    ],
    "X-Request-Id": [
      "2d19d382-fe81-4823-aab6-8f068343e487"
    ],
    "X-Request-Start": [
      "t=1773089965.426"
    ]
  },
  "host": "echoserver.127-0-0-101.nip.io",
  "method": "GET",
  "proto": "HTTP/1.1",
  "remote": "10.244.1.6:58188",
  "url": "/"
}

The upstream receives a clean "Content-Type": ["application/json"].

It seems the workaround works correctly after all?

[amanas91]

Thank you so much for testing this from your side,
but still it is not working for me, i am still getting 2 times duplicate application/json

i have like this httpproxy:
routes:
- conditions:
- prefix: /
services:
- name: enclaveapi-go-api
port: 80
requestHeadersPolicy:
remove:
- Content-Type
set:
- name: Content-Type
value: application/json

it did not work, i even restarted contour , envoy pod and application pod also but still not working.

what else can i try here, could you please suggest.

[tsaarni]

I could not reproduce the issue. I tested several prefix matches and it worked (and without requiring any restarts). I tested with Contour v1.33.2 + Envoy v1.35.8.

If possible, you could try with echoserver:

# Deploy echoserver
kubectl apply -f https://raw.githubusercontent.com/tsaarni/echoserver/refs/heads/main/manifests/echoserver.yaml

# Create HTTPProxy with requestHeadersPolicy
kubectl apply -f - <<'EOF'
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: echoserver
spec:
  virtualhost:
    fqdn: echoserver.127-0-0-101.nip.io
  routes:
    - conditions:
        - prefix: /
      services:
        - name: echoserver
          port: 80
      requestHeadersPolicy:
        remove:
          - Content-Type
        set:
          - name: Content-Type
            value: application/json
EOF

# Test with duplicate Content-Type headers and observe the echoed values.
curl -s -H "Content-Type: application/json" -H "Content-Type: application/json" http://echoserver.127-0-0-101.nip.io/

Of of course, apply this to a specific namespace if you are not using default and also update the fqdn and URL to match a host that works in your environment.

[amanas91]

I tested this application above and this one worked , but the one which we are using is not working.

Also, I have another question, if application returns multiple contents not just application/json, like xml etc, what would be the way to make this solution dynamic in contour ?

[amanas91]

also one thing i noticed here with app team is, when they tested the app locally, they only see one type content type json ,but when it goes via contour, it adds one more with comma separated. so they are saying their app is not adding this duplicate, but it is getting added when it goes via contour.

is there way this can be avoided globally because we did not see this issue with our existing k8s ingress controller, since we are doing poc with contour as we want to migrate to this from k8s ingress, we started seeing this issue.

[tsaarni]

I tested this application above and this one worked , but the one which we are using is not working.

I cannot think of the cause for this unless there are some other major differences between the application and echoserver HTTPProxies that could result in significantly different configuration for Envoy.

also one thing i noticed here with app team is, when they tested the app locally, they only see one type content type json ,but when it goes via contour, it adds one more with comma separated

If a client sends duplicate content type headers:

Content-Type: application/json
Content-Type: application/json

it is a malformed request. While the server could return a 400 Bad Request if it was very strict, it seems Envoy follows a general approach from RFC7230 section 3.2.2:

A recipient MAY combine multiple header fields with the same field
name into one "field-name: field-value" pair, without changing the
semantics of the message, by appending each subsequent field value to
the combined field value in order, separated by a comma. 

This results in an invalid header Content-Type: application/json,application/json.

When testing locally, your application or web server might drop the duplicate header. For example, the Go net/http server, which my echoserver uses, treats headers as arrays. If sending duplicate headers locally (without Contour/Envoy in between) response looks like this:

    "Content-Type": [
      "application/json",
      "application/json"
    ],

Go's request.Header.Get("Content-Type")) only returns the first value. If the application was calling that, it would not notice there is anything wrong. However request.Header.Values("Content-Type")) returns the full array.

Also, I have another question, if application returns multiple contents not just application/json, like xml etc, what would be the way to make this solution dynamic in contour ?

A request can only have one Content-Type, specifying what media type the request body is. The HTTPProxy.spec.routes[].requestHeadersPolicy setting specifically overrides the Content-Type for the incoming request.

If a client can handle multiple formats like JSON or XML, it should send an Accept header: Accept: application/json, application/xml. The server then chooses one format for the response content type. The requestHeadersPolicy only affects request headers and will not change the server's response.

[amanas91]

Thank you so much for the information, for now since httpproxy did not resolve an issue for our application, we were able to get the code modified by app team to send only one header instead of multiple headers which fixed the issue.

also your test application i mimic in our environment and for that issue got fixed with that change.

[tsaarni]

Great that you could fix it!

Remember that certgen job is one shot and the certificate it generated for contour-envoy connection will eventually expire :)

[amanas91]

thank you for that info, but if we keep that certgen job running, will it not actively monitor and renew the certificate automatiacally?

also i have another question, as by default behaviour for contour is to convert the custom header to lowercase, how can we preserve the header as it is in response, what we have to configure for that ?

[tsaarni]

if we keep that certgen job running, will it not actively monitor and renew the certificate automatiacally?

The certgen tool runs as a one-shot Job, so it cannot monitor or renew certificates automatically. You can find manual rotation instructions here: Rotate using the contour-certgen job. For automated management, using cert-manager is a better approach.

also i have another question, as by default behaviour for contour is to convert the custom header to lowercase, how can we preserve the header as it is in response, what we have to configure for that ?

Envoy detauls to lowercase which is aligned with the standards: HTTP/1.1 headers are case-insensitive, while HTTP/2 and HTTP/3 require lowercase. While Envoy has an optional feature to preserve header case in HTTP/1.1, this cannot be configured through Contour.

[amanas91]

We need this feature to preserve the header as we are doing poc on this product to migrate from existing k8s ingress controller and we really need to have this feature to support our existing application. is there any way this feature can be added in the contour also ?

[tsaarni]

No sorry, there are no plans this time for adding an option to change Envoy's header lowercase normalization .