Qfyilyi

I have an ALB with a target group and ECS cluster running PHP API.

I am trying to query the API for a CSV response but I am getting truncated results if the Request is coming through the ALB.

When I SSH into the EC2 instance running the cluster and try to run curl manually (going through the load balancer) the response gets truncated:

curl -sSL -D - 'https://my.domain.com/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' 

  -H 'Content-Type: application/json' 

  -H 'cache-control: no-cache' -o /dev/null

I am getting these headers:

HTTP/2 200 

date: Wed, 21 Nov 2018 20:25:27 GMT

content-type: text/csv; charset=utf-8

content-length: 173019

server: nginx

content-transfer-encoding: binary

content-description: File Transfer

content-disposition: attachment;filename=export.csv

cache-control: private, must-revalidate

etag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"

strict-transport-security: max-age=2592000; includeSubDomains; preload

content-security-policy-report-only: default-src 'self';

x-frame-options: DENY

x-xss-protection: 1; mode=block

x-content-type-options: nosniff

referrer-policy: origin



curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

If I try to run the same curl against the container (running locally - not through ALB)

curl -sSL -D - 'http://localhost:32776/api/export?token=foobar&start_date=01-01-2015&end_date=01-01-2019' 

  -H 'Content-Type: application/json' 

  -H 'cache-control: no-cache' -o /dev/null

Response:

HTTP/1.1 200 OK

Server: nginx

Content-Type: text/csv; charset=utf-8

Content-Length: 173019

Connection: keep-alive

Content-Transfer-Encoding: binary

Content-Description: File Transfer

content-disposition: attachment;filename=export.csv

Cache-Control: private, must-revalidate

Date: Wed, 21 Nov 2018 20:36:55 GMT

ETag: "b90d0da7b482da96e1a478d59eedd0d16552fbfd"

Strict-Transport-Security: max-age=2592000; includeSubDomains; preload

Content-Security-Policy-Report-Only: default-src 'self;

X-Frame-Options: DENY

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

Referrer-Policy: origin

When I compare them, there is a difference in the HTTP version. I tried switching to HTTP1 in ALB but still getting the same (or similar) issue: curl: (18) transfer closed with 130451 bytes remaining to read.

Another difference is the Keep-Alive option. I am not sure if this is an attribute I can enable on the ALB.

When I try to return a different response (complex web page/really long) the response goes through ALB without a problem (not truncated). According to the error message when ALB has HTTP/1.1 enabled the Response is truncated every time after 42568 bytes.

Any ideas?

UPDATE

If I leave out the Content-Type header in the response, it doesn't get truncated.

return new Response($content, Response::HTTP_OK, [

    # Works without this:

    # 'Content-Type' => 'text/csv; charset=utf-8',

    'Content-Transfer-Encoding' => 'binary',

    'Content-Description' => 'File Transfer',

    'Content-Disposition' => "attachment;filename=export.csv",

    'Content-Length' => strlen($content),  

]);

UPDATE 2

Changing the response Content-Type to be text/html returns the response properly.

So after some joyful debugging, I found this in the Nginx logs from the container:

nginx stderr | 2018/11/22 01:03:59 [warn] 39#39: *65 an upstream response is 

buffered to a temporary file /var/tmp/nginx/fastcgi/4/01/0000000014 while reading 

upstream, client: 10.1.1.163, server: _, request: "GET /api/export?

token=foobar&start_date=01-01-2015&end_date=01-01-2019 HTTP/1.1", upstream: 

"fastcgi://unix:/var/run/php-fpm.sock:", host: "my.domain.com"

Which can basically be solved by baking in these two lines into my nginx config:

client_body_temp_path /tmp 1 2;

fastcgi_temp_path /tmp 1 2;

The question why was this happening only for csv output will remain a mystery.

Thanks for the help!

You should enable keep-alive on your EC2 instances.

You can enable HTTP keep-alive in the web server settings for your EC2
instances.
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout

Also double check that the Content-Length header is accurate. An incorrect size here will result in the error you are seeing.