IOUtils.copy() with input and output streams is extremely slow
As part of my web service, I have a picture repository which retrieves an image from Amazon S3 (a datastore) then returns it. This is how the method that does this looks:
File getPicture(String path) throws IOException {
File file = File.createTempFile(path, ".png");
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, path));
IOUtils.copy(object.getObjectContent(), new FileOutputStream(file));
return file;
}
The problem is that it takes way too long to get a response from the service - (a 3MB image took 7.5 seconds to download). I notice that if I comment out the IOUtils.copy() line, the response time is significantly faster so it must be that particular method that's causing this delay.
I've seen this method used in almost all modern examples of converting an S3Object to a file but I seem to be a unique case. Am I missing a trick here?
Appreciate any help!
java amazon-web-services file amazon-s3 ioutils
asked Nov 23 at 0:31
eyes enberg
83621
add a comment |
As part of my web service, I have a picture repository which retrieves an image from Amazon S3 (a datastore) then returns it. This is how the method that does this looks:
File getPicture(String path) throws IOException {
File file = File.createTempFile(path, ".png");
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, path));
IOUtils.copy(object.getObjectContent(), new FileOutputStream(file));
return file;
}
The problem is that it takes way too long to get a response from the service - (a 3MB image took 7.5 seconds to download). I notice that if I comment out the IOUtils.copy() line, the response time is significantly faster so it must be that particular method that's causing this delay.
I've seen this method used in almost all modern examples of converting an S3Object to a file but I seem to be a unique case. Am I missing a trick here?
Appreciate any help!
java amazon-web-services file amazon-s3 ioutils
asked Nov 23 at 0:31
eyes enberg
83621
I can’t helo with your actual problem, but you might want to wrap the streams in try-with-resource statements. As per the IOUtils docs, it doesn’t close the Streams you pass in - commons.apache.org/proper/commons-io/javadocs/api-2.5/org/… - so who is responsible for closing the FileOutputStream?
– Jakg
Nov 23 at 0:35
Thanks - I've already tried closing the output stream but it didn't help
– eyes enberg
Nov 23 at 0:40
so it must be that particular method that's causing this delay or the fact that yo already downloaded the file before made the second download faster, due to proxies and caches.
– JB Nizet
Nov 23 at 0:46
I downloaded the file multiple times for each scenario (as well as multiple files).
– eyes enberg
Nov 23 at 1:27
Please indicate which IOUtils you are using. At the very least, include the package name. There are dozens of libraries with a class calledIOUtils- for example, the AWS Java SDK, but also the popular apache-commons library.
– Erwin Bolwidt
Nov 23 at 2:20
add a comment |
As part of my web service, I have a picture repository which retrieves an image from Amazon S3 (a datastore) then returns it. This is how the method that does this looks:
File getPicture(String path) throws IOException {
File file = File.createTempFile(path, ".png");
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, path));
IOUtils.copy(object.getObjectContent(), new FileOutputStream(file));
return file;
}
The problem is that it takes way too long to get a response from the service - (a 3MB image took 7.5 seconds to download). I notice that if I comment out the IOUtils.copy() line, the response time is significantly faster so it must be that particular method that's causing this delay.
I've seen this method used in almost all modern examples of converting an S3Object to a file but I seem to be a unique case. Am I missing a trick here?
Appreciate any help!
java amazon-web-services file amazon-s3 ioutils
asked Nov 23 at 0:31
eyes enberg
83621
As part of my web service, I have a picture repository which retrieves an image from Amazon S3 (a datastore) then returns it. This is how the method that does this looks:
File getPicture(String path) throws IOException {
File file = File.createTempFile(path, ".png");
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, path));
IOUtils.copy(object.getObjectContent(), new FileOutputStream(file));
return file;
}
The problem is that it takes way too long to get a response from the service - (a 3MB image took 7.5 seconds to download). I notice that if I comment out the IOUtils.copy() line, the response time is significantly faster so it must be that particular method that's causing this delay.
I've seen this method used in almost all modern examples of converting an S3Object to a file but I seem to be a unique case. Am I missing a trick here?
Appreciate any help!
java amazon-web-services file amazon-s3 ioutils
java amazon-web-services file amazon-s3 ioutils
asked Nov 23 at 0:31
eyes enberg
83621
asked Nov 23 at 0:31
eyes enberg
83621
asked Nov 23 at 0:31
eyes enberg
83621
asked Nov 23 at 0:31
eyes enberg
83621
asked Nov 23 at 0:31
eyes enberg
83621
83621
I can’t helo with your actual problem, but you might want to wrap the streams in try-with-resource statements. As per the IOUtils docs, it doesn’t close the Streams you pass in - commons.apache.org/proper/commons-io/javadocs/api-2.5/org/… - so who is responsible for closing the FileOutputStream?
– Jakg
Nov 23 at 0:35
Thanks - I've already tried closing the output stream but it didn't help
– eyes enberg
Nov 23 at 0:40
so it must be that particular method that's causing this delay or the fact that yo already downloaded the file before made the second download faster, due to proxies and caches.
– JB Nizet
Nov 23 at 0:46
I downloaded the file multiple times for each scenario (as well as multiple files).
– eyes enberg
Nov 23 at 1:27
Please indicate which IOUtils you are using. At the very least, include the package name. There are dozens of libraries with a class calledIOUtils- for example, the AWS Java SDK, but also the popular apache-commons library.
– Erwin Bolwidt
Nov 23 at 2:20
add a comment |
I can’t helo with your actual problem, but you might want to wrap the streams in try-with-resource statements. As per the IOUtils docs, it doesn’t close the Streams you pass in - commons.apache.org/proper/commons-io/javadocs/api-2.5/org/… - so who is responsible for closing the FileOutputStream?
– Jakg
Nov 23 at 0:35
Thanks - I've already tried closing the output stream but it didn't help
– eyes enberg
Nov 23 at 0:40
so it must be that particular method that's causing this delay or the fact that yo already downloaded the file before made the second download faster, due to proxies and caches.
– JB Nizet
Nov 23 at 0:46
I downloaded the file multiple times for each scenario (as well as multiple files).
– eyes enberg
Nov 23 at 1:27
Please indicate which IOUtils you are using. At the very least, include the package name. There are dozens of libraries with a class calledIOUtils- for example, the AWS Java SDK, but also the popular apache-commons library.
– Erwin Bolwidt
Nov 23 at 2:20
I can’t helo with your actual problem, but you might want to wrap the streams in try-with-resource statements. As per the IOUtils docs, it doesn’t close the Streams you pass in - commons.apache.org/proper/commons-io/javadocs/api-2.5/org/… - so who is responsible for closing the FileOutputStream?
– Jakg
Nov 23 at 0:35
I can’t helo with your actual problem, but you might want to wrap the streams in try-with-resource statements. As per the IOUtils docs, it doesn’t close the Streams you pass in - commons.apache.org/proper/commons-io/javadocs/api-2.5/org/… - so who is responsible for closing the FileOutputStream?
– Jakg
Nov 23 at 0:35
Thanks - I've already tried closing the output stream but it didn't help
– eyes enberg
Nov 23 at 0:40
Thanks - I've already tried closing the output stream but it didn't help
– eyes enberg
Nov 23 at 0:40
so it must be that particular method that's causing this delay or the fact that yo already downloaded the file before made the second download faster, due to proxies and caches.
– JB Nizet
Nov 23 at 0:46
so it must be that particular method that's causing this delay or the fact that yo already downloaded the file before made the second download faster, due to proxies and caches.
– JB Nizet
Nov 23 at 0:46
I downloaded the file multiple times for each scenario (as well as multiple files).
– eyes enberg
Nov 23 at 1:27
I downloaded the file multiple times for each scenario (as well as multiple files).
– eyes enberg
Nov 23 at 1:27
Please indicate which IOUtils you are using. At the very least, include the package name. There are dozens of libraries with a class called
IOUtils - for example, the AWS Java SDK, but also the popular apache-commons library.– Erwin Bolwidt
Nov 23 at 2:20
Please indicate which IOUtils you are using. At the very least, include the package name. There are dozens of libraries with a class called
IOUtils - for example, the AWS Java SDK, but also the popular apache-commons library.– Erwin Bolwidt
Nov 23 at 2:20
add a comment |
1 Answer
1
active
oldest
votes
From the AWS documentation:
public S3Object getObject(GetObjectRequest getObjectRequest)
the returned Amazon S3 object contains a direct stream of data from the HTTP connection. The underlying HTTP connection cannot be reused until the user finishes reading the data and closes the stream.
public S3ObjectInputStream getObjectContent()
Note: The method is a simple getter and does not actually create a stream. If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.
If you remove the IOUtils.copy line, then method exits quickly because you don't actually process the stream. If the file is large it will take time to download. You can't do much about that unless you can get a better connection to the AWS services.
answered Nov 23 at 0:52
flakes
6,51611850
That's so strange, the response time was fine for months until a few days ago - and I haven't touched this code for a very long time.
– eyes enberg
Nov 23 at 1:28
@eyesenberg There's a lot of external factors that can be slowing this down. Network speed, disk/storage R/W speed, factors on AWS side, etc. Maybe try reading all of the S3 object into just a byte array. That will at least tell you if the issue lies within theIOUtils/ disk speeds.
– flakes
Nov 23 at 1:31
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439403%2fioutils-copy-with-input-and-output-streams-is-extremely-slow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
From the AWS documentation:
public S3Object getObject(GetObjectRequest getObjectRequest)
the returned Amazon S3 object contains a direct stream of data from the HTTP connection. The underlying HTTP connection cannot be reused until the user finishes reading the data and closes the stream.
public S3ObjectInputStream getObjectContent()
Note: The method is a simple getter and does not actually create a stream. If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.
If you remove the IOUtils.copy line, then method exits quickly because you don't actually process the stream. If the file is large it will take time to download. You can't do much about that unless you can get a better connection to the AWS services.
answered Nov 23 at 0:52
flakes
6,51611850
That's so strange, the response time was fine for months until a few days ago - and I haven't touched this code for a very long time.
– eyes enberg
Nov 23 at 1:28
@eyesenberg There's a lot of external factors that can be slowing this down. Network speed, disk/storage R/W speed, factors on AWS side, etc. Maybe try reading all of the S3 object into just a byte array. That will at least tell you if the issue lies within theIOUtils/ disk speeds.
– flakes
Nov 23 at 1:31
add a comment |
From the AWS documentation:
public S3Object getObject(GetObjectRequest getObjectRequest)
the returned Amazon S3 object contains a direct stream of data from the HTTP connection. The underlying HTTP connection cannot be reused until the user finishes reading the data and closes the stream.
public S3ObjectInputStream getObjectContent()
Note: The method is a simple getter and does not actually create a stream. If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.
If you remove the IOUtils.copy line, then method exits quickly because you don't actually process the stream. If the file is large it will take time to download. You can't do much about that unless you can get a better connection to the AWS services.
answered Nov 23 at 0:52
flakes
6,51611850
That's so strange, the response time was fine for months until a few days ago - and I haven't touched this code for a very long time.
– eyes enberg
Nov 23 at 1:28
@eyesenberg There's a lot of external factors that can be slowing this down. Network speed, disk/storage R/W speed, factors on AWS side, etc. Maybe try reading all of the S3 object into just a byte array. That will at least tell you if the issue lies within theIOUtils/ disk speeds.
– flakes
Nov 23 at 1:31
add a comment |
From the AWS documentation:
public S3Object getObject(GetObjectRequest getObjectRequest)
the returned Amazon S3 object contains a direct stream of data from the HTTP connection. The underlying HTTP connection cannot be reused until the user finishes reading the data and closes the stream.
public S3ObjectInputStream getObjectContent()
Note: The method is a simple getter and does not actually create a stream. If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.
If you remove the IOUtils.copy line, then method exits quickly because you don't actually process the stream. If the file is large it will take time to download. You can't do much about that unless you can get a better connection to the AWS services.
answered Nov 23 at 0:52
flakes
6,51611850
From the AWS documentation:
public S3Object getObject(GetObjectRequest getObjectRequest)
the returned Amazon S3 object contains a direct stream of data from the HTTP connection. The underlying HTTP connection cannot be reused until the user finishes reading the data and closes the stream.
public S3ObjectInputStream getObjectContent()
Note: The method is a simple getter and does not actually create a stream. If you retrieve an S3Object, you should close this input stream as soon as possible, because the object contents aren't buffered in memory and stream directly from Amazon S3.
If you remove the IOUtils.copy line, then method exits quickly because you don't actually process the stream. If the file is large it will take time to download. You can't do much about that unless you can get a better connection to the AWS services.
answered Nov 23 at 0:52
flakes
6,51611850
answered Nov 23 at 0:52
flakes
6,51611850
answered Nov 23 at 0:52
flakes
6,51611850
answered Nov 23 at 0:52
flakes
6,51611850
6,51611850
That's so strange, the response time was fine for months until a few days ago - and I haven't touched this code for a very long time.
– eyes enberg
Nov 23 at 1:28
@eyesenberg There's a lot of external factors that can be slowing this down. Network speed, disk/storage R/W speed, factors on AWS side, etc. Maybe try reading all of the S3 object into just a byte array. That will at least tell you if the issue lies within theIOUtils/ disk speeds.
– flakes
Nov 23 at 1:31
add a comment |
That's so strange, the response time was fine for months until a few days ago - and I haven't touched this code for a very long time.
– eyes enberg
Nov 23 at 1:28
@eyesenberg There's a lot of external factors that can be slowing this down. Network speed, disk/storage R/W speed, factors on AWS side, etc. Maybe try reading all of the S3 object into just a byte array. That will at least tell you if the issue lies within theIOUtils/ disk speeds.
– flakes
Nov 23 at 1:31
That's so strange, the response time was fine for months until a few days ago - and I haven't touched this code for a very long time.
– eyes enberg
Nov 23 at 1:28
That's so strange, the response time was fine for months until a few days ago - and I haven't touched this code for a very long time.
– eyes enberg
Nov 23 at 1:28
@eyesenberg There's a lot of external factors that can be slowing this down. Network speed, disk/storage R/W speed, factors on AWS side, etc. Maybe try reading all of the S3 object into just a byte array. That will at least tell you if the issue lies within the
IOUtils / disk speeds.– flakes
Nov 23 at 1:31
@eyesenberg There's a lot of external factors that can be slowing this down. Network speed, disk/storage R/W speed, factors on AWS side, etc. Maybe try reading all of the S3 object into just a byte array. That will at least tell you if the issue lies within the
IOUtils / disk speeds.– flakes
Nov 23 at 1:31
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439403%2fioutils-copy-with-input-and-output-streams-is-extremely-slow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I can’t helo with your actual problem, but you might want to wrap the streams in try-with-resource statements. As per the IOUtils docs, it doesn’t close the Streams you pass in - commons.apache.org/proper/commons-io/javadocs/api-2.5/org/… - so who is responsible for closing the FileOutputStream?
– Jakg
Nov 23 at 0:35
Thanks - I've already tried closing the output stream but it didn't help
– eyes enberg
Nov 23 at 0:40
so it must be that particular method that's causing this delay or the fact that yo already downloaded the file before made the second download faster, due to proxies and caches.
– JB Nizet
Nov 23 at 0:46
I downloaded the file multiple times for each scenario (as well as multiple files).
– eyes enberg
Nov 23 at 1:27
Please indicate which IOUtils you are using. At the very least, include the package name. There are dozens of libraries with a class called
IOUtils- for example, the AWS Java SDK, but also the popular apache-commons library.– Erwin Bolwidt
Nov 23 at 2:20