Comments on bizo developer blog: google app engine (java) and s3

Thanks for the clarification. It sounds like a si...

2011-07-19T13:01:07.586-07:00

Thanks for the clarification. It sounds like a similar issue. There's nothing provided in the library to help with this, but I think you might be able to use s3's multipart upload support to break the file into smaller chunks on the client side...

Hi Larry, I have used a servelet to store file as ...

2011-07-18T21:14:14.291-07:00

Hi Larry,
I have used a servelet to store file as below.
public class UploaderServlet extends HttpServlet
{
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
S3Store s3 = new S3Store("s3.amazonaws.com", "public Key", "Private key");
s3.setBucket("eyetask");
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iterator = null;
try {
iterator = upload.getItemIterator(req);
} catch (FileUploadException e) {
e.printStackTrace();
}

FileItemStream item = null;
try {
item = iterator.next();
} catch (FileUploadException e) {
e.printStackTrace();
}
InputStream stream = item.openStream();
String filename = item.getName();

byte[] byetFile = IOUtils.toByteArray(stream);

s3.storeItem(filename, byetFile, "public-read");
}
}

The error I am getting when uploading is “Uncaught exception from servlet com.google.apphosting.api.ApiProxy$RequestTooLargeException: The request to API call urlfetch.Fetch() was too large”. This error occurs only if the size of the file I am going to upload exceeds 1Mb. Do you have solution for this?

Buddhika.jm, you're probably running into a p...

2011-07-18T10:38:40.154-07:00

Buddhika.jm, you're probably running into a problem with appengine request deadlines. By default all HTTP requests have a deadline of 5 seconds. This can be extended to a maximum of 10 seconds. See: http://code.google.com/appengine/docs/java/urlfetch/overview.html

Amazon has support for uploading files in chunks (see: http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadInitiate.html)

this may be able to help you... maybe you can schedule some chunks to upload using appengine's task api?

s3-simple does not currently have support for multipart upload. If you add it, please let me know and I will incorporate it.

I also tried this. That works nicely with small fi...

2011-07-17T06:48:13.394-07:00

I also tried this. That works nicely with small files with few Kilo bites. I wanted to store files with the size of about 10MB. I tried this with large size files. But program does not sent the file to amazon S3 bucket as I expected. Do u have any idea where the problem is? Please help.

Oliver, Are you using the 1.0.5 jar on github? (...

2010-04-13T14:07:33.397-07:00

Oliver,
Are you using the 1.0.5 jar on github? (http://github.com/downloads/ogrodnek/s3-simple/s3-shell-1.0.5.jar)?

It includes its own Base64 class, and doesn't reference the sun.misc.* stuff...

Hi there, I'm trying to use the jar you sugges...

2010-04-13T13:46:38.068-07:00

Hi there, I'm trying to use the jar you suggested but I'm running into trouble with GAE because it doesn't support the use of classes from the sun.misc package

This package is used in S3Store, line 565:

contentMD5 = new BASE64Encoder().encode(md.digest());

Any thoughts?

You don't really need to use the s3sh.java -- ...

2009-05-01T12:31:00.000-07:00

You don't really need to use the s3sh.java -- I left it in there, but in your application you should just use S3Store directly, e.g.:

S3Store s3 = new S3Store(AWS_HOST, AWS_KEY, AWS_SECRET);
List<String> buckets = s3.listBuckets();

And you won't have any problems.

hi i have followed the above procedure - java.io....

2009-05-01T10:45:00.000-07:00

hi i have followed the above procedure

- java.io.FileOutputStream is not supported by Google App Engine's Java runtime
environment in s3sh.java file