911 Timeline Moved to S3

Submitted by Bill St. Clair on Sun, 19 Mar 2006 15:45:15 GMT  <== Webmaster stuff ==> 

I finished moving my mirror of Paul Thompson's 911 Timeline to S3. billstclair.com/911timeline now redirects to a page documenting the move. All other links to the timeline automatically redirect. Apache's Redirect and RedirectMatch directives at work.

Next step in my S3 Ruby library: streaming upload of files and web URLs and download of files. The current Amazon-provided library requires objects to fit in memory.

Add comment Edit post Add post

Comments (2):

Streaming to S3

Submitted by Adam Groves on Thu, 27 Jul 2006 21:31:39 GMT

I've got streaming upload working (I think) by using the code posted here: http://blogs.missiondata.com/?p=29. I say I think, because I've no idea how to test if streaming is working. Any ideas?
The next step is to add an update progress bar, but that's looking rather complicated. I'd be interested to know how you are progressing.

Edit comment

I've got streaming upload

Submitted by Bill St. Clair on Fri, 28 Jul 2006 00:12:43 GMT

I've got streaming upload working in Ruby, but I haven't cleaned up the code for release. It works for what I need, uploading files to my S3 web sites. I have the following command line commands working well enough for limited use:

s3cd - change s3 working directory
s3pwd - print s3 working directory
s3ls - list files matching a prefix
s3rm - remove a file from s3
s3cp - upload local files to s3

My plan is for s3ls and s3rm to do full wildcard matching, and for s3cp to copy from s3 to disk and to copy from an http or ftp connection to s3 with no disk in between. This last isn't possible in general, since s3 requires that you know the object length at the beginning of the upload, but it will work for connections that give their length right away. Others need to be downloaded to disk first.

s3cp shows progress, like wget and scp. This is how I know that my streaming code is working; it shows upload progress once a second.

But I haven't done any work on it in a while, having enough to do what I need.

Edit comment