Box.com bulk transfer
https://docs.rc.uab.edu/
Please use the new documentation url https://docs.rc.uab.edu/ for all Research Computing documentation needs.
As a result of this move, we have deprecated use of this wiki for documentation. We are providing read-only access to the content to facilitate migration of bookmarks and to serve as an historical record. All content updates should be made at the new documentation site. The original wiki will not receive further updates.
Thank you,
The Research Computing Team
UAB has an Enterprise contract with Box.com, which is currently in BETA.
This page describes what we have learned about doing bulk-transfers of data.
Warning: size limitations
Box.com claims to have a 5G max filesize limit.
There was a rumor this would be increased in early 2016. Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307
If you need to work around this, you can use the Linux "split" utility
| # chop file into 4G pieces split \ |
| # record checksums of original and chunks md5sum \ |
Warning: time stamps
When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.
| FTP client | platform | modification | creation |
|---|---|---|---|
| SmartFTP | GUI/Win Only/$$ | yes | can be enabled |
| lftp | cmd_line/linux/free | yes | no |
| fileZilla | GUI/linux+win/free | can be enabled | no |
| ftp_ssl | cmd_line/linux/free | yes | no |
Filezilla on create times
- request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347
lftp mirror -R example
lftp mirror
- "mirror" copies directory hierarchies DOWN from box.com to local
- "mirror -R" copies directory hierarchies UP from local to box.com
error handling
- the box server frequently looses connection (fails) on particular files
- just re-run the "mirror -R" and it will upload only new/failed files..
Interactive lftp
| lftp ftp.box.com > user BLAZERID@uab.edu users_BOX_external_password |
scripted lftp
| cat > box_test.lftp << EOF open ftp.box.com |