storage/swift: upload fails on large objects

[buchanae]

Getting upload errors "Too Large Object" from swift.

Related info:
https://godoc.org/github.com/ncw/swift#Connection.DynamicLargeObjectCreate

rclone/rclone@cc7b9af

[adamstruck]

I was able to upload a 5 GB file without a problem using:

{
  "name": "Large file download/upload example",
  "executors": [
    {
      "image": "biocontainers/samtools:1.3.1",
      "command": ["sh", "-c", "samtools view /tmp/inputs/NA21097.cram | head -n 100 | md5sum"],
      "stdout": "/tmp/outputs/stdout"
    }
  ],
  "inputs": [
    {
      "url": "swift://funnel-test/inputs/NA21097.mapped.ILLUMINA.bwa.GIH.low_coverage.20120522.bam.cram",
      "path": "/tmp/inputs/NA21097.cram"
    }
  ],
  "outputs": [
    {
      "url": "swift://funnel-test/outputs/NA21097.cram",
      "path": "/tmp/inputs/NA21097.cram"
    },
    {
      "url": "swift://funnel-test/outputs/NA21097_100.md5sum",
      "path": "/tmp/outputs/stdout"
    }
  ]
}

UPDATE:

This test passed only because this object was just under the limit. According to the Openstack docs, an individual object cannot be larger than 5 GB.

https://docs.openstack.org/swift/latest/api/large_objects.html

[buchanae]

did you try it with #356 yet?

[adamstruck]

Yeah I tested it and have everything working in #356.