Openstack Swift Static Large Object (SLO)

Uploading Files Larger than 5 GB Using the REST API

Before You Begin

Purpose

This tutorial shows how to upload files larger than 5 GB to Oracle Cloud Infrastructure Object Storage Classic by using the Static Large Object approach, via the REST interface.

Time to Complete

Approximately 1 hour

Background

Oracle Cloud Infrastructure Object Storage Classic can support objects of virtually any size, but there is a maximum upload size of 5 GB. A single object can hold up to 5 GB of data, but multiple objects can be linked together to hold more than 5 GB of contiguous data. You can create small objects as segments and upload them as one large object by using a manifest object.

Any user with the Service Administrator role or a role that is specified in the X-Container-Write ACL of the container can perform this task.

Scenario

This tutorial shows how to upload files larger than 5 GB to the Oracle Cloud Infrastructure Object Storage Classic by using the REST API, via the cURL command line tool. First, if you don't already have cURL, you must install it. Next, you get an authorization token from Oracle Cloud Infrastructure Object Storage Classic. You use this authorization token to access Oracle Cloud Infrastructure Object Storage Classic to upload the files. 

To upload a file larger than 5 GB:

• Segment the large file locally into multiple sequential segment files, each smaller than 5 GB.
• Upload the individual file segments.
• Map the uploaded file segments together into a single object

What Do You Need?

• A tool to access the REST interface. This tutorial uses cURL, which is free, open software that runs under various operating systems. For the steps to install cURL on a Windows 64-bit machine, click here. Alternatively, click here to install cURL on Cygwin on Windows.
• A subscription (trial or paid) to Oracle Cloud Infrastructure Object Storage Classic.
• Your Oracle Cloud user name, password, and identity domain.

         You can find out these details from the New Account Information email that you received from Oracle Cloud when your user account was set up. If you don't have your New Account Information email, contact your account administrator for your Oracle Cloud user name, password, and identity domain.

• Service Administrator role.

 

Segmenting the Large File

1. Segment the large file locally into multiple sequential segment files, each smaller than 5 GB.

   On Linux, for example, you can use the following command:

   split -b 10m file_name segment_prefix

   For example, segment the large file "myLargeFile.zip".

   split -b 10m myLargeFile.zip segment_ 

2. List all the segment files:

   ls -al segment_* 

   segment_ab
   segment_ac
   segment_ad
   segment_ae
   segment_af
   segment_ag
   segment_ah
   segment_ai
   segment_aj

 

Requesting an Authentication Token

Oracle Cloud Infrastructure Object Storage Classic requires authentication for any operation against the service instance. Authentication is performed by using an authentication token. Authentication tokens are requested from the service by authenticating your user credentials with the service. All provisioned authentication tokens are temporary and expire in 30 minutes. You must include your current authentication token with every request to Oracle Cloud Infrastructure Object Storage Classic.

1. Request an authentication token by running the following cURL command:

   Note: This example command uses the credentials for the user jack.jones@example.com in a generalized identity domain named myIdentityDomain. If you wish to copy and adapt this example, change it to reflect your Oracle Cloud user name, password, and identity domain. The values that you must change are highlighted in bold in the command example.

   curl -v -s -X GET -H "X-Storage-User: myService-myIdentityDomain:john.doe@oracle.com" -H "X-Storage-Pass: xUs8M8rw" https://foo.storage.oraclecloud.com/auth/v1.0
   

   The following is an example of the output of this command, with certain key lines highlighted. Note that if the request includes the correct credentials, it returns the HTTP/1.1 200 OK response.

   > GET /auth/v1.0 HTTP/1.1
   > Host: foo.storage.oraclecloud.com
   > Accept: */*
   > X-Storage-User: Storage-myIdentityDomain:jack.jones@example.com
   > X-Storage-Pass: ft7)Dvjo
   < HTTP/1.1 200 OK
   < X-Storage-Url: https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain
   < X-Storage-Token: AUTH_tk209f7f2ea1265a0d3f29d28a2dc8ced6
   < X-Auth-Token: AUTH_tk23be53ac28c3e443de38367f87643eb9
   < X-Trans-Id: txba4aa8f776164c33b7aa587554c29fb6
   < Content-Length: 0
   < Cache-Control: no-cache
   < Pragma: no-cache
   < Content-Type: text/plain
   < Content-Language: en
   

Note: The URL of the account in this example in this tutorial is https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain.Replace this URL with the URL for your account. For the steps to find out your account URL, see URL for the Account in Using Oracle Cloud Infrastructure Object Storage Classic.

From the output of the command you just ran, note the value of the X-Auth-Token header. This value is the authentication token, which you'll use in the next step. Note that the authentication token expires after 30 minutes, after which you should request a fresh token.

In the example output, the authentication token is AUTH_tk23be53ac28c3e443de38367f87643eb9.

 

Creating a Container

Note: You can upload all the file segments to an existing container. This section shows you how to create a new container to upload all the file segments.

Create a new container to upload all the file segments:

curl -v -X PUT -H "X-Auth-Token: AUTH_tk23be53ac28c3e443de38367f87643eb9" https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain/FirstContainer

Note: When you run this command,
- Replace the value of the X-Auth-Token header with the authentication token that you obtained earlier.
- Change https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain to the X-Storage-Url header value that you noted while getting an authentication token.
- And change FirstContainer to the name of the container that you want to create.

If the request is completed successfully, it returns the HTTP/1.1 201 Created response, as shown in the following sample output.

* About to connect() to foo.storage.oraclecloud.com port 443 (#0)
* Trying 10.79.153.47...
* Connected to foo.storage.oraclecloud.com port 443 (#0)
> PUT /v1/Storage-myIdentityDomain/FirstContainer HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foo.storage.oraclecloud.com:443
> Accept: */*
> X-Auth-Token: AUTH_tk23be53ac28c3e443de38367f87643eb9
>
< HTTP/1.1 201 Created
< X-Trans-Id: tx1d8e9739df4a47bb847e0-005582875bga
< Date: Thu, 18 Jun 2015 08:54:51 GMT
< Content-Type: text/html;charset=UTF-8
< Content-Length: 0
< Server: Oracle-Storage-Cloud-Service
<
* Connection #0 to host foo.storage.oraclecloud.com left intact 

 

Uploading the Segmented Files

Create objects from each segment file (segment_aa, segment_ab...segment_aj), preserving the segment file names. Upload all the objects in the same container FirstContainer. Here’s an example for one of the segment files:

 curl -v -X PUT 
         -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" 
         -T segment_aa 
         https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain/FirstContainer/segment_aa

The following is the output of this command:

> User-Agent: curl/7.29.0
> Host: foo.storage.oraclecloud.com
> Accept: */*
> X-Auth-Token: AUTH_tkc9305a46ebaa0585c4c7ae063c844f0b
> Content-Length: 10485760
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Date: Tue, 15 Dec 2015 10:18:26 GMT
< Last-Modified: Tue, 15 Dec 2015 10:17:21 GMT
< X-Trans-Id: tx85da332ec5ae4852b7d8c-00566fe8b0ga
< Etag: f1c9645dbc14efddc7d8a322685f26eb
< Connection: keep-alive
< X-Last-Modified-Timestamp: 1450174640.10123
< Content-Type: text/html;charset=UTF-8
< Content-Length: 10
< Server: Oracle-Storage-Cloud-Service

                            

 

Creating a Manifest File

Create a manifest file in JSON format, where each element contains the following attributes:

• path: The container and object name in the format:

  containerName/segmentObjectName

• etag: MD5 checksum of the segment object.

  You can find the value from the Etag header that's returned when you upload each segment.

• size_bytes: Size of the segment object.

  You can find the value from the Content-Length header of the segment object.

Ensure that the manifest file contains these three attributes for each segment object.

Sample Manifest file:

[
    {
        "path": "FirstContainer/segment_aa",
        "etag": "f1c9645dbc14efddc7d8a322685f26eb",
        "size_bytes": 10485760
    },
    {
        "path": "FirstContainer/segment_ab",
        "etag": "f1c9645dbc14efddc7d8a322685f26eb",
        "size_bytes": 10485760
    },
    {
        "path": "FirstContainer/segment_ac",
        "etag": "f1c9645dbc14efddc7d8a322685f26eb",
        "size_bytes": 10485760
    },
    {
        "path": "FirstContainer/segment_ad",
        "etag": "f1c9645dbc14efddc7d8a322685f26eb",
        "size_bytes": 10485760
    },
    
    ...    
    {
        "path": "FirstContainer/segment_aj",
        "etag": "f1c9645dbc14efddc7d8a322685f26eb",
        "size_bytes": 10485760
    }
]

 

Uploading the Manifest File

Upload the manifest file that you just created:

curl -v -X PUT 
        -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" 
        "https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain/FirstContainer/myLargeFile?multipart-manifest=put" 
        -T ./manifest.json

• myLargeFile is the name of the static large object
• manifest.json is the name of the manifest file
• ?multipart-manifest=put is the query parameter to upload the manifest file

 

Checking the Size of the Large Object

Run a HEAD request to view the size of the large object (myLargeFile) that you created:

curl -v -X HEAD 
     -H "X-Auth-Token: AUTH_tkbaebb60dfa5b80d84e62b0d5d07031e5" 
https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain/FirstContainer/myLargeFile

The following is the output of this command:

> HEAD /v1/Storage-myIdentityDomain/FirstContainer/myLargeFile HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foo.storage.oraclecloud.com
> Accept: */*
> X-Auth-Token: AUTH_tkc9305a46ebaa0585c4c7ae063c844f0b
< Etag: "e6da53c20abee5c471fe8bf796abb1a4"
< Accept-Ranges: bytes
< Last-Modified: Tue, 15 Dec 2015 10:07:53 GMT
< X-Timestamp: 1455012472.56679
< X-Trans-Id: txcab964b91ba8474ca9193-0056b9bb6fga
< Date: Tue, 15 Dec 2015 10:12:00 GMT
< Connection: keep-alive
< X-Last-Modified-Timestamp: 1455012472.56679
< Content-Type: application/octet-stream;charset=UTF-8
< Content-Length: 104857600
< Server: Oracle-Storage-Cloud-Service
curl: (18) transfer closed with 52428800 bytes remaining to read

You can view the size of the static large object in the header Content-Length. The size of the static large object is the sum total of the sizes of the segment objects.

 

Downloading the Large Object

To download the large object, you must download the manifest object which returns all the segments concatenated as a single file.

curl -v -X GET 
        -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" 
        "https://foo.storage.oraclecloud.com/v1/Storage-myIdentityDomain/FirstContainer/myLargeFile" 
        -o ./myLargeFile1

> GET /v1/Storage-myIdentityDomain/myLargeFile HTTP/1.1
> User-Agent: curl/7.29.0
> Host: foo.storage.oraclecloud.com
> Accept: */*
> X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650
>
< HTTP/1.1 200 OK
< Etag: "6e9647a0cd268b9299e647d28f7027f7"
< Accept-Ranges: bytes
< Last-Modified: Tue, 15 Dec 2015 10:12:00 GMT
< X-Object-Manifest: FirstContainer/segment_
< X-Timestamp: 1453747701.91311
< X-Trans-Id: tx9f77e1e8b7b74de18dc53-0056a8a3f5ga
< Date: Wed, 20 Jan 2016 11:03:18 GMT
< Connection: keep-alive
< X-Last-Modified-Timestamp: 1453747701.91311
< Content-Type: application/zip;charset=UTF-8
< Content-Length: 52428800
< Server: Oracle-Storage-Cloud-Service

 

Want to Learn More?

• https://cloud.oracle.com/storage

• http://docs.oracle.com/cloud/latest/storagecs_common/storagecs_tutorials.htm

 

Credits

• Curriculum Developer: Lakshmi Mohan
• Contributors: Kumar Dhanagopal, Rehan Iftikhar

另外

Deleting a Large Object

A DELETE request will just delete the manifest object itself. The segment data referenced by the manifest will remain unchanged.

A DELETE with a query parameter:

?multipart-manifest=delete

例如

https://master:8080/v1/AUTH_b12f54d365e3416e864809fb7be2b624/con/myLargeFile?multipart-manifest=delete

will delete all the segments referenced in the manifest and then the manifest itself. The failure response will be similar to the bulk delete middleware.