Uploading a large firmware image

Device Management Update lets you upload large firmware images in multiple parts (chunks) where each request is less than or equal to 100 MiB (one hundred mebibytes). This is necessary if the final firmware image resource is greater than 100 MiB. You can do this using the Update Service APIs.

The minimum chunk size is 5 MiB (five mebibytes). In cases where your firmware chunk size is under 5 MiB, the upload process will not fail until the last packet is sent. This is because firmware images are stored in an S3 bucket, and the S3 APIs do not send a message of failure until the last packet.

Summary of API interfaces

You can click on each operation to see the full API reference.

Please consult the troubleshooting information on this page if you encounter any difficulty during a multi-part upload.

Creating an upload job

To create a multi-part upload job, use POST /v3/firmware-images/upload-jobs with header Content-Type: application/json:

    {
      "name": "<name>",
      "description": "<description>"
    }

The field name must be unique for each account, while the field description is optional; both default to an empty string.

Note: The response bodies in this tutorial are for demonstration purposes only. Your response bodies will differ.

The POST response consists of:

• A 201 Created status.
• A Location header containing the URI for the new firmware image upload job resource.
• A Content-Location header containing the same URI.
• A response body like:

    {
        "completed": false,
        "created_at": "2018-08-31T10:49:41.736515Z",
        "description": "<description>",
        "etag": "2018-08-31T10:49:41.736515Z",
        "firmware_image_id": "",
        "id": "01658f9a586800000000000100100023",
        "name": "<name>",
        "object": "upload-job",
        "status": "not_started",
        "updated_at": "2018-08-31T10:49:41.736515Z",
    }

The status is not_started because you have not yet uploaded any chunks for this job. The completed field is false until the upload job is finished.

The firmware_image_id is empty because you have not finished the upload. There is no corresponding firmware image resource until the job is complete.

Returning info on existing upload jobs

To see info on upload jobs, GET /v3/firmware-images/upload-jobs, which returns a 200 OK status, and a response like:

{
      "after": None,
      "data": [
          {   
              "completed": false,
              "created_at": "2018-08-31T15:09:06.659124Z",
              "description": "<description>",
              "etag": "2018-08-31T15:09:06.659124Z",
              "firmware_image_id": "",
              "id": "01659087d8e300000000000100100015",
              "name": "<name>",
              "object": "upload-job",
              "status": "in_progress",
              "updated_at": "2018-08-31T15:09:06.659124Z"
          },
          {
              "completed": false,
              "created_at": "2018-08-31T15:09:06.663841Z",
              "description": "<description>",
              "etag": "2018-08-31T15:09:06.663841Z",
              "firmware_image_id": "",
              "id": "01659087d8e900000000000100100016",
              "name": "<another name>",
              "object": "upload-job",
              "status": "not_started",
              "updated_at": "2018-08-31T15:09:06.663841Z"
          }
      ],
      "has_more": False,
      "limit": 50,
      "object": "list",
      "order": "ASC"
}

Uploading a new chunk

You must call the API each time you want to add a new chunk. The service appends data onto an existing upload.

Note: Since there is no way to specify chunk index, you cannot upload chunks in parallel. Doing so results in a corrupted firmware image. Wait for a 201 Created status before proceeding to the next chunk.

To do this, POST /v3/firmware-images/upload-jobs/{upload_job_id}/chunks with headers:

• Content-Type: binary/octet-stream.
• Content-Length: <data-length>.
• Content-MD5: <per-RFC1864>.

The body must contain the binary data for the chunk.

The response consists of:

• A 201 Created status.
• A Location header containing the URI for the new firmware image upload chunk resource.
• A Content-Location header containing the same URI.
• JSON like:

    {
        "created_at": "2018-08-31T10:49:41.742352Z",
        "etag": "2018-08-31T10:49:41.742352Z",
        "hash": "e3237bc98b00f204e8800999ecff316e",
        "id": 1,
        "length": 14,
        "updated_at": "2018-08-31T10:49:41.742352Z"
    }

The id is the chunk index, starting at 1 for the first chunk posted to the firmware image upload job, and increasing by one with each uploaded chunk.

The length reflects the number of bytes in the binary data. The binary data in a chunk must be at least 5 MB, except for the last chunk of the sequence (and ignoring the zero-length chunk that finishes the upload).

The hash field is the hexadecimal form of the Content-MD5 message integrity check.

Viewing metadata for all uploaded chunks

To view metadata for all uploaded chunks in a job, use GET /v3/firmware-images/upload-jobs/{upload_job_id}/chunks, which returns a 200 OK status and a response body like the one below.

This example shows two uploaded firmware image chunks. See Posting a chunk for more information on the various fields for each chunk below:

    {
        "after": None,
        "data": [
            {
                "created_at": "2018-08-31T14:44:25.023959Z",
                "etag": "2018-08-31T14:44:25.023959Z",
                "hash": "e3237bc98b00f204e8800999ecff316e",
                "id": 1,
                "length": 14,
                "updated_at": "2018-08-31T14:44:25.023959Z"
            },
            {
                "created_at": "2018-08-31T14:44:25.033691Z",
                "etag": "2018-08-31T14:44:25.033691Z",
                "hash": "d41d8cd98f00b204e9800998ecf8427e",
                "id": 2,
                "length": 19,
                "updated_at": "2018-08-31T14:44:25.033691Z"
            }
        ],
        "has_more": False,
        "limit": 50,
        "object": "list",
        "order": "ASC"
    }

Alternatively, if there are no chunks (either because you haven't uploaded any yet, or you completed an upload):

    {
        "after": None,
        "data": [],
        "has_more": False,
        "limit": 50,
        "object": "list",
        "order": "ASC"
    }

Viewing metadata for a specific chunk

To view metadata for a specific uploaded chunk, use GET /v3/firmware-images/upload-jobs/{upload_job_id}/chunks/{chunk_id}:

  {   
      "created_at": "2018-08-31T15:23:55.912834Z",
      "etag": "2018-08-31T15:23:55.912834Z",
      "hash": "d41d8cd98f00b204e9800998ecf8427e",
      "id": 1,
      "length": 14,
      "updated_at": "2018-08-31T15:23:55.912834Z"
  }

This returns a 200 OK status.

Ending a multi-part upload

Sending a chunk with zero data length ends the upload job and creates a record of the firmware image (firmware_image_id).

To do this, use POST /v3/firmware-images/upload-jobs/{upload_job_id}/chunks with headers:

• Content-Length: 0.
• Content-MD5: <per-RFC1864>.

Note: You still need to include headers when posting an empty response body.

This returns a 201 Created response and Location and Content-Location headers as for any other chunk.

For example, if you upload four data chunks, then the POST of a zero-length chunk returns JSON like:

    {
        "created_at": "2018-08-31T10:49:41.742352Z",
        "etag": "2018-08-31T10:49:41.742352Z",
        "hash": "d41d8cd98f00b204e9800998ecf8427e",
        "id": 5,
        "length": 0,
        "updated_at": "2018-08-31T10:49:41.742352Z"
    }

Note: You cannot complete an upload that has had no data. The first chunk uploaded cannot be of length zero. Attempting this gives a 400 Bad Request response. If you want to delete an upload job, see Delete a firmware image upload job.

You can then view information about the completed firmware image upload job using GET /v3/firmware-images/upload-jobs/{upload_job_id}, which returns JSON like:

  {
     "completed": true,
     "created_at": "2018-08-31T10:49:41.736515Z",
     "description": "<description>",
     "etag": "2018-08-31T10:49:41.756045Z",
     "firmware_image_id": "01658f9a587800000000000100100024",
     "id": "01658f9a586800000000000100100023",
     "name": "<name>",
     "object": "upload-job",
     "status": "completed",
     "updated_at": "2018-08-31T10:49:41.756045Z"
  }

View the newly created firmware image using GET /v3/firmware-images/{image_id}/. For example, GET /v3/firmware-images/01658f9a587800000000000100100024 returns the data for the firmware image resource, with the location of the image in the datafile field:

  {
     "completed:" true,
     "created_at": "2018-08-31T10:49:41.751725Z",
     "datafile": "http://example.com/00000000000000000000000000000001",
     "datafile_checksum": "1234",
     "datafile_size": 27,
     "description": "<description>",
     "etag": "2018-08-31T10:49:41.751725Z",
     "id": "01658f9a587800000000000100100024",
     "name": "<name>",
     "object": "firmware-image",
     "updated_at": "2018-08-31T10:49:41.755303Z"
  }

Deleting a firmware image upload job

Deleting an upload job cancels any ongoing upload job, and removes the upload job resource. You cannot undo or continue the job after this.

To do this, use DELETE /v3/firmware-images/upload-jobs/{upload_job_id}. The response is a 204 No Content status and no response body.

Troubleshooting

Posting to a completed upload job

If you try to POST to a firmware image upload job that has already finished (with status completed), the result is a 400 Bad Request.

409 Conflict

This conflict happens when you use POST /v3/firmware-images/upload-jobs but you already have either a /v3/firmware-images/:id with the same name as given, or a /v3/firmware-images/upload-jobs/:id with the same name as given.