Docs › Provisioning devices › Izuma Edge provisioning (pep) tool

Izuma Edge provisioning (pep) tool

The Izuma Edge provisioning (pep) tool lets you automate the gateway provisioning flow. The pep tool injects production-mode identity certificates and configuration information onto the gateways over IP. The automation is possible because of these features:

The gateway initiates certificate injection, instead of Factory Configurator Utility (FCU), which initiates the injection in the default Device Management provisioning flow.
The pep tool records the enrollment identity and configuration information of all provisioned gateways.

Use cases

Common use cases of the pep tool include the manufacturing floor, inventory management and testing with production credentials.

Manufacturing floor:

Consider a factory set up to provision Izuma Edge enabled gateways with 15 workstations running in parallel. The default provisioning flow, using FCU and FCC, requires accessing the terminal of the gateway to retrieve its IP address, serial number and the port on which FCC runs. You then have to enter the information into FCU, which typically runs outside the gateway on a local factory server. These manual steps reduce operational efficiency, increase complexity and are prone to user error. Therefore, the factory flow generally requires a headless environment in which you automate all steps, and the status is reported through the colors of the LED connected to the gateway.

Please note that this is not a factory tool that automates the complete factory flow process. It is intended to automate the Izuma provisioning flow and provide an easy integration with the factory tool, considering the factory tool is set up according to the customer's and factory's specific requirements.

After you configure the pep API server and run the pep tool successfully in the factory, you can provision any gateway on the same local network without any manual effort.
Inventory management:

If your factory dispatches gateways to various end users or tenants, you need a mechanism to track gateways dispatched to specific end users. The pep tool has a database that stores information about dispatched gateways and can later be exported to manage your inventory and the ownership of gateways.
Testing with production credentials:

You can also use the pep tool internally to iteratively test the Izuma Edge firmware in production mode without having to set up separate or individual FCUs, and your QA team can use the tool to automate regression testing and CI workflows.

Architecture

In the default Device Management gateway provisioning flow, FCU (running outside the gateway) injects the credentials in the gateway over IP. This requires knowing the serial number and IP address of the gateway. This flow cannot be automated because it requires manual effort to read the serial number and IP address off of the gateway. The pep tool inverts this flow by providing REST APIs that allow the gateway to send its serial number and IP address to the server and request the credentials on demand.

The server also saves the identity information and gateway configurations in the database, which you can later download or upload to a server for inventory management or to prime the system for the first-to-claim process.

The server is a multicontainer Docker application with two services:

pep API server: a node.js based RESTful web service that exposes APIs consumed by the client, pep CLI. You can find the API documentation in the pelion-edge-provisioner repository. This service internally interacts with FCU and leverages the ft-demo tool, which provides the IP interface to inject the certificates in the gateway. Therefore, the precursor to building the Docker image is to successfully download the factory_configurator_utility archive file from the Izuma Device Management Portal and configure it according to your use case. The details of that can be found in the Prerequisite section. To know more about the Docker image the instructions used to assemble it are stated in this Dockerfile.
Mongo: a service that deploys an instance of MongoDB. The database is used for record keeping. It stores the nonsensitive gateway information, such as gateway's serial number and its configuration parameter values. After the gateway is provisioned successfully in first-to-claim mode, it also stores the enrollment identity, which is the fingerprint of the device certificate of the gateway, but it never stores the device certificate itself. You can export the stored information into CSV format using pep CLI, which you can use for inventory management. When using the first-to-claim process, you can upload it to the Portal using the bulk upload feature.

You can use Docker's compose tool to configure and define the above service containers. It creates:

A default bridge network which setup for containers to communicate with each other.
A Docker volume mongo_data is created to persistently store the mongo data.

The environment file .env lists the versions of the services docker-compose uses.

Prerequisites

Physical setup requirements

The server can run on any Docker supported platform and client on any Izuma Edge supported platforms. Both the server and gateway need an IP address and to be connected to the same LAN network.
The server machine must allow inbound and outbound HTTP traffic on port 5151 and also TCP traffic on the port used by Factory Configurator Client (FCC), which is running on the client machine.

Note: We recommend you install the server on a secure machine, in a secure room or both to protect the device keys and certificates. Set up production flow in accordance with good security practices, and use secure processes and Hardware Security Module (HSM) hardware when possible.

Software requirements

Create a ./fcu_config_dir folder that contains the required files:

./fcu_config_dir
--- fcu.yml
--- factory_configurator_utility.zip
--- update-auth-certificate.der
--- keystore
--- --- CA_private.pem
--- --- CA_cert.pem

Place the fcu_config_dir folder in the root directory of the repository for the contents to be copied to the Docker container.

factory_configurator_utility.zip is the FCU archive.

We license FCU to Device Management customers who manufacture connected devices. Please contact us for more information. Authorized customers can download the tool and documentation from the Device Management Portal.

fcu.yml is the FCU configuration file.

Follow the documentation to configure FCU. You can find the default fcu.yml in the archive file at factory_configurator_utility.zip/config/fcu.yml.

Example: Typically, you configure fcu.yml with the following values; everything else remains unchanged.

Parameter	Value
use-bootstrap	true
time-sync	true
verify-on-device	true
first-to-claim	true
update-auth-certificate-file	<%= ENV['FCU_RESOURCES_DIR'] %>/update-auth-certificate.der
vendor-id	Valid UUID String (RFC4122)
class-id	Valid UUID String (RFC4122)
device-info	fill the information as per your organization
device-key-generation-mode	by_tool (or by_device when using TPM)
device-certificate	fill the information as per your organization
entropy-generation-mode	by_device
secure-storage-type	default value
secure-hardware-components	default values
device-private-keys-in-field-generation	by_cloud

If Trusted Platform Module (TPM) v2.0 is available on your gateway, consider securing Izuma Edge with Parsec and TPM. To make the TPM generate the private key, set the key generation mode in fcu.yml to by_device:

device-key-generation-mode: by_device

CA_private.pem and CA_cert.pem are the certificate authority (CA) private key certificate.

When your devices connect to Device Management, they need to trust the CA certificate that issued the device certificate or one of the CAs in the device certificate chain of trust. To set up a certificate authority, follow the documentation to create your own CA. Place the generated private key, CA_private.pem, and certificate, CA_cert.pem, in the keystore folder.

Note: As you are creating your own certificate authority, do not populate the X.509 properties in the certificate-authority section of the fcu.yml configuration file.

Caution: The OpenSSL commands in the docs are for reference only. You must adapt the commands to your own production setup and security requirements.

Note: You can also set up FCU as a CA, but that requires you to manually install the FCU and invoke the setup API. Because installation of FCU is automated in the Docker build process, we recommend you use generate your own CA.
update-auth-certificate.der is the firmware update authentication certificate (optional).

You only need this certificate if you intend to update your firmware. For more information about authentication certificate use and requirements, please see the authenticity certificate documentation.

For testing purposes, you can use the manifest tool to generate the authentication certificate. This enables you to develop the firmware updates for your gateway. These certificates generated above are not suitable for production environments. Only use them for testing purposes.

For example, you can run this command to generate the update certificate:
```
manifest-dev-tool init --vendor-id <Valid UUID String (RFC4122)>
```
You can find the DER formatted update certificate at .manifest-dev-tool/dev.cert.der. Rename it to update-auth-certificate.der, and place it in the ./fcu_config_dir folder.

Manually run the ./fcu-config-validator.sh validator script on fcu_config_dir to verify the files are in the correct structure.
Upload the CA_cert.pem certificate to Device Management.

Make sure to select Enrollment from the How will devices use this certificate? dropdown.

Installing the server using Docker

Use docker-compose to build and run the containers (these docker-compose commands have been tested with docker-compose version 1.25.0):

Clone this repository:

git clone https://github.com/PelionIoT/pelion-edge-provisioner

Build the pep server:

docker-compose build --build-arg fcu_config=<fcu_config_relative_dir_path>

Run the pep server:
```
docker-compose up -d
```
Verify that the mongo and pep API server containers are running:
```
docker-compose ps
```
If a container is restarting, view the logs to troubleshoot:
```
docker-compose logs
```
1. View the API server logs:
```
docker logs -f pep-api-server
```
2. Verify the mongo_data volume was created successfully:
```
docker volume inspect pelion-edge-provisioner_mongo_data
```
If the pep API server is running successfully, you can view the Swagger API docs at http://localhost:5151/docs/.

Installing the pep CLI tool

pep CLI is a command-line tool that interacts with the pep API server using cURL. To test, you can run it on the same machine as the server, but in production, it runs on the Izuma Edge supported platforms. The supported commands are detailed in the commands section.

Clone this repository on your gateway:

git clone https://github.com/PelionIoT/pelion-edge-provisioner

Install pep CLI to /usr/local/bin by creating a symlink:
```
cd pelion-edge-provisioner
sudo mkdir -p /usr/local/bin
sudo ln -s `pwd`/cli/bash/pep-cli.sh /usr/local/bin/pep-cli
sudo pep-cli --help
```
Note: Make sure /usr/local/bin is already in your PATH. To verify, run echo $PATH. If it is not in your PATH, add it.

Alternatively, go to:
```
cd cli/bash
./pep-cli.sh --help
```
To enable debug mode:
```
DEBUG=* pep-cli --help
```
When you run the CLI on the gateway, you can provide the PEP_SERVER_URL as an env variable:
```
PEP_SERVER_URL=http://<api-server-ip-address>:5151 pep-cli --help
```

Pep CLI commands

get-one-identity: Requests the pep API server to create a new identity based on the passed parameters. The prerequisite is to run FCC first. The steps on how to install and run FCC are documented in the typical production provisioning and onboarding flow section.
- In the request to the server, provide the gateway IP address and the port at which FCC is running, so the server can then invoke FCU to inject a new device certificate in the gateway.
- Run the help command to learn about the various parameters you can pass:
```
pep-cli get-one-identity --help
```
get-enrollment-id: After the gateway has been injected with a device certificate, you can request the server to provide the enrollment ID of the gateway, so you can upload it to the Device Management account with which you want the gateway to be associated:
```
pep-cli get-enrollment-id --help
```
get-verification-key: To verify logs on the gateway using journald's forward secure sealing (FSS) feature, you can request the server to provide the verification key associated with the gateway:
```
pep-cli get-verification-key --help
```
You can only use this when you have enabled journald's FSS while generating the new identity for the gateway.
list-enrollment-ids: Gets the enrollment identities of dispatched gateways in CSV and JSON format. The CSV formatted information is compatible with the bulk upload feature of Portal. Therefore, the information can directly be uploaded to a Izuma account through the Portal or using the bulk upload REST API. The command returns paginated data. By default, the command returns 50 entries in descending order. To enumerate through the list, you can provide the following command line parameters:
- limit: Number of results to return. Between 1 and 1000.
- order: The order in which the identities have been dispatched. ASC for ascending, DESC for descending.
- last: The ID of the last entry in the previous result. Marks the start of the next page.
For more information, run:
```
pep-cli list-enrollment-ids --help
```

Typical production provisioning and onboarding flow

Prepare:
1. Log in to the gateway.
2. Stop and mask the Edge Core service, and clear out the old provisioning files. For example, if you are running LmP Yocto Izuma Edge on Raspberry Pi 3 Model B+, run:
```
sudo systemctl mask edge-core
sudo systemctl stop edge-core
sudo rm -rf /userdata/mbed/mcc_config
sudo rm -rf /userdata/edge_gw_config
```
  Or run the ./example-scripts/meta-pelion-edge/prepare-for-new-identity.sh script. Please modify the commands or the scripts for your platform setup.
Run FCC:
1. On the gateway, run:
```
sudo ./factory-configurator-client-example.elf
```
  If you are using LmP with Izuma Edge, you can find the precompiled binary located at /wigwag/wwrelay-utils/I2C/factory-configurator-client-armcompiled.elf.
  
  By default, the interface is eth0. If required, you can change the interface name:
```
sudo ETHERNET_LINUX_IFACE_NAME=wlan0 ./factory-configurator-client-example.elf
```
  You can change the entropy source:
```
export ENTROPYSOURCE=/dev/urandom
```
2. Note down the TCP port of FCC: You will use it in the next step.
Fetch a new identity:
1. Open another terminal.
2. Log in to the gateway.
3. Run the following command. At minimum, you must pass in the serial number of the gateway, the IP address of the gateway and the TCP port at which FCC is running.
Note: There is no schema enforced on the serial_number, but we recommend using only the characters a-z, A-Z,0-9, '+,-.:/= and SPACE. For more information, see endpoint name. However, it has to be unique. The server will not provision two gateways with the same serial number.
1. (Optional) Enable the forward secure sealing feature of journald to verify logs on the gateway and detect tampering.
  1. Make sure journald supports this feature on your platform. Follow these prerequisites to enable and verify journald's FSS is supported.
  2. Use -k with the get-one-identity command. This generates a sealing key and a verification key.
  3. Use -e to specify the change interval for forward secure sealing, which is 10s by default.
```
sudo PEP_SERVER_URL=http://<api-server-ip-address>:5151 pep-cli get-one-identity -s <serial_number> -i <gateway_ip> -p <fcc_port>
```
Install a new identity for the Izuma Edge stack NOT secured by Parsec with TPM. After the above command runs successfully, the factory-configurator-client-example creates a pal folder, and pep CLI creates identity.json file.
1. Place and rename the pal folder to the location specified by the compile time flag PAL_FS_MOUNT_POINT_PRIMARY of Edge Core. By default, it is defined by this CMake flag. The meta-pelion-edge project overwrites that to /userdata/mbed/mcc_config and snap-pelion-edge to this.
  
  Note: Before copying to that location, make sure the folder is not present. Edge Core creates this folder only if this folder is not already present. Therefore, stop Edge Core, and clear out that folder before copying the pal folder.
2. Place identity.json at the location specified by platform_readers/params/identity_path in the Maestro configuration file. If you are using LmP with Izuma Edge, the identity.json location is /userdata/edge_gw_config/identity.json. If you are using snapcraft, use this location.
For example, when using LmP Izuma Edge, you can run the following commands on the gateway:
```
sudo mkdir -p /userdata/edge_gw_config
sudo mv identity.json /userdata/edge_gw_config/
sudo mv pal/ /userdata/mbed/mcc_config
```
The same steps are listed in the install-new-identity-without-parsec.sh script. You can modify the commands or the scripts for your platform setup.
Install a new identity for the Izuma Edge stack secured by Parsec with TPM. After the above command runs successfully, the factory-configurator-client-example creates a /userdata/mbed/mcc_config folder, and pep CLI creates identity.json file.
1. Place identity.json at the location specified by platform_readers/params/identity_path in the Maestro configuration file. If you are using LmP with Izuma Edge, place identity.json at this location. If you are using snapcraft with Izuma Edge, use this location.
For example, when using LmP Izuma Edge, you can run the following commands on the gateway:
```
sudo mkdir -p /userdata/edge_gw_config
sudo mv identity.json /userdata/edge_gw_config/
```
The same steps are listed in the ./example-scripts/meta-pelion-edge/install-new-identity-with-parsec.sh script. You can modify the commands or the scripts for your platform setup.

To verify if bootstrap key is stored in the TPM v2.0, use parsec-tool:
```
sudo parsec-tool list-keys
```
For example, successful output looks like:
```
[INFO ] Available keys:
* parsec-se-driver-key0 (Mbed Crypto provider, EccKeyPair { curve_family: SecpR1 }, 256 bits, permitted algorithm: AsymmetricSignature(Ecdsa { hash_alg: Specific(Sha256) }))
```
Note: A known issue in Parsec service 0.6.0 (which Izuma Edge uses) reports the default provider as Mbed Crypto instead of TPM.
Upload the enrollment identity:
1. To retrieve the enrollment ID of this gateway, send a request to the pep API server.
```
sudo pep-cli get-enrollment-id -s <serial_number>
```
1. Upload the enrollment identity to Device Management by following the First-to-Claim by enrollment list documentation
Connect to Device Management:
1. Unmask the Edge Core service:
```
sudo systemctl unmask edge-core
```
1. Reboot the gateway.
2. Verify the gateway is connected to Device Management. For example, in meta-pelion-edge, you can run this command to know the status of Edge Core:
```
curl localhost:9101/status
```

How to automate the provisioning flow

Pre-installed FCC with the Izuma Edge firmware image.
Create a bash script to:
1. Start FCC and capture the TCP port.
2. Read the serial number of the gateway. For example, you can find serial number in /proc/cpuinfo on Raspberry Pi.
3. Get the IP address of the gateway. For example, you can grep the IP address from ifconfig.
4. Delete the old identity files and reset the gateway configurations.
5. Run the pep CLI with the parameters above.
6. Install the new identity on the gateway and then reboot.

Typically, these steps are integrated with the factory tool, which is responsible for installing the gateway firmware image and validating the hardware.

Reprovision the gateway

To provision the same device twice, you must completely erase the provisioned contents because a device can't be provisioned twice:

sudo rm -rf /userdata/mbed/mcc_config
sudo rm -rf /userdata/edge_gw_config
sudo rm -rf /userdata/info/relaystatics.sh
sudo systemctl stop parsec
sudo systemctl stop swtpm
sudo rm -rf /userdata/parsec/
sudo systemctl start swtpm
sudo systemctl start parsec
sudo systemctl restart edge-core
sudo systemctl restart wait-for-pelion-identity

Troubleshooting

When compiling factory-configurator-client-example, if you see this error on the gateway:
```
ImportError: No module named site
```
Unset the Python path:
```
unset PYTHONPATH
unset PYTHONHOME
```
If you see bad-request error on mbed-edge, mbed-edge and factory-configurator-client-example have been compiled with different entropy sources.
If you see Connection error on mbed-edge, make sure the enrollment identity and the CA certificate has been uploaded successfully to your Device Management account.

When you run the FCC example with Parsec support, if you see error No such file or directory:

[2020-11-07T01:40:10Z ERROR parsec_se_driver] error getting available providers: Client(Ipc(Os { code: 2, kind: NotFound, message: "No such file or directory" })).
[ERR ][fcc ]: psa_driver_crypto.c:244:psa_drv_crypto_init:<=== Failed to initialize crypto module
[ERR ][fcc ]: psa_driver_common.c:50:psa_drv_translate_to_kcm_error:psa_status: -132, kcm_status: 0x1
[ERR ][fcc ]: key_slot_allocator.c:714:ksa_init:<=== Failed initializing PSA Crypto driver (1)
[ERR ][fcc ]: storage_psa.cpp:1086:storage_init:<=== Failed initializing KSA (kcm_status 1)
[ERR ][fcc ]: key_config_manager.c:46:kcm_init:<=== Failed initializing storage

[ERR ][fcc ]: storage_psa.cpp:760:storage_get_item_name_for_deleting:<=== KCM initialization failed

[ERR ][fcc ]: storage_psa.cpp:855:storage_factory_item_delete:<=== Failed to get storage name for item = mbed.BootstrapDevicePrivateKey len = 30
[ERR ][fcc ]: storage_psa.cpp:1144:storage_reset:<=== Failed to remove bootstrap private key
[ERR ][fcc ]: factory_configurator_client.c:146:fcc_storage_delete:<=== Failed for storage reset
[ERR ][fce ]: Failed to reset storage

Verify /run/parsec/parsec.sock is created by the Parsec service. This is the default socket file location, and it might not be defined in the Parsec service configuration file /etc/parsec/config.toml.

Documentation