Docs › Managing devices › Verified logging

Verified logging

Izuma Edge provides a verified logging feature you can use to verify whether the logs on a gateway have been tampered with. To do this, Izuma Edge uses the forward secure sealing (FSS) feature of journald.

The verified logging feature generates:
- A verification key, which you must store securely elsewhere.
- A sealing key, which stays on the gateway.
Through a cryptographic operation on log data, the sealing key periodically seals the logs handled by systemd journal. You can detect any tampering of log data before the seal.
A new sealing key is generated periodically.
The old sealing key is deleted after the change.

You can use the verification key to generate the sealing key for any time range, which you can use to verify previous log seals. Changing log file entries before the last seal results in verification failures.

Enabling forward secure sealing on a gateway

To enable the FSS feature of systemd on the gateway:

Enable the GCRYPT module in systemd. To verify and look for +GCRYPT, run:

$ journalctl --version

systemd 244 (244.3+)
-PAM -AUDIT -SELINUX +IMA -APPARMOR -SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=hybrid

Set Seal=yes and Storage=persistent in journald.conf on the gateway.

If you on platform running Izuma Edge Yocto image built using the meta-pelion-edge layers, then the feature was enabled by default in 2.2.0 release. GCRYPT module was added to systemd here and journald configuration file was updated here.

Set up verified logging on a gateway

To set up verified logging during the factory provisioning process, refer to and follow the Typical production provisioning and onboarding flow section of the Izuma Edge provisioning documentation.

Step 4 explains the options that enable the FSS feature. Use -k with the pep-cli get-one-identity command to generate and set up sealing key on the gateway. You can use the -e option to specify the change interval for FSS, which is 10s by default:

PEP_SERVER_URL=http://<api-server-ip-address>:5151 pep-cli get-one-identity -s <serial_number> -i <gateway_ip> -p <fcc_port> -k -e 10s

Verifying logs on the gateway

Get the verification key from the PEP server. On the gateway, run:
```
PEP_SERVER_URL=http://<api-server-ip-address>:5151 pep-cli get-verification-key -s <serial_number>
```
Use the same serial number used during the provisoning process.

Use the verification key obtained from the previous step to verify logs on the gateway:

journalctl –verify –verify-key=<verification_key>

This command runs the verification on the active log file /var/log/journal/<id>/system.journal. Because the journal file is append-only, there could be a race condition between verifying the logs and journald writing the new log entries to the file. Therefore, it leads to an odd behavior where consecutive verify commands could sometimes result in a false alarm of logs being corrupted, as shown:

$ journalctl --verify --verify-key=d0a9b2-923ea9-76d3c7-53a499/991f6ad-989680
079018: Data object references invalid entry at 52d240
File corruption detected at /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal:52cf70 (of 8388608 bytes, 64%).
FAIL: /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal (Bad message)

$ journalctl --verify --verify-key=d0a9b2-923ea9-76d3c7-53a499/991f6ad-989680
PASS: /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal

$ journalctl --verify --verify-key=d0a9b2-923ea9-76d3c7-53a499/991f6ad-989680
03a830: Data object references invalid entry at 52f410
File corruption detected at /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal:52f290 (of 8388608 bytes, 64%).
FAIL: /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal (Bad message)

$ journalctl --verify --verify-key=d0a9b2-923ea9-76d3c7-53a499/991f6ad-989680
PASS: /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal

$ journalctl --verify --verify-key=d0a9b2-923ea9-76d3c7-53a499/991f6ad-989680
PASS: /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal

$ journalctl --verify --verify-key=d0a9b2-923ea9-76d3c7-53a499/991f6ad-989680
PASS: /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal

To fix this, rotate the journal file before running the verify command:
```
journalctl --rotate
```
This creates a backup of the active log file system.journal and saves the new file at /var/log/journal/<id>/system@<id>.journal.
Run the verify command on the rotated file and not on the active log file:
```
journalctl --verify --verify-key=<key> --file=/var/log/journal/<id>/system@<id>.journal
```
This give a consistent result.

Verification successful output

$ journalctl --verify --verify-key=59726c-139807-191c78-8e2dfe/98f61ec-989680

PASS: /var/log/journal/5819ed35d81c4c42bd2b59396c576410/system.journal
=> Validated from Wed 2020-10-28 18:05:28 EDT to Wed 2020-10-28 18:17:56 EDT, final 12.270057s entries not sealed.
PASS: /var/log/journal/5819ed35d81c4c42bd2b59396c576410/system@ffdc51ad1f1e44efbdb09135e35f21d4-0000000000004b9a-0005b2c199ef5033.journal
=> Validated from Wed 2020-10-28 17:11:04 EDT to Wed 2020-10-28 18:05:18 EDT, final 10.141951s entries not sealed.
700ea0: Unused data (entry_offset==0)
PASS: /var/log/journal/5819ed35d81c4c42bd2b59396c576410/system@ffdc51ad1f1e44efbdb09135e35f21d4-0000000000002694-0005b2c09f9866ee.journal
7e8d98: Unused data (entry_offset==0)
PASS: /var/log/journal/5819ed35d81c4c42bd2b59396c576410/system@ffdc51ad1f1e44efbdb09135e35f21d4-0000000000000001-0005b2aebd945d98.journal

Verification failure output

$ journalctl --verify --verify-key=d0a9b2-923ea9-76d3c7-53a499/991f6ad-989680

Data object references invalid entry at 52f410
File corruption detected at /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal:52f290 (of 8388608 bytes, 64%).
FAIL: /var/log/journal/b14a872b3d8648c1baa289589a1d7bc5/system.journal (Bad message)

Documentation