Examining ZFS Pools with zdb

The zdb (ZFS Debug) command is a useful tool for examining the low-level structure and metadata of a ZFS pool or dataset, and can read the information from the on-disk data structures of exported pools as well as from the ZFS pool cache file. The output is subject to change over time as ZFS is further developed, which means that the exact content contained in the output may vary depending on which version of ZFS is being used.

This guide will not go into the details of the zdb command but will highlight one useful feature for examining ZFS metadata.

To read the configuration of a zpool that is imported on a host, use the following command:

zdb -C[C] <poolname>

If the -C option is used, the output will display the cached content from the zpool configuration cache. When -C is referenced more than once, zdb will also display on-disk configuration (called the Meta Object Set, or MOS).

For example:

[root@rh7z-mds1 ~]# zpool list mgspool
NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
mgspool  1008M  2.23M  1006M         -     1%     0%  1.00x  ONLINE  -
[root@rh7z-mds1 ~]# zdb -C mgspool

MOS Configuration:
        version: 5000
        name: 'mgspool'
        state: 0
        txg: 34022
        pool_guid: 11089712772589408485
        errata: 0
        hostid: 1489912803
        hostname: 'rh7z-mds1'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 11089712772589408485
            children[0]:
                type: 'mirror'
                id: 0
                guid: 10014780520217715169
                metaslab_array: 34
                metaslab_shift: 24
                ashift: 9
                asize: 1058537472
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 12166736421740210173
                    path: '/dev/sda1'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 8586593287004549671
                    path: '/dev/sdc1'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

If the pool has not been imported to the current host, or if there is no zpool cache file on the host for a currently imported pool, use the following command:

zdb -e <poolname>

This will provide a very large amount of output. To get the MOS configuration, use the following:

zdb -eC[C] <poolname>

For example:

[root@rh7z-mds1 ~]# zdb -eC mgspool

MOS Configuration:
        version: 5000
        name: 'mgspool'
        state: 1
        txg: 34189
        pool_guid: 11089712772589408485
        errata: 0
        hostid: 1489912803
        hostname: 'rh7z-mds1'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 11089712772589408485
            children[0]:
                type: 'mirror'
                id: 0
                guid: 10014780520217715169
                metaslab_array: 34
                metaslab_shift: 24
                ashift: 9
                asize: 1058537472
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 12166736421740210173
                    path: '/dev/sda1'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 8586593287004549671
                    path: '/dev/sdc1'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

Note: The MOS includes the hostid of the last host to import the zpool. If the hostid (as presented by the SPL) of the node that imported the zpool is 0 (zero), then this field will not be presented in the zdb MOS output. If there is no hostid in the MOS, then this strongly indicates that the hosts are not configured with proper hostids, and the ZFS volume will not be adequately protected from multiple concurrent accesses from multiple hosts.

Please see the article Protecting File System Volumes from Concurrent Access for details on the importance of correctly setting the hostid when working with high availability failover storage configurations.