Managing ZFS Storage Pools
This chapter describes how to create and administer ZFS storage pools.
The following sections are provided in this chapter:
Components of a ZFS Storage Pool
The following sections provide detailed information about the following storage pool components:
Using Disks in a ZFS Storage Pool
The most basic element of a storage pool is a piece of physical storage. Physical storage can be any block device of at least 128 Mbytes in size. Typically, this device is a hard drive that is visible to the system in the /dev/dsk directory.
A storage device can be a whole disk (c1t0d0) or an individual slice (c0t0d0s7). The recommended mode of operation is to use an entire disk, in which case the disk does not need to be specially formatted. ZFS formats the disk using an EFI label to contain a single, large slice. When used in this way, the partition table that is displayed by the format command appears similar to the following:
Current partition table (original): Total disk sectors available: 71670953 + 16384 (reserved sectors) Part Tag Flag First Sector Size Last Sector 0 usr wm 34 34.18GB 71670953 1 unassigned wm 0 0 0 2 unassigned wm 0 0 0 3 unassigned wm 0 0 0 4 unassigned wm 0 0 0 5 unassigned wm 0 0 0 6 unassigned wm 0 0 0 7 unassigned wm 0 0 0 8 reserved wm 71670954 8.00MB 71687337
To use whole disks, the disks must be named using the standard Solaris convention, such as /dev/dsk/cXtXdXsX. Some third-party drivers use a different naming convention or place disks in a location other than the /dev/dsk directory. To use these disks, you must manually label the disk and provide a slice to ZFS.
ZFS applies an EFI label when you create a storage pool with whole disks. Disks can be labeled with a traditional Solaris VTOC label when you create a storage pool with a disk slice.
Slices should only be used under the following conditions:
The device name is nonstandard.
A single disk is shared between ZFS and another file system, such as UFS.
A disk is used as a swap or a dump device.
Disks can be specified by using either the full path, such as /dev/dsk/c1t0d0, or a shorthand name that consists of the device name within the /dev/dsk directory, such as c1t0d0. For example, the following are valid disk names:
Using whole physical disks is the simplest way to create ZFS storage pools. ZFS configurations become progressively more complex, from management, reliability, and performance perspectives, when you build pools from disk slices, LUNs in hardware RAID arrays, or volumes presented by software-based volume managers. The following considerations might help you determine how to configure ZFS with other hardware or software storage solutions:
If you construct ZFS configurations on top of LUNs from hardware RAID arrays, you need to understand the relationship between ZFS redundancy features and the redundancy features offered by the array. Certain configurations might provide adequate redundancy and performance, but other configurations might not.
You can construct logical devices for ZFS using volumes presented by software-based volume managers, such as SolarisTM Volume Manager (SVM) or Veritas Volume Manager (VxVM). However, these configurations are not recommended. While ZFS functions properly on such devices, less-than-optimal performance might be the result.
For additional information about storage pool recommendations, see the ZFS best practices guide.
Disks are identified both by their path and by their device ID, if available. This method allows devices to be reconfigured on a system without having to update any ZFS state. If a disk is switched between controller 1 and controller 2, ZFS uses the device ID to detect that the disk has moved and should now be accessed using controller 2. The device ID is unique to the drive's firmware. While unlikely, some firmware updates have been known to change device IDs. If this situation happens, ZFS can still access the device by path and update the stored device ID automatically. If you inadvertently change both the path and the ID of the device, then export and re-import the pool in order to use it.
Using Files in a ZFS Storage Pool
ZFS also allows you to use UFS files as virtual devices in your storage pool. This feature is aimed primarily at testing and enabling simple experimentation, not for production use. The reason is that any use of files relies on the underlying file system for consistency. If you create a ZFS pool backed by files on a UFS file system, then you are implicitly relying on UFS to guarantee correctness and synchronous semantics.
However, files can be quite useful when you are first trying out ZFS or experimenting with more complicated layouts when not enough physical devices are present. All files must be specified as complete paths and must be at least 64 Mbytes in size. If a file is moved or renamed, the pool must be exported and re-imported in order to use it, as no device ID is associated with files by which they can be located.
Identifying Virtual Devices in a Storage Pool
Each storage pool is comprised of one or more virtual devices. A virtual device is an internal representation of the storage pool that describes the layout of physical storage and its fault characteristics. As such, a virtual device represents the disk devices or files that are used to create the storage pool.
Two top-level virtual devices provide data redundancy: mirror and RAID-Z virtual devices. These virtual devices consist of disks, disk slices, or files.
Disks, disk slices, or files that are used in pools outside of mirrors and RAID-Z virtual devices, function as top-level virtual devices themselves.
Storage pools typically contain multiple top-level virtual devices. ZFS dynamically stripes data among all of the top-level virtual devices in a pool.
Replication Features of a ZFS Storage Pool
ZFS provides data redundancy, as well as self-healing properties, in a mirrored and a RAID-Z configuration.
Mirrored Storage Pool Configuration
A mirrored storage pool configuration requires at least two disks, preferably on separate controllers. Many disks can be used in a mirrored configuration. In addition, you can create more than one mirror in each pool. Conceptually, a simple mirrored configuration would look similar to the following:
mirror c1t0d0 c2t0d0
Conceptually, a more complex mirrored configuration would look similar to the following:
mirror c1t0d0 c2t0d0 c3t0d0 mirror c4t0d0 c5t0d0 c6t0d0
For information about creating a mirrored storage pool, see Creating a Mirrored Storage Pool.
RAID-Z Storage Pool Configuration
In addition to a mirrored storage pool configuration, ZFS provides a RAID-Z configuration with either single or double parity fault tolerance. Single-parity RAID-Z is similar to RAID-5. Double-parity RAID-Z is similar to RAID-6.
All traditional RAID-5-like algorithms (RAID-4. RAID-5. RAID-6, RDP, and EVEN-ODD, for example) suffer from a problem known as the “RAID-5 write hole.” If only part of a RAID-5 stripe is written, and power is lost before all blocks have made it to disk, the parity will remain out of sync with the data, and therefore useless, forever (unless a subsequent full-stripe write overwrites it). In RAID-Z, ZFS uses variable-width RAID stripes so that all writes are full-stripe writes. This design is only possible because ZFS integrates file system and device management in such a way that the file system's metadata has enough information about the underlying data redundancy model to handle variable-width RAID stripes. RAID-Z is the world's first software-only solution to the RAID-5 write hole.
You need at least two disks for a RAID-Z configuration. Otherwise, no special hardware is required to create a RAID-Z configuration. Currently, RAID-Z provides single parity. For example, if you have three disks in a RAID-Z configuration, parity data occupies space equal to one of the three disks.
A RAID-Z configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised. You need at least two disks for a single-parity RAID-Z configuration and at least three disks for a double-parity RAID-Z configuration. For example, if you have three disks in a single-parity RAID-Z configuration, parity data occupies space equal to one of the three disks. Otherwise, no special hardware is required to create a RAID-Z configuration.
Conceptually, a RAID-Z configuration with three disks would look similar to the following:
raidz c1t0d0 c2t0d0 c3t0d0
A more complex conceptual RAID-Z configuration would look similar to the following:
raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 c5t0d0 c6t0d0 c7t0d0 raidz c8t0d0 c9t0d0 c10t0d0 c11t0d0 c12t0d0 c13t0d0 c14t0d0
If you are creating a RAID-Z configuration with many disks, as in this example, a RAID-Z configuration with 14 disks is better split into a two 7-disk groupings. RAID-Z configurations with single-digit groupings of disks should perform better.
For information about creating a RAID-Z storage pool, see Creating RAID-Z Storage Pools.
For more information about choosing between a mirrored configuration or a RAID-Z configuration based on performance and space considerations, see the blog post, When To (And Not To) Use RAID-Z. For additional information on RAID-Z storage pool recommendations, see the ZFS best practices guide.
Self-Healing Data in a Redundant Configuration
ZFS provides for self-healing data in a mirrored or RAID-Z configuration.
When a bad data block is detected, not only does ZFS fetch the correct data from another redundant copy, but it also repairs the bad data by replacing it with the good copy.
Dynamic Striping in a Storage Pool
For each virtual device that is added to the pool, ZFS dynamically stripes data across all available devices. The decision about where to place data is done at write time, so no fixed width stripes are created at allocation time.
When virtual devices are added to a pool, ZFS gradually allocates data to the new device in order to maintain performance and space allocation policies. Each virtual device can also be a mirror or a RAID-Z device that contains other disk devices or files. This configuration allows for flexibility in controlling the fault characteristics of your pool. For example, you could create the following configurations out of 4 disks:
Four disks using dynamic striping
One four-way RAID-Z configuration
Two two-way mirrors using dynamic striping
While ZFS supports combining different types of virtual devices within the same pool, this practice is not recommended. For example, you can create a pool with a two-way mirror and a three-way RAID-Z configuration. However, your fault tolerance is as good as your worst virtual device, RAID-Z in this case. The recommended practice is to use top-level virtual devices of the same type with the same redundancy level in each device.
Creating and Destroying ZFS Storage Pools
The following sections describe different scenarios for creating and destroying ZFS storage pools.
By design, creating and destroying pools is fast and easy. However, be cautious when doing these operations. Although checks are performed to prevent using devices known to be in use in a new pool, ZFS cannot always know when a device is already in use. Destroying a pool is even easier. Use zpool destroy with caution. This is a simple command with significant consequences.
Creating a ZFS Storage Pool
To create a storage pool, use the zpool create command. This command takes a pool name and any number of virtual devices as arguments. The pool name must satisfy the naming conventions outlined in ZFS Component Naming Requirements.
Creating a Basic Storage Pool
The following command creates a new pool named tank that consists of the disks c1t0d0 and c1t1d0:
# zpool create tank c1t0d0 c1t1d0
These whole disks are found in the /dev/dsk directory and are labelled appropriately by ZFS to contain a single, large slice. Data is dynamically striped across both disks.
Creating a Mirrored Storage Pool
To create a mirrored pool, use the
followed by any number of storage devices that will comprise the mirror. Multiple
mirrors can be specified by repeating the
on the command line. The following command creates a pool with two, two-way
# zpool create tank mirror c1d0 c2d0 mirror c3d0 c4d0
mirror keyword indicates that a new top-level
virtual device is being specified. Data is dynamically striped across both
mirrors, with data being redundant between each disk appropriately.
Currently, the following operations are supported on a ZFS mirrored configuration:
Adding another set of disks for an additional top-level
vdevto an existing mirrored configuration. For more information, see Adding Devices to a Storage Pool.
Attaching additional disks to an existing mirrored configuration. Or, attaching additional disks to a non-replicated configuration to create a mirrored configuration. For more information, see Attaching and Detaching Devices in a Storage Pool.
Replace a disk or disks in an existing mirrored configuration as long as the replacement disks are greater than or equal to the device to be replaced. For more information, see Replacing Devices in a Storage Pool.
Detach a disk or disk in a mirrored configuration as long as the remaining devices provide adequate redundancy for the configuration. For more information, see Attaching and Detaching Devices in a Storage Pool.
Currently, the following operations are not supported on a mirrored configuration:
You cannot outright remove a device from a mirrored storage pool. An RFE is filed for this feature.
You cannot split or break a mirror for backup purposes. An RFE is filed for this feature.
Creating RAID-Z Storage Pools
Creating a single-parity RAID-Z pool is identical to creating a mirrored
pool, except that the
is used instead of
mirror. The following example shows
how to create a pool with a single RAID-Z device that consists of five disks:
# zpool create tank raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 /dev/dsk/c5t0d0
This example demonstrates that disks can be specified by using their full paths. The /dev/dsk/c5t0d0 device is identical to the c5t0d0 device.
A similar configuration could be created with disk slices. For example:
# zpool create tank raidz c1t0d0s0 c2t0d0s0 c3t0d0s0 c4t0d0s0 c5t0d0s0
However, the disks must be preformatted to have an appropriately sized slice zero.
You can create a double-parity RAID-Z configuration by using the
raidz2 keyword when the pool is created. For example:
# zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0 # zpool status -v tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c1t0d0 ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c3t0d0 ONLINE 0 0 0 errors: No known data errors
Currently, the following operations are supported on a ZFS RAID-Z configuration:
Add another set of disks for an additional top-level
vdevto an existing RAID-Z configuration. For more information, see Adding Devices to a Storage Pool.
Replace a disk or disks in an existing RAID-Z configuration as long as the replacement disks are greater than or equal to the device to be replaced. For more information, see Replacing Devices in a Storage Pool.
Currently, the following operations are not supported on a RAID-Z configuration:
Attach an additional disk to an existing RAID-Z configuration.
Detach a disk from a RAID-Z configuration.
You cannot outright remove a device from a RAID-Z configuration. An RFE is filed for this feature.
For more information about a RAID-Z configuration, see RAID-Z Storage Pool Configuration.
Creating a ZFS Storage Pool with Log Devices
By default, the ZIL is allocated from blocks within the main pool. However, better performance might be possible by using separate intent log devices, such as NVRAM or a dedicated disk. For more information about ZFS log devices, see .
You can set up a ZFS logging device when the storage pool is created or after the pool is created.
For example, create a mirrored storage pool with mirrored log devices.
# zpool create datap mirror c1t1d0 c1t2d0 mirror c1t3d0 c1t4d0 log mirror c1t5d0 c1t8d0 # zpool status pool: datap state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM datap ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 logs ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 c1t8d0 ONLINE 0 0 0 errors: No known data errors
Creating a ZFS Storage Pool with Cache Devices
You can create a storage pool with cache devices to cache storage pool data. For example:
# zpool create tank mirror c2t0d0 c2t1d0 c2t3d0 cache c2t5d0 c2t8d0 # zpool status tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 cache c2t5d0 ONLINE 0 0 0 c2t8d0 ONLINE 0 0 0
Using cache devices provide the greatest performance improvement for random read-workloads of mostly static content.
Capacity and reads can be monitored by using the zpool iostat command.
Single or multiple cache devices can be added when the pool is created or added and removed after the pool is created. For more information, see Adding and Removing Cache Devices to Your ZFS Storage Pool.
Cache devices cannot be mirrored or be part of a RAID-Z configuration.
If a read error is encountered on a cache device, that read I/O is reissued to the original storage pool device, which might be part of a mirrored or RAID-Z configuration. The content of the cache devices is considered volatile, as is the case with other system caches.
Handling ZFS Storage Pool Creation Errors
Pool creation errors can occur for many reasons. Some of these reasons are obvious, such as when a specified device doesn't exist, while other reasons are more subtle.
Detecting in Use Devices
Before formatting a device, ZFS first determines if the disk is in use by ZFS or some other part of the operating system. If the disk is in use, you might see errors such as the following:
# zpool create tank c1t0d0 c1t1d0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c1t0d0s0 is currently mounted on /. Please see umount(1M). /dev/dsk/c1t0d0s1 is currently mounted on swap. Please see swap(1M). /dev/dsk/c1t1d0s0 is part of active ZFS pool zeepool. Please see zpool(1M).
Some of these errors can be overridden by using the
but most errors cannot. The following uses cannot be overridden by using the
-f option, and you must manually correct them:
- Mounted file system
The disk or one of its slices contains a file system that is currently mounted. To correct this error, use the umount command.
- File system in /etc/vfstab
The disk contains a file system that is listed in the /etc/vfstab file, but the file system is not currently mounted. To correct this error, remove or comment out the line in the /etc/vfstab file.
- Dedicated dump device
The disk is in use as the dedicated dump device for the system. To correct this error, use the dumpadm command.
- Part of a ZFS pool
The disk or file is part of an active ZFS storage pool. To correct this error, use the zpool command to destroy the pool.
The following in-use checks serve as helpful warnings and can be overridden
by using the
-f option to create the pool:
- Contains a file system
The disk contains a known file system, though it is not mounted and doesn't appear to be in use.
- Part of volume
The disk is part of an SVM volume.
- Live upgrade
The disk is in use as an alternate boot environment for Solaris Live Upgrade.
- Part of exported ZFS pool
The disk is part of a storage pool that has been exported or manually removed from a system. In the latter case, the pool is reported as
potentially active, as the disk might or might not be a network-attached drive in use by another system. Be cautious when overriding a potentially active pool.
The following example demonstrates how the
# zpool create tank c1t0d0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c1t0d0s0 contains a ufs filesystem. # zpool create -f tank c1t0d0
Ideally, correct the errors rather than use the
Mismatched Replication Levels
Creating pools with virtual devices of different replication levels is not recommended. The zpool command tries to prevent you from accidentally creating a pool with mismatched levels of redundancy. If you try to create a pool with such a configuration, you see errors similar to the following:
# zpool create tank c1t0d0 mirror c2t0d0 c3t0d0 invalid vdev specification use '-f' to override the following errors: mismatched replication level: both disk and mirror vdevs are present # zpool create tank mirror c1t0d0 c2t0d0 mirror c3t0d0 c4t0d0 c5t0d0 invalid vdev specification use '-f' to override the following errors: mismatched replication level: 2-way mirror and 3-way mirror vdevs are present
You can override these errors with the
-f option, though
this practice is not recommended. The command also warns you about creating
a mirrored or RAID-Z pool using devices of different sizes. While this configuration
is allowed, mismatched levels of redundancy result in unused space on the
larger device, and requires the
-f option to override the
Doing a Dry Run of Storage Pool Creation
Because creating a pool can fail unexpectedly in different ways, and
because formatting disks is such a potentially harmful action, the
zpool create command has an additional option,
simulates creating the pool without actually writing data to disk. This option
performs the device in-use checking and replication level validation, and
reports any errors in the process. If no errors are found, you see output
similar to the following:
# zpool create -n tank mirror c1t0d0 c1t1d0 would create 'tank' with the following layout: tank mirror c1t0d0 c1t1d0
Some errors cannot be detected without actually creating the pool. The most common example is specifying the same device twice in the same configuration. This error cannot be reliably detected without writing the data itself, so the create -n command can report success and yet fail to create the pool when run for real.
Default Mount Point for Storage Pools
When a pool is created, the default mount point for the root dataset
is /pool-name. This directory must either not exist
or be empty. If the directory does not exist, it is automatically created.
If the directory is empty, the root dataset is mounted on top of the existing
directory. To create a pool with a different default mount point, use the
-m option of the zpool create command:
# zpool create home c1t0d0 default mountpoint '/home' exists and is not empty use '-m' option to specify a different default # zpool create -m /export/zfs home c1t0d0
# zpool create home c1t0d0 default mountpoint '/home' exists and is not empty use '-m' option to provide a different default # zpool create -m /export/zfs home c1t0d0
This command creates a new pool
home and the
home dataset with a mount point of /export/zfs.
For more information about mount points, see Managing ZFS Mount Points.
Destroying ZFS Storage Pools
Pools are destroyed by using the zpool destroy command. This command destroys the pool even if it contains mounted datasets.
# zpool destroy tank
Be very careful when you destroy a pool. Make sure you are destroying the right pool and you always have copies of your data. If you accidentally destroy the wrong pool, you can attempt to recover the pool. For more information, see Recovering Destroyed ZFS Storage Pools.
Destroying a Pool With Faulted Devices
The act of destroying a pool requires that data be written to disk to indicate that the pool is no longer valid. This state information prevents the devices from showing up as a potential pool when you perform an import. If one or more devices are unavailable, the pool can still be destroyed. However, the necessary state information won't be written to these damaged devices.
These devices, when suitably repaired, are reported as potentially
active when you create a new pool, and appear as valid devices
when you search for pools to import. If a pool has enough faulted devices
such that the pool itself is faulted (meaning that a top-level virtual device
is faulted), then the command prints a warning and cannot complete without
-f option. This option is necessary because the pool cannot
be opened, so whether data is stored there or not is unknown. For example:
# zpool destroy tank cannot destroy 'tank': pool is faulted use '-f' to force destruction anyway # zpool destroy -f tank
For more information about pool and device health, see Determining the Health Status of ZFS Storage Pools.
For more information about importing pools, see Importing ZFS Storage Pools.
Managing Devices in ZFS Storage Pools
Most of the basic information regarding devices is covered in Components of a ZFS Storage Pool. Once a pool has been created, you can perform several tasks to manage the physical devices within the pool.
Adding Devices to a Storage Pool
You can dynamically add space to a pool by adding a new top-level virtual device. This space is immediately available to all datasets within the pool. To add a new virtual device to a pool, use the zpool add command. For example:
# zpool add zeepool mirror c2t1d0 c2t2d0
The format of the virtual devices is the same as for the
zpool create command, and the same rules apply. Devices are checked to
determine if they are in use, and the command cannot change the level of redundancy
-f option. The command also supports the
so that you can perform a dry run. For example:
# zpool add -n zeepool mirror c3t1d0 c3t2d0 would update 'zeepool' to the following configuration: zeepool mirror c1t0d0 c1t1d0 mirror c2t1d0 c2t2d0 mirror c3t1d0 c3t2d0
This command syntax would add mirrored devices
c3t2d0 to zeepool's existing configuration.
For more information about how virtual device validation is done, see Detecting in Use Devices.
Attaching and Detaching Devices in a Storage Pool
In addition to the zpool add command, you can use the zpool attach command to add a new device to an existing mirrored or non-mirrored device.
You can use the zpool detach command to detach a device from a mirrored storage pool. For example:
# zpool detach zeepool c2t1d0
However, this operation is refused if there are no other valid replicas of the data. For example:
# zpool detach newpool c1t2d0 cannot detach c1t2d0: only applicable to mirror and replacing vdevs
Onlining and Offlining Devices in a Storage Pool
ZFS allows individual devices to be taken offline or brought online. When hardware is unreliable or not functioning properly, ZFS continues to read or write data to the device, assuming the condition is only temporary. If the condition is not temporary, it is possible to instruct ZFS to ignore the device by bringing it offline. ZFS does not send any requests to an offlined device.
Devices do not need to be taken offline in order to replace them.
You can use the offline command when you need to temporarily disconnect storage. For example, if you need to physically disconnect an array from one set of Fibre Channel switches and connect the array to a different set, you could take the LUNs offline from the array that was used in ZFS storage pools. After the array was reconnected and operational on the new set of switches, you could then bring the same LUNs online. Data that had been added to the storage pools while the LUNs were offline would resilver to the LUNs after they were brought back online.
This scenario is possible assuming that the systems in question see the storage once it is attached to the new switches, possibly through different controllers than before, and your pools are set up as RAID-Z or mirrored configurations.
Taking a Device Offline
You can take a device offline by using the zpool offline command. The device can be specified by path or by short name, if the device is a disk. For example:
# zpool offline tank c1t0d0 bringing device c1t0d0 offline
Keep the following points in mind when taking a device offline:
You cannot take a pool offline to the point where it becomes faulted. For example, you cannot take offline two devices out of a RAID-Z configuration, nor can you take offline a top-level virtual device.
# zpool offline tank c1t0d0 cannot offline c1t0d0: no valid replicas
By default, the offline state is persistent. The device remains offline when the system is rebooted.
To temporarily take a device offline, use the zpool offline
-toption. For example:
# zpool offline -t tank c1t0d0 bringing device 'c1t0d0' offline
When the system is rebooted, this device is automatically returned to the
When a device is taken offline, it is not detached from the storage pool. If you attempt to use the offlined device in another pool, even after the original pool is destroyed, you will see a message similar to the following:
device is part of exported or potentially active ZFS pool. Please see zpool(1M)
If you want to use the offlined device in another storage pool after destroying the original storage pool, first bring the device back online, then destroy the original storage pool.
Another way to use a device from another storage pool if you want to keep the original storage pool is to replace the existing device in the original storage pool with another comparable device. For information about replacing devices, see Replacing Devices in a Storage Pool.
Offlined devices show up in the
OFFLINE state when
you query pool status. For information about querying pool status, see Querying ZFS Storage Pool Status.
For more information on device health, see Determining the Health Status of ZFS Storage Pools.
Bringing a Device Online
Once a device is taken offline, it can be restored by using the zpool online command:
# zpool online tank c1t0d0 bringing device c1t0d0 online
When a device is brought online, any data that has been written to the pool is resynchronized to the newly available device. Note that you cannot use device onlining to replace a disk. If you offline a device, replace the drive, and try to bring it online, it remains in the faulted state.
If you attempt to online a faulted device, a message similar to the following is displayed from fmd:
# zpool online tank c1t0d0 Bringing device c1t0d0 online # SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major EVENT-TIME: Thu Aug 31 11:13:59 MDT 2006 PLATFORM: SUNW,Ultra-60, CSN: -, HOSTNAME: neo SOURCE: zfs-diagnosis, REV: 1.0 EVENT-ID: e11d8245-d76a-e152-80c6-e63763ed7e4f DESC: A ZFS device failed. Refer to http://illumos.org/msg/ZFS-8000-D3 for more information. AUTO-RESPONSE: No automated response will occur. IMPACT: Fault tolerance of the pool may be compromised. REC-ACTION: Run 'zpool status -x' and replace the bad device.
For more information on replacing a faulted device, see Repairing a Missing Device.
Clearing Storage Pool Devices
If a device is taken offline due to a failure that causes errors to be listed in the zpool status output, you can clear the error counts with the zpool clear command.
If specified with no arguments, this command clears all device errors within the pool. For example:
# zpool clear tank
If one or more devices are specified, this command only clear errors associated with the specified devices. For example:
# zpool clear tank c1t0d0
For more information on clearing zpool errors, see Clearing Transient Errors.
Replacing Devices in a Storage Pool
You can replace a device in a storage pool by using the zpool replace command.
If you are physically replacing a device with another device in the
same location in a redundant pool, then you only need identify the replaced
device. ZFS recognizes that it is a different disk in the same location. For
example, to replace a failed disk (
c1t1d0) by removing
the disk and replacing it in the same location, use the syntax similar to
# zpool replace tank c1t1d0
If you are replacing a device in a non-redundant storage pool that contains only one device, you will need to specify both devices. For example:
# zpool replace tank c1t1d0 c1t2d0
Keep the following considerations in mind when replacing devices in a ZFS storage pool:
The replacement device must be greater than or equal to the minimum size of all the devices in a mirrored or RAID-Z configuration.
If the replacement device is larger, the pool capacity is increased when the replacement is complete. Currently, you must export and import the pool to see the expanded capacity. For example:
# zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 16.8G 94K 16.7G 0% ONLINE - # zpool replace tank c0t0d0 c0t4d0 # zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 16.8G 112K 16.7G 0% ONLINE - # zpool export tank # zpool import tank # zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 33.9G 114K 33.9G 0% ONLINE -
For more information about exporting and importing pools, see Migrating ZFS Storage Pools.
Currently, you must also perform the export and import steps when growing the size of an existing LUN that is part of a storage pool to see the expanded capacity.
Replacing many disks in a large pool is time consuming due to resilvering the data onto the new disks. In addition, you might consider running the zpool scrub command between disk replacements to ensure that the replacement devices are operational and the data is written correctly.
For more information about replacing devices, see Repairing a Missing Device and Repairing a Damaged Device.
Designating Hot Spares in Your Storage Pool
The hot spares feature enables you to identify disks that could be used to replace a failed or faulted device in one or more storage pools. Designating a device as a hot spare means that the device is not an active device in a pool, but if an active device in the pool fails, the hot spare automatically replaces the failed device.
Devices can be designated as hot spares in the following ways:
When the pool is created with the zpool create command
After the pool is created with the zpool add command
Hot spare devices can be shared between multiple pools
Designate devices as hot spares when the pool is created. For example:
# zpool create zeepool mirror c1t1d0 c2t1d0 spare c1t2d0 c2t2d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t2d0 AVAIL c2t2d0 AVAIL
Designate hot spares by adding them to a pool after the pool is created. For example:
# zpool add -f zeepool spare c1t3d0 c2t3d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 AVAIL
Multiple pools can share devices that are designated as hot spares. For example:
# zpool create zeepool mirror c1t1d0 c2t1d0 spare c1t2d0 c2t2d0 # zpool create tank raidz c3t1d0 c4t1d0 spare c1t2d0 c2t2d0
Hot spares can be removed from a storage pool by using the zpool remove command. For example:
# zpool remove zeepool c1t2d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t3d0 AVAIL
A hot spare cannot be removed if it is currently used by the storage pool.
Keep the following points in mind when using ZFS hot spares:
Currently, the zpool remove command can only be used to remove hot spares.
Add a disk as a spare that is equal to or larger than the size of the largest disk in the pool. Adding a smaller disk as a spare to a pool is allowed. However, when the smaller spare disk is activated, either automatically or with the zpool replace command, the operation fails with an error similar to the following:
cannot replace disk3 with disk4: device is too small
Activating and Deactivating Hot Spares in Your Storage Pool
Hot spares are activated in the following ways:
Manually replacement – Replace a failed device in a storage pool with a hot spare by using the zpool replace command.
Automatic replacement – When a fault is received, an FMA agent examines the pool to see if it has any available hot spares. If so, it replaces the faulted device with an available spare.
If a hot spare that is currently in use fails, the agent detaches the spare and thereby cancels the replacement. The agent then attempts to replace the device with another hot spare, if one is available. This feature is currently limited by the fact that the ZFS diagnosis engine only emits faults when a device disappears from the system.
Currently, no automated response is available to bring the original device back online. You must explicitly take one of the actions described in the example below. A future enhancement will allow ZFS to subscribe to hotplug events and automatically replace the affected device when it is replaced on the system.
Manually replace a device with a hot spare by using the zpool replace command. For example:
# zpool replace zeepool c2t1d0 c2t3d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: resilver completed with 0 errors on Fri Jun 2 13:44:40 2006 config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 spare ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 INUSE currently in use errors: No known data errors
A faulted device is automatically replaced if a hot spare is available. For example:
# zpool status -x pool: zeepool state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Fri Jun 2 13:56:49 2006 config: NAME STATE READ WRITE CKSUM zeepool DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c1t2d0 ONLINE 0 0 0 spare DEGRADED 0 0 0 c2t1d0 UNAVAIL 0 0 0 cannot open c2t3d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 INUSE currently in use errors: No known data errors
Currently, three ways to deactivate hot spares are available:
Canceling the hot spare by removing it from the storage pool
Replacing the original device with a hot spare
Permanently swapping in the hot spare
After the faulted device is replaced, use the zpool detach command to return the hot spare back to the spare set. For example:
# zpool detach zeepool c2t3d0 # zpool status zeepool pool: zeepool state: ONLINE scrub: resilver completed with 0 errors on Fri Jun 2 13:58:35 2006 config: NAME STATE READ WRITE CKSUM zeepool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 spares c1t3d0 AVAIL c2t3d0 AVAIL errors: No known data errors
Managing ZFS Storage Pool Properties
You can use the zpool get command to display pool property information. For example:
# zpool get all tank2 NAME PROPERTY VALUE SOURCE tank2 size 33.8G - tank2 used 158K - tank2 available 33.7G - tank2 capacity 0% - tank2 altroot - default tank2 health ONLINE - tank2 guid 8032621780930948264 - tank2 version 8 default tank2 bootfs - default tank2 delegation on default tank2 autoreplace off default tank2 temporary off default tank2 failmode wait default
Storage pool properties can be set with the zpool set command. For example:
# zpool set autoreplace=on tank # zpool get autoreplace tank NAME PROPERTY VALUE SOURCE tank autoreplace on default
Identifies an alternate root directory. If set, this directory is
prepended to any mount points within the pool. This property can be used
when examining an unknown pool, if the mount points cannot be trusted, or
in an alternate boot environment, where the typical paths are not valid.
Setting this property implies that the
Read-only value that identifies the amount of storage that is available within the pool.
This property can also be referred to by its shortened column name,
Controls automatic device replacement. If set to off, device replacement
must be initiated by the administrator by using the zpool replace command. If set to on, any new device, found in the same
physical location as a device that previously belonged to the pool, is automatically
formatted and replaced. The default behavior is off. This property can also be referred
to by its shortened column
Identifies the default bootable dataset for the root pool. This property is expected to be set mainly by the installation and upgrade programs.
Read-only value that identifies the percentage of pool space used.
This property can also be referred to by its shortened column name,
Controls whether a non-privileged user can be granted access permissions that are defined for the dataset. For more information, see ZFS Delegated Administration.
Read-only property that identifies the unique identifier for the pool.
Read-only property that identifies the current health of the pool, as either ONLINE, DEGRADED, FAULTED, OFFLINE, REMOVED, or UNAVAIL.
Read-only property that identifies the total size of the storage pool.
Read-only property that identifies the amount of storage space used within the pool.
Controls whether the pool is available temporarily. By default, all
pools are persistent, and are automatically opened when the system is rebooted.
Setting this property to
This property can also be referred to by its shortened column name,
Identifies the current on-disk version of the pool. The value of this
property can be increased, but never decreased. The preferred method of
updating pools is with the zpool upgrade command, although
this property can be used when a specific version is needed for backwards
compatibility. This property can be set to any number between 1 and the current
version reported by the zpool upgrade
Querying ZFS Storage Pool Status
The zpool list command provides a number of ways to request information regarding pool status. The information available generally falls into three categories: basic usage information, I/O statistics, and health status. All three types of storage pool information are covered in this section.
Displaying Basic ZFS Storage Pool Information
You can use the zpool list command to display basic information about pools.
Listing Information About All Storage Pools
With no arguments, the command displays all the fields for all pools on the system. For example:
# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 80.0G 22.3G 47.7G 28% ONLINE - dozer 1.2T 384G 816G 32% ONLINE -
This output displays the following information:
The name of the pool.
The total size of the pool, equal to the sum of the size of all top-level virtual devices.
The amount of space allocated by all datasets and internal metadata. Note that this amount is different from the amount of space as reported at the file system level.
For more information about determining available file system space, see ZFS Space Accounting.
The amount of unallocated space in the pool.
The amount of space used, expressed as a percentage of total space.
The current health status of the pool.
For more information about pool health, see Determining the Health Status of ZFS Storage Pools.
The alternate root of the pool, if any.
For more information about alternate root pools, see Using ZFS Alternate Root Pools.
You can also gather statistics for a specific pool by specifying the pool name. For example:
# zpool list tank NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 80.0G 22.3G 47.7G 28% ONLINE -
Listing Specific Storage Pool Statistics
Specific statistics can be requested by using the
This option allows for custom reports or a quick way to list pertinent information.
For example, to list only the name and size of each pool, you use the following
# zpool list -o name,size NAME SIZE tank 80.0G dozer 1.2T
The column names correspond to the properties that are listed in Listing Information About All Storage Pools.
Scripting ZFS Storage Pool Output
The default output for the zpool list command is
designed for readability, and is not easy to use as part of a shell script.
To aid programmatic uses of the command, the
-H option can
be used to suppress the column headings and separate fields by tabs, rather
than by spaces. For example, to request a simple list of all pool names on
# zpool list -Ho name tank dozer
Here is another example:
# zpool list -H -o name,size tank 80.0G dozer 1.2T
Viewing ZFS Storage Pool I/O Statistics
To request I/O statistics for a pool or specific virtual devices, use the zpool iostat command. Similar to the iostat command, this command can display a static snapshot of all I/O activity so far, as well as updated statistics for every specified interval. The following statistics are reported:
The amount of data currently stored in the pool or device. This figure differs from the amount of space available to actual file systems by a small amount due to internal implementation details.
For more information about the difference between pool space and dataset space, see ZFS Space Accounting.
The amount of space available in the pool or device. As with the
usedstatistic, this amount differs from the amount of space available to datasets by a small margin.
The number of read I/O operations sent to the pool or device, including metadata requests.
The number of write I/O operations sent to the pool or device.
The bandwidth of all read operations (including metadata), expressed as units per second.
The bandwidth of all write operations, expressed as units per second.
Listing Pool-Wide Statistics
With no options, the zpool iostat command displays the accumulated statistics since boot for all pools on the system. For example:
# zpool iostat capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- tank 100G 20.0G 1.2M 102K 1.2M 3.45K dozer 12.3G 67.7G 132K 15.2K 32.1K 1.20K
Because these statistics are cumulative since boot, bandwidth might appear low if the pool is relatively idle. You can request a more accurate view of current bandwidth usage by specifying an interval. For example:
# zpool iostat tank 2 capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- tank 100G 20.0G 1.2M 102K 1.2M 3.45K tank 100G 20.0G 134 0 1.34K 0 tank 100G 20.0G 94 342 1.06K 4.1M
In this example, the command displays usage statistics only for the
tank every two seconds until you type Ctrl-C. Alternately,
you can specify an additional
count parameter, which causes
the command to terminate after the specified number of iterations. For example, zpool iostat 2 3 would print a summary every two seconds for three
iterations, for a total of six seconds. If there is a single pool, then the
statistics are displayed on consecutive lines. If more than one pool exists,
then an additional dashed line delineates each iteration to provide visual
Listing Virtual Device Statistics
In addition to pool-wide I/O statistics, the zpool iostat command can display statistics for specific virtual devices. This command can be used to identify abnormally slow devices, or simply to observe the distribution of I/O generated by ZFS. To request the complete virtual device layout as well as all I/O statistics, use the zpool iostat -v command. For example:
# zpool iostat -v capacity operations bandwidth tank used avail read write read write ---------- ----- ----- ----- ----- ----- ----- mirror 20.4G 59.6G 0 22 0 6.00K c1t0d0 - - 1 295 11.2K 148K c1t1d0 - - 1 299 11.2K 148K ---------- ----- ----- ----- ----- ----- ----- total 24.5K 149M 0 22 0 6.00K
Note two important things when viewing I/O statistics on a virtual device basis.
First, space usage is only available for top-level virtual devices. The way in which space is allocated among mirror and RAID-Z virtual devices is particular to the implementation and not easily expressed as a single number.
Second, the numbers might not add up exactly as you would expect them to. In particular, operations across RAID-Z and mirrored devices will not be exactly equal. This difference is particularly noticeable immediately after a pool is created, as a significant amount of I/O is done directly to the disks as part of pool creation that is not accounted for at the mirror level. Over time, these numbers should gradually equalize, although broken, unresponsive, or offlined devices can affect this symmetry as well.
You can use the same set of options (interval and count) when examining virtual device statistics.
Determining the Health Status of ZFS Storage Pools
ZFS provides an integrated method of examining pool and device health. The health of a pool is determined from the state of all its devices. This state information is displayed by using the zpool status command. In addition, potential pool and device failures are reported by fmd and are displayed on the system console and the /var/adm/messages file. This section describes how to determine pool and device health. This chapter does not document how to repair or recover from unhealthy pools. For more information on troubleshooting and data recovery, see ZFS Troubleshooting and Data Recovery.
Each device can fall into one of the following states:
The device is in normal working order. While some transient errors might still occur, the device is otherwise in working order.
The virtual device has experienced failure but is still able to function. This state is most common when a mirror or RAID-Z device has lost one or more constituent devices. The fault tolerance of the pool might be compromised, as a subsequent fault in another device might be unrecoverable.
The virtual device is completely inaccessible. This status typically indicates total failure of the device, such that ZFS is incapable of sending or receiving data from it. If a top-level virtual device is in this state, then the pool is completely inaccessible.
The virtual device has been explicitly taken offline by the administrator.
The device or virtual device cannot be opened. In some cases, pools with
UNAVAILABLEdevices appear in
DEGRADEDmode. If a top-level virtual device is unavailable, then nothing in the pool can be accessed.
The device was physically removed while the system was running. Device removal detection is hardware-dependent and might not be supported on all platforms.
The health of a pool is determined from the health of all its top-level
virtual devices. If all virtual devices are
the pool is also
ONLINE. If any one of the virtual devices
UNAVAILABLE, then the
pool is also
DEGRADED. If a top-level virtual device is
OFFLINE, then the pool is also
FAULTED. A pool in the faulted state is completely inaccessible. No data
can be recovered until the necessary devices are attached or repaired. A pool
in the degraded state continues to run, but you might not achieve the same
level of data redundancy or data throughput than if the pool were online.
Basic Storage Pool Health Status
The simplest way to request a quick overview of pool health status is to use the zpool status command:
# zpool status -x all pools are healthy
Specific pools can be examined by specifying a pool name to the command.
Any pool that is not in the
ONLINE state should be investigated
for potential problems, as described in the next section.
Detailed Health Status
You can request a more detailed health summary by using the
# zpool status -v tank pool: tank state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-2Q scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c1t0d0 FAULTED 0 0 0 cannot open c1t1d0 ONLINE 0 0 0 errors: No known data errors
This output displays a complete description of why the pool is in its current state, including a readable description of the problem and a link to a knowledge article for more information. Each knowledge article provides up-to-date information on the best way to recover from your current problem. Using the detailed configuration information, you should be able to determine which device is damaged and how to repair the pool.
In the above example, the faulted device should be replaced. After the device is replaced, use the zpool online command to bring the device back online. For example:
# zpool online tank c1t0d0 Bringing device c1t0d0 online # zpool status -x all pools are healthy
If a pool has an offlined device, the command output identifies the problem pool. For example:
# zpool status -x pool: tank state: DEGRADED status: One or more devices has been taken offline by the adminstrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c1t0d0 ONLINE 0 0 0 c1t1d0 OFFLINE 0 0 0 errors: No known data errors
WRITE columns provides
a count of I/O errors seen on the device, while the
provides a count of uncorrectable checksum errors that occurred on the device.
Both of these error counts likely indicate potential device failure, and some
corrective action is needed. If non-zero errors are reported for a top-level
virtual device, portions of your data might have become inaccessible. The
errors count identifies any known data errors.
In the example output above, the offlined device is not causing data errors.
For more information about diagnosing and repairing faulted pools and data, see ZFS Troubleshooting and Data Recovery.
Migrating ZFS Storage Pools
Occasionally, you might need to move a storage pool between machines. To do so, the storage devices must be disconnected from the original machine and reconnected to the destination machine. This task can be accomplished by physically recabling the devices, or by using multiported devices such as the devices on a SAN. ZFS enables you to export the pool from one machine and import it on the destination machine, even if the machines are of different endianness. For information about replicating or migrating file systems between different storage pools, which might reside on different machines, see Saving and Restoring ZFS Data.
Preparing for ZFS Storage Pool Migration
Storage pools should be explicitly exported to indicate that they are ready to be migrated. This operation flushes any unwritten data to disk, writes data to the disk indicating that the export was done, and removes all knowledge of the pool from the system.
If you do not explicitly export the pool, but instead remove the disks manually, you can still import the resulting pool on another system. However, you might lose the last few seconds of data transactions, and the pool will appear faulted on the original machine because the devices are no longer present. By default, the destination machine refuses to import a pool that has not been explicitly exported. This condition is necessary to prevent accidentally importing an active pool that consists of network attached storage that is still in use on another system.
Exporting a ZFS Storage Pool
To export a pool, use the zpool export command. For example:
# zpool export tank
Once this command is executed, the pool
tank is no
longer visible on the system. The command attempts to unmount any mounted
file systems within the pool before continuing. If any of the file systems
fail to unmount, you can forcefully unmount them by using the
# zpool export tank cannot unmount '/export/home/eschrock': Device busy # zpool export -f tank
If devices are unavailable at the time of export, the disks cannot be
specified as cleanly exported. If one of these devices is later attached to
a system without any of the working devices, it appears as “potentially
active.” If ZFS volumes are in use in the pool, the pool cannot be exported,
even with the
-f option. To export a pool with an ZFS volume,
first make sure that all consumers of the volume are no longer active.
For more information about ZFS volumes, see ZFS Volumes.
Determining Available Storage Pools to Import
Once the pool has been removed from the system (either through export or by forcefully removing the devices), attach the devices to the target system. Although ZFS can handle some situations in which only a portion of the devices is available, all devices within the pool must be moved between the systems. The devices do not necessarily have to be attached under the same device name. ZFS detects any moved or renamed devices, and adjusts the configuration appropriately. To discover available pools, run the zpool import command with no options. For example:
# zpool import pool: tank id: 3778921145927357706 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: tank ONLINE mirror ONLINE c1t0d0 ONLINE c1t1d0 ONLINE
In this example, the pool
tank is available to be
imported on the target system. Each pool is identified by a name as well as
a unique numeric identifier. If multiple pools available to import have the
same name, you can use the numeric identifier to distinguish between them.
Similar to the zpool status command, the
zpool import command refers to a knowledge article available on the web
with the most up-to-date information regarding repair procedures for a problem
that is preventing a pool from being imported. In this case, the user can
force the pool to be imported. However, importing a pool that is currently
in use by another system over a storage network can result in data corruption
and panics as both systems attempt to write to the same storage. If some devices
in the pool are not available but enough redundancy is available to have a
usable pool, the pool appears in the
DEGRADED state. For
# zpool import pool: tank id: 3778921145927357706 state: DEGRADED status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://illumos.org/msg/ZFS-8000-2Q config: tank DEGRADED mirror DEGRADED c1t0d0 UNAVAIL cannot open c1t1d0 ONLINE
In this example, the first disk is damaged or missing, though you can still import the pool because the mirrored data is still accessible. If too many faulted or missing devices are present, the pool cannot be imported. For example:
# zpool import pool: dozer id: 12090808386336829175 state: FAULTED action: The pool cannot be imported. Attach the missing devices and try again. see: http://illumos.org/msg/ZFS-8000-6X config: raidz FAULTED c1t0d0 ONLINE c1t1d0 FAULTED c1t2d0 ONLINE c1t3d0 FAULTED
In this example, two disks are missing from a RAID-Z virtual device, which means that sufficient redundant data is not available to reconstruct the pool. In some cases, not enough devices are present to determine the complete configuration. In this case, ZFS doesn't know what other devices were part of the pool, though ZFS does report as much information as possible about the situation. For example:
# zpool import pool: dozer id: 12090808386336829175 state: FAULTED status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://illumos.org/msg/ZFS-8000-6X config: dozer FAULTED missing device raidz ONLINE c1t0d0 ONLINE c1t1d0 ONLINE c1t2d0 ONLINE c1t3d0 ONLINE Additional devices are known to be part of this pool, though their exact configuration cannot be determined.
Finding ZFS Storage Pools From Alternate Directories
By default, the zpool import command only searches
devices within the /dev/dsk directory. If devices exist
in another directory, or you are using pools backed by files, you must use
-d option to search different directories. For example:
# zpool create dozer mirror /file/a /file/b # zpool export dozer # zpool import -d /file pool: dozer id: 10952414725867935582 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: dozer ONLINE mirror ONLINE /file/a ONLINE /file/b ONLINE # zpool import -d /file dozer
If devices exist in multiple directories, you can specify multiple
Importing ZFS Storage Pools
Once a pool has been identified for import, you can import it by specifying the name of the pool or its numeric identifier as an argument to the zpool import command. For example:
# zpool import tank
If multiple available pools have the same name, you can specify which pool to import using the numeric identifier. For example:
# zpool import pool: dozer id: 2704475622193776801 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: dozer ONLINE c1t9d0 ONLINE pool: dozer id: 6223921996155991199 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: dozer ONLINE c1t8d0 ONLINE # zpool import dozer cannot import 'dozer': more than one matching pool import by numeric ID instead # zpool import 6223921996155991199
If the pool name conflicts with an existing pool name, you can import the pool under a different name. For example:
# zpool import dozer zeepool
This command imports the exported pool
the new name
zeepool. If the pool was not cleanly exported,
ZFS requires the
-f flag to prevent users from accidentally
importing a pool that is still in use on another system. For example:
# zpool import dozer cannot import 'dozer': pool may be in use on another system use '-f' to import anyway # zpool import -f dozer
Pools can also be imported under an alternate root by using the
For more information on alternate root pools, see Using ZFS Alternate Root Pools.
Recovering Destroyed ZFS Storage Pools
You can use the zpool import
to recover a storage pool that has been destroyed. For example:
# zpool destroy tank # zpool import -D pool: tank id: 3778921145927357706 state: ONLINE (DESTROYED) action: The pool can be imported using its name or numeric identifier. The pool was destroyed, but can be imported using the '-Df' flags. config: tank ONLINE mirror ONLINE c1t0d0 ONLINE c1t1d0 ONLINE
In the above zpool import output, you can identify this pool as the destroyed pool because of the following state information:
state: ONLINE (DESTROYED)
To recover the destroyed pool, issue the zpool import
-D command again with the pool to be recovered and the
# zpool import -Df tank # zpool status tank pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t0d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 errors: No known data errors
If one of the devices in the destroyed pool is faulted or unavailable, you might be able to recover the destroyed pool anyway. In this scenario, import the degraded pool and then attempt to fix the device failure. For example:
# zpool destroy dozer # zpool import -D pool: dozer id: state: DEGRADED (DESTROYED) status: One or more devices are missing from the system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. The pool was destroyed, but can be imported using the '-Df' flags. see: http://illumos.org/msg/ZFS-8000-2Q config: dozer DEGRADED raidz ONLINE c1t0d0 ONLINE c1t1d0 ONLINE c1t2d0 UNAVAIL cannot open c1t3d0 ONLINE # zpool import -Df dozer # zpool status -x pool: dozer state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Fri Mar 17 16:11:35 2006 config: NAME STATE READ WRITE CKSUM dozer DEGRADED 0 0 0 raidz ONLINE 0 0 0 c1t0d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c1t2d0 UNAVAIL 0 0 0 cannot open c1t3d0 ONLINE 0 0 0 errors: No known data errors # zpool online dozer c1t2d0 Bringing device c1t2d0 online # zpool status -x all pools are healthy
Upgrading ZFS Storage Pools
If you have ZFS storage pools from an earlier ZFS release, you can upgrade your pools with the zpool upgrade command to take advantage of any newer pool features. In addition, the zpool status command has been modified to notify you when your pools are running older versions. For example:
# zpool status pool: test state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 c1t27d0 ONLINE 0 0 0 errors: No known data errors
You can use the following syntax to identify additional information about a particular version and supported releases.
# zpool upgrade -v This system supports ZFS pool feature flags. The following features are supported: FEAT DESCRIPTION ------------------------------------------------------------- async_destroy (read-only compatible) Destroy filesystems asynchronously. empty_bpobj (read-only compatible) Snapshots use less space. lz4_compress LZ4 compression algorithm support. multi_vdev_crash_dump Crash dumps to multiple vdev pools. spacemap_histogram (read-only compatible) Spacemaps maintain space histograms. enabled_txg (read-only compatible) Record txg at which a feature is enabled hole_birth Retain hole birth txg for more precise zfs send extensible_dataset Enhanced dataset functionality, used by other features. embedded_data Blocks which compress very well use even less space. bookmarks (read-only compatible) "zfs bookmark" command filesystem_limits (read-only compatible) Filesystem and snapshot limits. large_blocks Support for blocks larger than 128KB. sha512 SHA-512/256 hash algorithm. skein Skein hash algorithm. edonr Edon-R hash algorithm. The following legacy versions are also supported: VER DESCRIPTION --- -------------------------------------------------------- 1 Initial ZFS version 2 Ditto blocks (replicated metadata) 3 Hot spares and double parity RAID-Z 4 zpool history 5 Compression using the gzip algorithm 6 bootfs pool property 7 Separate intent log devices 8 Delegated administration 9 refquota and refreservation properties 10 Cache devices 11 Improved scrub performance 12 Snapshot properties 13 snapused property 14 passthrough-x aclinherit 15 user/group space accounting 16 stmf property support 17 Triple-parity RAID-Z 18 Snapshot user holds 19 Log device removal 20 Compression using zle (zero-length encoding) 21 Deduplication 22 Received properties 23 Slim ZIL 24 System attributes 25 Improved scrub stats 26 Improved snapshot deletion performance 27 Improved snapshot creation performance 28 Multiple vdev replacements For more information on a particular version, including supported releases, see the ZFS Administration Guide.
Then, you can run the zpool upgrade command to upgrade your pools. For example:
# zpool upgrade -a
If you upgrade your pools to the latest version, they will not be accessible on systems that run older ZFS versions.