Josh-D. S. Davis

Xaminmo / Omnimax / Max Omni / Mad Scientist / Midnight Shadow / Radiation Master

Previous Entry Share Next Entry
AIX 4k sectors
Josh 2014
joshdavis
For AIX, it's best to stick with 512/528B format devices for internal disk.

DETAILS:
Raw, SAN, and ARRAY are 512b LBA (block or sector size). iSeries are 520. T10 RAID are 528. 4224b pdisks become 4k hdisks only.

For iSeries, vSCSI will remap 512 and 4096 from current PCIe3 SAS-RAID adapters to 520b sectors for hdisk VTDs.

All SAN LUNs, and most enterprise DDMs are 512/520/528b, even when presented through remote PV servers (GLVM/HAGEO) or Network Shared Disks (NSDs/GPFS).

There is no block-size translation for AIX, regardless of whether it's SSP, LV or PV backed. In other words, VGs, including an SSP, can only have one block size device inside.

Support for 4k disks:
AIX Announced 2013-11-16. SAS controller support 2014-06-06. AIX support 7.1.3.0 and 6.1.9.0. VIO support 2.2.3.3. iSeries support 7.1.9 and 7.2.1.

All of the POWER8 supported 4k parts have 512/528b counterparts:
http://www-01.ibm.com/support/knowledgecenter/P8DEA/p8ecs/p8ecs_drive_parts.htm

4k disks are okay for:
* Systems that will not have other storage attached.
* iSeries - 520b sector translation works on 4k PCIe3 SAS.
* Linux - MDADM is translates, and LVM can adjust data alignment.
* Windows 7/2008 dynamic arrays (not on POWER)
* Solaris ZFS ashift=12 can also mix (not on POWER).

References:
* http://www-01.ibm.com/support/knowledgecenter/P8DEA/p8ecs/p8ecs_drive_parts.htm
* http://public.dhe.ibm.com/systems/power/docs/hw/p8/p8ebj.pdf
* http://public.dhe.ibm.com/systems/power/docs/hw/p8/p8ed5.pdf
* http://www.redbooks.ibm.com/abstracts/redp5234.html
* https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/IBM%20i%20and%20Related%20Software
* https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/IBM%20i%20Technology%20Updates/page/SAS%20Adapter%20Performance%20Boost%20with%20VIOS
* http://www-01.ibm.com/support/knowledgecenter/POWER8/p8hb1/p8hb1_i5osrestrictions.htm
* https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/IBM%20i%20Technology%20Updates/page/IBM%20i%20IO%20Support%20Details

Sector sizes:
* Standard: 512 raw, 528 RAID, 520 iSeries
* Advanced: 4096 raw, 4224 RAID, 4160 iSeries.

extendvg error
0516-1980 /usr/sbin/extendvg: Block size of all disks in the volume group must be the same. Cannot mix disks with different block sizes.

extendvg manpage:
Note: You cannot mix physical volume (PV) of 4 KB block size with PV blocks of other sizes. The block size of all PVs in the volume group must be the same.

Considerations for SSD
* https://www-01.ibm.com/support/knowledgecenter/HW4L4/p8ebk/considerationsforssd.htm
* http://www-01.ibm.com/support/knowledgecenter/HW4M4/p8ebj/arebjsolidstatedrives.htm
* https://www-01.ibm.com/support/knowledgecenter/9117-MMB/p7ebkl/considerationsforssd.htm

While using SSDs, consider the following specifications:
* Intermixing of SSDs and HDDs within the same disk array is not allowed. A disk array must contain all SSDs or all HDDs.
* It is important to properly plan for hot-spare devices when using arrays of SSDs. An SSD hot-spare device is used to replace a failed device in an SSD disk array and an HDD hot spare is used for an HDD disk array
* Although SSDs can be used in a RAID 0 disk array, it is preferred that SSDs to be protected by RAID levels 5, 6, 10, 5T2, 6T2, or 10T2.
* See Installing and configuring Solid-state drives to identify specific configuration and placement requirements related to the SSD devices.
* Some adapters, known as RAID and SSD adapters, contain SSDs, which are integrated on the adapter. See the PCIe SAS RAID card comparison table for features and additional information for your specific adapter type.
* SSDs are supported only when formatted to a RAID block size and used as part of a RAID array.

During the controller boot process, any 528 bytes per sector (not 4224 bytes per secotor) SSD array candidate attached to a PCIe or PCIe2 SAS RAID Controller that is not already part of a disk array is automatically created as a single-drive RAID 0 disk array.
* The RAID 0 disk array can be migrated to a RAID 10 disk array by using the technique described in Migrating an existing disk array to a new RAID level.
* The automatically created RAID 0 disk array can be deleted (see Deleting a disk array) and a new SSD disk array can be created with a different level of RAID protection (see Creating a disk array).


Debug output looks like this:

# export LVMT _VERBOSE=9
# export LVMGS _VERBOSE=9
# export LVMT _OUT=stdout
# extendvg datavg hdisk4 hdisk5
[S 8519692 13697088 11/19/15-15:24:14:052 mkvg.c 424] extendvg datavg hdisk4 hdisk5
[7 8519692 0:000 mkvg.c 981] addto_pv_list: pv name=>hdisk4< =
[7 8519692 0:000 mkvg.c 981] addto_pv_list: pv name=>hdisk5< =
[7 8519692 0:000 comutl.c 1010] lvm_getvginfo: start
[7 8519692 0:000 configutl.c 1115] lvm_config: call to hd_cfg, cmd=147
[7 8519692 0:001 comutl.c 1203] lvm_getvginfo: end, rc=0
[7 8519692 0:001 utilities.c 1826] lvm_cfglock: name=datavg, pid=8519692, flags=0x5
[7 8519692 0:001 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:001 utilities.c 2007] lvm_cfglock_query: name=datavg, cfglock_state=-261, owner_pid=0, owner_ppid=0
[7 8519692 0:001 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:001 utilities.c 1826] lvm_cfglock: name=hdisk4, pid=8519692, flags=0x1
[7 8519692 0:001 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:001 utilities.c 2007] lvm_cfglock_query: name=hdisk4, cfglock_state=-261, owner_pid=0, owner_ppid=0
[7 8519692 0:001 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:001 utilodm.c 1288] lvmdb_chkpvcfg(), rc=0
[7 8519692 0:001 comutl.c 1557] lvm_pvio: start
[7 8519692 0:001 comutl.c 1565] lvm_pvio: loading
[7 8519692 0:001 comutl.c 1570] lvm_pvio: finished loading
[7 8519692 0:042 comutl.c 1426] lvm_thread_pvio: pv->name=hdisk4 hex blocks = 000000004dd00000 = 000000004dd00000
[7 8519692 0:042 comutl.c 1615] lvm_pvio: end, rc=0
[7 8519692 0:042 mkvg.c 1710] validate_pvs: PV type(DD_SCDISK/DD_SCRWOPT)
[2 8519692 0:042 mkvg.c 1718] validate_pvs: Mixed blk sizes! Other disks have 4096 block size. hdisk4 (512)
0516-1980 extendvg: Block size of all disks in the volume group must be the same.
Cannot mix disks with different block sizes.
[7 8519692 0:042 utilities.c 1826] lvm_cfglock: name=hdisk5, pid=8519692, flags=0x1
[7 8519692 0:042 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:042 utilities.c 2007] lvm_cfglock_query: name=hdisk5, cfglock_state=-261, owner_pid=0, owner_ppid=0
[7 8519692 0:042 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:042 utilodm.c 1288] lvmdb_chkpvcfg(), rc=0
[7 8519692 0:042 comutl.c 1557] lvm_pvio: start
[7 8519692 0:042 comutl.c 1565] lvm_pvio: loading
[7 8519692 0:042 comutl.c 1570] lvm_pvio: finished loading
[1 8519692 0:043 comutl.c 1358] lvm_thread_pvio: FAIL: invalid pv, name=hdisk4
[7 8519692 0:083 comutl.c 1426] lvm_thread_pvio: pv->name=hdisk5 hex blocks = 000000004dd00000 = 000000004dd00000
[7 8519692 0:083 comutl.c 1324] lvm_valid_pvs_remain: inv pv name=hdisk4, pv status=0, errno=0
[7 8519692 0:083 comutl.c 1615] lvm_pvio: end, rc=0
[7 8519692 0:083 mkvg.c 1710] validate_pvs: PV type(DD_SCDISK/DD_SCRWOPT)
[2 8519692 0:083 mkvg.c 1718] validate_pvs: Mixed blk sizes! Other disks have 4096 block size. hdisk5 (512)
0516-1980 extendvg: Block size of all disks in the volume group must be the same.
Cannot mix disks with different block sizes.
[7 8519692 0:083 mkvg.c 2244] num_invalid_pvs: name=hdisk4, status=0
[7 8519692 0:083 mkvg.c 2244] num_invalid_pvs: name=hdisk5, status=0
[1 8519692 0:083 mkvg.c 1887] validate_pvs: FAIL: Invalid PVs!, num_invalid_pvs failed, rc=2
[1 8519692 0:083 mkvg.c 740] main: FAIL: validate_pvs failed
[7 8519692 0:083 mkvg.c 161] cleanup_exit(), signal or line=741, cmd_progress=1
[1 8519692 0:083 mkvg.c 177] cleanup_exit: FAIL: pv_failures=2
0516-792 extendvg: Unable to extend volume group.
[7 8519692 0:083 utilities.c 1826] lvm_cfglock: name=hdisk4, pid=8519692, flags=0x2
[7 8519692 0:083 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:083 utilities.c 1826] lvm_cfglock: name=hdisk5, pid=8519692, flags=0x2
[7 8519692 0:083 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[7 8519692 0:083 utilities.c 1826] lvm_cfglock: name=data, pid=8519692, flags=0x2
[7 8519692 0:083 configutl.c 1115] lvm_config: call to hd_cfg, cmd=139
[E 8519692 0:110 mkvg.c 277] extendvg: exited with rc=1
# unset LVMT _OUT
# unset LVMGS _VERBOSE
# unset LVMT _VERBOSE
# exit



http://omnitech.net/reference/2015/11/19/aix-4k-sectors/

?

Log in

No account? Create an account