Fs3 Reservation Error Timeout
Contents |
Determining root cause for a SCSI Reservation Conflict Issue Posted on May 10, 2012 by Rick Blythe Previous deep dive
Scsi Reservation Conflict Vmware
posts have dealt with performance issues or faulty hardware. This week waiting for timed out hb Nathan Small (Twitter handle: vSphereStorage) takes us through the determination of root cause for a SCSI
Reservation State On Device Is Unknown.
Reservation Conflict issue: History of issue: Customer performed a firmware upgrade to their IBM SVC's. Upon completing the firmware update and bringing them back online, command failures and what is scsi reservation SCSI reservation conflicts were observed across all hosts. Issue was resolved by sending LUN reset (vmkfstools -L lunreset /vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxx) to the affected LUNs. Root cause analysis requested. We need to start by ensuring that all hosts saw the issue the same way, and that the firmware update sequence logically makes sense. Let's start by looking state in doubt; requested fast path state update at the firmware update sequence, and to determine when it began and finished. We have to locate when the paths to one target dropped: 2012-04-29T13:18:19.684Z cpu10:4768)<3> rport-5:0-4: blocked FC remote port time out: saving binding 2012-04-29T13:18:19.685Z cpu2:4794)<3>lpfc820 0000:04:00.0: 0:(0):0203 Devloss timeout on WWPN 50:05:07:68:01:30:59:fd NPort x3f010d Data: x0 x7 x0 2012-04-29T13:18:19.713Z cpu12:4759)<3> rport-6:0-4: blocked FC remote port time out: saving binding 2012-04-29T13:18:19.714Z cpu7:4795)<3>lpfc820 0000:04:00.1: 1:(0):0203 Devloss timeout on WWPN 50:05:07:68:01:10:59:fd NPort x5d0010 Data: x0 x7 x0 From the above we can see there were 2 paths dropped, 1 per HBA. Infact, these Emulex device loss messages even give you the WWPN of the IBM SVC targets: WWPN 50:05:07:68:01:30:59:fd WWPN 50:05:07:68:01:10:59:fd For our example, we will refer to LUN 42 when comparing WWPNs of targets and tracking path failures: naa.60050768018900725000000000000810 : IBM Fibre Channel Disk (naa.60050768018900725000000000000810) vmhba1:C0:T2:L42 LUN:42 state:active fc Adapter: WWNN: 20:00:00:00:c9:a2:26:87 WWPN: 10:00:00:00:c9:a2:26:87 Target: WWNN: 50:05:07:68:01:00:59:fd WWPN: 50:05:07:68:01:10:59:fd vmhba1:C0:T1:L42 LUN:42 state:active fc Adapter: WWNN: 20:00:00:00:c9:a2:26:87 WWPN: 10:00:00:00:c9:a2:26:87 Target: WWNN: 5
Tweet Storage Performance and Timeout Issues Posted on
Nmp_throttlelogfordevice
April 10, 2012 by Rick Blythe Here's our vmware hcl second log deep-dive from Nathan Small (Twitter handle: vSphereStorage). If you missed the first one, here it is. History of the issue General storage performance issue while experiencing timeouts and SCSI reservation issues (Timeout/conflict) http://blogs.vmware.com/kb/2012/05/determining-root-cause-for-a-scsi-reservation-conflict-issue.html on ESX 4.1 (VMware View) and ESX 5.0 environments. Need to determine why these events are being seen and how we can improve performance.The first step when troubleshooting this type of issue is to take a frame of reference that is visible across http://blogs.vmware.com/kb/2012/04/storage-performance-and-timeout-issues.html all hosts. Here is the issue observed on one of the ESX 4.1 host:ESXi 4.1:Mar 28 17:12:58 vmkernel: 34:11:37:08.013 cpu13:4262)NMP: nmp_PathDetermineFailure: SCSI cmd RESERVE failed on path vmhba1:C0:T2:L1, reservation state on device naa.60060480000190100554533032324644 is unknown.Mar 28 17:12:58 vmkernel: 34:11:37:08.013 cpu13:4262)ScsiDeviceIO: 1688: Command 0x16 to device "naa.60060480000190100554533032324644" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.Mar 28 17:12:58 vmkernel: 34:11:37:08.013 cpu13:4262)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.60060480000190100554533032324644" state in doubt; requested fast path state update…Mar 28 17:12:58 vmkernel: 34:11:37:08.013 cpu14:4134)WARNING: FS3: 7120: Reservation error: TimeoutWhen reviewing the logs from the ESX 5.0 host, we see the issue observed at the same time:ESXi 5.0:2012-03-28T17:12:56.481Z cpu11:4107)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba2:C0:T3:L23, reservation state on device naa.60060480000190100554533033364445 is unknown.2012-03-28T17:12:56.481Z cpu11:4107)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x16 (0x4124003d4740, 0) to dev "naa.60060480000190100554533033364445
VAAI API Symptoms:-All the esx host lost the datastores-When you login to esxhost and do "df -h" it hangs-You cannot shutdown the guest vm's http://pg8873.blogspot.com/2011/02/resolving-3parvmware-scsi-reservation.html properly, it will get stuck at say 95%-Your VM's won't be accessible to power onLogin to 3PAR3PAR_F200_PG1 cli% showrsvVVname Host Owner Port Reservation Typevv-datastore1 esx01 21000024 http://communities695.rssing.com/chan-22405585/all_p1179.html FF2FCA50 unknown SCSI-2vv-datastore2 esx01 21000024 FF2FCA50 unknown SCSI-2VV-datastore5-2MB esx01 21000024 FF2FCA50 unknown SCSI-2The above output confirms that there are scsi reservation on the lunsVMWARE KB Article to scsi reservation resolve the issuehttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002293Solution from 3PAR side:3PAR_F200_PG1 cli% setvv -clrrsv vv-datastore1WARNING: Active hosts may be disrupted.Are you sure you want to run setvv -clrrsv vv-datastore1select q=quit y=yes n=no: y3PAR_F200_PG1 cli% showrsvVVname Host Owner Port ReservationTypevv-datastore2 esx01 21000024FF2FCA50 unknown SCSI-2VV-datastore5-2MB esx01 21000024FF2FCA50 unknown SCSI-2(the above output shows it reset lock on vv-datastore1)3PAR_F200_PG1 cli% setvv -clrrsv vv-datastore2WARNING: fs3 reservation error Active hosts may be disrupted.Are you sure you want to run setvv -clrrsv vv-datastore2select q=quit y=yes n=no: y3PAR_F200_PG1 cli% showrsv VVname Host Owner Port ReservationTypeVV-datastore5-2MB esx01 21000024FF2FCA50 unknown SCSI-23PAR_F200_PG1 cli% setvv -clrrsv VV-datastore5-2MB(the above output shows it reset lock on vv-datastore2)WARNING: Active hosts may be disrupted.Are you sure you want to run setvv -clrrsv VV-datastore5-2MBselect q=quit y=yes n=no: y3PAR_F200_PG1 cli% showrsvno reservations found(the above output shows no reservations)Solution from VMware ESx host side:List all the pending reservation#tail -1 /proc/scsi/qla2xxx/[0-9]*Perform a LUN reset to clear the lock# vmkfstools --lock lunreset /vmfs/devices/disks/naa.50002ac000080a7e---------------ESXi /var/log/messages Error Logs:---------------Feb 18 19:11:32 Vpxa: [2011-02-18 19:11:32.385 1D695B90 warning 'App'] [VpxaHalStats] Unexpected return result. Expect 1 sample, receive 2Feb 18 19:11:35 Vpxa: [2011-02-18 19:11:35.216 1D613B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'Feb 18 19:11:35 Vpxa: [2011-02-18 19:11:35.216 1D613B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)Feb 18 19:11:35 Vpxa: [2011-02-18 19:11:35.216 1D5D2B90 info 'App' opID=task-internal-22417-dbfb04f9] [VpxLRO] -- BEGIN task-internal-22417 -- -- vpxapi.VpxaService.fetchQuickStats -- 52c7f745-06cb
about this channel Embed this content in your HTML Search confirm cancel Report adult content: click to rate: Account: (login) More Channels Showcase RSS Channel Showcase 6862243 RSS Channel Showcase 8220763 RSS Channel Showcase 6987990 RSS Channel Showcase 5469800 Channel Catalog Subsection Catalog Articles on this Page (showing articles 23561 to 23580 of 94404) 07/21/14--07:38: _Re: Not able to add... 07/21/14--07:35: _Re: vSAN implementa... 07/21/14--07:39: _Creating a lab envi... 07/21/14--07:47: _Re: How many days l... 07/21/14--07:48: _Re: Подскажите... н... 07/21/14--07:50: _Проблема с хостом в... 07/21/14--07:56: _Re: Getting "Call "... 07/21/14--07:48: _Re: vSphere Web Cli... 07/21/14--07:57: _Re: vSAN implementa... 07/21/14--07:59: _Re: Troubleshooting... 07/22/14--04:19: _Re: passing bulk vm... 07/22/14--04:20: _Power on vApp with ... 07/22/14--04:22: _Re: Recently took t... 07/22/14--04:34: _Re: custom property... 07/22/14--04:48: _Re: Backup of VM's ... 07/22/14--04:47: _Re: Cluster Microso... 07/22/14--04:50: _Re: vCAC 6 - how to... 07/22/14--04:56: _Запланированные зад... 07/22/14--04:57: _Re: vCAC running wr... 07/22/14--05:00: _Re: Email at the en... (showing articles 23561 to 23580 of 94404) Browse the Latest Snapshot Browsing All Articles (94404 Articles) Live Browser Channel Description: Most recent forum messages older | 1 | .... | 1176 | 1177 | 1178 | (Page 1179) | 1180 | 1181 | 1182 | .... | 4721 | newer 0 0 07/21/14--07:38: Re: Not able to add esxi to Vcenter appliance server Contact us about this article Also try installing ESXi2 from scratch. Perhaps, cloning ESXi is not supported (same id, keys, etc) Regards, Luis 0 0 07/21/14--07:35: Re: vSAN implementation with PERC h700 Contact us about this article @OP: STOP! Please read this first. Hu? Kmadnani please be aware: OP has 55 (without U1) running AFAIK thus before upgrading the hosts from ESXi55 to ESXi55U1 OP has to evacuate his VMs lying within VSAN Beta because there is no upgrade path from VSAN 55 to VSAN 55U1. Please also read VMware ESXi 5.5 Update 1 Release Notes Test Releases of Virtual SAN Upgrade of Virtual SAN cluster from Virtual SAN Beta to Virtual SAN 5.5 is not supported. Disable Virtual SAN Beta, and perform fresh installation of Virtual SAN 5.