SolidFire vs DelayedAck

As we follow up our previous post SolidFire vs ATS heartbeat about issues we had with SolidFire arrays, we had to disable another setting: DelayedAck Basically, this setting is used during TCP communication between different hosts, as the receiver of a TCP frame will wait to deliver acknowledgment segment (ACK) in order to be able to group these data segment and so decreasing network load (and at the end this is a good thing ^^).

The VMware [KB1002598][1] also explains this principle:

A central precept of the TCP network protocol is that data sent through TCP be acknowledged by the recipient. According to RFC 813, Very simply, when data arrives at the recipient, the protocol requires that it send back an acknowledgement of this data. The protocol specifies that the bytes of data are sequentially numbered, so that the recipient can acknowledge data by naming the highest numbered byte of data it has received, which also acknowledges the previous bytes.. The TCP packet that carries the acknowledgement is known as an ACK.

A host receiving a stream of TCP data segments can increase efficiency in both the network and the hosts by sending less than one ACK acknowledgment segment per data segment received. This is known as a delayed ACK. The common practice is to send an ACK for every other full-sized data segment and not to delay the ACK for a segment by more than a specified threshold. This threshold varies between 100ms and 500ms. ESXi/ESX uses delayed ACK because of its benefits, as do most other servers.

The main issue is that some iSCSI arrays are waiting to receive an ACK segment data to send the next frame, which can generate a lot of timeouts (you can easily see this with a Wireshark dump for instance), which lead to global performance decrease.

The affected iSCSI arrays in question take a slightly different approach to handling congestion. Instead of implementing either the slow start algorithm or congestion avoidance algorithm, or both, these arrays take the very conservative approach of retransmitting only one lost data segment at a time and waiting for the host’s ACK before retransmitting the next one. This process continues until all lost data segments have been recovered.

Later in the KB article, you can find some clarification about impact of DelayedAck over VMFS heartbeat:

Most notably, the VMFS heartbeat experiences a large volume of timeouts because VMFS uses a short timeout value

Compared to the array behavior, SolidFire support has asked us to disable this setting. This could be changed at any iSCSI chain level, i.e. at the adapter, target and/or devices level with inheritance of the value (which can be convenience to set it globally). As SolidFire support asked to disable it permanently we choose to set it at the adapter level:

Get-View -ViewType HostSystem -Property Name, Config.StorageDevice.HostBusAdapter | Select Name, @{Name="DelayedAck"; Expression={($_.Config.StorageDevice.HostBusAdapter.AdvancedOptions | ?{$_.key -eq "DelayedAck"}).value}}

Get-View -ViewType HostSystem -Property Config.StorageDevice.HostBusAdapter, ConfigManager.StorageSystem -SearchRoot (Get-View -ViewType ClusterComputeResource -Filter @{"Name" = "cluster name"}).MoRef | ?{ ($_.Config.StorageDevice.HostBusAdapter.AdvancedOptions | ?{$_.key -eq "DelayedAck"}).value} | %{ (Get-View ($_.ConfigManager.StorageSystem) ).UpdateInternetScsiAdvancedOptions( ($_.config.storagedevice.HostBusAdapter | ?{$_.Model -match "iSCSI Software"}).device, $null, (New-Object VMware.Vim.HostInternetScsiHbaParamValue -Property @{Key = "DelayedAck"; Value = $false}))}