Hi Kevin,
thanks, that helps. Let me answer your last post directly first, because we now have concrete data for both the SCSI and BlockStorage paths.
Regarding the ring size: agreed. We were deliberately conservative because we initially treated the shared region more like scarce driver memory. Your clarification changes that assumption. A substantially larger ring is clearly possible and may improve batching and headroom.
However, our measurements also support your other point: increasing the ring would not fundamentally remove the bottleneck while the original SCSI requests remain page-sized.
Using normal mounted-volume I/O on the same ~152 MiB file on our Intel test system, we measured:
Initial request-by-request WRITE: 0.32–0.36 MB/s
Shared-memory ring WRITE: 2.5–2.9 MB/s
Shared-memory ring READ: 5.8–8.0 MB/s
1 GbE payload ceiling: ~110 MB/s
The shared ring improved WRITE by roughly 7–9x. But almost all framework requests remained 4 KB:
WRITE:
4 KB 37,238
16 KB 18
64 KB 0
READ:
4 KB 38,315
16 KB 2
64 KB 22
Our application-side coalescer reduced 92,026 original requests to 91,876 wire commands, only about 0.16%. By the time requests reach us they are already separate active SCSI tasks, so a larger ring can keep more requests in flight but cannot remove the per-task framework lifecycle.
That gives the scale you asked for: on Intel, the current single-page behavior costs us roughly one to two orders of magnitude compared with normal 1 GbE NAS throughput. QD64, multiple ITTs, ImmediateData, larger Data-In PDUs and the shared ring all work, but they cannot compensate for paying the full SCSI/DriverKit lifecycle for almost every 4 KB request.
On Apple Silicon the 16 KB page size reduces the request count by 4x, but the same limitation remains. For a NAS initiator, that is still not product-viable.
So I agree that SCSIControllerDriverKit is probably the most useful path to fix first. If the kernel starts issuing larger requests while UserGetDataBuffer remains unchanged, our SCSI DEXT should require little architectural change.
There is, however, one new result from the BlockStorage path that may be relevant.
Our IOUserBlockStorageDevice implementation is essentially complete apart from one very small but critical missing bridge.
We already have dynamic geometry from the real iSCSI session, RMB=0 / native fixed-disk presentation, /dev/diskN, publish/unpublish, READ/WRITE/FLUSH orchestration, queue depth, shared-memory App/DEXT transport, real iSCSI READ/WRITE, completion handling and PR / single-writer arbitration.
The remaining problem is specifically this callback:
DoAsyncReadWrite(
bool isRead,
uint32_t requestID,
uint64_t dmaAddr,
uint64_t size,
uint64_t lba,
uint64_t numOfBlocks,
IOUserStorageOptions options)
DriverKit 25.5 documents dmaAddr only as:
DMA address of the data buffer
Unfortunately, the documentation does not explain how a software-backed BlockStorage driver is supposed to access the data behind that DMA address from CPU context.
We initially tested whether dmaAddr might be CPU-dereferenceable inside the DEXT. Runtime proved that assumption wrong.
READ completion repeatedly crashed the DEXT at:
memcpy(reinterpret_cast<void *>(dmaAddr), ...)
with:
EXC_BAD_ACCESS
KERN_INVALID_ADDRESS
After repeated IOUserServer crashes, macOS eventually panicked with:
Driver IOUserServer(com.aviontex.iscsi...) has crashed too many times
(reason 2:11)
We stopped testing and inspected the actual Xcode 26.5 / DriverKit 25.5 SDK instead of making further assumptions.
We checked IOUserBlockStorageDevice.iig, the generated header, the private StartDev interface, exported BlockStorageDeviceDriverKit symbols, the generic IODMACommand API and the wider DriverKit headers.
We could not find any public equivalent of:
GetDataBuffer(...)
GetDMACommand(...)
dmaAddr -> IOMemoryDescriptor
dmaAddr -> IODMACommand
Map/Resolve/LookupDMAAddress(...)
This is particularly noticeable because other DriverKit families expose descriptors explicitly when CPU-side access is intended. DoAsyncUnmap() in the same BlockStorage class receives an IOMemoryDescriptor *, and SCSIControllerDriverKit explicitly provides UserGetDataBuffer().
IODMACommand::PerformOperation() initially looked promising. DriverKit documents it as a way to perform CPU access to a prepared DMA mapping, for example to/from a driver-allocated bounce buffer.
But that method operates on a specific prepared IODMACommand instance. IOUserBlockStorageDevice::DoAsyncReadWrite() gives us only the resulting DMA address, not the IODMACommand or IOMemoryDescriptor that owns that mapping.
Interestingly, our older experimental BlockStorage implementation had already identified exactly this gap. It contained a proposed:
dmaAddr -> avx_descriptorForDMA() -> IOMemoryDescriptor
path, but avx_descriptorForDMA() was deliberately left as a stub returning nullptr until a real BlockStorage-family API could be identified.
So the BlockStorage path is now in a rather frustrating position: from our side it is almost finished, and the only missing connection is:
framework-created DMA mapping
|
dmaAddr
|
???
|
CPU-accessible request bytes
|
shared ring
|
iSCSI transport
We searched the public DriverKit 25.5 SDK for that bridge and cannot find one.
So the remaining question is now very narrow:
Is this absence intentional?
Is IOUserBlockStorageDevice designed on the assumption that dmaAddr is handed to DMA-capable hardware, with no supported CPU-access path for a software-backed device?
Or is there an intended BlockStorage-specific mechanism to access the already-prepared mapping that is not exposed or documented in the public SDK?
If such a mechanism exists, the BlockStorage path may genuinely be only one small missing API connection away from working and would avoid the single-page SCSI limitation entirely.
If it does not exist, then the picture is finally clear:
SCSIControllerDriverKit:
CPU-accessible request descriptor available,
but I/O is currently page-bound.
BlockStorageDeviceDriverKit:
the desired block-I/O model is available,
but READ/WRITE exposes only a DMA address
with no documented CPU-accessible descriptor path.
In that case, your comment that SCSIControllerDriverKit is the path most likely to be fixed first makes complete sense, and we would freeze the BlockStorage work rather than build another unsupported workaround around the DMA contract.
Thanks again for helping us narrow this down. At this point the remaining BlockStorage issue is no longer a large architectural problem on our side, but literally this one missing DMA-to-CPU access bridge.
Best regards,
Torsten
