BlockStorageDeviceDriverKit grant confirmed by support but shows "No Requests" in the portal. How to resolve?

Hi Kevin,

yes, I mean the initiator perspective, real iSCSI READ/WRITE over the network. Right now it is more annoying than technically critical. The important part is the DEXT does what it should, at least that is how it looks.

I completely agree with you. If WRITE runs but READ does not run cleanly, this is almost certainly a code issue, not a framework issue. My personal guess is that it comes down to a wrong or missing request ID, READs and WRITEs getting in each other's way during lookup. But that is only a theory. We will see who is right in the end. The bets are on. 😉

Nonetheless, thank you for your continued support so far. That is not something I take for granted. Thank you again!

I promise you, once it runs, you will be the first to know. I will send you valid numbers then. I am curious myself.

Best regards,

Torsten

Hi Kevin,

first off, thanks a lot for your pointers, and yes, macOS now has a DEXT-based iSCSI initiator for NAS. There's still a bit left to do, but the connection holds and traffic runs at full tilt. What more could you want. And as promised, here are some numbers:

Cache-independent RAW iSCSI measurements

These are cache-independent RAW whole-disk measurements against a real 4.29 TB NAS-backed iSCSI LUN, RAW and unformatted target. Every run completed deterministic read-back verification with zero data mismatches, so the VERIFY column is a full end-to-end integrity pass over the whole transfer.

Notes:

• Host queue depth was 256 for every run. The two 5G tests used sixteen 1 MiB requests, so the effective queue depth there was 16.
• Both 5G connections went straight to the NAS over independent mobile networks, no VPN. Throughput on those two rows is set by the radio path. What matters is that the initiator carries complete, read-back-verified iSCSI transfers over a live mobile link, the groundwork for our iSCSI-over-TLS layer in the initiator.
• The 400 GiB endurance run stayed at 111.592 MiB/s WRITE and 111.537 MiB/s READ, WRITE, READ and VERIFY each about 61 minutes, with a bounded DEXT proxy count throughout and no port-table growth.

This is all running on Intel now, and running well on older Intel hardware specifically mattered to us: for something like this it has to stay performant on aging systems, not just the latest ones. From next week we move to Apple Silicon (M5 Pro) and see how throughput holds up there. The initiator itself is not fully done either, getting a DEXT driver cleanly embedded inside an application turned out to be a real challenge. If anything else turns up during beta testing, I'll post it here, though I doubt it will.

One last thing, and I mean it. Thanks for the past few weeks. You revisited your own conclusions the moment the facts moved, and you went out of your way more than once, the entitlement request on our behalf being the clearest example. That kind of engagement is rare and made this a real pleasure.

So thank you, Kevin. Enjoy your weekend, and until the next posting. :)

Torsten

Hi Kevin,

I sent another Feedback Assistant report:

FB24597050

This one is about SCSI Task Management Functions (TMFs).

For our iSCSI initiator this is certainly annoying, but the good news is that it is not a showstopper for our initial SOHO/NAS target. In that environment, normal READ/WRITE operation and the basic recovery paths are much more important to us right now.

For enterprise use, however, I would consider proper TMF support essential.

The interesting part is what we found in the KDK. As far as I can see, most of the DriverKit-side plumbing already exists.

The generated User*TaskRequest IORPC paths are there for:

• Abort Task
• Abort Task Set
• Clear ACA
• Clear Task Set
• Logical Unit Reset
• Target Reset

But the corresponding IOUserSCSIParallelInterfaceController kernel entry points currently just return kSCSIServiceResponse_FUNCTION_REJECTED instead of forwarding into those generated RPCs.

So, from the outside, this looks less like a missing TMF architecture and more like a missing connection in the IOUserSCSIParallelInterfaceController kernel support layer.

That is actually the encouraging part: the missing piece appears to be quite localized, and much of the infrastructure seems to already be there.

One side note regarding BlockStorageDeviceDriverKit: TMF support does not appear to exist there at all, not even as an obvious equivalent API surface. For a SOHO/NAS-oriented software block device that may not matter much initially, but if that framework is ever intended to cover enterprise storage use cases as well, proper task-management and recovery semantics would probably need to be added there too.

I wish you a relaxed and quiet week despite all this. ;)

Best regards,

Torsten

So, from the outside, this looks less like a missing TMF architecture and more like a missing connection in the IOUserSCSIParallelInterfaceController kernel support layer.

Well... not exactly. It's not that DriverKit is missing that layer as much as it is that the mass storage layer itself doesn't really implement these methods.

More specifically, within the kernel, most of those methods were only ever implemented within the IOSCSIParallelFamily and/or the IOSCSIArchitectureModelFamily, without ever being called by the larger system. In other words, the driver stack may have implemented the methods, but that doesn't mean anything in the system actually called them.

The closest the system comes to that is the SCSITaskUserClient; however, that actually reinforces the larger issue. The only method in that list it implements is "AbortTask"[1] and, more importantly, the SCSITaskUserClient is a special-purpose interface that specifically ONLY works on things like disc burners and similar devices. It doesn't work on standard hard drives and never has.

In other words, if you're interacting with a mounted volume, the problem with the TMF functions isn't that DriverKit isn't sending them to your DEXT; it's that the entire mass storage stack doesn't actually generate them.

[1] Arguably, that does mean DriverKit should support AbortTask, but this is a MUCH narrow use case than the full TMF set.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hello Kevin,

Thank you very much! I really appreciate that, and I'd say that means:

FB24597050

is no longer an issue. ;-)

Thanks again!

Kind regards,

Torsten

Hi Kevin,

A milestone update from my side:

the world finally has an iSCSI initiator running on DEXT for macOS

Thank you again for your genuinely helpful support throughout this.

Data runs as it should up to 1 Gbit, and we're currently trying to win over a NAS vendor who can run the tests at 2.5 / 5 / 10 Gbit for us. We all agree internally that this kind of testing belongs in a lab, and we're curious to see whether someone will support us there.

As we already discussed, I'll keep this thread open for now. If anything turns up during beta testing, we may come back to you here. I'll let you know once we go to release, and we're happy to close the DTS ticket at that point.

Thanks again for all your help!

Best regards from Germany,

Torsten

Getting back to this after being out sick for a week...

the world finally has an iSCSI initiator running on DEXT for macOS

Wow. I'll be honest, I'm genuinely shocked that you've been able to get anything close to reasonable I/O performance with single page I/O. Well done!

Data runs as it should up to 1 Gbit, and we're currently trying to win over a NAS vendor who can run the tests at 2.5 / 5 / 10 Gbit for us.

One simple test I'd actually suggest here is a "null" driver where you "fake" every I/O request by succeeding every write (without doing anything) and return the same fixed data for every read. The main unanswered question I have here is how much of a bottleneck single page I/O is actually "being". My expectation was that it was going to be a pretty major issue, but that's not what the performance numbers are showing.

I'm not sure what's going on, but my guess is that the combination of these factors:

1. The network is slow enough that it's still the primary bottleneck.

2. The kernel is much more efficient at dividing and mapping requests than I'd expected.

3. Buffered I/O is constraining your callout count enough that the increased request count doesn't actually matter.

What I don't know is how well all this will hold out. My guess is that once the network is fast enough you'll pretty quickly hit a performance wall, but it's possible that 2 & 3 are fast enough that they'll keep up with the network instead.

Thanks again for all your help!

You're very welcome.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

after a short vacation last week, here is a first small update for you, just so you don't think I have forgotten about you. We are still tuning, but here are a few early numbers already (Intel based).

Well, if 1 Gbit already surprises you, then I do not know what you will say about these numbers. And a big compliment to the entire kernel developer team here, because it shows there is not just work behind this but also an enormous amount of know-how.

Calibration and warmup:

Data measurement (100-iteration loop):

I am not sure how you would rate these, but to me they look good enough to keep.

We also made a decision: the NULL driver goes into the initiator itself, as a built-in benchmark. That way the user can see right away whether something is the initiator or the network configuration. Thanks again for the pointer at exactly the right spot.

Based on your 1 Gbit remark, the numbers should come as a surprise. That is also why the 'nDEXT Benchmark' will stay in the app. Sure, we can claim a lot here, but some people will surely want to verify it. They are welcome to and a matching measurement script is no big deal. I think any AI can put one together in two minutes.

All in all we are happy with where this stands. The one thing we still want to get under control is the extreme outliers (compare High vs Low).

As a side note, we have already broken the 20 Gbit/s mark, although it hits the CPU hard. In short, the DEXT can hold its own.

Once this runs stably on Intel, we will move to the M5 for testing. Those still have delivery delays but they should (supposedly) arrive by the end of this week. I am curious.

Best,

Torsten

Well, if 1 Gbit already surprises you, then I do not know what you will say about these numbers. And a big compliment to the entire kernel developer team here, because it shows there is not just work behind this but also an enormous amount of know-how.

Thank you and I'll be sure to pass that along. One minor question I'm still curious about— what sort of slot count are you typically getting when we call UserProcessBundledParallelTasks? My suspicion is that it's the key element here, as it's letting us keep the number of call outs low enough that we don't drown.

As a side note, we have already broken the 20 Gbit/s mark, although it hits the CPU hard.

Yes, that’s what I'd expect to get hit after IPC cost. I'm optimistic that will get a bit better once we get a fix in place so you can use larger I/O sizes again.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

We’re already using UserProcessBundledParallelTasks, with no single-task callbacks. We’re just not getting as much batching out of it as we’d like.

In the latest run with roughly 1024 KiB I/O and host QD1 to 128, we counted about 3.21 million bundled callbacks for 3.26 million block requests: roughly 1.016 block requests per callback. That’s from the whole-run counters rather than a slot-count histogram.

We also looked at the kernel path. Tasks go through serial preparation, including DMA preparation, before entering the batching queue. The delayed dispatch uses a fixed 10us interval.

For now, though, we can live with it. Average RAW I/O latency at QD64 is around 20 to 30 ms. A bit annoying, but I guess it's OK. ;)

We’d rather wait for the DMA fix before spending more time on batching. Being able to use larger I/O sizes again means fewer tasks for the same amount of data. That could reduce the overhead enough that the limited batching no longer matters much for us, even if the batches themselves stay small.

Once that fix is available, we’ll take another look and see whether any further changes are needed at all. If they are, being able to adjust the 10us interval would be helpful. We have one or two other ideas as well, but we need to wait for the DMA fix before deciding whether they’re needed at all.

Best regards,

Torsten

In the latest run with roughly 1024 KiB I/O and host QD1 to 128, we counted about 3.21 million bundled callbacks for 3.26 million block requests: roughly 1.016 block requests per callback.

Huh. That's a lot fewer than I'd expected.

We’d rather wait for the DMA fix before spending more time on batching. Being able to use larger I/O sizes again means fewer tasks for the same amount of data. That could reduce the overhead enough that the limited batching no longer matters much for us, even if the batches themselves stay small.

One thing to be aware of is that you may see more batching happening under real world conditions where multiple readers/writers are running simultaneously.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

We seem to have found another little problem in the kernel. It's not a showstopper but it would be good to get this fixed.

We've been digging further into the latency spikes and traced one of the long stalls into the kernel submission path. I'd like to get your take before opening a Feedback report. Perhaps there's something we can do on the DEXT side that we're missing.

Our iSCSI initiator is running on macOS against a loopback null target. The controller reports a maximum of 256 parallel tasks. We're using synchronous 1-MiB pread calls from multiple threads. QD here is the number of outstanding host requests, not the number of commands active on the wire.

At QD1 to QD64, the latency already looked a bit off, but it wasn't obvious what was happening. We then deliberately pushed the test to QD512 and QD1024. That's when the long stalls really stood out and we could follow the affected thread into the kernel.

Above the task-pool depth, one raw read doesn't return for roughly the entire load run, while the other threads keep completing thousands of requests:

Even at QD256, the longest read is "only" 203 ms, so it doesn't look nearly as dramatic at first glance. With workloads involving millions of block I/Os, though, even shorter recurring delays can matter. It was only at QD512 and QD1024 that the sustained, multi-second stall became unmistakable.

We recorded the slow worker's OS thread ID and the exact start/end times of that read, then matched them to kernel stackshots. At QD1024/10 s, all 104 samples of that thread have this chain (leaving out the intermediate frames):

pread
  IOSCSIProtocolServices::ExecuteCommand
    IOSCSIProtocolServices::SendSCSITasksFromQueue
      IOSCSIParallelInterfaceDevice::SendSCSICommand
        IOSCSIParallelInterfaceController::GetSCSIParallelTask
          IOCommandPool::getCommand
            IOCommandPool::gatedGetCommand

The sampled PC in gatedGetCommand is the return address right after commandSleep. The 20-second runs show the same queue-draining path throughout the captured samples, mostly waiting for a task from the pool.

We also checked the disassembly of the installed Kernel Collection, with the SCSI images matched to the KDK by UUID. ExecuteCommand calls the shared queue drainer inline. That loop keeps picking up more queued tasks. SendSCSICommand can wait for a free parallel task when called outside the workloop thread.

Our reading is that one submitting thread ends up draining the queue for everyone else and gets held there as other callers keep adding work. Doubling the load duration from 10 to 20 seconds roughly doubles that read's latency too, which looks more like starvation under sustained load than a fixed timeout. One detail we haven't captured yet is exactly when that read's own SCSI task completes.

Does that reading make sense to you or have we missed something?

Is there a supported way to handle submission or completion in the DEXT that avoids holding up one caller like this while keeping the I/O concurrent? Or should I open a Feedback report?

Best regards,

Torsten

Hi Kevin!

We finally broke through 20 Gbit/s and only at that speed did another problem become visible. It is not a blocker for us, but we hope you have a solution for it.

Our DEXT moves each task's data between the IOBufferMemoryDescriptor from UserGetDataBuffer and its own shared ring. Tasks are 1 MiB. Both public API paths for that copy are expensive for us, though in different ways.

A: map the descriptor.

GetAddressRange, then memcpy. Every 1 MiB task causes 256 page faults in the DEXT (PROC_PIDTASKINFO pti_faults; the app takes almost none and the peer takes none, no COW faults, no pageins). 2047 KiB tasks cause 514. DEXT CPU per task: 0.62 ms user + 1.34 ms system (READ), 0.47 + 1.00 ms (WRITE). About 2 700 tasks/s with four copy queues, and CPU_Speed_Limit drops to around 60 during the run.

B: kernel copy, no mapping.

One IODMACommand per copy queue, Create(this, 0, {options 0, maxAddressBits 64}, &cmd). Per part: PrepareForDMA(0, hostDescriptor, offset, length, &flags, &count, segments), PerformOperation (OptionWrite for READ data, OptionRead for WRITE data, dmaOffset 0, length, ringOffset, ringDescriptor), CompleteDMA(0). All calls succeed, 582 435 parts, zero error returns, READ data verified (the test target discards WRITE data). The faults are gone, but PerformOperation takes about 1.375 ms of elapsed time per 1 MiB (prepare 10 µs, complete 6 µs) and DEXT CPU per task rises to 0.18 ms user + 3.29 ms system. Median READ at QD8 fell from 22.06 to 15.01 Gbit/s, WRITE from 20.39 to 15.84.

That leaves us with three questions for you:

1. Is there a supported way to access the task data without faulting every newly mapped page or copying through an intermediate buffer? For example a mapping that is populated up front (kIOMapPrefault is not among the public CreateMapping options and GetAddressRange takes no options).
2. Is PerformOperation intended for this use case? Its implementation in the KDK appears to allocate an intermediate buffer and copy through it. Is there a way to copy between the two descriptors without that buffer?
3. This looks like the same gap as in our already reported descriptor issues: every task still materializes a framework owned per task descriptor. Path A faults once per page of it, path B copies through an intermediate buffer. Does the planned software I/O path (task scoped, CPU accessible, multi page descriptor without hardware DMA preparation) cover this? If so, this measurement may be useful as the performance side of those reports.

Thanks,

Torsten

Does that reading make sense to you or have we missed something?

So, feeling a bit prophetic, in my last message, I said:

"My guess is that once the network is fast enough, you'll pretty quickly hit a performance wall." That leads to here:

Our reading is that one submitting thread ends up draining the queue for everyone else and gets held there as other callers keep adding work.

That's sort of what's going on, but not for the reason you think. I think what's actually going on here is that, as a general "rule", the system has a general preference for "unfair locks". What that refers to is how the system handles a sequence like this:

• Thread 1 & thread 2 both reach "lock A".

• Thread 1 "wins", taking lock A, and continues executing.

• Thread 2 blocks, waiting on lock A.

• Thread 1 releases lock A and continues executing.

• Thread 2 remains blocked, waiting for the scheduler to give it time.

• Thread 1 reaches "lock A" and attempts to acquire it.

With a traditional "fair" lock, thread 1 will now block waiting to acquire the lock again, ensuring that all lock holders eventually have some opportunity to run. However, with an unfair lock (on macOS), thread 1 will be allowed to acquire the lock and continue executing.

That advantage of this approach is that it allows thread 1 to continue executing through its full thread quanta, instead of artificially forcing a context switch at every lock hit. That might not seem important, but keep in mind the "proper" lock usage patterns often involve taking the same lock over and over again as it's briefly used to protect some critical data structure like the pool of SCSI commands the system reuses.

On the other hand, the risk with unfair locks is exactly what you're seeing. That is, under heavy lock contention, there's a tendency for a subset of threads to repeatedly "win" the same lock, starving other threads trying to access the same lock.

HOWEVER, the critical point here is that this:

Is there a supported way to handle submission or completion in the DEXT that avoids holding up one caller like this while keeping the I/O concurrent?

...won't actually solve the problem, at least not the way you'd want. That is, using a fair lock here would prevent any thread from starving; however, it would do so by massively increasing the churn between threads, slowing everything down. The actual solution to these issues is to reduce lock contention, as the problem only occurs when many threads are repeatedly contending for the same lock.

Or should I open a Feedback report?

Nah, I wouldn't bother. Reducing contention here means increasing the max transfer size, so that the system can use fewer SCSITasks per transfer... which is exactly the bug we're already working on fixing.

Moving to your second post, let me start by talking about choosing between these two choices:

A: map the descriptor.

B: kernel copy, no mapping.

Basically, I think your only choice here is to benchmark both and pick the fastest. My guess is that it will probably be "A", but there isn't really something that makes one of them inherently "better" than the other.

One comment here:

Every 1 MiB task causes 256 page faults in the DEXT

I don't think that number is all that meaningful here as, to the extent that "B" behaves differently, it's only doing so because it's shifted the same logic out of your DEXT and into the kernel.

Is there a supported way to access the task data without faulting every newly mapped page or copying through an intermediate buffer?

No, not within the current architecture. Within the kernel, you could map the descriptor directly into the target process, but DriverKit was intentionally designed NOT to allow it. That's partly for security reasons, but mostly because it's a great tool for making all sorts of interesting bugs. Even within KEXT, the preferred pattern in user client design had long been for the client to push memory into the driver, since that was much easier to conceptually manage.

Is PerformOperation intended for this use case?

Yes and no. It works because, in the high-level sense, its "job" is basically "copy data between two buffers". However, its actual reason for existence is tied into how DARTs are used in PCI transfers. Assuming it's slower than "A", I think that's actually because it ends up doing more work handling this as a "hardware" transfer, then ultimately ends up doing the equivalent of a basic memcpy.

Does the planned software I/O path (task-scoped, CPU accessible, multi-page descriptor without hardware DMA preparation) cover this?

No, I don't think those changes will really affect this. My guess is that using larger memory descriptors may change the faulting statistics a bit, but I don’t think the overall impact will be dramatic.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

Thanks, that helps. Two questions came to mind:

1. Larger transfers and lock contention

We’re already seeing actual 1 MiB SCSI tasks arrive at the DEXT.

Would you expect the planned larger-I/O fix to help with submission/completion contention at that size too, or would the main benefit be for workloads still split into page-sized tasks?

2. SuperPages and a software SCSI bridge

This bit caught our attention:

My guess is that using larger memory descriptors may change the faulting statistics a bit, but I don’t think the overall impact will be dramatic.

That made us wonder whether SuperPages could be useful as part of a lightweight software SCSI bridge. Since they already exist, could there be a supported way to put them to work for iSCSI with efficient CPU access and concurrent transfers?

Just an idea, but it looks promising to us if the framework could make them straightforward to use in this context. We’d certainly like to have that option.

Freedom for iSCSI! Give us the SuperPages! If we get that bridge, you’ll never hear from us again. Well, if it works. Promise. 😉

Seriously though, our small test worked with one SuperPage in flight and we verified the data byte for byte (QD1). When we tried to scale up, allocation failed before I/O started.

Could this become a practical option for concurrent iSCSI transfers? Or is there something about SuperPages or the SCSI path that makes them unsuitable here? We’d appreciate your take before pursuing the idea further.

Cheers and thanks again!

Torsten

We’re already seeing actual 1 MiB SCSI tasks arrive at the DEXT.

Is this when you're going through the VFS system/buffered dev node?

FYI, I would NOT ship that configuration on the current system. Buffered I/O is basically working by "accident" on current systems. I think what's going on is that the UBC is incidentally allocating physically contiguous pages, which then allows the SCSI controller to generate a contiguous buffer address. The problem is that I don't think that's actually reliable. My guess is one or more of:

• Sufficient VM pressure generating physical fragmentation.

• The right I/O patterns feeding physically fragmented memory.

• Dumb luck.

...would end up generating the same failure that kicked all of this off.

Would you expect the planned larger-I/O fix to help with submission/completion contention at that size too, or

Yes. I think contention all boils down to the volume of calls in and out of the DEXT, so anything that reduces that volume improves things.

However, one comment here:

Would the main benefit be for workloads still split into page-sized tasks?

Practically speaking, you'll basically only receive page-sized I/O requests. The idea behind the UBC is that it essentially treats the raw block device as a different kind of VM backing store, intentionally reading and writing in multiple pages whenever possible.

That made us wonder whether SuperPages could be useful as part of a lightweight software SCSI bridge. Since they already exist, could there be a supported way to put them to work for iSCSI with efficient CPU access and concurrent transfers?

Yeah... So, I'll be honest, I had to look Super Pages up when I saw this, as I only had the vaguest recollection of ever having heard about them. In any case, the short summary is that they were briefly implemented on x86_64, but have never been implemented on Apple Silicon and are now considered deprecated, at least as far as the system is concerned.

I don't know the exact history behind that, but my guess is that they were originally introduced because we thought they'd be "useful" but that experience showed that wasn't actually true, most likely because of the role the vm compressor plays in the "modern" VM system. There's a thread on this here you might find interesting, but the basic gist is that the system basically has a three-level memory system where a large portion of physical memory is actually storing compressed memory, with swap files being used as the backing store for compressed memory.

That architecture both benefits from a larger page size (it gives the compressor more data to work with) and mitigates the biggest downside (the unused space in a given page compresses really well), but it also makes arbitrarily large pages more problematic as they complicate the decompression process.

Could this become a practical option for concurrent iSCSI transfers? Or is there something about SuperPages or the SCSI path that makes them unsuitable here? We’d appreciate your take before pursuing the idea further.

Which part of the transfer path are you talking about here? Data transfer between your DEXT and your supporting daemon or something else? And what are you worried about actually happening?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

Thank you for the straight answer and for taking this seriously. I won't pretend that was an easy message to read. This is pretty devastating news for us and puts our release plans for this year in serious doubt. We absolutely take your warning seriously.

To answer your question directly: the 1 MiB measurements came from pread and pwrite against /dev/rdisk2, with F_NOCACHE enabled using ordinary page-aligned buffers. They did not come from VFS file I/O or the buffered /dev/disk2 node.

We have also completed multiple continuous 400 GB RAW write, read and verification runs all cleanly. These were not just short throughput measurements.

Given that, does your explanation about the UBC incidentally providing physically contiguous pages also apply to this RAW path? Or could we be looking at a different set of conditions there? I want to make sure we distinguish the paths correctly and understand exactly what could trigger the failure despite those successful runs.

Our current SCSI implementation already includes the shared-memory ring, bundled task handling and parallel copy processing. It accepts large tasks, and our RAW benchmarks exceed 20 Gbit/s. The concern now is whether we can reliably ship that configuration on current macOS.

Is there any supported workaround for the DMA preparation issue?

As we understand it, the kernel checks the DMA segment layout before handing the task to our DEXT. If that check rejects the request, allocating a different buffer inside our task handler comes too late.

Could an existing DMA or HBA configuration make the kernel provide a suitable physically contiguous bounce buffer before that check? Or is there another supported mechanism that removes the dependency on the original buffer's physical layout?

We can accept an extra copy or a throughput penalty if it gives us a reliable native SCSI path. Our IODMACommand::PerformOperation experiment eliminated the mapping faults in the DEXT, but we understand that this does not remove the earlier DMA preparation.

If there is no supported workaround, do we need the kernel fix you discussed with the team before shipping this large-I/O configuration?

And regarding that software-I/O switch: if you can share even a rough estimate, are we talking about weeks, months or potentially years?

I know you cannot promise a release date, and I won't hold you to an estimate. We just need some sense of the timeframe to plan around. Even a temporary, supported workaround would solve a huge number of problems for us.

Thanks again for sticking with this and discussing it with the team. This is really difficult news, but we appreciate your honesty and your help in finding a way forward.

Best regards,

Torsten

Given that, does your explanation about the UBC incidentally providing physically contiguous pages also apply to this RAW path? Or could we be looking at a different set of conditions there? I want to make sure we distinguish the paths correctly and understand exactly what could trigger the failure despite those successful runs.

SO, just to make sure we're talking about the same thing, the critical issue here is that, on the current system, the only safe value that can be passed into "maxTransferSize" passed into "UserGetDMASpecification" is 1 page. Larger sizes may work under some circumstances, but those circumstances are outside of your DEXT’s control and not really predictable.

Even worse, the most critical factors here are things like:

1. Overall memory load, which determines the degree of physical memory fragmentation.

2. The "history" of the underlying memory pages and whether or not that history has allowed it to fragment.

...basically "all" of which mean problems are more likely to occur under real-world use and less likely to occur on synthetic benchmarks.

Putting that in more concrete terms, if you reboot a device, allocate a large buffer, and then run a long series of tests against that buffer, it's very likely that test will work fine. The memory was originally allocated as a physically contiguous block (because memory was "empty") and it then stayed that way (because ongoing use kept all those pages "live"). Unfortunately, that kind of usage pattern is nothing like real-world use.

Is there any supported workaround for the DMA preparation issue?

The only option is to set maxTransferSize to 1 page. If your performance is unacceptable when maxTransferSize is set to 1 page, then my recommendation would be that you wait for us to address the issue on our side.

As we understand it, the kernel checks the DMA segment layout before handing the task to our DEXT. If that check rejects the request, allocating a different buffer inside our task handler comes too late.

Yes.

More specifically, it attempts to convert it to a bus address and it assumes that conversion will result in a single segment because, if a bus-level DART were present, it WOULD have. However, that fails because you’re attached to IOUserResource, so you don't have an underlying DART.

Could an existing DMA or HBA configuration make the kernel provide a suitable physically contiguous bounce buffer before that check? Or is there another supported mechanism that removes the dependency on the original buffer's physical layout?

There are lots of ways the kernel could fix this and nothing you DEXT can do about it.

If there is no supported workaround, do we need the kernel fix you discussed with the team before shipping this large-I/O configuration?

The fix I mentioned earlier is something we'd need to implement, not you. Even if you could implement it, distributing a KEXT at this point isn't really an option.

And regarding that software-I/O switch: if you can share even a rough estimate, are we talking about weeks, months, or potentially years?

I'm extremely limited in what I can share, but what I'll say is the following:

• There is at least one totally different approach[1] outside of DriverKit that COULD solve your same basic "problem.” The reason I haven't mentioned it is that, given what I know, I don't think it would be a good use of your time to try and get it working.

• The mass storage team has a clear understanding of the issue and considers it a very serious problem that needs to be addressed.

• The bug is obviously not fixed in macOS 27.0 and can't be fixed in macOS 27.2.

Expanding on the last point, if you look at the release history of our last few releases, there's a consistent pattern where the "B" release happens shortly (~1 month) after the initial release, and the "C" release has then followed a few months later in December. That happens for exactly the reason you'd expect—the "B" release is narrowly focused on fixing any critical bugs we've found, while the "C" release has more time for testing, so it can accept a broader set of changes.

Note that the choice to "skip" 27.1 has made this conversation a bit more confusing, since 27.2 (which just entered beta) is a "B" release, but, historically, "x.2" was a "C" release.

[1] I don't remember if I talked about it on this thread or a different thread, but the basic idea would be to use a custom file system to present a "fake" DiskImage to the system, mount that DiskImage, then use your custom file system to route all I/O requests to that image file over to your remote iSCSI target. The "file system" implementation could either be an FSKit extension or an smb server running on the local machine.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware