smbfs silently zero-fills already-written data after cached file size regresses on reopen

Question

Created 2w

Replies 4

Boosts 0

Participants 3

Hello everyone,

I've been chasing a nasty silent data corruption bug in the macOS SMB client (smbfs.kext) and wanted to share what I found, in case someone else has hit it and in the hope that someone from the SMB team sees it.

What happens

Under concurrent writes with repeated reopens, the client can regress its cached file size (np->n_size) to an earlier, smaller value - behind data it has already written and flushed to the server. The next write then treats the already-written range as a hole, zero-fills it via IO_HEADZEROFILL, and sends the zeros to the server, right on top of the correct bytes it transmitted moments earlier. No write(2) fails and nothing is logged - the file just quietly comes back with a chunk-aligned run of zeros in the middle, at the correct overall length.

Environment

macOS 26.5 (Darwin 25.5.0), Apple silicon (16 KiB VM pages), SMB 2.1
Sources referenced: SMB client 538.121.1, xnu 12377.121.6

How to reproduce

Mount an SMB 2.1(2.0.2 has the same issue as well) share.
Have several threads write the same files in 8 KiB chunks, each chunk via its own open/lseek/write/close (so the file is reopened constantly as it fills), while the files are concurrently resolved by name (stat / directory enumeration).
Read the files back through a cache-cold path (second mount, or F_NOCACHE) and compare.

Roughly 1 file in several hundred came back corrupted for me. The core of the write pattern:

CHUNK = 8192  # 2 chunks per 16 KiB page

def write_chunk(path, data, offset):
    fd = os.open(path, os.O_CREAT | os.O_RDWR)  # own handle per chunk
    try:
        os.lseek(fd, offset, os.SEEK_SET)
        os.write(fd, data)
    finally:
        os.close(fd)

# per file: content = os.urandom(random.randint(265000, 300000));
# chunks written in batches of 8 threads, joined between batches;
# each file written twice from the same buffer: NAME, then NAME.copy

In my runs the corruption always landed on the second (.copy) write.

One caveat: I reproduced this against a third-party SMB server, not against macOS File Sharing (smbd), and I don't expect it to reproduce against smbd directly. The stale size arrives via reopen-via-lookup (smbfs_update_size <- smbfs_nget <- smbfs_vnop_lookup) on a freshly instantiated vnode, whose n_sizetime lets the freshness guard pass. smbd instead reopens via vnop_compound_open -> smbfs_attr_cacheenter (warm vnode; the guard rejects it) - the same stale-size candidates occur, they just all get rejected. The server merely steers the client onto the vulnerable path; the bug itself is entirely client-side.

What I observed

I captured the kernel side with dtrace fbt probes on smbfs_setsize() / smbfs_update_size() (os_log drops events under this load). Timeline for one corrupted file, correlating pcap and dtrace (dtrace has whole-second resolution, marked ".x"):

[pcap] = network packet capture of the SMB traffic between client and server
[dtrace] = kernel-side trace of the smbfs size-update functions; timestamps only have whole-second resolution, so ".x" marks an unknown sub-second time

:39.778 [pcap] — client sends WRITE off=32768 len=32768 with the correct data, covering [40960:65536).
:39.777–.860 [pcap] — throughout, the server's CREATE/CLOSE responses report a strictly monotonic EOF: 0, 8192, 40960, 65536, ... 288255.
:39.x [dtrace] — on a reopen, smbfs_update_size applies EOF 40960 (a superseded value), regressing n_size from 65536 to 40960.
:39.x [dtrace] — the next write starts past the regressed size, so zero_head_off = 40960 and IO_HEADZEROFILL is set.
:39.804 [pcap] — client sends WRITE off=32768 len=57344, ALL ZEROS over [40960:65536), on top of the correct data it sent 26 ms earlier.

End result: the file is 288255 bytes (correct length) with 24 KiB of zeros at [40960:65536) - three consecutive 8 KiB chunks, i.e. 1.5 x 16 KiB VM pages. Worth stressing: the server's own responses reported a strictly monotonic EOF the whole time, so the regression to 40960 was purely the client applying a superseded value.

Expected, obviously: the file reads back byte-for-byte identical to what was written.

Where I think the bug is

From reading the smbfs and xnu sources, three things combine:

np->n_size isn't consistently synchronized - read under the node lock only (smbfs_vnops.c:7329/7387/7391) but written under f_clusterWriteLock (:7411) and by smbfs_vnop_strategy under the cluster lock, so the reader deciding the zero-fill has no ordering guarantee. Possible fix: read it once under f_clusterWriteLock in smbfs_vnop_write so the snapshot, extend, and zero_head_off stay consistent.
The freshness guard checks the wrong thing - smbfs_update_size's reqtime <= n_sizetime guard validates the reply's request time, not whether the value is still current, so a superseded (smaller) size applied later still passes and calls smbfs_setsize(smaller). Possible fix: never shrink n_size from fa_size while the vnode has dirty pages or in-flight writes beyond that size.
The zero-fill is destructive - zero_head_off = np->n_size (smbfs_vnops.c:7391) feeds IO_HEADZEROFILL, and cluster_write zeros [n_size, uio_offset) without checking whether the UBC already holds those pages as valid/dirty (vfs_cluster.c), then flushes the zeros to the server. A defensive check there would neutralize the corruption regardless of cause.

Has anyone else seen silent zero-runs in files written over SMB under concurrent access?

Thanks!

Answer 1

Hoffman OP

2w

What happens with an os.fsync(fd) ahead of the close()?

Answer 2

mladendinev OP

2w

Good suggestion, tested it. I added os.fsync(fd) between the last write() and close() of every chunk, so each 8 KiB chunk's handle was fsync'd before closing.

The corruption still reproduces. It hit at 37 minutes in (previous runs ranged 10 min to 3 h, so well within normal variance). Same signature: exactly one 8 KiB chunk of zeros, [204800:212992), at the correct total file length.

The fsync had no measurable effect on the mechanism, and the dtrace capture from this run shows why. The zeros don't come from unflushed data. The correct bytes were already transmitted and acknowledged by the server (visible in the packet capture); the zeros come from a later write consulting a stale cached file size.

In this run the trace actually caught the trigger directly: the file's vnode was reclaimed and re-instantiated three times within one second, mid-write. On one of the freshly created vnodes (smbfs_nget path), smbfs_update_size applied a stale EOF of 196608 eight times in a row while the file had already grown past 212992; the next write then zero-filled the "gap" [204800:212992) via IO_HEADZEROFILL and flushed the zeros over data the server already had.

So fsync guarantees the data reaches the server, but this bug doesn't lose unflushed data. It actively overwrites already-acknowledged data, driven by the attribute-cache / vnode-lifetime race.

Answer 3

DTS Engineer OP

Apple

2w

I've been chasing a nasty silent data corruption bug in the macOS SMB client (smbfs.kext) and wanted to share what I found, in case someone else has hit it and in the hope that someone from the SMB team sees it.

Have you filed a bug and, if so, what's the bug number?

FYI, on this point:

Have several threads write the same files in 8 KiB chunks,

While I agree that there seems to be a bug here, I think you're also creating what's basically the worst possible I/O pattern. You're writing in 1/2 page chunks from multiple threads, which means you're specifically pushing on details of how the UBC resolves conflicting activity from multiple threads. You're also extending the size of an existing file, which means race conditions between threads are going to force zero fills across parts of the file.

Putting that in more concrete terms, if you issue writes to three chunks, "canonical" ordering would be this:

1-> 8192 to position 1 
2-> 8192 to position 2 
3-> 8192 to position 3

However, when race conditions reorder those writes, that would then become:

1-> 8192 to position 1 
3-> 8192 to zero fill position 2, 8192 to position 3
2-> 8192 to position 2

Of course, being a new network file system means that its race conditions all the way down, so another reorder later gets you:

1-> 8192 to position 1 
2-> 8192 to position 2 
3-> 8192 to zero fill position 2, 8192 to position 3

...creating exactly the problem you're seeing. I don't know if that's what's happening here and I'm not saying this isn't a bug, but I will say that what you're doing doesn't sound like a great idea.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Answer 4

mladendinev OP

1w

Thank you for looking into this case, Kevin.

Indeed, we have filed a report (with pcap, DTrace output, byte-level diffs, etc.) - it is FB24552903 (https://feedbackassistant.apple.com/feedback/24552903). If it would help you narrow the case down, please let me know, and I can run variants of the workload. Thank you in advance!

Regarding your comment on the workload - it is a fair point. Let me give some context on why it looks this way.

We have an SMB enterprise solution, and our customers run it under just about every application imaginable - video editing, audio workstations, render pipelines, build systems. So we don't get to choose the I/O patterns; we have to survive all of them. We therefore run long, multi-node simulations that exercise the whole chain and independently verify every byte written, because data integrity is the one property we can't compromise on. The pattern I posted is a distilled reproducer from those simulations: deliberately adversarial, because its job is to compress a months-scale field risk into hours.

On the reordering example: I agree that's a real interleaving, and we see that transient zero-fill constantly - but it's self-healing. In your third ordering, thread 2's write(2) has already returned successfully, so its bytes sit in the UBC as dirty pages and flush afterward; the final file is whole. That case is not harmful in any way, and it isn't what I'm reporting.

I would say there are three things that separate the observed failure from that race:

Nothing was in flight against the zeroed range.The writer runs 8 threads per batch over disjoint 8 KiB ranges and joins all of them before the next batch starts. The zeroed range [147456:196608) belonged to a completed batch: every write(2) covering it had returned, the client had flushed it (smbfs_do_strategy writes at offsets 147456/163840/180224), and the server had acknowledged a strictly monotonic EOF through 196608. The zeros arrived ~50 ms later, on the same ordered SMB2 session, from the next batch. There was no pending write left to reorder with.
The discriminator is the size regression, which no legal interleaving produces. 46 ms after the last correct flush, the kernel logged smbfs_update_size: UBC_PUSHDIRTY, UBC_INVALIDATE ... due to file size changed, and the next vnop_write entered with old eof = 147456 - backwards from 196608. That eviction is what destroys the already-flushed pages (so nothing can re-cover the range), and the regressed n_size is what aims the zero-fill at committed data. 147456 is a value the client itself held 84 ms earlier; it appears in none of the server's size-bearing replies - CREATE, CLOSE, QUERY_INFO, or QUERY_DIRECTORY - all checked in the capture.
The failure discriminates in a way an ordering race can't. Each file is written twice from the same buffer by the same code - NAME, then NAME.copy. Across every instance we've captured, NAME is intact and only the second carries the hole, and the hole is always exactly [regressed n_size : next write-aligned ...]. A reorder-driven zero-fill would have no reason to respect that structure. So while the workload is intentionally unkind, a server-acknowledged write can be silently zeroed afterward because the client's cached EOF moved backwards.