[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By dkruckemyer-ventana:

A quick one to address the specification: For management CBOs, the spec does state what happens with the accessed bit and, by omission, what happens with the dirty bit (don't check and don't change). I agree that the latter should probably be made more explicit in the specification, especially since the dirty bit behaviors are explicit for the other two extensions.

The other issue of permissions is a longer topic that I have promised multiple people I would get back to (see issue #44 - we certainly owe the community a lengthy description of the rationale for posterity). Please be patient, and I promise I'll have something in the new year.

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

Not checking and not changing sounds terrifying...

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By dkruckemyer-ventana:

One perspective is that management CBOs are not explicit stores, and that the dirty bit is only set when an explicit store is executed. Flush and clean do not change the values visible to the harts, and although invalidate might, it's not actually "dirtying" memory. Certainly, those instructions have performed a memory access, so checking and setting the accessed bit seems appropriate.

In what way is the above "terrifying"?

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

Because an invalidate is a programmer-initiated instruction to potentially change memory, which is a dirtying action in my book

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

The result being that it's basically impossible to delegate CBO.INVAL permission to entities that sit above the layer doing swap, for example, even if you make sure to clean all pages before handing them out. Otherwise there's this potential sequence:

1. Userspace writes value 1 to page
2. Kernel marks page as dirty
3. Userspace writes 2 to page
4. Kernel swaps page out
5. Userspace reads 2 from page
6. Kernel swaps page in but leaves page on disk
7. Userspace invalidates page,
8. Userspace reads back the original 1 because that never left the LLC (unlikely, but possible)
9. Kernel evicts page again, but doesn't re-write out to disk because still dirty
10. Userspace reads back the 2, despite having seen the 1

Now, yes, that's a bit of a contrived sequence, but the point is to demonstrate the nature of the problem. I think the irony is that, by relaxing the preconditions of the instruction, you've actually constrained its potential use.

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

The result being that it's basically impossible to delegate CBO.INVAL permission to entities that sit above the layer doing swap, for example, even if you make sure to clean all pages before handing them out. Otherwise there's this potential sequence:

1. Userspace writes value 1 to page
2. Kernel marks page as dirty
3. Userspace writes 2 to page
4. Kernel swaps page out
5. Userspace reads 2 from page (kernel swaps page in but leaves page on disk)
6. Userspace invalidates page
7. Userspace reads back the original 1 because that never left the LLC (unlikely, but possible)
8. Kernel evicts page again, but doesn't re-write out to disk because still dirty
9. Userspace reads back the 2, despite having seen the 1 (kernel swaps page in)

Now, yes, that's a bit of a contrived sequence, but the point is to demonstrate the nature of the problem. I think the irony is that, by relaxing the preconditions of the instruction, you've actually constrained its potential use.

What is the purpose of using CBO.INVAL on normal pageable memory?  A determined Userspace always has the option of corrupting its own data by simply storing random garbage, so I don't see that the CBO.INVAL is any more terrifying (arguably less so, because the compiler won't generate CBO.INVAL from C or higher-level languages).

Maybe a more robust answer is that Userspace should only use CBO.INVAL under controlled circumstances (e.g. a non-coherent I/O buffer, where the Kernel, Userspace, and I/O device are all in agreement that the buffer won't be paged out while it is in use).  If the kernel doesn't trust Userspace to not shoot itself in the foot, it always has the option to disable or demote the CBO.INAL operation.

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By dkruckemyer-ventana:

Now, yes, that's a bit of a contrived sequence, but the point is to demonstrate the nature of the problem. I think the irony is that, by relaxing the preconditions of the instruction, you've actually constrained its potential use.

I like contrived sequences. They make architecture fun. :)

What I don't understand is how the old value of 1 remains in a cache after the second store of 2 (step 3) or the page has been swapped out (step 4) and then back in (step 5). Swapping out and in pages would require some sort of hardware coherent DMA or cache management plus non-coherent DMA, but in either case, the value of 2 should be stored to disk and all copies with a value of 1 should be invalidated, updated, or inaccessible by some other means.

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

1 isn't in the cache, it's in DRAM. 2 is in the cache, until you invalidate it, at which point it's 1 again.

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

and then reappears on subsequent swap-in because it's re-written to the cache by the kernel

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

and then reappears on subsequent swap-in (step 9) because it's re-written to the cache by the kernel

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

In the former case, the hart is a consumer (i.e. reader) of data produced by the non-coherent agent

The sequence for this is:

1. invalidate
2. initiate + wait for DMA to finish
3. read data

The act of DMA'ing into the buffer is notionally a write. Yes, this is an external agent, so the MMU doesn't apply here, but whoever gave you access to that DMA-capable device is already granting you the ability to write to the pages, so I don't see why giving write permission from the MMU's perspective is a problem. Not to mention that in a "real" system you'll also have an IOMMU in the way that you'll have to ask the kernel to set up for you with write permission for those pages. But CBO.INVAL allows you to totally bypass that for any page you can read.

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

5. Userspace reads 2 from page (kernel swaps page in but leaves page on disk)

So the value 2 should be in memory (and any cached copies) and on disk, and the page is clean

This is the point where you go wrong: there's no reason it needs to be in both memory and cache. Polled IO will put it into the cache, as will boring DMA that acts as an LLC client (it probably wouldn't on miss, but on hit sure). Yes, in the "I did a DMA from disk to the actual physical page via the interconnect behind the LLC" case the actual point of coherence will have 2, but that isn't the only case that can happen.

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

5. Userspace reads 2 from page (kernel swaps page in but leaves page on disk)

So the value 2 should be in memory (and any cached copies) and on disk, and the page is clean

This is the point where you go wrong: there's no reason it needs to be in both memory and cache. Polled IO will put it into the cache, as will boring DMA that acts as an LLC client (it probably wouldn't on miss, but on hit sure), but that line won't necessarily be written back to replace the 1 in memory. Yes, in the "I did a DMA from disk to the actual physical page via the interconnect behind the LLC" case the actual point of coherence will have 2, but that isn't the only case that can happen.

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

8. Kernel evicts page again, but doesn't re-write out to disk because still dirty

Do you mean still clean (or not dirty)?

(yes, wording got butchered there along the way...)

---

[riscv-CMOs:master] New Comment on Issue #45 Interaction between management instructions and dirty bit
By jrtc27:

5. Userspace reads 2 from page (kernel swaps page in but leaves page on disk)

So the value 2 should be in memory (and any cached copies) and on disk, and the page is clean

This is the point where you go wrong: there's no reason it needs to be in both memory and cache. Polled IO will put it into the cache, as will boring DMA that acts as an LLC client (it probably wouldn't on miss, but on hit sure it'll just update the line), but that line won't necessarily be written back to replace the 1 in memory. Yes, in the "I did a DMA from disk to the actual physical page via the interconnect behind the LLC" case the actual point of coherence will have 2, but that isn't the only case that can happen.