Another avenue to the same goal of eliminating the need for the "virtual I/O" bit in G-stage PTEs,

rather than changing the guest device driver to add otherwise unnecessary FENCEs to Strongly Ordered IO regions or upgrade FENCE [IO],[IO] to RWIO,RWIO,

require the Hypervisor to not break Guest execution (by telling the Guest one thing and doing another)!

Example text:

"When the Hypervisor informs the Guest that a memory region is Main Memory, IO, or Strongly Ordered IO, then it must not remap those addresses to a memory region of a different type (Main Memory, IO, or Strongly Ordered IO), otherwise memory ordering consistency could be lost by the Guest:

• If the Hypervisor informs the Guest that a memory region is IO but remaps it to a Main Memory region, then a FENCE [IO],[IO] executed by the guest might not order the desired accesses.
• If the Hypervisor informs the Guest that a memory region is Main Memory but remaps it to an IO region, then a FENCE [RW],[RW] executed by the guest might not order the desired accesses.
• If the Hypervisor informs the Guest that a memory region is Strongly Ordered but remaps it to a Weakly Ordered region, then only weak memory ordering might be provided."

-- John

On Sun, May 30, 2021 at 9:06 AM John Hauser <jh.riscv@...> wrote:

Greg Favor wrote:
>   It seems like this G-stage "I/O" bit is going down a questionable rabbit
> hole that:
> [...]

I may have found the formula to defuse the issue.  I think we can
eliminate the need for a "virtual I/O" bit in G-stage page tables by
simply requiring device drivers to act more conservatively.  To that
end, I propose inserting some version of the following two paragraphs
into the Unprivileged ISA's section 1.4, "Memory".

First:

    A naturally aligned 256-byte subregion of the address space is
    called a _paragraph_.  The minimum unit of contiguous main memory
    is a complete paragraph.  That is, if any byte within a paragraph
    is main memory, then every byte of the paragraph must be main
    memory; and conversely, if any byte of a paragraph is vacant or
    I/O, then every non-vacant byte of the paragraph must be considered
    I/O.

Second:

    If an I/O device has memory that is accessible in the address
    space, and if any paragraph of that memory has the properties
    that permit the system to label it as main memory, an execution
    environment may choose each such paragraph to be either main
    memory or I/O.  When the same type of I/O device exists in multiple
    systems, portable RISC-V software must assume that device memory
    that is considered main memory in one execution environment may be
    considered I/O in another execution environment, and vice versa.

It may also be appropriate to add a comment to the FENCE section
reminding that software may not know whether some device memory is
considered main memory or I/O, in which case it will need to fence
conservatively for either possibility.

I welcome comments.

    - John Hauser