1. Tech-Privileged
  2. Topics

Handling faults on new HLV/HSV instructions in Hypervisor Extension draft 0.6

Anup Patel
 

-----Original Message-----
From: tech-privileged@... <tech-privileged@...> On
Behalf Of John Hauser
Sent: 14 April 2020 08:17
To: tech-privileged@...
Subject: Re: [RISC-V] [tech-privileged] Handling faults on new HLV/HSV
instructions in Hypervisor Extension draft 0.6

Greg Favor wrote:
When one of the new HLV/HSV instructions faults, what virtualization
and privilege modes are recorded in mstatus.mpp/mpv, or in
sstatus.spp/spv and hstatus.spvp? Are they based on the actual modes
from within which the instruction executes (i.e. on HS or U, and V=0),
or on the effective modes used by the instruction as it executes (i.e. on
spvp and V=1).

The actual virtualization and privilege modes, same as always.

Consider the analogy with memory accesses made in M mode when
mstatus.MPRV = 1. The document says that such memory accesses occur "as
though the current privilege mode were set to MPP". If such a memory
access causes a trap, mstatus.MPP gets set to 3, the actual mode at the time
of the trap, not the "as-though" mode. As far as I know, there's never been
a question about this for MPRV.

Likewise, HLV and HSV are defined as performing memory accesses "as
though V = 1". Sounds the same to me.

I also think tables 5.6 and 5.7 in section 5.7.2, "Trap Entry", are reasonably
unambiguous on this point. Since HLV and HSV aren't said to actually change
the current virtualization or privilege modes, I feel it's evident they don't
affect what's written to SPV and SPP on a trap.

If instead the "effective modes applied", as you put it, then note that SRET
would no longer be sufficient to resume from a trap caused by HLV/HSV.
(Nor would MRET, if the trap is taken in M mode.)
This is our understanding as well.

The SRET usage will certainly break for hypervisors if STATUS.SPP and
HSTATUS.SPV don't point to mode when trap happened.


FWIW, there's another clue hidden in this comment in section 5.2.1,
"Hypervisor Status Register (hstatus)":

For memory faults, GVA is redundant with field SPV (the two bits
are set the same) except when the explicit memory access of an HLV,
HLVX, or HSV instruction causes a fault. In that case, SPV = 0 but
GVA = 1.

Note, it says SPV gets set to 0, not 1.

Assuming, for example, the trap is taken into HS-mode:

- If the actual modes apply, then hstatus.spvp remains unchanged and
sstatus.spp/spv are set to reflect the actual privilege mode and V=0.
The hypervisor would then presumably figure out from htinst what
caused this trap?
Yes. Bit GVA in hstatus might also be helpful.

(In certain cases would the hypervisor need to save the original
'spp/spv' values before doing any HLV/HSV instructions so that it can
restore them afterwards?)
It is generally the case, whenever nested traps might be taken in HS mode,
that the hypervisor may need to save sstatus and hstatus before the nested
trap could occur, and restore them afterward. That's no different than when
an operating system might trigger a nested S-mode-handled trap (like a page
fault) by a memory access executed in S mode: the OS may need to save and
restore sstatus around such possibilities. The specific situation determines
exactly what must be saved and restored.
Yes, both Xvisor RISC-V and KVM RISC-V will save SSTATUS and HSTATUS
In low-level trap entry path.

Regards,
Anup
Greg Favor
 

John, thanks for the full responses.  I had suspected the former.  But as can sometimes be the case, we were looking at certain parts of the spec and weren't looking at the tables in section 5.7.2.

Given that the general style of the arch spec is to not do heavy cross-referencing, I won't suggest that.  And ultimately it was our own fault in not broadly searching for and noticing those tables while in the heat of the moment of answering a question about "hstatus" section 5.2.1 that was raised by one of our designers.

Thanks,
Greg


On Mon, Apr 13, 2020 at 7:48 PM John Hauser <jh.riscv@...> wrote:
Greg Favor wrote:
> When one of the new HLV/HSV instructions faults, what virtualization and
> privilege modes are recorded in mstatus.mpp/mpv, or in sstatus.spp/spv and
> hstatus.spvp?  Are they based on the actual modes from within which the
> instruction executes (i.e. on HS or U, and V=0), or on the effective modes
> used by the instruction as it executes (i.e. on spvp and V=1).

The actual virtualization and privilege modes, same as always.

Consider the analogy with memory accesses made in M mode when
mstatus.MPRV = 1.  The document says that such memory accesses occur
"as though the current privilege mode were set to MPP".  If such a
memory access causes a trap, mstatus.MPP gets set to 3, the actual mode
at the time of the trap, not the "as-though" mode.  As far as I know,
there's never been a question about this for MPRV.

Likewise, HLV and HSV are defined as performing memory accesses "as
though V = 1".  Sounds the same to me.

I also think tables 5.6 and 5.7 in section 5.7.2, "Trap Entry", are
reasonably unambiguous on this point.  Since HLV and HSV aren't said to
actually change the current virtualization or privilege modes, I feel
it's evident they don't affect what's written to SPV and SPP on a trap.

If instead the "effective modes applied", as you put it, then note
that SRET would no longer be sufficient to resume from a trap caused by
HLV/HSV.  (Nor would MRET, if the trap is taken in M mode.)

FWIW, there's another clue hidden in this comment in section 5.2.1,
"Hypervisor Status Register (hstatus)":

    For memory faults, GVA is redundant with field SPV (the two bits
    are set the same) except when the explicit memory access of an HLV,
    HLVX, or HSV instruction causes a fault.  In that case, SPV = 0 but
    GVA = 1.

Note, it says SPV gets set to 0, not 1.

> Assuming, for example, the trap is taken into HS-mode:
>
> - If the actual modes apply, then hstatus.spvp remains unchanged and
> sstatus.spp/spv are set to reflect the actual privilege mode and V=0.  The
> hypervisor would then presumably figure out from htinst what caused this
> trap?

Yes.  Bit GVA in hstatus might also be helpful.

>  (In certain cases would the hypervisor need to save the original
> 'spp/spv' values before doing any HLV/HSV instructions so that it can
> restore them afterwards?)

It is generally the case, whenever nested traps might be taken in
HS mode, that the hypervisor may need to save sstatus and hstatus
before the nested trap could occur, and restore them afterward.  That's
no different than when an operating system might trigger a nested
S-mode-handled trap (like a page fault) by a memory access executed
in S mode:  the OS may need to save and restore sstatus around such
possibilities.  The specific situation determines exactly what must be
saved and restored.

    - John Hauser
Jonathan Behrens <behrensj@...>
 

Having SPP/SPV hold the real values makes the most sense to me. The strategy I'd expect hypervisors to use would be to set a bit before issuing any HLV or HSV instructions and clear it after. Then in their page fault handler they'd check if it is set in order to "blame" that fault on the guest and take appropriate action instead of resuming normal execution.

Jonathan


On Mon, Apr 13, 2020 at 10:48 PM John Hauser via lists.riscv.org <jh.riscv=jhauser.us@...> wrote:
Greg Favor wrote:
> When one of the new HLV/HSV instructions faults, what virtualization and
> privilege modes are recorded in mstatus.mpp/mpv, or in sstatus.spp/spv and
> hstatus.spvp?  Are they based on the actual modes from within which the
> instruction executes (i.e. on HS or U, and V=0), or on the effective modes
> used by the instruction as it executes (i.e. on spvp and V=1).

The actual virtualization and privilege modes, same as always.

Consider the analogy with memory accesses made in M mode when
mstatus.MPRV = 1.  The document says that such memory accesses occur
"as though the current privilege mode were set to MPP".  If such a
memory access causes a trap, mstatus.MPP gets set to 3, the actual mode
at the time of the trap, not the "as-though" mode.  As far as I know,
there's never been a question about this for MPRV.

Likewise, HLV and HSV are defined as performing memory accesses "as
though V = 1".  Sounds the same to me.

I also think tables 5.6 and 5.7 in section 5.7.2, "Trap Entry", are
reasonably unambiguous on this point.  Since HLV and HSV aren't said to
actually change the current virtualization or privilege modes, I feel
it's evident they don't affect what's written to SPV and SPP on a trap.

If instead the "effective modes applied", as you put it, then note
that SRET would no longer be sufficient to resume from a trap caused by
HLV/HSV.  (Nor would MRET, if the trap is taken in M mode.)

FWIW, there's another clue hidden in this comment in section 5.2.1,
"Hypervisor Status Register (hstatus)":

    For memory faults, GVA is redundant with field SPV (the two bits
    are set the same) except when the explicit memory access of an HLV,
    HLVX, or HSV instruction causes a fault.  In that case, SPV = 0 but
    GVA = 1.

Note, it says SPV gets set to 0, not 1.

> Assuming, for example, the trap is taken into HS-mode:
>
> - If the actual modes apply, then hstatus.spvp remains unchanged and
> sstatus.spp/spv are set to reflect the actual privilege mode and V=0.  The
> hypervisor would then presumably figure out from htinst what caused this
> trap?

Yes.  Bit GVA in hstatus might also be helpful.

>  (In certain cases would the hypervisor need to save the original
> 'spp/spv' values before doing any HLV/HSV instructions so that it can
> restore them afterwards?)

It is generally the case, whenever nested traps might be taken in
HS mode, that the hypervisor may need to save sstatus and hstatus
before the nested trap could occur, and restore them afterward.  That's
no different than when an operating system might trigger a nested
S-mode-handled trap (like a page fault) by a memory access executed
in S mode:  the OS may need to save and restore sstatus around such
possibilities.  The specific situation determines exactly what must be
saved and restored.

    - John Hauser
John Hauser
 

Greg Favor wrote:
When one of the new HLV/HSV instructions faults, what virtualization and
privilege modes are recorded in mstatus.mpp/mpv, or in sstatus.spp/spv and
hstatus.spvp? Are they based on the actual modes from within which the
instruction executes (i.e. on HS or U, and V=0), or on the effective modes
used by the instruction as it executes (i.e. on spvp and V=1).
The actual virtualization and privilege modes, same as always.

Consider the analogy with memory accesses made in M mode when
mstatus.MPRV = 1. The document says that such memory accesses occur
"as though the current privilege mode were set to MPP". If such a
memory access causes a trap, mstatus.MPP gets set to 3, the actual mode
at the time of the trap, not the "as-though" mode. As far as I know,
there's never been a question about this for MPRV.

Likewise, HLV and HSV are defined as performing memory accesses "as
though V = 1". Sounds the same to me.

I also think tables 5.6 and 5.7 in section 5.7.2, "Trap Entry", are
reasonably unambiguous on this point. Since HLV and HSV aren't said to
actually change the current virtualization or privilege modes, I feel
it's evident they don't affect what's written to SPV and SPP on a trap.

If instead the "effective modes applied", as you put it, then note
that SRET would no longer be sufficient to resume from a trap caused by
HLV/HSV. (Nor would MRET, if the trap is taken in M mode.)

FWIW, there's another clue hidden in this comment in section 5.2.1,
"Hypervisor Status Register (hstatus)":

For memory faults, GVA is redundant with field SPV (the two bits
are set the same) except when the explicit memory access of an HLV,
HLVX, or HSV instruction causes a fault. In that case, SPV = 0 but
GVA = 1.

Note, it says SPV gets set to 0, not 1.

Assuming, for example, the trap is taken into HS-mode:

- If the actual modes apply, then hstatus.spvp remains unchanged and
sstatus.spp/spv are set to reflect the actual privilege mode and V=0. The
hypervisor would then presumably figure out from htinst what caused this
trap?
Yes. Bit GVA in hstatus might also be helpful.

(In certain cases would the hypervisor need to save the original
'spp/spv' values before doing any HLV/HSV instructions so that it can
restore them afterwards?)
It is generally the case, whenever nested traps might be taken in
HS mode, that the hypervisor may need to save sstatus and hstatus
before the nested trap could occur, and restore them afterward. That's
no different than when an operating system might trigger a nested
S-mode-handled trap (like a page fault) by a memory access executed
in S mode: the OS may need to save and restore sstatus around such
possibilities. The specific situation determines exactly what must be
saved and restored.

- John Hauser
Greg Favor
 

When one of the new HLV/HSV instructions faults, what virtualization and privilege modes are recorded in mstatus.mpp/mpv, or in sstatus.spp/spv and hstatus.spvp?  Are they based on the actual modes from within which the instruction executes (i.e. on HS or U, and V=0), or on the effective modes used by the instruction as it executes (i.e. on spvp and V=1).

Assuming, for example, the trap is taken into HS-mode:

- If the actual modes apply, then hstatus.spvp remains unchanged and sstatus.spp/spv are set to reflect the actual privilege mode and V=0.  The hypervisor would then presumably figure out from htinst what caused this trap?   (In certain cases would the hypervisor need to save the original 'spp/spv' values before doing any HLV/HSV instructions so that it can restore them afterwards?)

- If the effective modes apply, then sstatus.spp and hstatus.spvp are set to the effective privilege mode of the HLV/HSV instruction (as specified by spvp) and sstatus.spv is set to reflect V=1.  The hypervisor would then figure out in some way (such as from htinst?) that this was a re-entry into the hypervisor due to its own actions?  (Typically all three of these fields would end up not changing in their values.  But in certain cases would the hypervisor need to save the original 'spp/spv' values before doing any HLV/HSV instructions so that it can restore them afterwards?)

In any case, which is the intended behavior (which should probably then be clarified in the spec)?

Thanks,
Greg

1 - 5 of 5