proposal to add "virtual instruction exception" to the hypervisor extension


Paolo Bonzini
 

On 06/05/20 18:56, John Hauser wrote:
Paolo Bonzini wrote:
On 05/05/20 23:15, John Hauser wrote:
- attempts to execute WFI, unless the instruction completes within an
implementation-specific, bounded time;
It would be great to have this controlled by a bit in the hstatus CSR.
For example a hypervisor that does not overcommit CPUs will probably
want to delegate WFI to the guest. Guest-delegated interrupts then
would not incur the overhead of world switching, while host-handled
interrupts would exit to the hypervisor anyway.
I'm afraid this makes no sense to me. Ordinary user-mode applications
don't execute WFI.
But this would be a VS-mode application, not an ordinary user-mode
application. I'm not sure why U-mode matters. Anyway...

And if a guest OS itself executes WFI, I find it hard to believe it
expects the WFI to trap as an illegal instruction, much less that it
has a performance-sensitive response coded in its illegal instruction
trap handler.

So I don't understand "delegate WFI to the guest". What's the scenario
where a guest OS expects WFI to trap?
... I guess this is the misunderstanding. I'm not proposing to delegate
the WFI trap to the guest, but rather *the WFI instruction*: HS-mode
could optionally let WFI run in VS-mode, even if it wouldn't complete
within a bounded time. This is because, if you don't overcommit CPUs,
there's no advantage in getting out of VS-mode and doing the wait for
interrupts in HS-mode.

When an interrupt arrives, it would either be delivered to VS-mode or
cause an HS-mode trap, depending on the contents of hideleg.

Nested virtualization does not matter.

Is this clearer?

Paolo


John Hauser
 

Paolo Bonzini wrote:
On 05/05/20 23:15, John Hauser wrote:
- attempts to execute WFI, unless the instruction completes within an
implementation-specific, bounded time;
It would be great to have this controlled by a bit in the hstatus CSR.
For example a hypervisor that does not overcommit CPUs will probably
want to delegate WFI to the guest. Guest-delegated interrupts then
would not incur the overhead of world switching, while host-handled
interrupts would exit to the hypervisor anyway.
I'm afraid this makes no sense to me. Ordinary user-mode applications
don't execute WFI. In U mode, WFI is useful only in connection
with support for user-level interrupts provided by the may-never-be-
standardized N extension.

And if a guest OS itself executes WFI, I find it hard to believe it
expects the WFI to trap as an illegal instruction, much less that it
has a performance-sensitive response coded in its illegal instruction
trap handler.

So I don't understand "delegate WFI to the guest". What's the scenario
where a guest OS expects WFI to trap?

The only thing I can think of is when the guest is a nested
hypervisor. But I'm not sure that specifically focusing on optimizing
WFI for a nested hypervisor is really on the agenda. Why not make many
other improvements in support of nested hypervisors, besides this one
thing? We intend eventually to propose a whole package of optional
added support for nested hypervisors. But outside of that add-on
extension, nested hypervisors aren't catered to beyond a bare-bones
level of support.

- John Hauser


Anup Patel
 

-----Original Message-----
From: tech-privileged@... <tech-privileged@...> On
Behalf Of Paolo Bonzini
Sent: 06 May 2020 14:34
To: John Hauser <jh.riscv@...>; tech-privileged@...
Subject: Re: [RISC-V] [tech-privileged] proposal to add "virtual instruction
exception" to the hypervisor extension

On 05/05/20 23:15, John Hauser wrote:
- attempts to execute WFI, unless the instruction completes within an
implementation-specific, bounded time;
It would be great to have this controlled by a bit in the hstatus CSR.
For example a hypervisor that does not overcommit CPUs will probably want
to delegate WFI to the guest. Guest-delegated interrupts then would not
incur the overhead of world switching, while host-handled interrupts would
exit to the hypervisor anyway.

(It may even make sense to add this to mstatus, but I wouldn't care much
about it right now except perhaps when choosing which bit to use in
hstatus).
Good suggestion.

A HSTATUS.VTW bit for trapping WFI when executed with V=1 would
be good (just like HSTATUS.VTSR and HSTATUS.VTVM bits) ?

Regards,
Anup


Paolo Bonzini
 

On 05/05/20 23:15, John Hauser wrote:
- attempts to execute WFI, unless the instruction completes within an
implementation-specific, bounded time;
It would be great to have this controlled by a bit in the hstatus CSR.
For example a hypervisor that does not overcommit CPUs will probably
want to delegate WFI to the guest. Guest-delegated interrupts then
would not incur the overhead of world switching, while host-handled
interrupts would exit to the hypervisor anyway.

(It may even make sense to add this to mstatus, but I wouldn't care much
about it right now except perhaps when choosing which bit to use in
hstatus).

Paolo


Jonathan Behrens <behrensj@...>
 

It mostly just comes down to crossing out "In VS-mode" in your list of cases for trapping when V=1:

  - attempts to execute an HFENCE instruction or to access
    an implemented hypervisor CSR or VS CSR;

  - attempts to execute SRET when hstatus.VTSR = 1 or in VU-mode;

  - attempts to execute an SFENCE instruction or to access
    satp, when hstatus.VTVM = 1 or in VU-mode

A few more cases I'm less sure if they make sense:

  - attempts to access an unimplemented hypervisor CSR

  - attempts to access a supervisor CSR in VU-mode

  - attempts to execute MRET or access an M-mode CSR

Jonathan


On Tue, May 5, 2020 at 8:31 PM John Hauser via lists.riscv.org <jh.riscv=jhauser.us@...> wrote:
Jonathan Behrens wrote:
> I like this plan. The one comment I have is that it seems unnecessarily
> opinionated about which operations trigger virtual instruction traps vs
> illegal instruction traps when run in VU mode. I think we should error more
> on having things trigger virtual instruction traps everywhere that it is
> unlikely to require M-mode emulation. To give one example of where the
> current design might go wrong: analogously to HLV and friends, HFENCE from
> U-mode might be allowed via a CSR in the future, in which case it would now
> require hypervisor emulation when a nested VM tried to run it in VU-mode.

Point taken.  Would you like to take a stab at listing every case you
think should be included, besides HFENCE?

    - John Hauser




John Hauser
 

Jonathan Behrens wrote:
I like this plan. The one comment I have is that it seems unnecessarily
opinionated about which operations trigger virtual instruction traps vs
illegal instruction traps when run in VU mode. I think we should error more
on having things trigger virtual instruction traps everywhere that it is
unlikely to require M-mode emulation. To give one example of where the
current design might go wrong: analogously to HLV and friends, HFENCE from
U-mode might be allowed via a CSR in the future, in which case it would now
require hypervisor emulation when a nested VM tried to run it in VU-mode.
Point taken. Would you like to take a stab at listing every case you
think should be included, besides HFENCE?

- John Hauser


Jonathan Behrens <behrensj@...>
 

I like this plan. The one comment I have is that it seems unnecessarily opinionated about which operations trigger virtual instruction traps vs illegal instruction traps when run in VU mode. I think we should error more on having things trigger virtual instruction traps everywhere that it is unlikely to require M-mode emulation. To give one example of where the current design might go wrong: analogously to HLV and friends, HFENCE from U-mode might be allowed via a CSR in the future, in which case it would now require hypervisor emulation when a nested VM tried to run it in VU-mode.

Jonathan


On Tue, May 5, 2020 at 5:16 PM John Hauser via lists.riscv.org <jh.riscv=jhauser.us@...> wrote:
Hello tech-privileged guys,

I've created a pull request for the RISC-V privileged spec in response
to requests from our hypervisor software authors:
https://github.com/riscv/riscv-isa-manual/pull/518

For those with an interest, please review.

This change would add to the hypervisor extension a new kind of
exception, "virtual instruction exception".  The following is copied
from new text added to the hypervisor chapter:

--------------------

When V = 1, a virtual instruction trap (not an illegal instruction
trap) is taken for:

  - attempts to access a counter CSR when the corresponding bit in
    hcounteren is 0 and the same bit in mcounteren is 1;

  - attempts to execute WFI, unless the instruction completes within an
    implementation-specific, bounded time;

  - attempts to execute a virtual-machine load/store instruction, HLV,
    HLVX, or HSV;

  - in VS-mode, attempts to execute an HFENCE instruction or to access
    an implemented hypervisor CSR or VS CSR;

  - in VS-mode, attempts to execute SRET when hstatus.VTSR = 1; or

  - in VS-mode, attempts to execute an SFENCE instruction or to access
    satp, when hstatus.VTVM = 1.

On a virtual instruction trap, mtval or stval is written the same as
for an illegal instruction trap.

\begin{commentary}
When V = 1, circumstances that might otherwise cause an illegal
instruction trap instead cause a virtual instruction trap if a
hypervisor is normally expected to emulate the instruction.  Notably,
for VS-mode this includes the hypervisor instructions (HLV, HLVX, HSV,
and HFENCE) and accesses to the hypervisor-level CSRs, all of which
must be emulated for nested hypervisors.  A hypervisor that does not
support nested hypervisors should convert many virtual instruction
traps into illegal instruction exceptions for the guest virtual
machine.

Machine level is expected ordinarily to delegate virtual instruction
traps directly to HS-level, whereas illegal instruction traps are
likely to be processed first in M-mode before being conditionally
delegated (by software) to HS-level.  Consequently, virtual instruction
traps are expected typically to be handled faster than illegal
instruction traps.
\end{commentary}

--------------------

Regards,

    - John Hauser




John Hauser
 

Hello tech-privileged guys,

I've created a pull request for the RISC-V privileged spec in response
to requests from our hypervisor software authors:
https://github.com/riscv/riscv-isa-manual/pull/518

For those with an interest, please review.

This change would add to the hypervisor extension a new kind of
exception, "virtual instruction exception". The following is copied
from new text added to the hypervisor chapter:

--------------------

When V = 1, a virtual instruction trap (not an illegal instruction
trap) is taken for:

- attempts to access a counter CSR when the corresponding bit in
hcounteren is 0 and the same bit in mcounteren is 1;

- attempts to execute WFI, unless the instruction completes within an
implementation-specific, bounded time;

- attempts to execute a virtual-machine load/store instruction, HLV,
HLVX, or HSV;

- in VS-mode, attempts to execute an HFENCE instruction or to access
an implemented hypervisor CSR or VS CSR;

- in VS-mode, attempts to execute SRET when hstatus.VTSR = 1; or

- in VS-mode, attempts to execute an SFENCE instruction or to access
satp, when hstatus.VTVM = 1.

On a virtual instruction trap, mtval or stval is written the same as
for an illegal instruction trap.

\begin{commentary}
When V = 1, circumstances that might otherwise cause an illegal
instruction trap instead cause a virtual instruction trap if a
hypervisor is normally expected to emulate the instruction. Notably,
for VS-mode this includes the hypervisor instructions (HLV, HLVX, HSV,
and HFENCE) and accesses to the hypervisor-level CSRs, all of which
must be emulated for nested hypervisors. A hypervisor that does not
support nested hypervisors should convert many virtual instruction
traps into illegal instruction exceptions for the guest virtual
machine.

Machine level is expected ordinarily to delegate virtual instruction
traps directly to HS-level, whereas illegal instruction traps are
likely to be processed first in M-mode before being conditionally
delegated (by software) to HS-level. Consequently, virtual instruction
traps are expected typically to be handled faster than illegal
instruction traps.
\end{commentary}

--------------------

Regards,

- John Hauser