Date   

Re: Hypervisor interrupt enables

Paolo Bonzini
 

There are both "HU" and VU-modes indeed.

There were very old discussions on the public isa-dev mailing list about the advantages of this method (shared with s390 and x86 among others) to the "pure H-mode" used by ARM (especially before the introduction of VHE) and POWER. H-mode was reserved by early version of the RISC-V privileged specification, but those discuss ultimately led to the removal of all mentiond of H-mode and the introduction of HS-mode in its stead.


Thanks,

Paolo


Il mar 9 mar 2021, 23:33 Allen Baum <allen.baum@...> ha scritto:
This is a stupid question: if VS mode is enabled - can a core ever be in Umode, or must it be in VU mode?
IF the answer is: when virtualization is on, the core cannot be in S mode or U mode, but only in VS and modes (and M...) then the question is moot.


Re: Hypervisor interrupt enables

Scott Johnson
 

When hypervisor is enabled, S-mode becomes HS-mode, and VS-mode and VU-mode are newly available.

U-mode is still available.


On Tue, Mar 9, 2021 at 02:29 PM, Allen Baum wrote:
This is a stupid question: if VS mode is enabled - can a core ever be in Umode, or must it be in VU mode?
IF the answer is: when virtualization is on, the core cannot be in S mode or U mode, but only in VS and modes (and M...) then the question is moot.

On Tue, Mar 9, 2021 at 7:43 AM Scott Johnson <scott.johnson@...> wrote:
The question is about VS-mode interrupts, relevant even without the N extension.

Are interrupts destined for VS-mode globally enabled when in U-mode?

If U-mode is less privileged than VS-mode, then yes.
If U-mode is the same privilege as VS-mode, then vsstatus.SIE applies.
If U-mode is more privileged than VS-mode, then they are disabled. (Priv spec 3.1.6.1 says "Interrupts for lower-privilege modes, w<x, are always globally disabled regardless of the setting of any global wIE bit for the lower-privilege mode.)

So the relative privilege of U vs VS is important, and apparently unspecified.

The graph of privileges is how I had pictured it in my head before this issue came up. But it doesn't address this question.

 

 


Re: Hypervisor interrupt enables

Allen Baum
 

This is a stupid question: if VS mode is enabled - can a core ever be in Umode, or must it be in VU mode?
IF the answer is: when virtualization is on, the core cannot be in S mode or U mode, but only in VS and modes (and M...) then the question is moot.

On Tue, Mar 9, 2021 at 7:43 AM Scott Johnson <scott.johnson@...> wrote:
The question is about VS-mode interrupts, relevant even without the N extension.

Are interrupts destined for VS-mode globally enabled when in U-mode?

If U-mode is less privileged than VS-mode, then yes.
If U-mode is the same privilege as VS-mode, then vsstatus.SIE applies.
If U-mode is more privileged than VS-mode, then they are disabled. (Priv spec 3.1.6.1 says "Interrupts for lower-privilege modes, w<x, are always globally disabled regardless of the setting of any global wIE bit for the lower-privilege mode.)

So the relative privilege of U vs VS is important, and apparently unspecified.

The graph of privileges is how I had pictured it in my head before this issue came up. But it doesn't address this question.


Re: Hypervisor interrupt enables

Scott Johnson
 

The question is about VS-mode interrupts, relevant even without the N extension.

Are interrupts destined for VS-mode globally enabled when in U-mode?

If U-mode is less privileged than VS-mode, then yes.
If U-mode is the same privilege as VS-mode, then vsstatus.SIE applies.
If U-mode is more privileged than VS-mode, then they are disabled. (Priv spec 3.1.6.1 says "Interrupts for lower-privilege modes, w<x, are always globally disabled regardless of the setting of any global wIE bit for the lower-privilege mode.)

So the relative privilege of U vs VS is important, and apparently unspecified.

The graph of privileges is how I had pictured it in my head before this issue came up. But it doesn't address this question.


Re: Correspondence between hedeleg and medeleg writeable bits?

Scott Johnson
 

Funny, I was just asking the same question. https://github.com/riscv/riscv-isa-sim/issues/668


Correspondence between hedeleg and medeleg writeable bits?

James Robinson
 

Riscv-privileged_1.12-draft, Table 6.2 gives a list of hedeleg bits which must be writable, and it is stated:

"Requiring that certain bits of hedeleg be writable reduces some of the burden on a hypervisor to handle variations of implementation."

I see that the list of hedeleg writable bits includes bits which are not writable in medeleg in the spike implementation (Illegal Instruction, Load/Store address misaligned, Load/Store/Inst Access Fault).

Is it intended that these bits should be writable even if they are not writable in medeleg, and that the writability the corresponding bits in medeleg remains an implementation choice?

In which case, some exceptions, although writable in hedeleg, will never actually be delegated to VS-mode by the hardware?

Is machine mode software expected (under certain circumstances) to emulate delegation of the exception by setting up the correct state and executing mret?

Thanks,
James


Re: Hypervisor interrupt enables

John Hauser
 

Scott Johnson wrote:
What about U-mode? Are VS-mode interrupts enabled when in U-mode?

I would guess not...so the hierarchy is M > S > U > VS > VU.
The intended hierarchy is what mathematicians call a partial order:

M > HS > U and HS > VS > VU

In other words, HS has two children in the graph.

Interrupts are globally enabled for VS level if the current operating
mode has less privilege than VS or if the current operating mode is
VS and vsstatus.SIE = 1. Because U and VS modes are unordered in the
privilege hierarchy, U does not have less privilege than VS, or vice
versa.

This would become painfully relevant if the N extension for user-level
interrupts were ever made fully compatible with the hypervisor
extension. The original intent was that U-level interrupts not be able
to interrupt VS mode, and VS-level interrupts not be able to interrupt
U mode.

However, as it currently stands, the incomplete N extension has not
been updated to be fully compatible with the hypervisor extension, and
what's more, there's a movement to discourage the N extension, or at
the very least, to discourage it when supervisor mode is implemented.
If the N extension and S mode were forever mutually exclusive, we could
more easily adopt the full order you propose, with U > VS.

- John Hauser


Re: Hypervisor interrupt enables

Jonathan Behrens <behrensj@...>
 

Oh, good point. I'd forgotten about the hstatus.HU bit which gives U-mode the ability to do loads and stores in the VS-mode address space. U-mode still doesn't have access to the VS-mode CSRs (or virtual memory fence instructions AFAIK), but this does complicate the story.

On Mon, Mar 8, 2021 at 12:01 PM Scott Johnson via lists.riscv.org <scott.johnson=arilinc.com@...> wrote:
There is talk in the spec of running portions of the hypervisor code in U-mode.

On Mon, Mar 8, 2021 at 08:50 AM, Jonathan Behrens wrote:
I don't think calling U-mode more privileged than VS-mode is quite right either. U-mode software cannot access VS-mode CSRs for instance.
 
A different way of looking at it would be that VS-mode only exists when virtualization is enabled, so if you are running in U-mode (meaning virtualization is disabled) then VS-mode software cannot receive interrupts because there is no trap handler or other software configured for that mode.
 
Jonathan

On Mon, Mar 8, 2021 at 11:32 AM Scott Johnson via lists.riscv.org <scott.johnson=arilinc.com@...> wrote:
Ugh, correction: M > HS > U > VS > VU.

On Mon, Mar 8, 2021 at 08:27 AM, Scott Johnson wrote:

I would guess not...so the hierarchy is M > S > U > VS > VU.

 

 


Re: Hypervisor interrupt enables

Scott Johnson
 

There is talk in the spec of running portions of the hypervisor code in U-mode.


On Mon, Mar 8, 2021 at 08:50 AM, Jonathan Behrens wrote:
I don't think calling U-mode more privileged than VS-mode is quite right either. U-mode software cannot access VS-mode CSRs for instance.
 
A different way of looking at it would be that VS-mode only exists when virtualization is enabled, so if you are running in U-mode (meaning virtualization is disabled) then VS-mode software cannot receive interrupts because there is no trap handler or other software configured for that mode.
 
Jonathan

On Mon, Mar 8, 2021 at 11:32 AM Scott Johnson via lists.riscv.org <scott.johnson=arilinc.com@...> wrote:
Ugh, correction: M > HS > U > VS > VU.

On Mon, Mar 8, 2021 at 08:27 AM, Scott Johnson wrote:

I would guess not...so the hierarchy is M > S > U > VS > VU.

 

 


Re: Hypervisor interrupt enables

Jonathan Behrens <behrensj@...>
 

I don't think calling U-mode more privileged than VS-mode is quite right either. U-mode software cannot access VS-mode CSRs for instance.

A different way of looking at it would be that VS-mode only exists when virtualization is enabled, so if you are running in U-mode (meaning virtualization is disabled) then VS-mode software cannot receive interrupts because there is no trap handler or other software configured for that mode.

Jonathan


On Mon, Mar 8, 2021 at 11:32 AM Scott Johnson via lists.riscv.org <scott.johnson=arilinc.com@...> wrote:
Ugh, correction: M > HS > U > VS > VU.

On Mon, Mar 8, 2021 at 08:27 AM, Scott Johnson wrote:

I would guess not...so the hierarchy is M > S > U > VS > VU.


Re: Hypervisor interrupt enables

Scott Johnson
 

Ugh, correction: M > HS > U > VS > VU.


On Mon, Mar 8, 2021 at 08:27 AM, Scott Johnson wrote:

I would guess not...so the hierarchy is M > S > U > VS > VU.


Re: Hypervisor interrupt enables

Scott Johnson
 

I think it makes more sense this way, though I concur that a spec clarification is needed.

I'm not sure I understand Andrew's explanation though. How could VS-mode code block HS-mode interrupts, since VS-mode has no control over sstatus.SIE?

What about U-mode? Are VS-mode interrupts enabled when in U-mode?

I would guess not...so the hierarchy is M > S > U > VS > VU.

When explained this way, my confusion about sstatus vs vsstatus disappears, since when V=1, HS-mode interrupts are globally enabled regardless of any *status.SIE bit.


On Sun, Mar 7, 2021 at 12:14 PM, John Hauser wrote:
I wrote:
Interrupts are always
enabled/disabled globally for HS level by the HS-level sstatus.SIE.
Correction: Interrupts are globally enabled for HS level if the
current privilege mode has less privilege than HS (U, VS, or VU mode)
or if the current privilege mode is HS and the HS-level sstatus.SIE
= 1.

- John Hauser

 
d-        state.mip = set_field(state.mip, MIP_MSIP, bytes[0]);


Re: Hypervisor interrupt enables

Sandro Pinto
 


Scott Johnson wrote:
> I expect, and have observed in Spike, that `sstatus.SIE` will apply
> to interrupts delegated to HS-mode, and `vsstatus.SIE` will apply to
> interrupts delegated to VS-mode.
>
> But I cannot find this explained anywhere in the Hypervisor 0.6.1
> spec. Am I missing it somewhere? There is this statement
> ( https://github.com/riscv/riscv-isa-manual/blob/5ca937811c13ae142f783e69f61f6a402694ebe0/src/hypervisor.tex#L1154-L1156
> ) about `vsstatus` substituting for `sstatus` when V=1, but that
> seems incomplete because `sstatus.SIE` apparently still applies to
> some interrupts even when in VS-mode.

Your expectation about the roles of sstatus.SIE and vsstatus.SIE
is correct, and you're probably also right to say that this isn't
explained clearly in the current Privileged Architecture document.

Definitely the Privileged Architecture document needs additional
clarification with regard to this topic.

A few months ago we had a similar problem with QEMU.
We needed to cross-check with Andrew and provide a fix to QEMU:

Sandro


Re: Hypervisor interrupt enables

John Hauser
 

I wrote:
Interrupts are always
enabled/disabled globally for HS level by the HS-level sstatus.SIE.
Correction: Interrupts are globally enabled for HS level if the
current privilege mode has less privilege than HS (U, VS, or VU mode)
or if the current privilege mode is HS and the HS-level sstatus.SIE
= 1.

- John Hauser


Re: Hypervisor interrupt enables

John Hauser
 

Scott Johnson wrote:
I expect, and have observed in Spike, that `sstatus.SIE` will apply
to interrupts delegated to HS-mode, and `vsstatus.SIE` will apply to
interrupts delegated to VS-mode.

But I cannot find this explained anywhere in the Hypervisor 0.6.1
spec. Am I missing it somewhere? There is this statement
( https://github.com/riscv/riscv-isa-manual/blob/5ca937811c13ae142f783e69f61f6a402694ebe0/src/hypervisor.tex#L1154-L1156
) about `vsstatus` substituting for `sstatus` when V=1, but that
seems incomplete because `sstatus.SIE` apparently still applies to
some interrupts even when in VS-mode.
Your expectation about the roles of sstatus.SIE and vsstatus.SIE
is correct, and you're probably also right to say that this isn't
explained clearly in the current Privileged Architecture document.

Related question: it seems that (in Spike, at least) interrupts
delegated to HS-mode are blocked by `sstatus.SIE` even from VS-mode.
I suppose that is spec-compliant based on this statement
( https://github.com/riscv/riscv-isa-manual/blob/5ca937811c13ae142f783e69f61f6a402694ebe0/src/supervisor.tex#L372-L374
) since the current privilege mode when in VS-mode is not less than S
(right?).
I'm afraid I don't understand what you're asking here, since to my eyes
it seems to be a repeat of your first point. Interrupts are always
enabled/disabled globally for HS level by the HS-level sstatus.SIE.

If VS-mode was considered less privileged than HS-mode then my first
question above would be moot.
Whether or not the manual explicitly says so, it definitely intends
that VS mode is less privileged than HS mode.

- John Hauser


Hypervisor interrupt enables

Scott Johnson
 

I expect, and have observed in Spike, that `sstatus.SIE` will apply to interrupts delegated to HS-mode, and `vsstatus.SIE` will apply to interrupts delegated to VS-mode.
 
But I cannot find this explained anywhere in the Hypervisor 0.6.1 spec. Am I missing it somewhere? There is this statement about `vsstatus` substituting for `sstatus` when V=1, but that seems incomplete because `sstatus.SIE` apparently still applies to some interrupts even when in VS-mode.
 
Related question: it seems that (in Spike, at least) interrupts delegated to HS-mode are blocked by `sstatus.SIE` even from VS-mode. I suppose that is spec-compliant based on this statement since the current privilege mode when in VS-mode is not less than S (right?).
 
But is that really the intended/desired behavior?
 
If VS-mode was considered less privileged than HS-mode then my first question above would be moot.
 
Scott
 


Clarification on writing MXL field of the MISA CSR

Joseph Rahmeh <joseph.rahmeh@...>
 

Hi All,
 
On a 64-bit implementation RISCV that supports a writable XML field in the MISA CSR and that supports writing 1 to that field (to turn on 32-bit mode), if we apply section 2.4 (CSR width modulation) to the MISA CSR then the XML field would end up with a 0 no matter what we attempt to write to it.  This does not seem right. 
 
Say we are in 32-bit mode and XML has the value 1.  If we write 1 to XML, this will change MXLEN to 64.  Following section 2.4 we get the following:
    Top 32 bits of the temporary register are set to zero since new width is larger than old width.
    Writable field XML in its new position (bits 62 and 63) is copied from corresponding location in temporary register: New XML gets 0.
 
Say we are in 64-bit mode and XML has the value 2.  If we write 1  to XML, this will change MXLEN to 32.  Following section 2,4 we get the following:
    Bits 31 and 30 are read only in the previous width MISA. Consequently bits 31 and 30 will be zero in the temporary register. When writable fields in the new width MISA are copied from the temporary register, the new XML fields gets 0.
 
Is it implied that software should write the XML field twice in order to put in it a value consistent with MXLEN? Once in its previous width MISA and again after MXLEN changes?
 
Joe Rahmeh
 


Re: [RISC-V] [tech-unixplatformspec] [RISC-V] [tech-privileged] [Announcement] Successful KVM RISC-V bring up on FPGA (Rocket core with H extension)

Sandro Pinto
 

Hi Sean,

The rocket-chip repo with our H-extension (a.k.a. Rocket-H) implementation can be found here:

Here you can find instructions on how to build and run the Rocket-H on FireSim with our hypervisor (Bao):

Feel free to ping us if you have any doubt or question..

Sandro

On Sat, Feb 27, 2021 at 4:56 AM Sean Halle <seanhalle@...> wrote:

Thank you for sending out the announcement, and congratulations :-)

I was wondering about the Rocket-H RTL.  Did some searching, which didn't turn up much.  Is that in the rocket-chip repo?  If not, is it open source?  If so, I would really appreciate a pointer to the implementation.

Thanks,

Sean


On Tue, Feb 2, 2021 at 11:18 AM Atish Patra <atish.patra@...> wrote:
On Tue, 2021-02-02 at 03:14 -0800, mark wrote:
> congrats!
>
> can we send something out to tech-announce about it?
>

Sure. I will send that once I have the detailed instructions available
in public domain.

> --------
> sent from a mobile device. please forgive any typos.
>
> > On Feb 2, 2021, at 12:40 AM, Atish Patra <atish.patra@...>
> > wrote:
> >
> > Hi,
> > We are glad to announce that we are able to boot Linux in KVM guest
> > on
> > a FPGA (Rocket chip + H extension v0.6.1). We now have three
> > hypervisors working on a Hardware with H extension.
> >
> > 1. KVM [1]
> > 2. Xvisor [2]
> > 3. Bao [3]
> >
> > KVM bring up was done using Firesim and the detailed instructions
> > will
> > be available very soon. Here are the software versions used for
> > bringup. Please find the attached boot log.
> >
> > OpenSBI: v0.9
> > Linux kernel: 5.11-rc5 + KVM patches(v16) + few kernel fixes [4].
> > Kvmtool: Upstream + RISC-V KVMTOOL patches (v6) [5]
> >
> >
> > We would like to thank Sandro & Jose who implemented the H
> > extension.
> > The Rocket-H design is available as a AFI image to be used within
> > Firesim or a stand alone FPGA board.
> >
> > We would also like to thank Andrew/John/Greg & others for defining
> > such
> > a clean specification as we did not discover any significant issues
> > while doing the bring up. As there are no changes proposed to the H
> > extension in the last year, we believe the current version of the H
> > extension can be considered as a freeze candidate. Please let us
> > know
> > if that is not the case.
> >
> > --
> > Regards,
> > Atish
> >
> > [1] https://github.com/kvm-riscv/howto/wiki
> > [2] https://github.com/xvisor/xvisor
> > [3] https://github.com/sandro2pinto/bao-rocket-h-firesim
> > [4] https://github.com/atishp04/linux/commits/rocket_kvm_working
> > [5] https://github.com/kvm-riscv/kvmtool
> >
> >
> >
> >
> >
> > <kvm_fpga_boot_log.txt>
>
>
>
>
>

--
Regards,
Atish






Re: [RISC-V] [tech-unixplatformspec] [RISC-V] [tech-privileged] [Announcement] Successful KVM RISC-V bring up on FPGA (Rocket core with H extension)

Sean Halle
 


Thank you for sending out the announcement, and congratulations :-)

I was wondering about the Rocket-H RTL.  Did some searching, which didn't turn up much.  Is that in the rocket-chip repo?  If not, is it open source?  If so, I would really appreciate a pointer to the implementation.

Thanks,

Sean


On Tue, Feb 2, 2021 at 11:18 AM Atish Patra <atish.patra@...> wrote:
On Tue, 2021-02-02 at 03:14 -0800, mark wrote:
> congrats!
>
> can we send something out to tech-announce about it?
>

Sure. I will send that once I have the detailed instructions available
in public domain.

> --------
> sent from a mobile device. please forgive any typos.
>
> > On Feb 2, 2021, at 12:40 AM, Atish Patra <atish.patra@...>
> > wrote:
> >
> > Hi,
> > We are glad to announce that we are able to boot Linux in KVM guest
> > on
> > a FPGA (Rocket chip + H extension v0.6.1). We now have three
> > hypervisors working on a Hardware with H extension.
> >
> > 1. KVM [1]
> > 2. Xvisor [2]
> > 3. Bao [3]
> >
> > KVM bring up was done using Firesim and the detailed instructions
> > will
> > be available very soon. Here are the software versions used for
> > bringup. Please find the attached boot log.
> >
> > OpenSBI: v0.9
> > Linux kernel: 5.11-rc5 + KVM patches(v16) + few kernel fixes [4].
> > Kvmtool: Upstream + RISC-V KVMTOOL patches (v6) [5]
> >
> >
> > We would like to thank Sandro & Jose who implemented the H
> > extension.
> > The Rocket-H design is available as a AFI image to be used within
> > Firesim or a stand alone FPGA board.
> >
> > We would also like to thank Andrew/John/Greg & others for defining
> > such
> > a clean specification as we did not discover any significant issues
> > while doing the bring up. As there are no changes proposed to the H
> > extension in the last year, we believe the current version of the H
> > extension can be considered as a freeze candidate. Please let us
> > know
> > if that is not the case.
> >
> > --
> > Regards,
> > Atish
> >
> > [1] https://github.com/kvm-riscv/howto/wiki
> > [2] https://github.com/xvisor/xvisor
> > [3] https://github.com/sandro2pinto/bao-rocket-h-firesim
> > [4] https://github.com/atishp04/linux/commits/rocket_kvm_working
> > [5] https://github.com/kvm-riscv/kvmtool
> >
> >
> >
> >
> >
> > <kvm_fpga_boot_log.txt>
>
>
>
>
>

--
Regards,
Atish






Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

On Thu, Feb 25, 2021 at 10:03 PM Sanjay Patel <spatel@...> wrote:

One last question(s). I assume hideleg[13] is supported (as well as mideleg[13], which has been confirmed). This allows the interrupt to be taken in VS-mode and thus vsip[13] and vsie[13] are required.


As I mentioned in my 2/25 response:  "This manner of supporting delegation involves defining a bunch of bits.  Which is not a clean and scalable solution as other bits in the ip/*ie registers come along with the same need.  The preferred and better solution will be provided by the "interrupt filtering" functionality that will be in the AIA (Advanced Interrupt Architecture) spec that is being developed to fully support interrupt virtualization (among other goals)."
 

And scountovf is common to both HS and VS-modes – the read is dependent on which context is interrupted.  If you can confirm, then I’ll be done.


Both HS and VS modes execute the exact same binary instruction to read scountovf (and intrinsically have the same rights to do so) - with the exact same behavior modulo just the effects of hcounteren. 

 Otherwise I understand that local interrupts will need to be prioritized *locally* wrt to interrupts from the proposed external interrupt controller. This is different from the scheme that I’ve used in the past. Its good though to keep top-level wires to a minimum.


All pending interrupts present in mip/sip get prioritized wrt each other, with the three "external" interrupts simply being three among that set of pending interrupts.  That is a function simply of the current Priv spec.  Where the three "external" interrupt requests come from is an orthogonal issue.  The AIA and PLIC are just two possible external interrupt controller architectures that one could use to generate these three per-hart request signals.
 
Greg

681 - 700 of 1210