RISC-V H-extension freeze consideration


Anup Patel
 

Hi All,

The RISC-V H-extension v0.6.1 draft was released almost a year back in
May 2020. There has been no changes in the H-extension specification
since then.

Meanwhile, we have RISC-V H-extension v0.6.1 implemented in QEMU,
Spike, and Rocket FPGA. We also have three different hypervisors ported
to RISC-V H-extension v0.6.1:
1. Xvisor RISC-V (Works on QEMU, Spike, and Rocket FPGA)
2. KVM RISC-V (Works on QEMU, Spike and Rocket FPGA)
3. Bao (Works on Rocket FPGA)

Unfortunately, RISC-V H-extension not being in freeze state is now gating
further software development because major open source projects (such
as Linux RISC-V and GCC RISC-V) have adopted a policy of accepting patches
only for frozen or ratified RISC-V extensions.

Few examples of gated software development:
1. KVM RISC-V not merged in upstream Linux RISC-V. The KVM RISC-V
patches are already reviewed and acked by maintainers in July 2019.
Currently, we are rebasing KVM RISC-V patches with every kernel
release since 1.5+ years.
2. GCC RISC-V not accepting patches for H-extension related instructions
3. KVMTOOL RISC-V not merged because KVM RISC-V is not merged in
upstream Linux RISC-V
4. QEMU KVM RISC-V acceleration not merged because KVM RISC-V is
not merged in upstream Linux RISC-V
5. Various feature additions (such as SBI v0.2, nested, etc) can't happen
(or can't be merged) until KVM RISC-V is merged in Linux RISC-V
6. Libvirt library blocked on QEMU KVM RISC-V acceleration being
available. The Libvirt library is a crucial piece in open-source cloud
solutions (such as open-stack).
7. As time passes more items (such as KVM RISC-V AIA support) will
get blocked if KVM RISC-V is not merged upstream.

We would request the TSC to consider freezing RISC-V H-extension v0.6.1
draft specification. Remaining items in done checklist for ratification can
certainly be completed while H-extension is in the frozen state.

Best Regards,
Anup Patel


Andrew Waterman
 

I’m not in support of freezing it yet. My concern is that development of virtualization-aware interrupt controllers and IOMMUs will lead to reconsideration of some of the details. All of these items are logically interlocking, even if physically disjoint separate. It’s entirely possible that we will make no changes as a result of that further development, but it’s far from certain.

Furthermore, the hypervisor extension is of substantially greater with those other items completed, so we aren’t losing out as much as it might seem by postponing the freeze.

On Tue, Feb 2, 2021 at 7:47 PM Anup Patel <Anup.Patel@...> wrote:
Hi All,



The RISC-V H-extension v0.6.1 draft was released almost a year back in

May 2020. There has been no changes in the H-extension specification

since then.



Meanwhile, we have RISC-V H-extension v0.6.1 implemented in QEMU,

Spike, and Rocket FPGA. We also have three different hypervisors ported

to RISC-V H-extension v0.6.1:

1. Xvisor RISC-V (Works on QEMU, Spike, and Rocket FPGA)

2. KVM RISC-V (Works on QEMU, Spike and Rocket FPGA)

3. Bao (Works on Rocket FPGA)



Unfortunately, RISC-V H-extension not being in freeze state is now gating

further software development because major open source projects (such

as Linux RISC-V and GCC RISC-V) have adopted a policy of accepting patches

only for frozen or ratified RISC-V extensions.



Few examples of gated software development:

1. KVM RISC-V not merged in upstream Linux RISC-V. The KVM RISC-V

    patches are already reviewed and acked by maintainers in July 2019.

    Currently, we are rebasing KVM RISC-V patches with every kernel

    release since 1.5+ years.

2. GCC RISC-V not accepting patches for H-extension related instructions

3. KVMTOOL RISC-V not merged because KVM RISC-V is not merged in

    upstream Linux RISC-V

4. QEMU KVM RISC-V acceleration not merged because KVM RISC-V is

    not merged in upstream Linux RISC-V

5. Various feature additions (such as SBI v0.2, nested, etc) can't happen

   (or can't be merged) until KVM RISC-V is merged in Linux RISC-V

6. Libvirt library blocked on QEMU KVM RISC-V acceleration being

    available. The Libvirt library is a crucial piece in open-source cloud

    solutions (such as open-stack).

7. As time passes more items (such as KVM RISC-V AIA support) will

    get blocked if KVM RISC-V is not merged upstream.



We would request the TSC to consider freezing RISC-V H-extension v0.6.1

draft specification. Remaining items in done checklist for ratification can

certainly be completed while H-extension is in the frozen state.



Best Regards,

Anup Patel


Andrew Waterman
 



On Tue, Feb 2, 2021 at 7:54 PM Andrew Waterman <andrew@...> wrote:
I’m not in support of freezing it yet. My concern is that development of virtualization-aware interrupt controllers and IOMMUs will lead to reconsideration of some of the details. All of these items are logically interlocking, even if physically disjoint separate. It’s entirely possible that we will make no changes as a result of that further development, but it’s far from certain.

Furthermore, the hypervisor extension is of substantially greater with those other items completed, so we aren’t losing out as much as it might seem by postponing the freeze.

* substantially greater utility

Shouldn’t write emails on phone...


On Tue, Feb 2, 2021 at 7:47 PM Anup Patel <Anup.Patel@...> wrote:
Hi All,



The RISC-V H-extension v0.6.1 draft was released almost a year back in

May 2020. There has been no changes in the H-extension specification

since then.



Meanwhile, we have RISC-V H-extension v0.6.1 implemented in QEMU,

Spike, and Rocket FPGA. We also have three different hypervisors ported

to RISC-V H-extension v0.6.1:

1. Xvisor RISC-V (Works on QEMU, Spike, and Rocket FPGA)

2. KVM RISC-V (Works on QEMU, Spike and Rocket FPGA)

3. Bao (Works on Rocket FPGA)



Unfortunately, RISC-V H-extension not being in freeze state is now gating

further software development because major open source projects (such

as Linux RISC-V and GCC RISC-V) have adopted a policy of accepting patches

only for frozen or ratified RISC-V extensions.



Few examples of gated software development:

1. KVM RISC-V not merged in upstream Linux RISC-V. The KVM RISC-V

    patches are already reviewed and acked by maintainers in July 2019.

    Currently, we are rebasing KVM RISC-V patches with every kernel

    release since 1.5+ years.

2. GCC RISC-V not accepting patches for H-extension related instructions

3. KVMTOOL RISC-V not merged because KVM RISC-V is not merged in

    upstream Linux RISC-V

4. QEMU KVM RISC-V acceleration not merged because KVM RISC-V is

    not merged in upstream Linux RISC-V

5. Various feature additions (such as SBI v0.2, nested, etc) can't happen

   (or can't be merged) until KVM RISC-V is merged in Linux RISC-V

6. Libvirt library blocked on QEMU KVM RISC-V acceleration being

    available. The Libvirt library is a crucial piece in open-source cloud

    solutions (such as open-stack).

7. As time passes more items (such as KVM RISC-V AIA support) will

    get blocked if KVM RISC-V is not merged upstream.



We would request the TSC to consider freezing RISC-V H-extension v0.6.1

draft specification. Remaining items in done checklist for ratification can

certainly be completed while H-extension is in the frozen state.



Best Regards,

Anup Patel




Anup Patel
 

On all major architectures (x86 and ARM64), the virtualization-aware interrupt controllers and IOMMUs are totally independent from ISA virtualization support.

 

We already the required ISA support in H-extension for virtualization-aware interrupt controller.

 

The IOMMUs are totally independent of CPU virtualization support on all major architectures and I don’t see how H-extension need to change for IOMMU support.

 

Regards,

Anup

 

From: Andrew Waterman <andrew@...>
Sent: 03 February 2021 09:24
To: Anup Patel <Anup.Patel@...>
Cc: Alistair Francis <Alistair.Francis@...>; Allen Baum <allen.baum@...>; Atish Patra <Atish.Patra@...>; Greg Favor <gfavor@...>; John Hauser <jh.riscv@...>; Krste Asanovic <krste@...>; tech-privileged@...; tech-unixplatformspec@...
Subject: Re: RISC-V H-extension freeze consideration

 

I’m not in support of freezing it yet. My concern is that development of virtualization-aware interrupt controllers and IOMMUs will lead to reconsideration of some of the details. All of these items are logically interlocking, even if physically disjoint separate. It’s entirely possible that we will make no changes as a result of that further development, but it’s far from certain.

 

Furthermore, the hypervisor extension is of substantially greater with those other items completed, so we aren’t losing out as much as it might seem by postponing the freeze.

 

On Tue, Feb 2, 2021 at 7:47 PM Anup Patel <Anup.Patel@...> wrote:

Hi All,



The RISC-V H-extension v0.6.1 draft was released almost a year back in

May 2020. There has been no changes in the H-extension specification

since then.



Meanwhile, we have RISC-V H-extension v0.6.1 implemented in QEMU,

Spike, and Rocket FPGA. We also have three different hypervisors ported

to RISC-V H-extension v0.6.1:

1. Xvisor RISC-V (Works on QEMU, Spike, and Rocket FPGA)

2. KVM RISC-V (Works on QEMU, Spike and Rocket FPGA)

3. Bao (Works on Rocket FPGA)



Unfortunately, RISC-V H-extension not being in freeze state is now gating

further software development because major open source projects (such

as Linux RISC-V and GCC RISC-V) have adopted a policy of accepting patches

only for frozen or ratified RISC-V extensions.



Few examples of gated software development:

1. KVM RISC-V not merged in upstream Linux RISC-V. The KVM RISC-V

    patches are already reviewed and acked by maintainers in July 2019.

    Currently, we are rebasing KVM RISC-V patches with every kernel

    release since 1.5+ years.

2. GCC RISC-V not accepting patches for H-extension related instructions

3. KVMTOOL RISC-V not merged because KVM RISC-V is not merged in

    upstream Linux RISC-V

4. QEMU KVM RISC-V acceleration not merged because KVM RISC-V is

    not merged in upstream Linux RISC-V

5. Various feature additions (such as SBI v0.2, nested, etc) can't happen

   (or can't be merged) until KVM RISC-V is merged in Linux RISC-V

6. Libvirt library blocked on QEMU KVM RISC-V acceleration being

    available. The Libvirt library is a crucial piece in open-source cloud

    solutions (such as open-stack).

7. As time passes more items (such as KVM RISC-V AIA support) will

    get blocked if KVM RISC-V is not merged upstream.



We would request the TSC to consider freezing RISC-V H-extension v0.6.1

draft specification. Remaining items in done checklist for ratification can

certainly be completed while H-extension is in the frozen state.



Best Regards,

Anup Patel


Andrew Waterman
 

In other architectures, those devices are needlessly complex in part because they weren’t co-designed with the ISA. Yes, they can be independently designed, but possibly with regrettable consequences.

On Tue, Feb 2, 2021 at 8:06 PM Anup Patel <Anup.Patel@...> wrote:
















On all major architectures (x86 and ARM64), the virtualization-aware interrupt controllers and IOMMUs are totally independent from ISA virtualization support.



 



We already the required ISA support in H-extension for virtualization-aware interrupt controller.



 



The IOMMUs are totally independent of CPU virtualization support on all major architectures and I don’t see how H-extension need to change for IOMMU support.



 



Regards,



Anup



 









From: Andrew Waterman <andrew@...>


Sent: 03 February 2021 09:24


To: Anup Patel <Anup.Patel@...>


Cc: Alistair Francis <Alistair.Francis@...>; Allen Baum <allen.baum@...>; Atish Patra <Atish.Patra@...>; Greg Favor <gfavor@...>; John Hauser <jh.riscv@...>; Krste Asanovic <krste@...>; tech-privileged@...;

tech-unixplatformspec@...


Subject: Re: RISC-V H-extension freeze consideration







 







I’m not in support of freezing it yet. My concern is that development of virtualization-aware interrupt controllers and IOMMUs will lead to reconsideration of some of the details. All of these items are logically interlocking, even if physically

disjoint separate. It’s entirely possible that we will make no changes as a result of that further development, but it’s far from certain.









 







Furthermore, the hypervisor extension is of substantially greater with those other items completed, so we aren’t losing out as much as it might seem by postponing the freeze.







 







On Tue, Feb 2, 2021 at 7:47 PM Anup Patel <Anup.Patel@...> wrote:







Hi All,











The RISC-V H-extension v0.6.1 draft was released almost a year back in





May 2020. There has been no changes in the H-extension specification





since then.











Meanwhile, we have RISC-V H-extension v0.6.1 implemented in QEMU,





Spike, and Rocket FPGA. We also have three different hypervisors ported





to RISC-V H-extension v0.6.1:





1. Xvisor RISC-V (Works on QEMU, Spike, and Rocket FPGA)





2. KVM RISC-V (Works on QEMU, Spike and Rocket FPGA)





3. Bao (Works on Rocket FPGA)











Unfortunately, RISC-V H-extension not being in freeze state is now gating





further software development because major open source projects (such





as Linux RISC-V and GCC RISC-V) have adopted a policy of accepting patches





only for frozen or ratified RISC-V extensions.











Few examples of gated software development:





1. KVM RISC-V not merged in upstream Linux RISC-V. The KVM RISC-V





    patches are already reviewed and acked by maintainers in July 2019.





    Currently, we are rebasing KVM RISC-V patches with every kernel





    release since 1.5+ years.





2. GCC RISC-V not accepting patches for H-extension related instructions





3. KVMTOOL RISC-V not merged because KVM RISC-V is not merged in





    upstream Linux RISC-V





4. QEMU KVM RISC-V acceleration not merged because KVM RISC-V is





    not merged in upstream Linux RISC-V





5. Various feature additions (such as SBI v0.2, nested, etc) can't happen





   (or can't be merged) until KVM RISC-V is merged in Linux RISC-V





6. Libvirt library blocked on QEMU KVM RISC-V acceleration being





    available. The Libvirt library is a crucial piece in open-source cloud





    solutions (such as open-stack).





7. As time passes more items (such as KVM RISC-V AIA support) will





    get blocked if KVM RISC-V is not merged upstream.











We would request the TSC to consider freezing RISC-V H-extension v0.6.1





draft specification. Remaining items in done checklist for ratification can





certainly be completed while H-extension is in the frozen state.











Best Regards,





Anup Patel


















Greg Favor
 

I generally agree with Andrew.  At the same time I'll also observe that, practically speaking, the AIA is coming soon and it very much directly interacts with the H extension.  So waiting a little longer to see that at least stabilize is a good (and probably necessary) compromise.  (Past that we can then come back to arguing when to draw the line on freezing the H extension spec.)  Further, there are other extensions happening now and the next few months (from the virt-mem group, pointer masking from the J group, and a couple of fast-track extensions) that it would be good to stabilize if not freeze in conjunction with the H extension.  So, in my own opinion, we're getting close.  Not a few weeks, but not quarters either.  (I'll also say that the "pressure is on" to intelligently try and get through this period of time sooner than later.)

Having all these things that interact with virtualization being finalized together (I'm being loose for now wrt official "stable" versus "freeze" milestones, to focus on the general idea) is a good thing for the reasons Andrew mentioned.  Most important (risk-wise) to me is seeing the virt-mem extensions and AIA stuff stabilize.

Now, when it comes to the IOMMU, its architecture needs to (or strongly should) follow the CPU virtualization architecture.  But I think it is an acceptable compromise to not hold up all the preceding because of the IOMMU architecture.  I see very low risk of realizing from the IOMMU architecture that something in the H extension should have been done differently.  Maybe some extra feature will be identified, but that could be done as an extension to the H extension (and I think that is also low risk of happening).  I'll also note that the AIA will cover how an IOMMU handles virtualization of I/O interrupts (aka MSIs).  Which leaves normal translation of I/O addresses to follow the mold of the Supervisor and Hypervisor architectures.  (And, for completeness, many of the other "interesting" aspects of an IOMMU architecture I believe can and should comport with the Supervisor and Hypervisor architectures as needed.)

In short, I think a reasonable compromise is to wait a "little" bit longer for most of the above "coming soon" things, and to decouple the IOMMU timeline from freezing the H extension and related extensions.

Greg


On Tue, Feb 2, 2021 at 8:13 PM Andrew Waterman <andrew@...> wrote:
In other architectures, those devices are needlessly complex in part because they weren’t co-designed with the ISA. Yes, they can be independently designed, but possibly with regrettable consequences.

On Tue, Feb 2, 2021 at 8:06 PM Anup Patel <Anup.Patel@...> wrote:
















On all major architectures (x86 and ARM64), the virtualization-aware interrupt controllers and IOMMUs are totally independent from ISA virtualization support.



 



We already the required ISA support in H-extension for virtualization-aware interrupt controller.



 



The IOMMUs are totally independent of CPU virtualization support on all major architectures and I don’t see how H-extension need to change for IOMMU support.



 



Regards,



Anup



 









From: Andrew Waterman <andrew@...>


Sent: 03 February 2021 09:24


To: Anup Patel <Anup.Patel@...>


Cc: Alistair Francis <Alistair.Francis@...>; Allen Baum <allen.baum@...>; Atish Patra <Atish.Patra@...>; Greg Favor <gfavor@...>; John Hauser <jh.riscv@...>; Krste Asanovic <krste@...>; tech-privileged@...;

tech-unixplatformspec@...


Subject: Re: RISC-V H-extension freeze consideration







 







I’m not in support of freezing it yet. My concern is that development of virtualization-aware interrupt controllers and IOMMUs will lead to reconsideration of some of the details. All of these items are logically interlocking, even if physically

disjoint separate. It’s entirely possible that we will make no changes as a result of that further development, but it’s far from certain.









 







Furthermore, the hypervisor extension is of substantially greater with those other items completed, so we aren’t losing out as much as it might seem by postponing the freeze.







 







On Tue, Feb 2, 2021 at 7:47 PM Anup Patel <Anup.Patel@...> wrote:







Hi All,











The RISC-V H-extension v0.6.1 draft was released almost a year back in





May 2020. There has been no changes in the H-extension specification





since then.











Meanwhile, we have RISC-V H-extension v0.6.1 implemented in QEMU,





Spike, and Rocket FPGA. We also have three different hypervisors ported





to RISC-V H-extension v0.6.1:





1. Xvisor RISC-V (Works on QEMU, Spike, and Rocket FPGA)





2. KVM RISC-V (Works on QEMU, Spike and Rocket FPGA)





3. Bao (Works on Rocket FPGA)











Unfortunately, RISC-V H-extension not being in freeze state is now gating





further software development because major open source projects (such





as Linux RISC-V and GCC RISC-V) have adopted a policy of accepting patches





only for frozen or ratified RISC-V extensions.











Few examples of gated software development:





1. KVM RISC-V not merged in upstream Linux RISC-V. The KVM RISC-V





    patches are already reviewed and acked by maintainers in July 2019.





    Currently, we are rebasing KVM RISC-V patches with every kernel





    release since 1.5+ years.





2. GCC RISC-V not accepting patches for H-extension related instructions





3. KVMTOOL RISC-V not merged because KVM RISC-V is not merged in





    upstream Linux RISC-V





4. QEMU KVM RISC-V acceleration not merged because KVM RISC-V is





    not merged in upstream Linux RISC-V





5. Various feature additions (such as SBI v0.2, nested, etc) can't happen





   (or can't be merged) until KVM RISC-V is merged in Linux RISC-V





6. Libvirt library blocked on QEMU KVM RISC-V acceleration being





    available. The Libvirt library is a crucial piece in open-source cloud





    solutions (such as open-stack).





7. As time passes more items (such as KVM RISC-V AIA support) will





    get blocked if KVM RISC-V is not merged upstream.











We would request the TSC to consider freezing RISC-V H-extension v0.6.1





draft specification. Remaining items in done checklist for ratification can





certainly be completed while H-extension is in the frozen state.











Best Regards,





Anup Patel