[PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform


Kumar Sankaran
 

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
* ISA Profile
** The OS-A platform is required to comply with the RVA22 profile.
* Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

==== Debug
The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
==== Architecture
The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

==== Debug
The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
respect to all harts connected to the DM
* Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
==== Boot and Runtime Requirements
===== Firmware
The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar


Andrew Waterman
 

+ Dan

I'm not sure it makes sense to include the cache coherence requirement--not because we wish to permit incoherent caches, but because it's a constraint specified at the wrong level of abstraction.  The fundamental requirement is that the harts adhere to the RVWMO memory model.  The concept of caches and the concept of coherence are implementation details.  (Yes, in practice, if you have caches, you'll need cache coherence to implement RVWMO, but that doesn't mean it should be in a list of requirements.)

On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...> wrote:
This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar






Kumar Sankaran
 

Hi Andrew,

Agree with your comment on the RVWMO memory model requirement. Since this requirement is for the OS-A platform, providing an additional level of clarity will help since all implementations will be having a cache sub-system. So how about the below wording?

- All harts must adhere to the RVWMO memory model. Consequently, any hart-related caches must all be hardware coherent with respect to each other, and must appear to software as physically-indexed, physically-tagged (PIPT) caches.

- Memory accesses by I/O masters can be coherent or non-coherent with respect to any hart-related caches 

Regards

Kumar

From: Andrew Waterman <andrew@...>
Sent: Thursday, June 24, 2021 11:24 PM
To: Kumar Sankaran <ksankaran@...>
Cc: tech-unixplatformspec@...; Daniel Lustig <dlustig@...>
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

+ Dan

 

I'm not sure it makes sense to include the cache coherence requirement--not because we wish to permit incoherent caches, but because it's a constraint specified at the wrong level of abstraction.  The fundamental requirement is that the harts adhere to the RVWMO memory model.  The concept of caches and the concept of coherence are implementation details.  (Yes, in practice, if you have caches, you'll need cache coherence to implement RVWMO, but that doesn't mean it should be in a list of requirements.)

 

On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...> wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar





Daniel Lustig
 

On 6/25/2021 4:04 AM, Kumar Sankaran wrote:
Hi Andrew,

Agree with your comment on the RVWMO memory model requirement. Since this
requirement is for the OS-A platform, providing an additional level of
clarity will help since all implementations will be having a cache
sub-system. So how about the below wording?

- All harts must adhere to the RVWMO memory model. Consequently, any
hart-related caches must all be hardware coherent with respect to each
other, and must appear to software as physically-indexed, physically-tagged
(PIPT) caches.
I agree with Andrew. What does it mean for caches to appear to software
in any particular way? Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

In any case, what's wrong with VIPT caches?

- Memory accesses by I/O masters can be coherent or non-coherent with
respect to any hart-related caches
Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs? IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

Dan

Regards

Kumar

*From:* Andrew Waterman <andrew@...>
*Sent:* Thursday, June 24, 2021 11:24 PM
*To:* Kumar Sankaran <ksankaran@...>
*Cc:* tech-unixplatformspec@...; Daniel Lustig <
dlustig@...>
*Subject:* Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency
and ASID Requirements for OS-A platform



+ Dan



I'm not sure it makes sense to include the cache coherence requirement--not
because we wish to permit incoherent caches, but because it's a constraint
specified at the wrong level of abstraction. The fundamental requirement
is that the harts adhere to the RVWMO memory model. The concept of caches
and the concept of coherence are implementation details. (Yes, in
practice, if you have caches, you'll need cache coherence to implement
RVWMO, but that doesn't mean it should be in a list of requirements.)



On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...>
wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
* ISA Profile
** The OS-A platform is required to comply with the RVA22 profile.
* Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with
respect
+to the HART related caches
+
+==== PMU

==== Debug
The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
==== Architecture
The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

==== Debug
The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
respect to all harts connected to the DM
* Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
==== Boot and Runtime Requirements
===== Firmware
The boot and system firmware for the RV64I server platforms required to be


Philipp Tomsich
 



On Fri, Jun 25, 2021 at 2:01 PM Daniel Lustig <dlustig@...> wrote:
On 6/25/2021 4:04 AM, Kumar Sankaran wrote:
> Hi Andrew,
>
> Agree with your comment on the RVWMO memory model requirement. Since this
> requirement is for the OS-A platform, providing an additional level of
> clarity will help since all implementations will be having a cache
> sub-system. So how about the below wording?
>
> - All harts must adhere to the RVWMO memory model. Consequently, any
> hart-related caches must all be hardware coherent with respect to each
> other, and must appear to software as physically-indexed, physically-tagged
> (PIPT) caches.

I agree with Andrew.  What does it mean for caches to appear to software
in any particular way?  Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

The intent seems to be that caches shall appear (to software) to be non-aliasing.
 
In any case, what's wrong with VIPT caches?

Depends on the internal organisation of a VIPT cache (and its size/number of index bits)...
The requirement clearly is that the cache appears to be non-aliasing, by whatever design choices.
A simple rewording of the original statement to this effect (instead of referencing PIPT) should address this.
 
> - Memory accesses by I/O masters can be coherent or non-coherent with
> respect to any hart-related caches

Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs?  IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

Dan

> Regards
>
> Kumar
>
> *From:* Andrew Waterman <andrew@...>
> *Sent:* Thursday, June 24, 2021 11:24 PM
> *To:* Kumar Sankaran <ksankaran@...>
> *Cc:* tech-unixplatformspec@...; Daniel Lustig <
> dlustig@...>
> *Subject:* Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency
> and ASID Requirements for OS-A platform
>
>
>
> + Dan
>
>
>
> I'm not sure it makes sense to include the cache coherence requirement--not
> because we wish to permit incoherent caches, but because it's a constraint
> specified at the wrong level of abstraction.  The fundamental requirement
> is that the harts adhere to the RVWMO memory model.  The concept of caches
> and the concept of coherence are implementation details.  (Yes, in
> practice, if you have caches, you'll need cache coherence to implement
> RVWMO, but that doesn't mean it should be in a list of requirements.)
>
>
>
> On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...>
> wrote:
>
> This patch adds the following
> Cache coherency and ASID requirements
> Interrupt Controller and PMU chapter sub-sections for OS-A base and
> Server Extension
>
> diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
> index 87ea6d5..b985f50 100644
> --- a/riscv-platform-spec.adoc
> +++ b/riscv-platform-spec.adoc
> @@ -88,8 +88,12 @@ The M platform has the following extensions:
>  * ISA Profile
>  ** The OS-A platform is required to comply with the RVA22 profile.
>  * Cache Coherency
> -* PMU
> -* ASID
> +** All HART related caches must be hardware coherent and must appear to
> +software as Physically Indexed, Physically Tagged (PIPT) caches
> +** Memory accesses by I/O masters can be coherent or non-coherent with
> respect
> +to the HART related caches
> +
> +==== PMU
>
>  ==== Debug
>  The OS-A base platform requirements are -
> @@ -287,10 +291,12 @@ base with the additional requirements as below.
>  ==== Architecture
>  The platforms which conform to server extension are required to implement +
>
> -- RISC-V Hypervisor-level H Instruction-Set Extensions
> -- IOMMU with support for memory resident interrupt files
> -- PMU
> -- ASID
> +- RV64 support
> +- RISC-V H ISA extension
> +- ASID support
> +- VMID support
> +
> +==== PMU
>
>  ==== Debug
>  The OS-A server platform requirements are all of the base above plus:
> @@ -305,6 +311,8 @@ above.
>  respect to all harts connected to the DM
>    * Rationale: Debuggers must be able to view memory coherently
>
> +==== Interrupt Controller
> +
>  ==== Boot and Runtime Requirements
>  =====  Firmware
>  The boot and system firmware for the RV64I server platforms required to be
>






Greg Favor
 

On Fri, Jun 25, 2021 at 5:22 AM Philipp Tomsich <ptomsich@...> wrote:
I agree with Andrew.  What does it mean for caches to appear to software
in any particular way?  Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

The intent seems to be that caches shall appear (to software) to be non-aliasing.

ARM went through this issue with the variety of CPU designs that people have done (both outside and inside ARM).  In particular, as people went to larger and/or more power-efficient L1 caches, they implemented VIPT caches.  Which exposes VA aliasing to software.  So either software has to manage cache consistency, or hardware has to (i.e hardware has to make the VIPT cache appear to behave like a PIPT cache).  That's non-trivial hardware, but not unreasonable.  The result is that while ARM architecture allows VIVT, VIPT, and PIPT caches, in A class cores everything has moved to caches that appear to be PIPT while many actually are VIPT designs.

One can say that RVWMO compliance implies a requirement that caches appear to behave like PIPT caches.  But, given the growing use of VIPT caches, it's worth half a sentence to make clear the implications of RVWMO.
 
 
In any case, what's wrong with VIPT caches?

Depends on the internal organisation of a VIPT cache (and its size/number of index bits)...
The requirement clearly is that the cache appears to be non-aliasing, by whatever design choices.
A simple rewording of the original statement to this effect (instead of referencing PIPT) should address this.

Agreed.  I'll come up with some better words for Kumar to use.
 
 
> - Memory accesses by I/O masters can be coherent or non-coherent with
> respect to any hart-related caches

Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs?  IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

People tend to presume this in OS/A-class systems, but yes this should be explicitly stated.

Greg


Jonathan Behrens <behrensj@...>
 

Is this supposed to include more detail than just "ASID support" and "VMID support"? Not sure if those are just placeholder section headings for now, but I'd expect to at least see a requirement of how many bits are required for each.

Also not entirely sure how to specify it, but it would be really nice if implementations weren't allowed to have "fake" ASID support by just clearing the TLB whenever the ASID changes. That's the strategy QEMU uses (at least the last time I checked) and it has the impact of leading to worse performance than not having ASIDs at all.

Jonathan


On Thu, Jun 24, 2021 at 8:01 PM Kumar Sankaran via lists.riscv.org <ksankaran=ventanamicro.com@...> wrote:
This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar






Kumar Sankaran
 

Hi Jonathan,

Regarding ASID and VMID support, we are mandating those only for the Server Extension within the OS-A platform and not for OS-A platform base.

We could consider a minimum of 12-bits for the ASID support for the server extension, but then most server platforms would in reality only implement 4 bits of ASID. So simply mandating ASID support without any additional requirement seemed to be the best approach.

Is there any particular number of ASID bits you suggest for the server extension? Same for the VMID as well.

 

Regarding your second comment, yes, I agree that implementations shouldn’t be allowed to “fake” ASID support by clearing the entire TLB when the ASID changes. This will be a big performance penalty. Having said that, the approach the platform spec takes is to have a minimal set of features for hardware and software compatibility without mandating any performance requirements. So we can add the below line as an Informational or Application Note below the requirement. If you have an alternative wording for this line, feel free to send that over.

 

Regards

Kumar

From: Jonathan Behrens <behrensj@...>
Sent: Friday, June 25, 2021 9:18 AM
To: Kumar Sankaran <ksankaran@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

Is this supposed to include more detail than just "ASID support" and "VMID support"? Not sure if those are just placeholder section headings for now, but I'd expect to at least see a requirement of how many bits are required for each.

 

Also not entirely sure how to specify it, but it would be really nice if implementations weren't allowed to have "fake" ASID support by just clearing the TLB whenever the ASID changes. That's the strategy QEMU uses (at least the last time I checked) and it has the impact of leading to worse performance than not having ASIDs at all.

 

Jonathan

 

On Thu, Jun 24, 2021 at 8:01 PM Kumar Sankaran via lists.riscv.org <ksankaran=ventanamicro.com@...> wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar





Jonathan Behrens <behrensj@...>
 

Requiring ASID support without specifying the number of bits means that even 1-bit ASIDs would be allowed, even though they'd be essentially useless. That said, I don't have a strong feeling about a particular number of bits to pick. Some software designs can get by on 4 bits, while others may benefit from 8 or even 16.

Is there anyone who would object to mandating the full 16 ASID bits for the Server Extension of the OS-A platform? I'm a software person so I really don't have a good sense of whether more bits would meaningfully increase costs/complexity.

Jonathan


On Fri, Jun 25, 2021 at 4:55 PM Kumar Sankaran <ksankaran@...> wrote:

Hi Jonathan,

Regarding ASID and VMID support, we are mandating those only for the Server Extension within the OS-A platform and not for OS-A platform base.

We could consider a minimum of 12-bits for the ASID support for the server extension, but then most server platforms would in reality only implement 4 bits of ASID. So simply mandating ASID support without any additional requirement seemed to be the best approach.

Is there any particular number of ASID bits you suggest for the server extension? Same for the VMID as well.

 

Regarding your second comment, yes, I agree that implementations shouldn’t be allowed to “fake” ASID support by clearing the entire TLB when the ASID changes. This will be a big performance penalty. Having said that, the approach the platform spec takes is to have a minimal set of features for hardware and software compatibility without mandating any performance requirements. So we can add the below line as an Informational or Application Note below the requirement. If you have an alternative wording for this line, feel free to send that over.

 

Regards

Kumar

From: Jonathan Behrens <behrensj@...>
Sent: Friday, June 25, 2021 9:18 AM
To: Kumar Sankaran <ksankaran@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

Is this supposed to include more detail than just "ASID support" and "VMID support"? Not sure if those are just placeholder section headings for now, but I'd expect to at least see a requirement of how many bits are required for each.

 

Also not entirely sure how to specify it, but it would be really nice if implementations weren't allowed to have "fake" ASID support by just clearing the TLB whenever the ASID changes. That's the strategy QEMU uses (at least the last time I checked) and it has the impact of leading to worse performance than not having ASIDs at all.

 

Jonathan

 

On Thu, Jun 24, 2021 at 8:01 PM Kumar Sankaran via lists.riscv.org <ksankaran=ventanamicro.com@...> wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar





Andrew Waterman
 


On Fri, Jun 25, 2021 at 4:31 PM Jonathan Behrens <behrensj@...> wrote:
Requiring ASID support without specifying the number of bits means that even 1-bit ASIDs would be allowed, even though they'd be essentially useless. That said, I don't have a strong feeling about a particular number of bits to pick. Some software designs can get by on 4 bits, while others may benefit from 8 or even 16.

Is there anyone who would object to mandating the full 16 ASID bits for the Server Extension of the OS-A platform? I'm a software person so I really don't have a good sense of whether more bits would meaningfully increase costs/complexity.

Significant HW cost in TLBs.

My take is we should avoid imposing this requirement altogether. Kernel software needs to support the no-ASID case, anyway, so we aren’t simplifying software by imposing this requirement.



Jonathan

On Fri, Jun 25, 2021 at 4:55 PM Kumar Sankaran <ksankaran@...> wrote:

Hi Jonathan,

Regarding ASID and VMID support, we are mandating those only for the Server Extension within the OS-A platform and not for OS-A platform base.

We could consider a minimum of 12-bits for the ASID support for the server extension, but then most server platforms would in reality only implement 4 bits of ASID. So simply mandating ASID support without any additional requirement seemed to be the best approach.

Is there any particular number of ASID bits you suggest for the server extension? Same for the VMID as well.

 

Regarding your second comment, yes, I agree that implementations shouldn’t be allowed to “fake” ASID support by clearing the entire TLB when the ASID changes. This will be a big performance penalty. Having said that, the approach the platform spec takes is to have a minimal set of features for hardware and software compatibility without mandating any performance requirements. So we can add the below line as an Informational or Application Note below the requirement. If you have an alternative wording for this line, feel free to send that over.

 

Regards

Kumar

From: Jonathan Behrens <behrensj@...>
Sent: Friday, June 25, 2021 9:18 AM
To: Kumar Sankaran <ksankaran@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

Is this supposed to include more detail than just "ASID support" and "VMID support"? Not sure if those are just placeholder section headings for now, but I'd expect to at least see a requirement of how many bits are required for each.

 

Also not entirely sure how to specify it, but it would be really nice if implementations weren't allowed to have "fake" ASID support by just clearing the TLB whenever the ASID changes. That's the strategy QEMU uses (at least the last time I checked) and it has the impact of leading to worse performance than not having ASIDs at all.

 

Jonathan

 

On Thu, Jun 24, 2021 at 8:01 PM Kumar Sankaran via lists.riscv.org <ksankaran=ventanamicro.com@...> wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar

















Greg Favor
 

On Fri, Jun 25, 2021 at 8:00 PM Andrew Waterman <andrew@...> wrote:
My take is we should avoid imposing this requirement altogether. Kernel software needs to support the no-ASID case, anyway, so we aren’t simplifying software by imposing this requirement.

Recent platform discussion about this question concluded the same.

Greg

 


Kumar Sankaran
 

OK agreed. We can remove the ASID requirement all together.

 

Regards

Kumar

From: Greg Favor <gfavor@...>
Sent: Friday, June 25, 2021 8:05 PM
To: Andrew Waterman <andrew@...>
Cc: Jonathan Behrens <behrensj@...>; Kumar Sankaran <ksankaran@...>; tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

On Fri, Jun 25, 2021 at 8:00 PM Andrew Waterman <andrew@...> wrote:

My take is we should avoid imposing this requirement altogether. Kernel software needs to support the no-ASID case, anyway, so we aren’t simplifying software by imposing this requirement.

 

Recent platform discussion about this question concluded the same.

 

Greg