On 6/25/2021 4:04 AM, Kumar Sankaran wrote:
> Hi Andrew,
>
> Agree with your comment on the RVWMO memory model requirement. Since this
> requirement is for the OS-A platform, providing an additional level of
> clarity will help since all implementations will be having a cache
> sub-system. So how about the below wording?
>
> - All harts must adhere to the RVWMO memory model. Consequently, any
> hart-related caches must all be hardware coherent with respect to each
> other, and must appear to software as physically-indexed, physically-tagged
> (PIPT) caches.

I agree with Andrew.  What does it mean for caches to appear to software
in any particular way?  Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

In any case, what's wrong with VIPT caches?

> - Memory accesses by I/O masters can be coherent or non-coherent with
> respect to any hart-related caches

Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs?  IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

Dan

> Regards
>
> Kumar
>
> *From:* Andrew Waterman <andrew@...>
> *Sent:* Thursday, June 24, 2021 11:24 PM
> *To:* Kumar Sankaran <ksankaran@...>
> *Cc:* tech-unixplatformspec@...; Daniel Lustig <
> dlustig@...>
> *Subject:* Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency
> and ASID Requirements for OS-A platform
>
>
>
> + Dan
>
>
>
> I'm not sure it makes sense to include the cache coherence requirement--not
> because we wish to permit incoherent caches, but because it's a constraint
> specified at the wrong level of abstraction.  The fundamental requirement
> is that the harts adhere to the RVWMO memory model.  The concept of caches
> and the concept of coherence are implementation details.  (Yes, in
> practice, if you have caches, you'll need cache coherence to implement
> RVWMO, but that doesn't mean it should be in a list of requirements.)
>
>
>
> On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...>
> wrote:
>
> This patch adds the following
> Cache coherency and ASID requirements
> Interrupt Controller and PMU chapter sub-sections for OS-A base and
> Server Extension
>
> diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
> index 87ea6d5..b985f50 100644
> --- a/riscv-platform-spec.adoc
> +++ b/riscv-platform-spec.adoc
> @@ -88,8 +88,12 @@ The M platform has the following extensions:
>  * ISA Profile
>  ** The OS-A platform is required to comply with the RVA22 profile.
>  * Cache Coherency
> -* PMU
> -* ASID
> +** All HART related caches must be hardware coherent and must appear to
> +software as Physically Indexed, Physically Tagged (PIPT) caches
> +** Memory accesses by I/O masters can be coherent or non-coherent with
> respect
> +to the HART related caches
> +
> +==== PMU
>
>  ==== Debug
>  The OS-A base platform requirements are -
> @@ -287,10 +291,12 @@ base with the additional requirements as below.
>  ==== Architecture
>  The platforms which conform to server extension are required to implement +
>
> -- RISC-V Hypervisor-level H Instruction-Set Extensions
> -- IOMMU with support for memory resident interrupt files
> -- PMU
> -- ASID
> +- RV64 support
> +- RISC-V H ISA extension
> +- ASID support
> +- VMID support
> +
> +==== PMU
>
>  ==== Debug
>  The OS-A server platform requirements are all of the base above plus:
> @@ -305,6 +311,8 @@ above.
>  respect to all harts connected to the DM
>    * Rationale: Debuggers must be able to view memory coherently
>
> +==== Interrupt Controller
> +
>  ==== Boot and Runtime Requirements
>  =====  Firmware
>  The boot and system firmware for the RV64I server platforms required to be
>

I agree with Andrew.  What does it mean for caches to appear to software
in any particular way?  Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

The intent seems to be that caches shall appear (to software) to be non-aliasing.

 

In any case, what's wrong with VIPT caches?

Depends on the internal organisation of a VIPT cache (and its size/number of index bits)...

The requirement clearly is that the cache appears to be non-aliasing, by whatever design choices.

A simple rewording of the original statement to this effect (instead of referencing PIPT) should address this.

 

> - Memory accesses by I/O masters can be coherent or non-coherent with
> respect to any hart-related caches

Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs?  IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

Requiring ASID support without specifying the number of bits means that even 1-bit ASIDs would be allowed, even though they'd be essentially useless. That said, I don't have a strong feeling about a particular number of bits to pick. Some software designs can get by on 4 bits, while others may benefit from 8 or even 16.

Is there anyone who would object to mandating the full 16 ASID bits for the Server Extension of the OS-A platform? I'm a software person so I really don't have a good sense of whether more bits would meaningfully increase costs/complexity.

Jonathan

On Fri, Jun 25, 2021 at 4:55 PM Kumar Sankaran <ksankaran@...> wrote:

Hi Jonathan,

Regarding ASID and VMID support, we are mandating those only for the Server Extension within the OS-A platform and not for OS-A platform base.

We could consider a minimum of 12-bits for the ASID support for the server extension, but then most server platforms would in reality only implement 4 bits of ASID. So simply mandating ASID support without any additional requirement seemed to be the best approach.

Is there any particular number of ASID bits you suggest for the server extension? Same for the VMID as well.

 

Regarding your second comment, yes, I agree that implementations shouldn’t be allowed to “fake” ASID support by clearing the entire TLB when the ASID changes. This will be a big performance penalty. Having said that, the approach the platform spec takes is to have a minimal set of features for hardware and software compatibility without mandating any performance requirements. So we can add the below line as an Informational or Application Note below the requirement. If you have an alternative wording for this line, feel free to send that over.

 

Regards

Kumar

From: Jonathan Behrens <behrensj@...>
Sent: Friday, June 25, 2021 9:18 AM
To: Kumar Sankaran <ksankaran@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

Is this supposed to include more detail than just "ASID support" and "VMID support"? Not sure if those are just placeholder section headings for now, but I'd expect to at least see a requirement of how many bits are required for each.

 

Also not entirely sure how to specify it, but it would be really nice if implementations weren't allowed to have "fake" ASID support by just clearing the TLB whenever the ASID changes. That's the strategy QEMU uses (at least the last time I checked) and it has the impact of leading to worse performance than not having ASIDs at all.

 

Jonathan

 

On Thu, Jun 24, 2021 at 8:01 PM Kumar Sankaran via lists.riscv.org <ksankaran=ventanamicro.com@...> wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar

My take is we should avoid imposing this requirement altogether. Kernel software needs to support the no-ASID case, anyway, so we aren’t simplifying software by imposing this requirement.