Date   

Re: [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Kumar Sankaran
 

Hi Jonathan,

Regarding ASID and VMID support, we are mandating those only for the Server Extension within the OS-A platform and not for OS-A platform base.

We could consider a minimum of 12-bits for the ASID support for the server extension, but then most server platforms would in reality only implement 4 bits of ASID. So simply mandating ASID support without any additional requirement seemed to be the best approach.

Is there any particular number of ASID bits you suggest for the server extension? Same for the VMID as well.

 

Regarding your second comment, yes, I agree that implementations shouldn’t be allowed to “fake” ASID support by clearing the entire TLB when the ASID changes. This will be a big performance penalty. Having said that, the approach the platform spec takes is to have a minimal set of features for hardware and software compatibility without mandating any performance requirements. So we can add the below line as an Informational or Application Note below the requirement. If you have an alternative wording for this line, feel free to send that over.

 

Regards

Kumar

From: Jonathan Behrens <behrensj@...>
Sent: Friday, June 25, 2021 9:18 AM
To: Kumar Sankaran <ksankaran@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

Is this supposed to include more detail than just "ASID support" and "VMID support"? Not sure if those are just placeholder section headings for now, but I'd expect to at least see a requirement of how many bits are required for each.

 

Also not entirely sure how to specify it, but it would be really nice if implementations weren't allowed to have "fake" ASID support by just clearing the TLB whenever the ASID changes. That's the strategy QEMU uses (at least the last time I checked) and it has the impact of leading to worse performance than not having ASIDs at all.

 

Jonathan

 

On Thu, Jun 24, 2021 at 8:01 PM Kumar Sankaran via lists.riscv.org <ksankaran=ventanamicro.com@...> wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar





Next Platform HSC Meeting on Mon Jun 28 2021 8AM PST

Kumar Sankaran
 

Hi All,
The next platform HSC meeting is scheduled on Mon Jun 28th at 8AM PST.

Here are the details:

Agenda and minutes kept on the github wiki:
https://github.com/riscv/riscv-platform-specs/wiki

Here are the slides:
https://docs.google.com/presentation/d/1BnCDjSxAFqpRlEbyQ4zOrGqoOEp0aQzzoJgTxgDygLk/edit#slide=id.ge09029d4aa_0_0

Meeting info
Zoom meeting: https://zoom.us/j/2786028446
Passcode: 901897

Or iPhone one-tap :
US: +16465588656,,2786028466# or +16699006833,,2786028466# Or Telephone:
Dial(for higher quality, dial a number based on your current location):
US: +1 646 558 8656 or +1 669 900 6833
Meeting ID: 278 602 8446
International numbers available:
https://zoom.us/zoomconference?m=_R0jyyScMETN7-xDLLRkUFxRAP07A-_

Regards
Kumar


Re: [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Jonathan Behrens <behrensj@...>
 

Is this supposed to include more detail than just "ASID support" and "VMID support"? Not sure if those are just placeholder section headings for now, but I'd expect to at least see a requirement of how many bits are required for each.

Also not entirely sure how to specify it, but it would be really nice if implementations weren't allowed to have "fake" ASID support by just clearing the TLB whenever the ASID changes. That's the strategy QEMU uses (at least the last time I checked) and it has the impact of leading to worse performance than not having ASIDs at all.

Jonathan


On Thu, Jun 24, 2021 at 8:01 PM Kumar Sankaran via lists.riscv.org <ksankaran=ventanamicro.com@...> wrote:
This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar






Re: [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Greg Favor
 

On Fri, Jun 25, 2021 at 5:22 AM Philipp Tomsich <ptomsich@...> wrote:
I agree with Andrew.  What does it mean for caches to appear to software
in any particular way?  Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

The intent seems to be that caches shall appear (to software) to be non-aliasing.

ARM went through this issue with the variety of CPU designs that people have done (both outside and inside ARM).  In particular, as people went to larger and/or more power-efficient L1 caches, they implemented VIPT caches.  Which exposes VA aliasing to software.  So either software has to manage cache consistency, or hardware has to (i.e hardware has to make the VIPT cache appear to behave like a PIPT cache).  That's non-trivial hardware, but not unreasonable.  The result is that while ARM architecture allows VIVT, VIPT, and PIPT caches, in A class cores everything has moved to caches that appear to be PIPT while many actually are VIPT designs.

One can say that RVWMO compliance implies a requirement that caches appear to behave like PIPT caches.  But, given the growing use of VIPT caches, it's worth half a sentence to make clear the implications of RVWMO.
 
 
In any case, what's wrong with VIPT caches?

Depends on the internal organisation of a VIPT cache (and its size/number of index bits)...
The requirement clearly is that the cache appears to be non-aliasing, by whatever design choices.
A simple rewording of the original statement to this effect (instead of referencing PIPT) should address this.

Agreed.  I'll come up with some better words for Kumar to use.
 
 
> - Memory accesses by I/O masters can be coherent or non-coherent with
> respect to any hart-related caches

Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs?  IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

People tend to presume this in OS/A-class systems, but yes this should be explicitly stated.

Greg


Re: [PATCH 1/2] riscv-platform-spec: Real-time Clock to server extension

Jonathan Behrens <behrensj@...>
 

Any chance that we can ban EFI_UNSPECIFIED_TIMEZONE and/or require that time is always UTC?

Jonathan


On Fri, Jun 25, 2021 at 10:35 AM Abner Chang via lists.riscv.org <renba.chang=gmail.com@...> wrote:
From: Abner Chang <abner.chang@...>

RTC (Real-time Clock)
   Real-time clock is the server basic system peripheral to provide the real date/time information for server to manage the system date, time and time zones settings for different regions through the local POST time firmware utility, NTP or the remote management such as Redfish.

Signed-off-by: Abner Chang <renba.chang@...>
Signed-off-by: Abner Chang <abner.chang@...>
---
 riscv-platform-spec.adoc | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..d611d69 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -442,9 +442,9 @@ The UEFI run time services listed below are required to be implemented.
 |SetVariable               | 8.2        | A dedicated storage for firmware is
 required so that there is no conflict in access by both firmware and the OS.
 |QueryVariableInfo         | 8.2        |
-|GetTime                   | 8.3        | RTC Access by the OS
-|SetTime                   | 8.3        | If it is not possible to set the RTC,
-the SetTime() can return an error.
+|GetTime                   | 8.3        | RTC Access by the OS and firmware
+|SetTime                   | 8.3        | RTC configured by the OS and
+firmware
 |GetWakeupTime             | 8.3        | Interface is required to be
 implemented but it can return EFI_UNSUPPORTED.
 |SetWakeupTime             | 8.3        | Interface is required to be
@@ -469,6 +469,17 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
 ** Platforms are required to delegate the supervisor timer interrupt to 'S'
 mode. If the 'H' extension is implemented then the platforms are required to
 delegate the virtual supervisor timer interrupt to 'VS' mode.
+
+* Real-time Clock (RTC)
+The Real-time clock must be provided to the server extension platform to
+facilitate the time management (Date, time and the time zone) and RTC wake up
+when the server is in the power down state for the server manageability.
+The GetTime()and SetTime() UEFI runtime services must be implemented by
+firmware to incorporate with RTC and flexibly support RTC use cases.
+GetWakeupTime() and SetWakeupTime() UEFI runtime services are required to be
+implemented but it can return EFI_UNSUPPORTED if the wake up from RTC is not
+supported.
+
 * PCI-E

 ==== Secure Boot
--
2.19.0.windows.1







[PATCH 2/2] contributors: Add Abner as contributor

Abner Chang
 

From: Abner Chang <abner.chang@...>

Signed-off-by: Abner Chang <renba.chang@...>
Signed-off-by: Abner Chang <abner.chang@...>
---
contributors.adoc | 1 +
1 file changed, 1 insertion(+)

diff --git a/contributors.adoc b/contributors.adoc
index a1ea5ef..57ac8bc 100644
--- a/contributors.adoc
+++ b/contributors.adoc
@@ -30,6 +30,7 @@ Sunil V L,
Rahul Pathak,
Mayuresh Chitale,
Paul Donahue
+Abner Chang

If you have contributed and are not listed, do let us know. We haven't
forgotten you, but we may have forgotten to edit this list.
--
2.19.0.windows.1


[PATCH 1/2] riscv-platform-spec: Real-time Clock to server extension

Abner Chang
 

From: Abner Chang <abner.chang@...>

RTC (Real-time Clock)
Real-time clock is the server basic system peripheral to provide the real date/time information for server to manage the system date, time and time zones settings for different regions through the local POST time firmware utility, NTP or the remote management such as Redfish.

Signed-off-by: Abner Chang <renba.chang@...>
Signed-off-by: Abner Chang <abner.chang@...>
---
riscv-platform-spec.adoc | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..d611d69 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -442,9 +442,9 @@ The UEFI run time services listed below are required to be implemented.
|SetVariable | 8.2 | A dedicated storage for firmware is
required so that there is no conflict in access by both firmware and the OS.
|QueryVariableInfo | 8.2 |
-|GetTime | 8.3 | RTC Access by the OS
-|SetTime | 8.3 | If it is not possible to set the RTC,
-the SetTime() can return an error.
+|GetTime | 8.3 | RTC Access by the OS and firmware
+|SetTime | 8.3 | RTC configured by the OS and
+firmware
|GetWakeupTime | 8.3 | Interface is required to be
implemented but it can return EFI_UNSUPPORTED.
|SetWakeupTime | 8.3 | Interface is required to be
@@ -469,6 +469,17 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
** Platforms are required to delegate the supervisor timer interrupt to 'S'
mode. If the 'H' extension is implemented then the platforms are required to
delegate the virtual supervisor timer interrupt to 'VS' mode.
+
+* Real-time Clock (RTC)
+The Real-time clock must be provided to the server extension platform to
+facilitate the time management (Date, time and the time zone) and RTC wake up
+when the server is in the power down state for the server manageability.
+The GetTime()and SetTime() UEFI runtime services must be implemented by
+firmware to incorporate with RTC and flexibly support RTC use cases.
+GetWakeupTime() and SetWakeupTime() UEFI runtime services are required to be
+implemented but it can return EFI_UNSUPPORTED if the wake up from RTC is not
+supported.
+
* PCI-E

==== Secure Boot
--
2.19.0.windows.1


Re: [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Philipp Tomsich
 



On Fri, Jun 25, 2021 at 2:01 PM Daniel Lustig <dlustig@...> wrote:
On 6/25/2021 4:04 AM, Kumar Sankaran wrote:
> Hi Andrew,
>
> Agree with your comment on the RVWMO memory model requirement. Since this
> requirement is for the OS-A platform, providing an additional level of
> clarity will help since all implementations will be having a cache
> sub-system. So how about the below wording?
>
> - All harts must adhere to the RVWMO memory model. Consequently, any
> hart-related caches must all be hardware coherent with respect to each
> other, and must appear to software as physically-indexed, physically-tagged
> (PIPT) caches.

I agree with Andrew.  What does it mean for caches to appear to software
in any particular way?  Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

The intent seems to be that caches shall appear (to software) to be non-aliasing.
 
In any case, what's wrong with VIPT caches?

Depends on the internal organisation of a VIPT cache (and its size/number of index bits)...
The requirement clearly is that the cache appears to be non-aliasing, by whatever design choices.
A simple rewording of the original statement to this effect (instead of referencing PIPT) should address this.
 
> - Memory accesses by I/O masters can be coherent or non-coherent with
> respect to any hart-related caches

Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs?  IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

Dan

> Regards
>
> Kumar
>
> *From:* Andrew Waterman <andrew@...>
> *Sent:* Thursday, June 24, 2021 11:24 PM
> *To:* Kumar Sankaran <ksankaran@...>
> *Cc:* tech-unixplatformspec@...; Daniel Lustig <
> dlustig@...>
> *Subject:* Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency
> and ASID Requirements for OS-A platform
>
>
>
> + Dan
>
>
>
> I'm not sure it makes sense to include the cache coherence requirement--not
> because we wish to permit incoherent caches, but because it's a constraint
> specified at the wrong level of abstraction.  The fundamental requirement
> is that the harts adhere to the RVWMO memory model.  The concept of caches
> and the concept of coherence are implementation details.  (Yes, in
> practice, if you have caches, you'll need cache coherence to implement
> RVWMO, but that doesn't mean it should be in a list of requirements.)
>
>
>
> On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...>
> wrote:
>
> This patch adds the following
> Cache coherency and ASID requirements
> Interrupt Controller and PMU chapter sub-sections for OS-A base and
> Server Extension
>
> diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
> index 87ea6d5..b985f50 100644
> --- a/riscv-platform-spec.adoc
> +++ b/riscv-platform-spec.adoc
> @@ -88,8 +88,12 @@ The M platform has the following extensions:
>  * ISA Profile
>  ** The OS-A platform is required to comply with the RVA22 profile.
>  * Cache Coherency
> -* PMU
> -* ASID
> +** All HART related caches must be hardware coherent and must appear to
> +software as Physically Indexed, Physically Tagged (PIPT) caches
> +** Memory accesses by I/O masters can be coherent or non-coherent with
> respect
> +to the HART related caches
> +
> +==== PMU
>
>  ==== Debug
>  The OS-A base platform requirements are -
> @@ -287,10 +291,12 @@ base with the additional requirements as below.
>  ==== Architecture
>  The platforms which conform to server extension are required to implement +
>
> -- RISC-V Hypervisor-level H Instruction-Set Extensions
> -- IOMMU with support for memory resident interrupt files
> -- PMU
> -- ASID
> +- RV64 support
> +- RISC-V H ISA extension
> +- ASID support
> +- VMID support
> +
> +==== PMU
>
>  ==== Debug
>  The OS-A server platform requirements are all of the base above plus:
> @@ -305,6 +311,8 @@ above.
>  respect to all harts connected to the DM
>    * Rationale: Debuggers must be able to view memory coherently
>
> +==== Interrupt Controller
> +
>  ==== Boot and Runtime Requirements
>  =====  Firmware
>  The boot and system firmware for the RV64I server platforms required to be
>






Re: [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Daniel Lustig
 

On 6/25/2021 4:04 AM, Kumar Sankaran wrote:
Hi Andrew,

Agree with your comment on the RVWMO memory model requirement. Since this
requirement is for the OS-A platform, providing an additional level of
clarity will help since all implementations will be having a cache
sub-system. So how about the below wording?

- All harts must adhere to the RVWMO memory model. Consequently, any
hart-related caches must all be hardware coherent with respect to each
other, and must appear to software as physically-indexed, physically-tagged
(PIPT) caches.
I agree with Andrew. What does it mean for caches to appear to software
in any particular way? Unless you're talking about something like CMOs,
aren't caches entirely transparent to software?

In any case, what's wrong with VIPT caches?

- Memory accesses by I/O masters can be coherent or non-coherent with
respect to any hart-related caches
Since we're aiming for clarity here, if we do keep this wording (and
I'm not saying we should), can we also clarify whether this implies
anything about the coherence PMAs? IIUC the PMAs represent the harts'
perspective on memory, so those would generally (or possibly always
in the OS-A platform?) mark main memory regions as coherent, even if
external devices may access that same memory non-coherently?

Dan

Regards

Kumar

*From:* Andrew Waterman <andrew@...>
*Sent:* Thursday, June 24, 2021 11:24 PM
*To:* Kumar Sankaran <ksankaran@...>
*Cc:* tech-unixplatformspec@...; Daniel Lustig <
dlustig@...>
*Subject:* Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency
and ASID Requirements for OS-A platform



+ Dan



I'm not sure it makes sense to include the cache coherence requirement--not
because we wish to permit incoherent caches, but because it's a constraint
specified at the wrong level of abstraction. The fundamental requirement
is that the harts adhere to the RVWMO memory model. The concept of caches
and the concept of coherence are implementation details. (Yes, in
practice, if you have caches, you'll need cache coherence to implement
RVWMO, but that doesn't mean it should be in a list of requirements.)



On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...>
wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
* ISA Profile
** The OS-A platform is required to comply with the RVA22 profile.
* Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with
respect
+to the HART related caches
+
+==== PMU

==== Debug
The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
==== Architecture
The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

==== Debug
The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
respect to all harts connected to the DM
* Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
==== Boot and Runtime Requirements
===== Firmware
The boot and system firmware for the RV64I server platforms required to be


Re: [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Kumar Sankaran
 

Hi Andrew,

Agree with your comment on the RVWMO memory model requirement. Since this requirement is for the OS-A platform, providing an additional level of clarity will help since all implementations will be having a cache sub-system. So how about the below wording?

- All harts must adhere to the RVWMO memory model. Consequently, any hart-related caches must all be hardware coherent with respect to each other, and must appear to software as physically-indexed, physically-tagged (PIPT) caches.

- Memory accesses by I/O masters can be coherent or non-coherent with respect to any hart-related caches 

Regards

Kumar

From: Andrew Waterman <andrew@...>
Sent: Thursday, June 24, 2021 11:24 PM
To: Kumar Sankaran <ksankaran@...>
Cc: tech-unixplatformspec@...; Daniel Lustig <dlustig@...>
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

 

+ Dan

 

I'm not sure it makes sense to include the cache coherence requirement--not because we wish to permit incoherent caches, but because it's a constraint specified at the wrong level of abstraction.  The fundamental requirement is that the harts adhere to the RVWMO memory model.  The concept of caches and the concept of coherence are implementation details.  (Yes, in practice, if you have caches, you'll need cache coherence to implement RVWMO, but that doesn't mean it should be in a list of requirements.)

 

On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...> wrote:

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar





Re: [PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Andrew Waterman
 

+ Dan

I'm not sure it makes sense to include the cache coherence requirement--not because we wish to permit incoherent caches, but because it's a constraint specified at the wrong level of abstraction.  The fundamental requirement is that the harts adhere to the RVWMO memory model.  The concept of caches and the concept of coherence are implementation details.  (Yes, in practice, if you have caches, you'll need cache coherence to implement RVWMO, but that doesn't mean it should be in a list of requirements.)

On Thu, Jun 24, 2021 at 7:00 PM Kumar Sankaran <ksankaran@...> wrote:
This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
 * ISA Profile
 ** The OS-A platform is required to comply with the RVA22 profile.
 * Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

 ==== Debug
 The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
 ==== Architecture
 The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

 ==== Debug
 The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
 respect to all harts connected to the DM
   * Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
 ==== Boot and Runtime Requirements
 =====  Firmware
 The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar






[PATCH 1/1] Cache Coherency and ASID Requirements for OS-A platform

Kumar Sankaran
 

This patch adds the following
Cache coherency and ASID requirements
Interrupt Controller and PMU chapter sub-sections for OS-A base and
Server Extension

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 87ea6d5..b985f50 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -88,8 +88,12 @@ The M platform has the following extensions:
* ISA Profile
** The OS-A platform is required to comply with the RVA22 profile.
* Cache Coherency
-* PMU
-* ASID
+** All HART related caches must be hardware coherent and must appear to
+software as Physically Indexed, Physically Tagged (PIPT) caches
+** Memory accesses by I/O masters can be coherent or non-coherent with respect
+to the HART related caches
+
+==== PMU

==== Debug
The OS-A base platform requirements are -
@@ -287,10 +291,12 @@ base with the additional requirements as below.
==== Architecture
The platforms which conform to server extension are required to implement +

-- RISC-V Hypervisor-level H Instruction-Set Extensions
-- IOMMU with support for memory resident interrupt files
-- PMU
-- ASID
+- RV64 support
+- RISC-V H ISA extension
+- ASID support
+- VMID support
+
+==== PMU

==== Debug
The OS-A server platform requirements are all of the base above plus:
@@ -305,6 +311,8 @@ above.
respect to all harts connected to the DM
* Rationale: Debuggers must be able to view memory coherently

+==== Interrupt Controller
+
==== Boot and Runtime Requirements
===== Firmware
The boot and system firmware for the RV64I server platforms required to be

--
Regards
Kumar


Re: [PATCH 1/1] RAS features for OS-A platform server extension

Kumar Sankaran
 

Patch merged with all the changes requested.

 

Regards

Kumar

From: Greg Favor <gfavor@...>
Sent: Wednesday, June 23, 2021 11:02 PM
To: Abner Chang <renba.chang@...>
Cc: Kumar Sankaran <ksankaran@...>; tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] [PATCH 1/1] RAS features for OS-A platform server extension

 

On Wed, Jun 23, 2021 at 8:25 PM Abner Chang <renba.chang@...> wrote:

Please review the below sentence. 

If the RAS event is configured as the firmware first model, the platform should be able to trigger the higest priority of M-mode interrupt to all HARTs in the physical RV processor. This prevents the subsequent RAS errors are propagated by other HARTs that access the problematic hardware (PCIe, Memory, I/O and etc.)

 

Note that the priority of any RAS interrupts would be software configurable in the interrupt controller.  Also note that there are other common techniques for preventing the propagation of errors and for isolating the impact of errors (e.g. precise hart exceptions on attempted use of corrupted data, data poisoning, I/O flow termination, ...).

 

One question:

Besides those RAS events come from the interrupt controller,

 

In a typical enterprise-class RAS architecture, "error events" are logged in RAS registers, which then optionally generate RAS interrupt requests.  These then go to the system interrupt controller, which prioritizes and routes requests to appropriate harts.  

 

how about the HART or Memory RAS events?

 

One would typically have RAS registers (for logging and reporting errors) spread around the system, ideally at all points in the system where errors can be detected and at all points where corrupted data can be consumed.  

 

Are those RAS events in the scope of exception? or they would be also routed to  interrupt controller?

 

RAS errors generally result in RAS interrupts, but when a hart tries to consume corrupted data, the ideal RAS behavior is for the hart to take a precise exception on the load instruction that is trying to consume corrupted data.

 

Or we don't have to worry about this, RAS TG will have the solution?

 

All this would be covered by a proper RAS architecture (to hopefully be developed by a TG next year).

 

Greg 


Re: [PATCH 1/1] Initial commit of PLIC

Alistair Francis
 

On Sun, 2021-06-20 at 21:32 +0800, Abner Chang wrote:
From: Abner Chang <abner.chang@...>

This is the commit for creating the patches for
widely review in Platform Spec HSC task group

Signed-off-by: Abner Chang <abner.chang@...>
Reviewed-by: Alistair Francis <alistair.francis@...>

Alistair

---
 riscv-plic.adoc | 306 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 306 insertions(+)
 create mode 100644 riscv-plic.adoc

diff --git a/riscv-plic.adoc b/riscv-plic.adoc
new file mode 100644
index 0000000..b770e0e
--- /dev/null
+++ b/riscv-plic.adoc
@@ -0,0 +1,306 @@
+= *RISC-V Platform-Level Interrupt Controller Specification*
+
+== Copyright and license information
+
+This RISC-V PLIC specification is
+
+[%hardbreaks]
+(C) 2017 Drew Barbier <drew@...>
+(C) 2018-2019 Palmer Dabbelt <palmer@...>
+(C) 2019 Abner Chang, Hewlett Packard Enterprise <abner.chang@...>
+
+It is licensed under the Creative Commons Attribution 4.0
International
+License (CC-BY 4.0).  The full license text is available at
+https://creativecommons.org/licenses/by/4.0/.
+
+== Introduction
+
+This document contains the RISC-V platform-level interrupt controller
(PLIC)
+specification, which defines an interrupt controller specifically
designed to
+work in the context of RISC-V systems.  The PLIC multiplexes various
device
+interrupts onto the external interrupt lines of Hart contexts, with
+hardware support for interrupt priorities. +
+This specification defines the general PLIC architecture and operation
parameters.
+The PLIC which claimed as PLIC-Compliant standard PLIC should follow
the
+implementations mentioned in sections below.
+
+.Figure 1 RISC-V PLIC Interrupt Architecture Block Diagram
+image::Images/PLIC.jpg[GitHub,1000,643,
link=https://github.com/riscv/riscv-plic-spec/blob/master/Images/PLIC.jpg
]
+
+== RISC-V PLIC Operation Parameters
+
+General PLIC operation parameter register blocks are defined in this
spec, those are: +
+
+- *Interrupt Priorities registers:* +
+   The interrupt priority for each interrupt source. +
+
+- *Interrupt Pending Bits registers:* +
+   The interrupt pending status of each interrupt source. +
+  
+- *Interrupt Enables registers:* +
+   The enablement of interrupt source of each context. +
+
+- *Priority Thresholds registers:* +
+   The interrupt priority threshold of each context. +
+
+- *Interrupt Claim registers:* +
+   The register to acquire interrupt source ID of each context. +
+  
+- *Interrupt Completion registers:* +
+   The register to send interrupt completion message to the associated
gateway. +
+
++
+
+Below is the figure of PLIC Operation Parameter Block Diagram,
+
+.Figure 2 PLIC Operation Parameter Block Diagram
+image::Images/PLICArch.jpg[GitHub,
link=https://github.com/riscv/riscv-plic-spec/blob/master/Images/PLICArch.jpg
]
+
+== Memory Map
+
+The `base address of PLIC Memory Map` is platform implementation-
specific.
+
+*PLIC Memory Map*
+
+       base + 0x000000: Reserved (interrupt source 0 does not exist)
+       base + 0x000004: Interrupt source 1 priority
+       base + 0x000008: Interrupt source 2 priority
+       ...
+       base + 0x000FFC: Interrupt source 1023 priority
+       base + 0x001000: Interrupt Pending bit 0-31
+       base + 0x00107C: Interrupt Pending bit 992-1023
+       ...     
+       base + 0x002000: Enable bits for sources 0-31 on context 0
+       base + 0x002004: Enable bits for sources 32-63 on context 0
+       ...
+       base + 0x00207F: Enable bits for sources 992-1023 on context 0
+       base + 0x002080: Enable bits for sources 0-31 on context 1
+       base + 0x002084: Enable bits for sources 32-63 on context
1     
+       ...
+       base + 0x0020FF: Enable bits for sources 992-1023 on context 1
+       base + 0x002100: Enable bits for sources 0-31 on context 2
+       base + 0x002104: Enable bits for sources 32-63 on context
2     
+       ...
+       base + 0x00217F: Enable bits for sources 992-1023 on context 2
+       ...
+       base + 0x1F1F80: Enable bits for sources 0-31 on context 15871
+       base + 0x1F1F84: Enable bits for sources 32-63 on context
15871         
+       base + 0x1F1FFF: Enable bits for sources 992-1023 on context
15871
+       ...     
+       base + 0x1FFFFC: Reserved
+       base + 0x200000: Priority threshold for context 0
+       base + 0x200004: Claim/complete for context 0
+       base + 0x200008: Reserved
+       ...
+       base + 0x200FFC: Reserved
+       base + 0x201000: Priority threshold for context 1
+       base + 0x201004: Claim/complete for context 1
+       ...
+       base + 0x3FFE000: Priority threshold for context 15871
+       base + 0x3FFE004: Claim/complete for context 15871
+       base + 0x3FFE008: Reserved
+       ...     
+       base + 0x3FFFFFC: Reserved
+       
+Sections below describe the control register blocks of PLIC operation
parameters.
+
+== Register Width
+
+The memory map register width is in 32-bit.
+
+== Interrupt Priorities
+
+If PLIC supports Interrupt Priorities, then each PLIC interrupt source
can be assigned a priority by writing to its 32-bit
+memory-mapped `priority` register.  A priority value of 0 is reserved
to mean ''never interrupt'' and effectively
+disables the interrupt. Priority 1 is the lowest active priority while
the maximum level of priority depends on
+PLIC implementation. Ties between global interrupts of the same
priority are broken by the Interrupt ID; interrupts
+with the lowest ID have the highest
+effective priority. +
+ +
+The base address of Interrupt Source Priority block within PLIC Memory
Map region is fixed at 0x000000.
+
+[cols="15%,20%,20%,45%"]
+|===
+| *PLIC Register Block Name*| *Function*|*Register Block Size in
Byte*| *Description*
+|Interrupt Source Priority
+|Interrupt Source Priority #0 to #1023
+|1024 * 4 = 4096(0x1000) bytes
+|This is a continuously memory block which contains PLIC Interrupt
Source Priority. Total 1024 Interrupt Source Priority
+in this memory block. Interrupt Source Priority #0 is reserved which
indicates it does not exist.
+|===
+
+*PLIC Interrupt Source Priority Memory Map* +
+
+       0x000000: Reserved (interrupt source 0 does not exist)
+       0x000004: Interrupt source 1 priority
+       0x000008: Interrupt source 2 priority
+       ...
+       0x000FFC: Interrupt source 1023 priority
+
+== Interrupt Pending Bits
+
+The current status of the interrupt source pending bits in the PLIC
core can be
+read from the pending array, organized as 32-bit register.  The
pending bit
+for interrupt ID N is stored in bit (N mod 32) of word (N/32).  Bit 0
+of word 0, which represents the non-existent interrupt source 0, is
hardwired
+to zero.
+
+A pending bit in the PLIC core can be cleared by setting the
associated enable
+bit then performing a claim. +
+ +
+The base address of Interrupt Pending Bits block within PLIC Memory
Map region is fixed at 0x001000.
+
+[cols="15%,20%,20%,45%"]
+|===
+| *PLIC Register Block Name* | *Function*|*Register Block Size in
Byte*| *Description*
+|Interrupt Pending Bits
+|Interrupt Pending Bit of Interrupt Source #0 to #N
+|1024 / 8 = 128(0x80) bytes
+|This is a continuously memory block contains PLIC Interrupt Pending
Bits. Each Interrupt Pending Bit occupies 1-bit from this register
block.
+|===
+
+*PLIC Interrupt Pending Bits Memory Map* +
+
+       0x001000: Interrupt Source #0 to #31 Pending Bits
+       ...
+       0x00107C: Interrupt Source #992 to #1023 Pending Bits
+
+
+== Interrupt Enables
+
+Each global interrupt can be enabled by setting the corresponding bit
in the
+`enables` register. The `enables` registers are accessed as a
contiguous array
+of 32-bit registers, packed the same way as the `pending` bits. Bit 0
of enable
+register 0 represents the non-existent interrupt ID 0 and is hardwired
to 0.
+PLIC has 15872 Interrupt Enable blocks for the contexts. The `context`
is referred
+to the specific privilege mode in the specific Hart of specific RISC-V
processor
+instance. How PLIC organizes interrupts for the contexts (Hart and
privilege mode)
+is out of RISC-V PLIC specification scope, however it must be spec-out
in vendor's
+PLIC specification. +
+ +
+The base address of Interrupt Enable Bits block within PLIC Memory Map
region is fixed at 0x002000. +
+ +
+[cols="15%,20%,20%,45%"]
+|===
+| *PLIC Register Block Name* | *Function*|*Register Block Size in
Byte*| *Description*
+|Interrupt Enable Bits
+|Interrupt Enable Bit of Interrupt Source #0 to #1023 for 15872
contexts
+|(1024 / 8) * 15872 = 2031616(0x1f0000) bytes
+|This is a continuously memory block contains PLIC Interrupt Enable
Bits of 15872 contexts.
+Each Interrupt Enable Bit occupies 1-bit from this register block and
total 15872 Interrupt
+Enable Bit blocks
+|===
+
+*PLIC Interrupt Enable Bits Memory Map* +
+
+       0x002000: Interrupt Source #0 to #31 Enable Bits on context 0
+       ...
+       0x00207F: Interrupt Source #992 to #1023 Enable Bits on context
0
+       0x002080: Interrupt Source #0 to #31 Enable Bits on context 1
+       ...
+       0x0020FF: Interrupt Source #992 to #1023 Enable Bits on context
1
+       0x002100: Interrupt Source #0 to #31 Enable Bits on context 2
+       ...
+       0x00217F: Interrupt Source #992 to #1023 Enable Bits on context
2
+       0x002180: Interrupt Source #0 to #31 Enable Bits on context 3
+       ...
+       0x0021FF: Interrupt Source #992 to #1023 Enable Bits on context
3
+       ...
+       ...
+       ...
+       0x1F1F80: Interrupt Source #0 to #31 on context 15871   
+       ...     
+       0x1F1F80: Interrupt Source #992 to #1023 on context 15871
+       
+== Priority Thresholds
+
+PLIC provides context based `threshold register` for the settings of a
interrupt priority
+threshold of each context. The `threshold register` is a WARL field.
The PLIC will mask all
+PLIC interrupts of a priority less than or equal to `threshold`.  For
example,
+a`threshold` value of zero permits all interrupts with non-zero
priority. +
+ +
+The base address of Priority Thresholds register block is located at
4K alignement starts
+from offset 0x200000.
+
+[cols="15%,20%,20%,45%"]
+|===
+| *PLIC Register Block Name* | *Function*|*Register Block Size in
Byte*| *Description*
+|Priority Threshold
+|Priority Threshold for 15872 contexts
+|4096 * 15872 = 65011712(0x3e00000) bytes
+|This is the register of Priority Thresholds setting for each conetxt
+|===
+
+*PLIC Interrupt Priority Thresholds Memory Map* +
+
+       0x200000: Priority threshold for context 0
+       0x201000: Priority threshold for context 1
+       0x202000: Priority threshold for context 2
+       0x203000: Priority threshold for context 3
+       ...
+       ...
+       ...
+       0x3FFF000: Priority threshold for context 15871
+       
+== Interrupt Claim Process
+
+The PLIC can perform an interrupt claim by reading the
`claim/complete`
+register, which returns the ID of the highest priority pending
interrupt or
+zero if there is no pending interrupt.  A successful claim will also
atomically
+clear the corresponding pending bit on the interrupt source. +
+The PLIC can perform a claim at any time and the claim operation is
not affected
+by the setting of the priority threshold register. +
+The Interrupt Claim Process register is context based and is located
at
+(4K alignement + 4) starts from offset 0x200000.
+
+[cols="15%,20%,20%,45%"]
+|===
+| *PLIC Register Block Name* | *Function*|*Register Block Size in
Byte*| *Description*
+|Interrupt Claim Register
+|Interrupt Claim Process for 15872 contexts
+|4096 * 15872 = 65011712(0x3e00000) bytes
+|This is the register used to acquire interrupt ID for each conetxt
+|===
+
+*PLIC Interrupt Claim Process Memory Map* +
+
+       0x200004: Interrupt Claim Process for context 0
+       0x201004: Interrupt Claim Process for context 1
+       0x202004: Interrupt Claim Process for context 2
+       0x203004: Interrupt Claim Process for context 3
+       ...
+       ...
+       ...
+       0x3FFF004: Interrupt Claim Process for context 15871
+       
+## Interrupt Completion
+
+The PLIC signals it has completed executing an interrupt handler by
writing the
+interrupt ID it received from the claim to the `claim/complete`
register.  The
+PLIC does not check whether the completion ID is the same as the last
claim ID
+for that target.  If the completion ID does not match an interrupt
source that
+is currently enabled for the target, the completion is silently
ignored. +
+The Interrupt Completion registers are context based and located at
the same address
+with Interrupt Claim Process register, which is at (4K alignement + 4)
starts from
+offset 0x200000.
+ +
+[cols="15%,20%,20%,45%"]
+|===
+| *PLIC Register Block Name* | *Registers*|*Register Block Size in
Byte*| *Description*
+|Interrupt Completion Register
+|Interrupt Completion  for 15872 contexts
+|4096 * 15872 = 65011712(0x3e00000) bytes
+|This is register to write to complete Interrupt process
+|===
+
+*PLIC Interrupt Completion Memory Map* +
+
+       0x200004: Interrupt Completion for context 0
+       0x201004: Interrupt Completion for context 1
+       0x202004: Interrupt Completion for context 2
+       0x203004: Interrupt Completion for context 3
+       ...
+       ...
+       ...
+       0x3FFF004: Interrupt Completion for context 15871
+


Re: [PATCH 1/1] RAS features for OS-A platform server extension

Greg Favor
 

On Wed, Jun 23, 2021 at 8:25 PM Abner Chang <renba.chang@...> wrote:
Please review the below sentence. 
If the RAS event is configured as the firmware first model, the platform should be able to trigger the higest priority of M-mode interrupt to all HARTs in the physical RV processor. This prevents the subsequent RAS errors are propagated by other HARTs that access the problematic hardware (PCIe, Memory, I/O and etc.)

Note that the priority of any RAS interrupts would be software configurable in the interrupt controller.  Also note that there are other common techniques for preventing the propagation of errors and for isolating the impact of errors (e.g. precise hart exceptions on attempted use of corrupted data, data poisoning, I/O flow termination, ...).
 
One question:
Besides those RAS events come from the interrupt controller,

In a typical enterprise-class RAS architecture, "error events" are logged in RAS registers, which then optionally generate RAS interrupt requests.  These then go to the system interrupt controller, which prioritizes and routes requests to appropriate harts.  
 
how about the HART or Memory RAS events?

One would typically have RAS registers (for logging and reporting errors) spread around the system, ideally at all points in the system where errors can be detected and at all points where corrupted data can be consumed.  
 
Are those RAS events in the scope of exception? or they would be also routed to  interrupt controller?

RAS errors generally result in RAS interrupts, but when a hart tries to consume corrupted data, the ideal RAS behavior is for the hart to take a precise exception on the load instruction that is trying to consume corrupted data.
 
Or we don't have to worry about this, RAS TG will have the solution?

All this would be covered by a proper RAS architecture (to hopefully be developed by a TG next year).

Greg 


Re: [PATCH 1/1] RAS features for OS-A platform server extension

Greg Favor
 

On Wed, Jun 23, 2021 at 7:17 PM Abner Chang <renba.chang@...> wrote:
Do we need to define what is the RAS error signals output to the interrupt controller? (The signal could be classified by the error severities such as CE, UC_FATAL, UC_NONFATAL or classified by the RAS error categories such as RAS_MEM_ERROR, RAS_IO_ERROR and etc.)

This just starts down the path of defining a small bit of a RAS architecture - which we shouldn't do without developing a full RAS architecture is developed (next year).
 
I think we can just leave it to RAS TG because we just define what server platform needs on RAS, right?

Exactly.
 
Without the hardware signal to trigger TEE. The alternative would be triggering the M-mode exception and jump to TEE in the M-mode exception handler?
So the scenario of triggering TEE would be,
For software management mode interface:
     S-mode-> sbi ecall to M-mode->TEE jump vector->TEE

Effectively the same as with ARM.
 
For the hardware management mode interface:
Hardware interrupt -> M-mode handler-> TEE jump vector->TEE
What firmware or software resides in TEE is implementation-specific. For example on edk2, we will load the management mode core into TEE.
I am just trying to get more understanding of the future design of TEE on RV.

I think the tech-tee TG has done some pieces of things around TEE, but I'm not sure what (and certainly there isn't anything heading to ratification this year).

Greg


Re: [PATCH 1/1] RAS features for OS-A platform server extension

Abner Chang
 



Kumar Sankaran <ksankaran@...> 於 2021年6月24日 週四 上午5:11寫道:
On Wed, Jun 23, 2021 at 9:00 AM Greg Favor <gfavor@...> wrote:
>
> On Wed, Jun 23, 2021 at 7:59 AM Abner Chang <renba.chang@...> wrote:
>>>
>>> Yes.  Which is just a software matter of configuring the interrupt controller accordingly.
>>
>> Does this mean the interrupt controller would integrate all RAS events (HART, PCI, I/O, memory and etc.)?
>> Or there would be a separate hardware box that manages all RAS error events, and maybe some error signals output from that box and connected to the interrupt controller? The interrupt controller just provides the mechanism to morph those error signals to FFM or OSF interrupt?
>
>
> To the extent that "RAS interrupts" are literally that, i.e. interrupt request signals, then they go to the system interrupt controller just like all other interrupt request signals.  (Some system designs might also have a "platform microcontroller" that has its own local interrupt controller and may receive some of these interrupt request signals.)
>
> Maybe part of what you're trying to get at is that RAS error events in many architectures get logged in and reported from hardware RAS registers.  RAS registers "report" errors by outputting RAS interrupt request signals.  Software then comes back around and reads the RAS registers to gather info about logged errors.
>
>>>>
>>>> Can we summarize the requirement to
>>>>
>>>> - RAS errors should be capable of interrupting TEE.
>>
>> This is ok for now because there is no hardware signal defined for triggering TEE right? I have more comments on this below.
>
>
> I expect RV will have similarities to ARM in this matter - and ARM doesn't have a hardware signal defined for triggering TEE either (and hasn't felt the need to define such).
>
>>>
>>>
>>> This implies a requirement to have a TEE - and defining what constitutes a compliant TEE in the platform spec.  Btw, what distinguishes the TEE from "firmware"?
>>
>> Please correct me on ARM part if I am wrong.
>> The equivalent mechanism to TEE is SMM on X86 and TZ on ARM. I don't quite understand how ARM TZ works, however on X86 system, all cores are brought to SMM environment when SMI is triggered. ARM has the equivalent event which is SMC, right?
>
>
> Neither ARM nor RISC-V has a direct equivalent of SMM.  So I'll pick on what ARM has - which is rather like RV.  At a hardware level ARM has EL3 and Secure ELx, and RV as M-mode and secure partitions of S/U-mode (using PMP).  At a software level one has a Secure monitor running in EL3/M-mode and tbd whether other parts run in SELx/partitions.  TZ as a TEE is a combination of these hardware features and the secure software that runs on it.  ARM TZ doesn't specify the actual software TEE, it just provides the hardware architectural features and framework for creating and running a TEE.  There is no one standard ARM TEE (although ARM has developed their ATF as a reference secure boot flow; although maybe it has expanded in scope in recent years?).
>
> In short, RV first needs to define, develop, and specify a software TEE.  The hardware components are falling into place (e.g. PMP, ePMP, Zkr), and OpenSBI is working towards supporting secure partitions.  So, until there is a concrete RISC-V TEE standard (or even a standard framework), we shouldn't be stating requirements tied with having a TEE.  Also keep in mind that things like secure boot will be required in the Server extension - which is part of the overall topic of TEE.
>
>>
>> The above is called management mode (MM) which is defined in UEFI PI spec. MM has the highest privilege than CR0 on X86 and EL3 on ARM. The MM is OS agnostic and the MM event halts any processes and gets the core into management mode to run the firmware code. The environment of MM (data and code) can only be accessed when the core is in MM. Firmware always uses this for the secure stuff, power management, and of course the RAS.
>
>
> What you describe, for RV, is M-mode - a pretty direct analog of ARM EL3.
>
>>
>>
>> I would like to add one more thing to the RAS requirement but I don't know how to describe it properly because seems we don't have the MM event on RISC-V such as SMI and SMC which can bring the system to MM.
>
>
> RV has ECALL, just like ARM has SMC.
>
>>
>> So there are two scenarios for RAS on the firmware first model.
>> - If the platform doesn't have TEE and the hardware event to trigger TEE:
>>   If the RAS event is configured to firmware first mode, the platform should be able to trigger M-Mode exception to all harts in the physical processor. This prevents the subsequent RAS error propagated by other harts that access the problematic hardware (PCI, memory and etc.)
>>
>> - If the platform has TEE and the hardware event to trigger TEE:
>>     If the RAS event is configured to firmware first mode, the platform should be able to trigger TEE event to all harts in the physical processor and bring all harts into TEE. This prevents the subsequent RAS error propagated by other cores which access the problematic hardware (PCI, memory and etc.)
>
>
> I think part of what complicates this discussion is the nebulous nature of what exactly is the "TEE" in any given architecture.  At a hardware level x86/ARM/RV have SMM/EL3/M-mode and they have ways to "call" into that secure environment.  The software TEE architecture is what is rather nebulous.  There isn't a standard software TEE architecture for x86; RV doesn't have something (yet), and ARM has just ATF (which one may or may not fully equate to being a "TEE").
>
> Greg
>

Given where we are currently with the lack of a proper definition for
TEE, I suggest we simply remove the requirement for TEE for now and
add it later when the TEE spec is finalized.
Suggest we remove the line "RAS errors should be capable of
interrupting TEE" and leave it at that.
I agree with this Kumar.

Please review the below sentence. 
If the RAS event is configured as the firmware first model, the platform should be able to trigger the higest priority of M-mode interrupt to all HARTs in the physical RV processor. This prevents the subsequent RAS errors are propagated by other HARTs that access the problematic hardware (PCIe, Memory, I/O and etc.)

One question:
Besides those RAS events come from the interrupt controller, how about the HART or Memory RAS events? Are those RAS events in the scope of exception? or they would be also routed to  interrupt controller? Or we don't have to worry about this, RAS TG will have the solution?

Abner
 

--
Regards
Kumar


Re: [PATCH 1/1] RAS features for OS-A platform server extension

Abner Chang
 



Greg Favor <gfavor@...> 於 2021年6月24日 週四 上午12:00寫道:
On Wed, Jun 23, 2021 at 7:59 AM Abner Chang <renba.chang@...> wrote:
Yes.  Which is just a software matter of configuring the interrupt controller accordingly.
Does this mean the interrupt controller would integrate all RAS events (HART, PCI, I/O, memory and etc.)? 
Or there would be a separate hardware box that manages all RAS error events, and maybe some error signals output from that box and connected to the interrupt controller? The interrupt controller just provides the mechanism to morph those error signals to FFM or OSF interrupt?

To the extent that "RAS interrupts" are literally that, i.e. interrupt request signals, then they go to the system interrupt controller just like all other interrupt request signals.  (Some system designs might also have a "platform microcontroller" that has its own local interrupt controller and may receive some of these interrupt request signals.)

Maybe part of what you're trying to get at is that RAS error events in many architectures get logged in and reported from hardware RAS registers.  RAS registers "report" errors by outputting RAS interrupt request signals.  Software then comes back around and reads the RAS registers to gather info about logged errors.
Yes, something likes that.

Do we need to define what is the RAS error signals output to the interrupt controller? (The signal could be classified by the error severities such as CE, UC_FATAL, UC_NONFATAL or classified by the RAS error categories such as RAS_MEM_ERROR, RAS_IO_ERROR and etc.)
I think we can just leave it to RAS TG because we just define what server platform needs on RAS, right?
  
Can we summarize the requirement to
- RAS errors should be capable of interrupting TEE.
This is ok for now because there is no hardware signal defined for triggering TEE right? I have more comments on this below. 

I expect RV will have similarities to ARM in this matter - and ARM doesn't have a hardware signal defined for triggering TEE either (and hasn't felt the need to define such).
Ok,  I thought there is a similar hardware signal.

Without the hardware signal to trigger TEE. The alternative would be triggering the M-mode exception and jump to TEE in the M-mode exception handler?
So the scenario of triggering TEE would be,
For software management mode interface:
     S-mode-> sbi ecall to M-mode->TEE jump vector->TEE
For the hardware management mode interface:
Hardware interrupt -> M-mode handler-> TEE jump vector->TEE
What firmware or software resides in TEE is implementation-specific. For example on edk2, we will load the management mode core into TEE.
I am just trying to get more understanding of the future design of TEE on RV.

 

This implies a requirement to have a TEE - and defining what constitutes a compliant TEE in the platform spec.  Btw, what distinguishes the TEE from "firmware"?
Please correct me on ARM part if I am wrong.
The equivalent mechanism to TEE is SMM on X86 and TZ on ARM. I don't quite understand how ARM TZ works, however on X86 system, all cores are brought to SMM environment when SMI is triggered. ARM has the equivalent event which is SMC, right?

Neither ARM nor RISC-V has a direct equivalent of SMM.  So I'll pick on what ARM has - which is rather like RV.  At a hardware level ARM has EL3 and Secure ELx, and RV as M-mode and secure partitions of S/U-mode (using PMP).  At a software level one has a Secure monitor running in EL3/M-mode and tbd whether other parts run in SELx/partitions.  TZ as a TEE is a combination of these hardware features and the secure software that runs on it.  ARM TZ doesn't specify the actual software TEE, it just provides the hardware architectural features and framework for creating and running a TEE.  There is no one standard ARM TEE (although ARM has developed their ATF as a reference secure boot flow; although maybe it has expanded in scope in recent years?).

In short, RV first needs to define, develop, and specify a software TEE.  The hardware components are falling into place (e.g. PMP, ePMP, Zkr), and OpenSBI is working towards supporting secure partitions.  So, until there is a concrete RISC-V TEE standard (or even a standard framework), we shouldn't be stating requirements tied with having a TEE.  Also keep in mind that things like secure boot will be required in the Server extension - which is part of the overall topic of TEE.
Thanks for the above explanation. 
 
The above is called management mode (MM) which is defined in UEFI PI spec. MM has the highest privilege than CR0 on X86 and EL3 on ARM. The MM is OS agnostic and the MM event halts any processes and gets the core into management mode to run the firmware code. The environment of MM (data and code) can only be accessed when the core is in MM. Firmware always uses this for the secure stuff, power management, and of course the RAS.

What you describe, for RV, is M-mode - a pretty direct analog of ARM EL3.
 

I would like to add one more thing to the RAS requirement but I don't know how to describe it properly because seems we don't have the MM event on RISC-V such as SMI and SMC which can bring the system to MM.

RV has ECALL, just like ARM has SMC.
Thanks for the correction. I thought SMC is the hardware signal.  
 
So there are two scenarios for RAS on the firmware first model.
- If the platform doesn't have TEE and the hardware event to trigger TEE:
  If the RAS event is configured to firmware first mode, the platform should be able to trigger M-Mode exception to all harts in the physical processor. This prevents the subsequent RAS error propagated by other harts that access the problematic hardware (PCI, memory and etc.)

- If the platform has TEE and the hardware event to trigger TEE:
    If the RAS event is configured to firmware first mode, the platform should be able to trigger TEE event to all harts in the physical processor and bring all harts into TEE. This prevents the subsequent RAS error propagated by other cores which access the problematic hardware (PCI, memory and etc.) 

I think part of what complicates this discussion is the nebulous nature of what exactly is the "TEE" in any given architecture.  At a hardware level x86/ARM/RV have SMM/EL3/M-mode and they have ways to "call" into that secure environment.  The software TEE architecture is what is rather nebulous.  There isn't a standard software TEE architecture for x86; RV doesn't have something (yet), and ARM has just ATF (which one may or may not fully equate to being a "TEE").
Agreed. 

Greg


Re: [PATCH 1/1] RAS features for OS-A platform server extension

Kumar Sankaran
 

On Wed, Jun 23, 2021 at 9:00 AM Greg Favor <gfavor@...> wrote:

On Wed, Jun 23, 2021 at 7:59 AM Abner Chang <renba.chang@...> wrote:

Yes. Which is just a software matter of configuring the interrupt controller accordingly.
Does this mean the interrupt controller would integrate all RAS events (HART, PCI, I/O, memory and etc.)?
Or there would be a separate hardware box that manages all RAS error events, and maybe some error signals output from that box and connected to the interrupt controller? The interrupt controller just provides the mechanism to morph those error signals to FFM or OSF interrupt?

To the extent that "RAS interrupts" are literally that, i.e. interrupt request signals, then they go to the system interrupt controller just like all other interrupt request signals. (Some system designs might also have a "platform microcontroller" that has its own local interrupt controller and may receive some of these interrupt request signals.)

Maybe part of what you're trying to get at is that RAS error events in many architectures get logged in and reported from hardware RAS registers. RAS registers "report" errors by outputting RAS interrupt request signals. Software then comes back around and reads the RAS registers to gather info about logged errors.


Can we summarize the requirement to

- RAS errors should be capable of interrupting TEE.
This is ok for now because there is no hardware signal defined for triggering TEE right? I have more comments on this below.

I expect RV will have similarities to ARM in this matter - and ARM doesn't have a hardware signal defined for triggering TEE either (and hasn't felt the need to define such).



This implies a requirement to have a TEE - and defining what constitutes a compliant TEE in the platform spec. Btw, what distinguishes the TEE from "firmware"?
Please correct me on ARM part if I am wrong.
The equivalent mechanism to TEE is SMM on X86 and TZ on ARM. I don't quite understand how ARM TZ works, however on X86 system, all cores are brought to SMM environment when SMI is triggered. ARM has the equivalent event which is SMC, right?

Neither ARM nor RISC-V has a direct equivalent of SMM. So I'll pick on what ARM has - which is rather like RV. At a hardware level ARM has EL3 and Secure ELx, and RV as M-mode and secure partitions of S/U-mode (using PMP). At a software level one has a Secure monitor running in EL3/M-mode and tbd whether other parts run in SELx/partitions. TZ as a TEE is a combination of these hardware features and the secure software that runs on it. ARM TZ doesn't specify the actual software TEE, it just provides the hardware architectural features and framework for creating and running a TEE. There is no one standard ARM TEE (although ARM has developed their ATF as a reference secure boot flow; although maybe it has expanded in scope in recent years?).

In short, RV first needs to define, develop, and specify a software TEE. The hardware components are falling into place (e.g. PMP, ePMP, Zkr), and OpenSBI is working towards supporting secure partitions. So, until there is a concrete RISC-V TEE standard (or even a standard framework), we shouldn't be stating requirements tied with having a TEE. Also keep in mind that things like secure boot will be required in the Server extension - which is part of the overall topic of TEE.


The above is called management mode (MM) which is defined in UEFI PI spec. MM has the highest privilege than CR0 on X86 and EL3 on ARM. The MM is OS agnostic and the MM event halts any processes and gets the core into management mode to run the firmware code. The environment of MM (data and code) can only be accessed when the core is in MM. Firmware always uses this for the secure stuff, power management, and of course the RAS.

What you describe, for RV, is M-mode - a pretty direct analog of ARM EL3.



I would like to add one more thing to the RAS requirement but I don't know how to describe it properly because seems we don't have the MM event on RISC-V such as SMI and SMC which can bring the system to MM.

RV has ECALL, just like ARM has SMC.


So there are two scenarios for RAS on the firmware first model.
- If the platform doesn't have TEE and the hardware event to trigger TEE:
If the RAS event is configured to firmware first mode, the platform should be able to trigger M-Mode exception to all harts in the physical processor. This prevents the subsequent RAS error propagated by other harts that access the problematic hardware (PCI, memory and etc.)

- If the platform has TEE and the hardware event to trigger TEE:
If the RAS event is configured to firmware first mode, the platform should be able to trigger TEE event to all harts in the physical processor and bring all harts into TEE. This prevents the subsequent RAS error propagated by other cores which access the problematic hardware (PCI, memory and etc.)

I think part of what complicates this discussion is the nebulous nature of what exactly is the "TEE" in any given architecture. At a hardware level x86/ARM/RV have SMM/EL3/M-mode and they have ways to "call" into that secure environment. The software TEE architecture is what is rather nebulous. There isn't a standard software TEE architecture for x86; RV doesn't have something (yet), and ARM has just ATF (which one may or may not fully equate to being a "TEE").

Greg
Given where we are currently with the lack of a proper definition for
TEE, I suggest we simply remove the requirement for TEE for now and
add it later when the TEE spec is finalized.
Suggest we remove the line "RAS errors should be capable of
interrupting TEE" and leave it at that.

--
Regards
Kumar


Re: [PATCH 1/1] RAS features for OS-A platform server extension

Greg Favor
 

On Wed, Jun 23, 2021 at 7:59 AM Abner Chang <renba.chang@...> wrote:
Yes.  Which is just a software matter of configuring the interrupt controller accordingly.
Does this mean the interrupt controller would integrate all RAS events (HART, PCI, I/O, memory and etc.)? 
Or there would be a separate hardware box that manages all RAS error events, and maybe some error signals output from that box and connected to the interrupt controller? The interrupt controller just provides the mechanism to morph those error signals to FFM or OSF interrupt?

To the extent that "RAS interrupts" are literally that, i.e. interrupt request signals, then they go to the system interrupt controller just like all other interrupt request signals.  (Some system designs might also have a "platform microcontroller" that has its own local interrupt controller and may receive some of these interrupt request signals.)

Maybe part of what you're trying to get at is that RAS error events in many architectures get logged in and reported from hardware RAS registers.  RAS registers "report" errors by outputting RAS interrupt request signals.  Software then comes back around and reads the RAS registers to gather info about logged errors.
 
Can we summarize the requirement to
- RAS errors should be capable of interrupting TEE.
This is ok for now because there is no hardware signal defined for triggering TEE right? I have more comments on this below. 

I expect RV will have similarities to ARM in this matter - and ARM doesn't have a hardware signal defined for triggering TEE either (and hasn't felt the need to define such).
 

This implies a requirement to have a TEE - and defining what constitutes a compliant TEE in the platform spec.  Btw, what distinguishes the TEE from "firmware"?
Please correct me on ARM part if I am wrong.
The equivalent mechanism to TEE is SMM on X86 and TZ on ARM. I don't quite understand how ARM TZ works, however on X86 system, all cores are brought to SMM environment when SMI is triggered. ARM has the equivalent event which is SMC, right?

Neither ARM nor RISC-V has a direct equivalent of SMM.  So I'll pick on what ARM has - which is rather like RV.  At a hardware level ARM has EL3 and Secure ELx, and RV as M-mode and secure partitions of S/U-mode (using PMP).  At a software level one has a Secure monitor running in EL3/M-mode and tbd whether other parts run in SELx/partitions.  TZ as a TEE is a combination of these hardware features and the secure software that runs on it.  ARM TZ doesn't specify the actual software TEE, it just provides the hardware architectural features and framework for creating and running a TEE.  There is no one standard ARM TEE (although ARM has developed their ATF as a reference secure boot flow; although maybe it has expanded in scope in recent years?).

In short, RV first needs to define, develop, and specify a software TEE.  The hardware components are falling into place (e.g. PMP, ePMP, Zkr), and OpenSBI is working towards supporting secure partitions.  So, until there is a concrete RISC-V TEE standard (or even a standard framework), we shouldn't be stating requirements tied with having a TEE.  Also keep in mind that things like secure boot will be required in the Server extension - which is part of the overall topic of TEE.
 
The above is called management mode (MM) which is defined in UEFI PI spec. MM has the highest privilege than CR0 on X86 and EL3 on ARM. The MM is OS agnostic and the MM event halts any processes and gets the core into management mode to run the firmware code. The environment of MM (data and code) can only be accessed when the core is in MM. Firmware always uses this for the secure stuff, power management, and of course the RAS.

What you describe, for RV, is M-mode - a pretty direct analog of ARM EL3.
 

I would like to add one more thing to the RAS requirement but I don't know how to describe it properly because seems we don't have the MM event on RISC-V such as SMI and SMC which can bring the system to MM.

RV has ECALL, just like ARM has SMC.
 
So there are two scenarios for RAS on the firmware first model.
- If the platform doesn't have TEE and the hardware event to trigger TEE:
  If the RAS event is configured to firmware first mode, the platform should be able to trigger M-Mode exception to all harts in the physical processor. This prevents the subsequent RAS error propagated by other harts that access the problematic hardware (PCI, memory and etc.)

- If the platform has TEE and the hardware event to trigger TEE:
    If the RAS event is configured to firmware first mode, the platform should be able to trigger TEE event to all harts in the physical processor and bring all harts into TEE. This prevents the subsequent RAS error propagated by other cores which access the problematic hardware (PCI, memory and etc.) 

I think part of what complicates this discussion is the nebulous nature of what exactly is the "TEE" in any given architecture.  At a hardware level x86/ARM/RV have SMM/EL3/M-mode and they have ways to "call" into that secure environment.  The software TEE architecture is what is rather nebulous.  There isn't a standard software TEE architecture for x86; RV doesn't have something (yet), and ARM has just ATF (which one may or may not fully equate to being a "TEE").

Greg

741 - 760 of 1847