[RFC PATCH 1/1] server extension: PCIe requirements


Abner Chang
 

Hi   Mayuresh,
As I mentioned in the platform meeting, we missed the requirement for firmware. I added in below section and please rephrase it if you want.

Regards,
Abner

Mayuresh Chitale <mchitale@...> 於 2021年6月10日 週四 上午2:27寫道:
This patch adds requirements for PCIe support for the server extension

Signed-off-by: Mayuresh Chitale <mchitale@...>

Signed-off-by: Mayuresh Chitale <mchitale@...>
---
 riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 4418788..9de487e 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
 ** Platforms are required to delegate the supervisor timer interrupt to 'S'
 mode. If the 'H' extension is implemented then the platforms are required to
 delegate the virtual supervisor timer interrupt to 'VS' mode.
-* PCI-E
+
+===== PCIe
+Platforms are required to support PCIe
+footnote:[https://pcisig.com/specifications].Following are the requirements:
+
+====== PCIe Config Space
+* Platforms shall support access to the PCIe config space via ECAM as described
+in the PCI Express Base specification.
+* The entire config space for a single PCIe domain should be accessible via a
+single ECAM I/O region.
+* Platform firmware should implement the MCFG table to allow the operating
+systems to discover the supported PCIe domains and map the ECAM I/O region for
+each domain.
+* ECAM I/O regions shall be configured as channel 0 I/O regions.
+
+====== PCIe Memory Space
+* PCIe Outbound region +
+Platforms are required to provide atleast two I/O regions for mapping the
+memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
+The first I/O region is required to be located below 4G physical address to
+map the memory requested by non-prefetchabe BARs. This region shall be
+configured as channel 0 I/O region. The second I/O region is required to be
+located above 4G physical address to map the memory requested by prefetchable
+BARs. This region may be configured as I/O region or as memory region.
+
+* PCIe Inbound region +
+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.
+
+====== PCIe Interrupts
+* Platforms shall support both INTx and MSI/MSI-x interrupts.
+* Integration with AIA +
+TBD
+
+====== PCIe I/O coherency
+Following are the requirements:
+
+* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
+outbound TLP.
+* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
+set then hardware shall generate a snoop request.
+* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
+hardware coherency is required. Software coherency may be required via CMOs.
+
+====== PCIe Topology
+Platforms are required to implement atleast one of the following topologies and
+the components required in that topology.
+
+[ditaa]
+....
+
+            +----------+                             +----------+
+            |   CPU    |                             |   CPU    |
+            |          |                             |          |
+            +-----|----+                             +-----|----+
+                  |                                        |
+                  |                                        |
+    +-------------|------------+             +-------------|------------+
+    |        ROOT | COMPLEX    |             |        ROOT | COMPLEX    |
+    |                          |             |                          |
+    |      +------|-------+    |             |      +------|-------+    |
+    |      |  Host Bridge |    |             |      |  Host Bridge |    |
+    |      +------|-------+    |             |      +------|-------+    |
+    |             |            |             |             |            |
+    |             | BUS 0      |             |             | BUS 0      |
+    |     |-------|------|     |             |       +-----|-------+    |
+    |     |              |     |             |       | ROOT  PORT  |    |
+    |     |              |     |             |       +-----|-------+    |
+    | +---|---+      +---|---+ |             |             |            |
+    | | RCEIP |      | RCEC  | |             |             | PCIe Link  |
+    | +-------+      +-------+ |             |             |            |
+    |                          |             +-------------|------------+
+    +--------------------------+                           |
+                                                           |  BUS 1
+    RCEIP - Root complex integrated endpoint
+    RCEC - Root complex event collector
+....
+
+* Host Bridge +
+Following are the requirements for host bridges:
+
+** Any read or write access by a hart to an ECAM I/O region shall be converted
+by the host bridge into the corresponding PCIe config read or config write
+request.
+** Any read or write access by a hart to a PCIe outbound region shall be
+forwarded by the host bridge to a BAR or prefetch/non-prefetch memory window,
+if the address falls within the region claimed by the BAR or prefetch/
+non-prefetch memory window. Otherwise the host bridge shall return an error.
+
+** Host bridge shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on root bus.
+*** Config reads that receive Unsupported Request response from functions and
+devices on the root bus.
+* Root ports +
+Following are the requirements for root ports.
+** Root ports shall appear as PCI-PCI bridge to software.
+** Root ports shall implememnt all registers of Type 1 header.
+** Root ports shall implement all capabilities specified in the PCI Express
+Base specification for a root port.
+** Root ports shall forward type 1 configuration access when the bus number in
+the TLP is greater than the root port's secondary bus number and less than or
+equal to the root port's subordinate bus number.
+** Root ports shall convert type 1 configuration access to a type 0
+configuration acess when bus number in the TLP is equal to the root port's
+secondary bus number.
+** Root ports shall respond to any type 0 configuration accesses it receives.
+** Root ports shall forward memory accesses targeting its prefetch/non-prefetch
+memory windows to downstream components. If address of the transaction does not
+fall within the regions claimed by prefetch/non-prefetch memory windows then
+the root port shall generate a Unsupported Request.
+** Root port requester id or completer id shall be formed using the bdf of the
+root port.
+** The root ports shall support the CRS software visbility.
+** Root ports shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on seconday bus.
+*** Config reads that receive Unsupported Request from downstream components.
+*** Config read when root port's link is down.
+** The root port shall implement the AER capability.
+
+* RCEIP +
+All the requirements for RCEIP in the PCI Express Base specification shall be implemented.
+In addition the following requirements shall be met:
+** If RCEIP is implemented then RCEC shall be implemented as well. All
+requrirements for RCEC specified in the PCI Express Base specification shall be
+implemented. RCEC is required to terminate the AER and PME messages from RCEIP.
+** If both the topologies mentioned above are supported then RCEIP and RCEC
+shall be implemented in a separate PCIe domain and shall be addressable via a
+separate ECAM I/O region.
+
+====== PCIe peer to peer transactions +
+TBD

====== PCIe Device Firmware Requirement 
PCI expansion ROM code type 3 (UEFI) image must be provided by PCIe device for OS/A
server extension platform accroding to
if that PCIe device is utilized during UEFI firmware boot process. The image stored in PCI
expansion ROM is an UEFI driver that must be compliant with https://uefi.org/specifications[UEFI specification 2.9]
14.4.2 PCI Option ROMs.


 ==== Secure Boot
 * TEE
--
2.17.1







Greg Favor
 

On Mon, Jun 14, 2021 at 5:23 PM Josh Scheid <jscheid@...> wrote:

I understand that IOPMP is not an IOMMU, but to the extent that it is a general "bus master memory protection" widget, it can be used by M-mode to ensure simple things, such as that S-mode-SW-controlled PCIe initiators can not access address regions not accessible by S-mode. 

Yes, most likely IOPMP can be used to do this.
 
For example, the platform spec could avoid mentioning the IOPMP proposal, but state that the platform is required to have a mechanism to allow M-mode SW to control (including prevent) PCIe initiator access to regions of system address space.  While remaining open to custom implementations, it's clear on the functional intent.

That would be appropriate.  And, for example, one much simpler implementation approach (than the IOPMP proposals) would be to replicate a CPU PMP block as an "IO PMP" in front of each I/O device or group of devices.

That would allow M-mode software to only have to deal with one PMP software programming model across all masters in the system.

Greg


Josh Scheid
 

On Mon, Jun 14, 2021 at 4:02 PM Greg Favor <gfavor@...> wrote:
On Mon, Jun 14, 2021 at 2:28 PM Josh Scheid <jscheid@...> wrote:
+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.

While a standard IOMMU is further off, is the current opinion that the IOPMP is not in a position to be required or suggested as an implementation of the above requirement?  If not, then it's hard to check for compliance.

I'm not sure if an IOPMP could be used for this particular purpose, but more generally IOPMP is being driven by embedded people and isn't consciously thinking about functionality requirements implied by H-style virtualization, or PCIe MSIs, or other PCIe features.  In this regard IOPMP is analogous to PLIC and CLIC - and not generally suitable for OS/A platforms (and presumably is well-suited for M platforms).

I understand that IOPMP is not an IOMMU, but to the extent that it is a general "bus master memory protection" widget, it can be used by M-mode to ensure simple things, such as that S-mode-SW-controlled PCIe initiators can not access address regions not accessible by S-mode.  There's value in memory protection even without full virtualization support.  I'm questioning how vague the memory protection "requirement" should be to the extent that it ends up being usable and sufficient to provide a defined level of assurance. 

For example, the platform spec could avoid mentioning the IOPMP proposal, but state that the platform is required to have a mechanism to allow M-mode SW to control (including prevent) PCIe initiator access to regions of system address space.  While remaining open to custom implementations, it's clear on the functional intent.

-Josh


Greg Favor
 

On Mon, Jun 14, 2021 at 2:28 PM Josh Scheid <jscheid@...> wrote:
+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.

While a standard IOMMU is further off, is the current opinion that the IOPMP is not in a position to be required or suggested as an implementation of the above requirement?  If not, then it's hard to check for compliance.

I'm not sure if an IOPMP could be used for this particular purpose, but more generally IOPMP is being driven by embedded people and isn't consciously thinking about functionality requirements implied by H-style virtualization, or PCIe MSIs, or other PCIe features.  In this regard IOPMP is analogous to PLIC and CLIC - and not generally suitable for OS/A platforms (and presumably is well-suited for M platforms).
 
Is this mechanism expected to be M-mode SW controlled, or is it also expected to be controlled by S-mode (either directly or via SBI)?

IOPMP has continued to evolve significantly, so I only casually/loosely watch what's going on.  But they don't appear to be thinking much yet about this aspect (unless they started in the past couple of weeks).  Although at a hardware level the flexibility is probably there for control by either since IOPMP registers will presumably be memory-mapped.

Greg


Josh Scheid
 

On Wed, Jun 9, 2021 at 11:27 AM Mayuresh Chitale <mchitale@...> wrote:
This patch adds requirements for PCIe support for the server extension

Signed-off-by: Mayuresh Chitale <mchitale@...>

Signed-off-by: Mayuresh Chitale <mchitale@...>
---
 riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 4418788..9de487e 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
 ** Platforms are required to delegate the supervisor timer interrupt to 'S'
 mode. If the 'H' extension is implemented then the platforms are required to
 delegate the virtual supervisor timer interrupt to 'VS' mode.

Is this an M-mode SW requirement or a HW requirement that these interrupts are delegatable (writeable) in HW?

Why require the delegation by M-mode instead of allowing for M-mode to trap and pass down?  Is this just a performance benefit?

-* PCI-E
+
+===== PCIe
+Platforms are required to support PCIe
+footnote:[https://pcisig.com/specifications].Following are the requirements:

Any particular baseline PCIe version and/or extensions?

+
+====== PCIe Config Space
+* Platforms shall support access to the PCIe config space via ECAM as described
+in the PCI Express Base specification.
+* The entire config space for a single PCIe domain should be accessible via a
+single ECAM I/O region.
+* Platform firmware should implement the MCFG table to allow the operating
+systems to discover the supported PCIe domains and map the ECAM I/O region for
+each domain.
+* ECAM I/O regions shall be configured as channel 0 I/O regions.
+
+====== PCIe Memory Space
+* PCIe Outbound region +
+Platforms are required to provide atleast two I/O regions for mapping the
+memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
+The first I/O region is required to be located below 4G physical address to
+map the memory requested by non-prefetchabe BARs. This region shall be
+configured as channel 0 I/O region. The second I/O region is required to be
+located above 4G physical address to map the memory requested by prefetchable
+BARs.

Is there any guidance needed about the amount of total space available (below 4G), or that space needs to be allocated for each domain?

I think that this is only necessary in the platform because of the current lack of an IOMMU requirement or standard.  With an IOMMU, that component can be used to locate 32-bit BARS anywhere in the system address space.  So at least keep in mind the requirement can be dropped at that time.
 
This region may be configured as I/O region or as memory region.

Is an SBI call needed to support S-mode configuration?  What is the default expected to be if there is no SBI call or no call is made?

IIRC, some older versions of some HCI standards (USB, SATA?) only had device support for 32-bit addresses.  I mention this to check if the requirement is really just that non-prefetchable BARs need to be supported <4GB, or that it's also needed for other 32-bit BAR support.  Thus it may need to support prefetchable BARs located <4GB.
 
+
+* PCIe Inbound region +
+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.

While a standard IOMMU is further off, is the current opinion that the IOPMP is not in a position to be required or suggested as an implementation of the above requirement?  If not, then it's hard to check for compliance.

Is this mechanism expected to be M-mode SW controlled, or is it also expected to be controlled by S-mode (either directly or via SBI)?

+
+====== PCIe Interrupts
+* Platforms shall support both INTx and MSI/MSI-x interrupts.
+* Integration with AIA +
+TBD

While TBD, one question interesting to me is whether or not it matters if the PCI RC implements it's own INTx to MSI bridge, or if an AIA APLIC is required for that.

+
+====== PCIe I/O coherency
+Following are the requirements:
+
+* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
+outbound TLP.

Is it implicit here if this mechanism is provided to M-mode SW only, or also to S-mode?

+* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
+set then hardware shall generate a snoop request.
+* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
+hardware coherency is required. Software coherency may be required via CMOs.

I read this as primarily stating that inbound NoSnoop controls the "coherent" access attribute.  But why this instead of focusing on control of the "cacheable" vs "non-cacheable" attribute?  With the latter, it seems more apparent how harts would then manages coherency: by controlling accesses to use the same "cacheable" attribute.

+
+====== PCIe Topology
+Platforms are required to implement atleast one of the following topologies and
+the components required in that topology.
+
+[ditaa]
+....
+
+            +----------+                             +----------+
+            |   CPU    |                             |   CPU    |
+            |          |                             |          |
+            +-----|----+                             +-----|----+
+                  |                                        |
+                  |                                        |
+    +-------------|------------+             +-------------|------------+
+    |        ROOT | COMPLEX    |             |        ROOT | COMPLEX    |
+    |                          |             |                          |
+    |      +------|-------+    |             |      +------|-------+    |
+    |      |  Host Bridge |    |             |      |  Host Bridge |    |
+    |      +------|-------+    |             |      +------|-------+    |
+    |             |            |             |             |            |
+    |             | BUS 0      |             |             | BUS 0      |
+    |     |-------|------|     |             |       +-----|-------+    |
+    |     |              |     |             |       | ROOT  PORT  |    |
+    |     |              |     |             |       +-----|-------+    |
+    | +---|---+      +---|---+ |             |             |            |
+    | | RCEIP |      | RCEC  | |             |             | PCIe Link  |
+    | +-------+      +-------+ |             |             |            |
+    |                          |             +-------------|------------+
+    +--------------------------+                           |
+                                                           |  BUS 1
+    RCEIP - Root complex integrated endpoint
+    RCEC - Root complex event collector
+....


Have we considered the option of requiring EPs to be behind virtual integrated RPs, instead of being RCiEPs?  This seems to bypass some of the unique limitations of RCiEPs, including the RCEC.

Do we need to ban or allow for impl-spec address mapping capabilities between PCI and system addresses?

Do we need say anything about peer-to-peer support, or requirements if a system enables it?  Including ACS?

Should the system mtimer counter also be the source for PCIe PTP?

-Josh


Mayuresh Chitale
 





On Thu, Jun 10, 2021 at 5:19 AM Bin Meng <bmeng.cn@...> wrote:
On Thu, Jun 10, 2021 at 2:27 AM Mayuresh Chitale
<mchitale@...> wrote:
>
> This patch adds requirements for PCIe support for the server extension
>
> Signed-off-by: Mayuresh Chitale <mchitale@...>
>
> Signed-off-by: Mayuresh Chitale <mchitale@...>

nits: 2 SoB here

Thanks. I will fix this and the typos below in the next version. 
> ---
>  riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 132 insertions(+), 1 deletion(-)
>
> diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
> index 4418788..9de487e 100644
> --- a/riscv-platform-spec.adoc
> +++ b/riscv-platform-spec.adoc
> @@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
>  ** Platforms are required to delegate the supervisor timer interrupt to 'S'
>  mode. If the 'H' extension is implemented then the platforms are required to
>  delegate the virtual supervisor timer interrupt to 'VS' mode.
> -* PCI-E
> +
> +===== PCIe
> +Platforms are required to support PCIe
> +footnote:[https://pcisig.com/specifications].Following are the requirements:
> +
> +====== PCIe Config Space
> +* Platforms shall support access to the PCIe config space via ECAM as described
> +in the PCI Express Base specification.
> +* The entire config space for a single PCIe domain should be accessible via a
> +single ECAM I/O region.
> +* Platform firmware should implement the MCFG table to allow the operating

Is ACPI mandatory?

Yes,  ACPI is mandatory for server extension.

> +systems to discover the supported PCIe domains and map the ECAM I/O region for
> +each domain.
> +* ECAM I/O regions shall be configured as channel 0 I/O regions.
> +
> +====== PCIe Memory Space
> +* PCIe Outbound region +
> +Platforms are required to provide atleast two I/O regions for mapping the

at least

> +memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
> +The first I/O region is required to be located below 4G physical address to
> +map the memory requested by non-prefetchabe BARs. This region shall be
> +configured as channel 0 I/O region. The second I/O region is required to be
> +located above 4G physical address to map the memory requested by prefetchable
> +BARs. This region may be configured as I/O region or as memory region.
> +
> +* PCIe Inbound region +
> +For security reasons, platforms are required to provide a mechanism to

Is this mechanism a standard one, or platform specific?

I am not sure if we have a standard mechanism yet.
 
> +restrict the inbound accesses over PCIe to certain specific regions in
> +the address space such as the DRAM.
> +
> +====== PCIe Interrupts
> +* Platforms shall support both INTx and MSI/MSI-x interrupts.
> +* Integration with AIA +
> +TBD
> +
> +====== PCIe I/O coherency
> +Following are the requirements:
> +
> +* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
> +outbound TLP.
> +* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
> +set then hardware shall generate a snoop request.
> +* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
> +hardware coherency is required. Software coherency may be required via CMOs.
> +
> +====== PCIe Topology
> +Platforms are required to implement atleast one of the following topologies and

at least

> +the components required in that topology.
> +
> +[ditaa]
> +....
> +
> +            +----------+                             +----------+
> +            |   CPU    |                             |   CPU    |
> +            |          |                             |          |
> +            +-----|----+                             +-----|----+
> +                  |                                        |
> +                  |                                        |
> +    +-------------|------------+             +-------------|------------+
> +    |        ROOT | COMPLEX    |             |        ROOT | COMPLEX    |
> +    |                          |             |                          |
> +    |      +------|-------+    |             |      +------|-------+    |
> +    |      |  Host Bridge |    |             |      |  Host Bridge |    |
> +    |      +------|-------+    |             |      +------|-------+    |
> +    |             |            |             |             |            |
> +    |             | BUS 0      |             |             | BUS 0      |
> +    |     |-------|------|     |             |       +-----|-------+    |
> +    |     |              |     |             |       | ROOT  PORT  |    |
> +    |     |              |     |             |       +-----|-------+    |
> +    | +---|---+      +---|---+ |             |             |            |
> +    | | RCEIP |      | RCEC  | |             |             | PCIe Link  |
> +    | +-------+      +-------+ |             |             |            |
> +    |                          |             +-------------|------------+
> +    +--------------------------+                           |
> +                                                           |  BUS 1
> +    RCEIP - Root complex integrated endpoint
> +    RCEC - Root complex event collector
> +....
> +
> +* Host Bridge +
> +Following are the requirements for host bridges:
> +
> +** Any read or write access by a hart to an ECAM I/O region shall be converted
> +by the host bridge into the corresponding PCIe config read or config write
> +request.
> +** Any read or write access by a hart to a PCIe outbound region shall be
> +forwarded by the host bridge to a BAR or prefetch/non-prefetch memory window,
> +if the address falls within the region claimed by the BAR or prefetch/
> +non-prefetch memory window. Otherwise the host bridge shall return an error.
> +
> +** Host bridge shall return all 1s in the following cases:
> +*** Config read to non existent functions and devices on root bus.
> +*** Config reads that receive Unsupported Request response from functions and
> +devices on the root bus.
> +* Root ports +
> +Following are the requirements for root ports.
> +** Root ports shall appear as PCI-PCI bridge to software.
> +** Root ports shall implememnt all registers of Type 1 header.

typo: implement

> +** Root ports shall implement all capabilities specified in the PCI Express
> +Base specification for a root port.
> +** Root ports shall forward type 1 configuration access when the bus number in
> +the TLP is greater than the root port's secondary bus number and less than or
> +equal to the root port's subordinate bus number.
> +** Root ports shall convert type 1 configuration access to a type 0
> +configuration acess when bus number in the TLP is equal to the root port's

typo: access

> +secondary bus number.
> +** Root ports shall respond to any type 0 configuration accesses it receives.
> +** Root ports shall forward memory accesses targeting its prefetch/non-prefetch
> +memory windows to downstream components. If address of the transaction does not
> +fall within the regions claimed by prefetch/non-prefetch memory windows then
> +the root port shall generate a Unsupported Request.
> +** Root port requester id or completer id shall be formed using the bdf of the
> +root port.
> +** The root ports shall support the CRS software visbility.

typo: visibility

> +** Root ports shall return all 1s in the following cases:
> +*** Config read to non existent functions and devices on seconday bus.

typo: secondary

> +*** Config reads that receive Unsupported Request from downstream components.
> +*** Config read when root port's link is down.
> +** The root port shall implement the AER capability.
> +
> +* RCEIP +
> +All the requirements for RCEIP in the PCI Express Base specification shall be implemented.
> +In addition the following requirements shall be met:
> +** If RCEIP is implemented then RCEC shall be implemented as well. All
> +requrirements for RCEC specified in the PCI Express Base specification shall be
> +implemented. RCEC is required to terminate the AER and PME messages from RCEIP.
> +** If both the topologies mentioned above are supported then RCEIP and RCEC
> +shall be implemented in a separate PCIe domain and shall be addressable via a
> +separate ECAM I/O region.
> +
> +====== PCIe peer to peer transactions +
> +TBD

Regards,
Bin


Bin Meng
 

On Thu, Jun 10, 2021 at 2:27 AM Mayuresh Chitale
<mchitale@...> wrote:

This patch adds requirements for PCIe support for the server extension

Signed-off-by: Mayuresh Chitale <mchitale@...>

Signed-off-by: Mayuresh Chitale <mchitale@...>
nits: 2 SoB here

---
riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 4418788..9de487e 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
** Platforms are required to delegate the supervisor timer interrupt to 'S'
mode. If the 'H' extension is implemented then the platforms are required to
delegate the virtual supervisor timer interrupt to 'VS' mode.
-* PCI-E
+
+===== PCIe
+Platforms are required to support PCIe
+footnote:[https://pcisig.com/specifications].Following are the requirements:
+
+====== PCIe Config Space
+* Platforms shall support access to the PCIe config space via ECAM as described
+in the PCI Express Base specification.
+* The entire config space for a single PCIe domain should be accessible via a
+single ECAM I/O region.
+* Platform firmware should implement the MCFG table to allow the operating
Is ACPI mandatory?

+systems to discover the supported PCIe domains and map the ECAM I/O region for
+each domain.
+* ECAM I/O regions shall be configured as channel 0 I/O regions.
+
+====== PCIe Memory Space
+* PCIe Outbound region +
+Platforms are required to provide atleast two I/O regions for mapping the
at least

+memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
+The first I/O region is required to be located below 4G physical address to
+map the memory requested by non-prefetchabe BARs. This region shall be
+configured as channel 0 I/O region. The second I/O region is required to be
+located above 4G physical address to map the memory requested by prefetchable
+BARs. This region may be configured as I/O region or as memory region.
+
+* PCIe Inbound region +
+For security reasons, platforms are required to provide a mechanism to
Is this mechanism a standard one, or platform specific?

+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.
+
+====== PCIe Interrupts
+* Platforms shall support both INTx and MSI/MSI-x interrupts.
+* Integration with AIA +
+TBD
+
+====== PCIe I/O coherency
+Following are the requirements:
+
+* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
+outbound TLP.
+* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
+set then hardware shall generate a snoop request.
+* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
+hardware coherency is required. Software coherency may be required via CMOs.
+
+====== PCIe Topology
+Platforms are required to implement atleast one of the following topologies and
at least

+the components required in that topology.
+
+[ditaa]
+....
+
+ +----------+ +----------+
+ | CPU | | CPU |
+ | | | |
+ +-----|----+ +-----|----+
+ | |
+ | |
+ +-------------|------------+ +-------------|------------+
+ | ROOT | COMPLEX | | ROOT | COMPLEX |
+ | | | |
+ | +------|-------+ | | +------|-------+ |
+ | | Host Bridge | | | | Host Bridge | |
+ | +------|-------+ | | +------|-------+ |
+ | | | | | |
+ | | BUS 0 | | | BUS 0 |
+ | |-------|------| | | +-----|-------+ |
+ | | | | | | ROOT PORT | |
+ | | | | | +-----|-------+ |
+ | +---|---+ +---|---+ | | | |
+ | | RCEIP | | RCEC | | | | PCIe Link |
+ | +-------+ +-------+ | | | |
+ | | +-------------|------------+
+ +--------------------------+ |
+ | BUS 1
+ RCEIP - Root complex integrated endpoint
+ RCEC - Root complex event collector
+....
+
+* Host Bridge +
+Following are the requirements for host bridges:
+
+** Any read or write access by a hart to an ECAM I/O region shall be converted
+by the host bridge into the corresponding PCIe config read or config write
+request.
+** Any read or write access by a hart to a PCIe outbound region shall be
+forwarded by the host bridge to a BAR or prefetch/non-prefetch memory window,
+if the address falls within the region claimed by the BAR or prefetch/
+non-prefetch memory window. Otherwise the host bridge shall return an error.
+
+** Host bridge shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on root bus.
+*** Config reads that receive Unsupported Request response from functions and
+devices on the root bus.
+* Root ports +
+Following are the requirements for root ports.
+** Root ports shall appear as PCI-PCI bridge to software.
+** Root ports shall implememnt all registers of Type 1 header.
typo: implement

+** Root ports shall implement all capabilities specified in the PCI Express
+Base specification for a root port.
+** Root ports shall forward type 1 configuration access when the bus number in
+the TLP is greater than the root port's secondary bus number and less than or
+equal to the root port's subordinate bus number.
+** Root ports shall convert type 1 configuration access to a type 0
+configuration acess when bus number in the TLP is equal to the root port's
typo: access

+secondary bus number.
+** Root ports shall respond to any type 0 configuration accesses it receives.
+** Root ports shall forward memory accesses targeting its prefetch/non-prefetch
+memory windows to downstream components. If address of the transaction does not
+fall within the regions claimed by prefetch/non-prefetch memory windows then
+the root port shall generate a Unsupported Request.
+** Root port requester id or completer id shall be formed using the bdf of the
+root port.
+** The root ports shall support the CRS software visbility.
typo: visibility

+** Root ports shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on seconday bus.
typo: secondary

+*** Config reads that receive Unsupported Request from downstream components.
+*** Config read when root port's link is down.
+** The root port shall implement the AER capability.
+
+* RCEIP +
+All the requirements for RCEIP in the PCI Express Base specification shall be implemented.
+In addition the following requirements shall be met:
+** If RCEIP is implemented then RCEC shall be implemented as well. All
+requrirements for RCEC specified in the PCI Express Base specification shall be
+implemented. RCEC is required to terminate the AER and PME messages from RCEIP.
+** If both the topologies mentioned above are supported then RCEIP and RCEC
+shall be implemented in a separate PCIe domain and shall be addressable via a
+separate ECAM I/O region.
+
+====== PCIe peer to peer transactions +
+TBD
Regards,
Bin


Mayuresh Chitale
 

This patch adds requirements for PCIe support for the server extension

Signed-off-by: Mayuresh Chitale <mchitale@...>

Signed-off-by: Mayuresh Chitale <mchitale@...>
---
riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 4418788..9de487e 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
** Platforms are required to delegate the supervisor timer interrupt to 'S'
mode. If the 'H' extension is implemented then the platforms are required to
delegate the virtual supervisor timer interrupt to 'VS' mode.
-* PCI-E
+
+===== PCIe
+Platforms are required to support PCIe
+footnote:[https://pcisig.com/specifications].Following are the requirements:
+
+====== PCIe Config Space
+* Platforms shall support access to the PCIe config space via ECAM as described
+in the PCI Express Base specification.
+* The entire config space for a single PCIe domain should be accessible via a
+single ECAM I/O region.
+* Platform firmware should implement the MCFG table to allow the operating
+systems to discover the supported PCIe domains and map the ECAM I/O region for
+each domain.
+* ECAM I/O regions shall be configured as channel 0 I/O regions.
+
+====== PCIe Memory Space
+* PCIe Outbound region +
+Platforms are required to provide atleast two I/O regions for mapping the
+memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
+The first I/O region is required to be located below 4G physical address to
+map the memory requested by non-prefetchabe BARs. This region shall be
+configured as channel 0 I/O region. The second I/O region is required to be
+located above 4G physical address to map the memory requested by prefetchable
+BARs. This region may be configured as I/O region or as memory region.
+
+* PCIe Inbound region +
+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.
+
+====== PCIe Interrupts
+* Platforms shall support both INTx and MSI/MSI-x interrupts.
+* Integration with AIA +
+TBD
+
+====== PCIe I/O coherency
+Following are the requirements:
+
+* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
+outbound TLP.
+* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
+set then hardware shall generate a snoop request.
+* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
+hardware coherency is required. Software coherency may be required via CMOs.
+
+====== PCIe Topology
+Platforms are required to implement atleast one of the following topologies and
+the components required in that topology.
+
+[ditaa]
+....
+
+ +----------+ +----------+
+ | CPU | | CPU |
+ | | | |
+ +-----|----+ +-----|----+
+ | |
+ | |
+ +-------------|------------+ +-------------|------------+
+ | ROOT | COMPLEX | | ROOT | COMPLEX |
+ | | | |
+ | +------|-------+ | | +------|-------+ |
+ | | Host Bridge | | | | Host Bridge | |
+ | +------|-------+ | | +------|-------+ |
+ | | | | | |
+ | | BUS 0 | | | BUS 0 |
+ | |-------|------| | | +-----|-------+ |
+ | | | | | | ROOT PORT | |
+ | | | | | +-----|-------+ |
+ | +---|---+ +---|---+ | | | |
+ | | RCEIP | | RCEC | | | | PCIe Link |
+ | +-------+ +-------+ | | | |
+ | | +-------------|------------+
+ +--------------------------+ |
+ | BUS 1
+ RCEIP - Root complex integrated endpoint
+ RCEC - Root complex event collector
+....
+
+* Host Bridge +
+Following are the requirements for host bridges:
+
+** Any read or write access by a hart to an ECAM I/O region shall be converted
+by the host bridge into the corresponding PCIe config read or config write
+request.
+** Any read or write access by a hart to a PCIe outbound region shall be
+forwarded by the host bridge to a BAR or prefetch/non-prefetch memory window,
+if the address falls within the region claimed by the BAR or prefetch/
+non-prefetch memory window. Otherwise the host bridge shall return an error.
+
+** Host bridge shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on root bus.
+*** Config reads that receive Unsupported Request response from functions and
+devices on the root bus.
+* Root ports +
+Following are the requirements for root ports.
+** Root ports shall appear as PCI-PCI bridge to software.
+** Root ports shall implememnt all registers of Type 1 header.
+** Root ports shall implement all capabilities specified in the PCI Express
+Base specification for a root port.
+** Root ports shall forward type 1 configuration access when the bus number in
+the TLP is greater than the root port's secondary bus number and less than or
+equal to the root port's subordinate bus number.
+** Root ports shall convert type 1 configuration access to a type 0
+configuration acess when bus number in the TLP is equal to the root port's
+secondary bus number.
+** Root ports shall respond to any type 0 configuration accesses it receives.
+** Root ports shall forward memory accesses targeting its prefetch/non-prefetch
+memory windows to downstream components. If address of the transaction does not
+fall within the regions claimed by prefetch/non-prefetch memory windows then
+the root port shall generate a Unsupported Request.
+** Root port requester id or completer id shall be formed using the bdf of the
+root port.
+** The root ports shall support the CRS software visbility.
+** Root ports shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on seconday bus.
+*** Config reads that receive Unsupported Request from downstream components.
+*** Config read when root port's link is down.
+** The root port shall implement the AER capability.
+
+* RCEIP +
+All the requirements for RCEIP in the PCI Express Base specification shall be implemented.
+In addition the following requirements shall be met:
+** If RCEIP is implemented then RCEC shall be implemented as well. All
+requrirements for RCEC specified in the PCI Express Base specification shall be
+implemented. RCEC is required to terminate the AER and PME messages from RCEIP.
+** If both the topologies mentioned above are supported then RCEIP and RCEC
+shall be implemented in a separate PCIe domain and shall be addressable via a
+separate ECAM I/O region.
+
+====== PCIe peer to peer transactions +
+TBD

==== Secure Boot
* TEE
--
2.17.1