On Thu, Jun 10, 2021 at 2:27 AM Mayuresh Chitale
<mchitale@...> wrote:
>
> This patch adds requirements for PCIe support for the server extension
>
> Signed-off-by: Mayuresh Chitale <mchitale@...>
>
> Signed-off-by: Mayuresh Chitale <mchitale@...>

nits: 2 SoB here

> ---
>  riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 132 insertions(+), 1 deletion(-)
>
> diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
> index 4418788..9de487e 100644
> --- a/riscv-platform-spec.adoc
> +++ b/riscv-platform-spec.adoc
> @@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
>  ** Platforms are required to delegate the supervisor timer interrupt to 'S'
>  mode. If the 'H' extension is implemented then the platforms are required to
>  delegate the virtual supervisor timer interrupt to 'VS' mode.
> -* PCI-E
> +
> +===== PCIe
> +Platforms are required to support PCIe
> +footnote:[https://pcisig.com/specifications].Following are the requirements:
> +
> +====== PCIe Config Space
> +* Platforms shall support access to the PCIe config space via ECAM as described
> +in the PCI Express Base specification.
> +* The entire config space for a single PCIe domain should be accessible via a
> +single ECAM I/O region.
> +* Platform firmware should implement the MCFG table to allow the operating

Is ACPI mandatory?

> +systems to discover the supported PCIe domains and map the ECAM I/O region for
> +each domain.
> +* ECAM I/O regions shall be configured as channel 0 I/O regions.
> +
> +====== PCIe Memory Space
> +* PCIe Outbound region +
> +Platforms are required to provide atleast two I/O regions for mapping the

at least

> +memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
> +The first I/O region is required to be located below 4G physical address to
> +map the memory requested by non-prefetchabe BARs. This region shall be
> +configured as channel 0 I/O region. The second I/O region is required to be
> +located above 4G physical address to map the memory requested by prefetchable
> +BARs. This region may be configured as I/O region or as memory region.
> +
> +* PCIe Inbound region +
> +For security reasons, platforms are required to provide a mechanism to

Is this mechanism a standard one, or platform specific?

> +restrict the inbound accesses over PCIe to certain specific regions in
> +the address space such as the DRAM.
> +
> +====== PCIe Interrupts
> +* Platforms shall support both INTx and MSI/MSI-x interrupts.
> +* Integration with AIA +
> +TBD
> +
> +====== PCIe I/O coherency
> +Following are the requirements:
> +
> +* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
> +outbound TLP.
> +* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
> +set then hardware shall generate a snoop request.
> +* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
> +hardware coherency is required. Software coherency may be required via CMOs.
> +
> +====== PCIe Topology
> +Platforms are required to implement atleast one of the following topologies and

at least

> +the components required in that topology.
> +
> +[ditaa]
> +....
> +
> +            +----------+                             +----------+
> +            |   CPU    |                             |   CPU    |
> +            |          |                             |          |
> +            +-----|----+                             +-----|----+
> +                  |                                        |
> +                  |                                        |
> +    +-------------|------------+             +-------------|------------+
> +    |        ROOT | COMPLEX    |             |        ROOT | COMPLEX    |
> +    |                          |             |                          |
> +    |      +------|-------+    |             |      +------|-------+    |
> +    |      |  Host Bridge |    |             |      |  Host Bridge |    |
> +    |      +------|-------+    |             |      +------|-------+    |
> +    |             |            |             |             |            |
> +    |             | BUS 0      |             |             | BUS 0      |
> +    |     |-------|------|     |             |       +-----|-------+    |
> +    |     |              |     |             |       | ROOT  PORT  |    |
> +    |     |              |     |             |       +-----|-------+    |
> +    | +---|---+      +---|---+ |             |             |            |
> +    | | RCEIP |      | RCEC  | |             |             | PCIe Link  |
> +    | +-------+      +-------+ |             |             |            |
> +    |                          |             +-------------|------------+
> +    +--------------------------+                           |
> +                                                           |  BUS 1
> +    RCEIP - Root complex integrated endpoint
> +    RCEC - Root complex event collector
> +....
> +
> +* Host Bridge +
> +Following are the requirements for host bridges:
> +
> +** Any read or write access by a hart to an ECAM I/O region shall be converted
> +by the host bridge into the corresponding PCIe config read or config write
> +request.
> +** Any read or write access by a hart to a PCIe outbound region shall be
> +forwarded by the host bridge to a BAR or prefetch/non-prefetch memory window,
> +if the address falls within the region claimed by the BAR or prefetch/
> +non-prefetch memory window. Otherwise the host bridge shall return an error.
> +
> +** Host bridge shall return all 1s in the following cases:
> +*** Config read to non existent functions and devices on root bus.
> +*** Config reads that receive Unsupported Request response from functions and
> +devices on the root bus.
> +* Root ports +
> +Following are the requirements for root ports.
> +** Root ports shall appear as PCI-PCI bridge to software.
> +** Root ports shall implememnt all registers of Type 1 header.

typo: implement

> +** Root ports shall implement all capabilities specified in the PCI Express
> +Base specification for a root port.
> +** Root ports shall forward type 1 configuration access when the bus number in
> +the TLP is greater than the root port's secondary bus number and less than or
> +equal to the root port's subordinate bus number.
> +** Root ports shall convert type 1 configuration access to a type 0
> +configuration acess when bus number in the TLP is equal to the root port's

typo: access

> +secondary bus number.
> +** Root ports shall respond to any type 0 configuration accesses it receives.
> +** Root ports shall forward memory accesses targeting its prefetch/non-prefetch
> +memory windows to downstream components. If address of the transaction does not
> +fall within the regions claimed by prefetch/non-prefetch memory windows then
> +the root port shall generate a Unsupported Request.
> +** Root port requester id or completer id shall be formed using the bdf of the
> +root port.
> +** The root ports shall support the CRS software visbility.

typo: visibility

> +** Root ports shall return all 1s in the following cases:
> +*** Config read to non existent functions and devices on seconday bus.

typo: secondary

> +*** Config reads that receive Unsupported Request from downstream components.
> +*** Config read when root port's link is down.
> +** The root port shall implement the AER capability.
> +
> +* RCEIP +
> +All the requirements for RCEIP in the PCI Express Base specification shall be implemented.
> +In addition the following requirements shall be met:
> +** If RCEIP is implemented then RCEC shall be implemented as well. All
> +requrirements for RCEC specified in the PCI Express Base specification shall be
> +implemented. RCEC is required to terminate the AER and PME messages from RCEIP.
> +** If both the topologies mentioned above are supported then RCEIP and RCEC
> +shall be implemented in a separate PCIe domain and shall be addressable via a
> +separate ECAM I/O region.
> +
> +====== PCIe peer to peer transactions +
> +TBD

Regards,
Bin

This patch adds requirements for PCIe support for the server extension

Signed-off-by: Mayuresh Chitale <mchitale@...>

Signed-off-by: Mayuresh Chitale <mchitale@...>
---
 riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 4418788..9de487e 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
 ** Platforms are required to delegate the supervisor timer interrupt to 'S'
 mode. If the 'H' extension is implemented then the platforms are required to
 delegate the virtual supervisor timer interrupt to 'VS' mode.

-* PCI-E
+
+===== PCIe
+Platforms are required to support PCIe
+footnote:[https://pcisig.com/specifications].Following are the requirements:

+
+====== PCIe Config Space
+* Platforms shall support access to the PCIe config space via ECAM as described
+in the PCI Express Base specification.
+* The entire config space for a single PCIe domain should be accessible via a
+single ECAM I/O region.
+* Platform firmware should implement the MCFG table to allow the operating
+systems to discover the supported PCIe domains and map the ECAM I/O region for
+each domain.
+* ECAM I/O regions shall be configured as channel 0 I/O regions.
+
+====== PCIe Memory Space
+* PCIe Outbound region +
+Platforms are required to provide atleast two I/O regions for mapping the
+memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
+The first I/O region is required to be located below 4G physical address to
+map the memory requested by non-prefetchabe BARs. This region shall be
+configured as channel 0 I/O region. The second I/O region is required to be
+located above 4G physical address to map the memory requested by prefetchable
+BARs.

This region may be configured as I/O region or as memory region.

+
+* PCIe Inbound region +
+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.

+
+====== PCIe Interrupts
+* Platforms shall support both INTx and MSI/MSI-x interrupts.
+* Integration with AIA +
+TBD

+
+====== PCIe I/O coherency
+Following are the requirements:
+
+* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
+outbound TLP.

+* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
+set then hardware shall generate a snoop request.
+* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
+hardware coherency is required. Software coherency may be required via CMOs.

+
+====== PCIe Topology
+Platforms are required to implement atleast one of the following topologies and
+the components required in that topology.
+
+[ditaa]
+....
+
+            +----------+                             +----------+
+            |   CPU    |                             |   CPU    |
+            |          |                             |          |
+            +-----|----+                             +-----|----+
+                  |                                        |
+                  |                                        |
+    +-------------|------------+             +-------------|------------+
+    |        ROOT | COMPLEX    |             |        ROOT | COMPLEX    |
+    |                          |             |                          |
+    |      +------|-------+    |             |      +------|-------+    |
+    |      |  Host Bridge |    |             |      |  Host Bridge |    |
+    |      +------|-------+    |             |      +------|-------+    |
+    |             |            |             |             |            |
+    |             | BUS 0      |             |             | BUS 0      |
+    |     |-------|------|     |             |       +-----|-------+    |
+    |     |              |     |             |       | ROOT  PORT  |    |
+    |     |              |     |             |       +-----|-------+    |
+    | +---|---+      +---|---+ |             |             |            |
+    | | RCEIP |      | RCEC  | |             |             | PCIe Link  |
+    | +-------+      +-------+ |             |             |            |
+    |                          |             +-------------|------------+
+    +--------------------------+                           |
+                                                           |  BUS 1
+    RCEIP - Root complex integrated endpoint
+    RCEC - Root complex event collector
+....

+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.

While a standard IOMMU is further off, is the current opinion that the IOPMP is not in a position to be required or suggested as an implementation of the above requirement?  If not, then it's hard to check for compliance.

Is this mechanism expected to be M-mode SW controlled, or is it also expected to be controlled by S-mode (either directly or via SBI)?

On Mon, Jun 14, 2021 at 2:28 PM Josh Scheid <jscheid@...> wrote:

+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.

While a standard IOMMU is further off, is the current opinion that the IOPMP is not in a position to be required or suggested as an implementation of the above requirement?  If not, then it's hard to check for compliance.

I'm not sure if an IOPMP could be used for this particular purpose, but more generally IOPMP is being driven by embedded people and isn't consciously thinking about functionality requirements implied by H-style virtualization, or PCIe MSIs, or other PCIe features.  In this regard IOPMP is analogous to PLIC and CLIC - and not generally suitable for OS/A platforms (and presumably is well-suited for M platforms).

I understand that IOPMP is not an IOMMU, but to the extent that it is a general "bus master memory protection" widget, it can be used by M-mode to ensure simple things, such as that S-mode-SW-controlled PCIe initiators can not access address regions not accessible by S-mode. 

For example, the platform spec could avoid mentioning the IOPMP proposal, but state that the platform is required to have a mechanism to allow M-mode SW to control (including prevent) PCIe initiator access to regions of system address space.  While remaining open to custom implementations, it's clear on the functional intent.

This patch adds requirements for PCIe support for the server extension

Signed-off-by: Mayuresh Chitale <mchitale@...>

Signed-off-by: Mayuresh Chitale <mchitale@...>
---
 riscv-platform-spec.adoc | 133 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/riscv-platform-spec.adoc b/riscv-platform-spec.adoc
index 4418788..9de487e 100644
--- a/riscv-platform-spec.adoc
+++ b/riscv-platform-spec.adoc
@@ -363,7 +363,138 @@ https://lists.riscv.org/g/tech-privileged/message/404[Sstc] extension.
 ** Platforms are required to delegate the supervisor timer interrupt to 'S'
 mode. If the 'H' extension is implemented then the platforms are required to
 delegate the virtual supervisor timer interrupt to 'VS' mode.
-* PCI-E
+
+===== PCIe
+Platforms are required to support PCIe
+footnote:[https://pcisig.com/specifications].Following are the requirements:
+
+====== PCIe Config Space
+* Platforms shall support access to the PCIe config space via ECAM as described
+in the PCI Express Base specification.
+* The entire config space for a single PCIe domain should be accessible via a
+single ECAM I/O region.
+* Platform firmware should implement the MCFG table to allow the operating
+systems to discover the supported PCIe domains and map the ECAM I/O region for
+each domain.
+* ECAM I/O regions shall be configured as channel 0 I/O regions.
+
+====== PCIe Memory Space
+* PCIe Outbound region +
+Platforms are required to provide atleast two I/O regions for mapping the
+memory requested by PCIe endpoints and PCIe bridges/switches through BARs.
+The first I/O region is required to be located below 4G physical address to
+map the memory requested by non-prefetchabe BARs. This region shall be
+configured as channel 0 I/O region. The second I/O region is required to be
+located above 4G physical address to map the memory requested by prefetchable
+BARs. This region may be configured as I/O region or as memory region.
+
+* PCIe Inbound region +
+For security reasons, platforms are required to provide a mechanism to
+restrict the inbound accesses over PCIe to certain specific regions in
+the address space such as the DRAM.
+
+====== PCIe Interrupts
+* Platforms shall support both INTx and MSI/MSI-x interrupts.
+* Integration with AIA +
+TBD
+
+====== PCIe I/O coherency
+Following are the requirements:
+
+* Platforms shall provide a mechanism to control the `NoSnoop` bit for any
+outbound TLP.
+* If the host bridge/root port receives a TLP which does not have `NoSnoop` bit
+set then hardware shall generate a snoop request.
+* If the host bridge/root port receives a TLP which has `NoSnoop` set then no
+hardware coherency is required. Software coherency may be required via CMOs.
+
+====== PCIe Topology
+Platforms are required to implement atleast one of the following topologies and
+the components required in that topology.
+
+[ditaa]
+....
+
+            +----------+                             +----------+
+            |   CPU    |                             |   CPU    |
+            |          |                             |          |
+            +-----|----+                             +-----|----+
+                  |                                        |
+                  |                                        |
+    +-------------|------------+             +-------------|------------+
+    |        ROOT | COMPLEX    |             |        ROOT | COMPLEX    |
+    |                          |             |                          |
+    |      +------|-------+    |             |      +------|-------+    |
+    |      |  Host Bridge |    |             |      |  Host Bridge |    |
+    |      +------|-------+    |             |      +------|-------+    |
+    |             |            |             |             |            |
+    |             | BUS 0      |             |             | BUS 0      |
+    |     |-------|------|     |             |       +-----|-------+    |
+    |     |              |     |             |       | ROOT  PORT  |    |
+    |     |              |     |             |       +-----|-------+    |
+    | +---|---+      +---|---+ |             |             |            |
+    | | RCEIP |      | RCEC  | |             |             | PCIe Link  |
+    | +-------+      +-------+ |             |             |            |
+    |                          |             +-------------|------------+
+    +--------------------------+                           |
+                                                           |  BUS 1
+    RCEIP - Root complex integrated endpoint
+    RCEC - Root complex event collector
+....
+
+* Host Bridge +
+Following are the requirements for host bridges:
+
+** Any read or write access by a hart to an ECAM I/O region shall be converted
+by the host bridge into the corresponding PCIe config read or config write
+request.
+** Any read or write access by a hart to a PCIe outbound region shall be
+forwarded by the host bridge to a BAR or prefetch/non-prefetch memory window,
+if the address falls within the region claimed by the BAR or prefetch/
+non-prefetch memory window. Otherwise the host bridge shall return an error.
+
+** Host bridge shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on root bus.
+*** Config reads that receive Unsupported Request response from functions and
+devices on the root bus.
+* Root ports +
+Following are the requirements for root ports.
+** Root ports shall appear as PCI-PCI bridge to software.
+** Root ports shall implememnt all registers of Type 1 header.
+** Root ports shall implement all capabilities specified in the PCI Express
+Base specification for a root port.
+** Root ports shall forward type 1 configuration access when the bus number in
+the TLP is greater than the root port's secondary bus number and less than or
+equal to the root port's subordinate bus number.
+** Root ports shall convert type 1 configuration access to a type 0
+configuration acess when bus number in the TLP is equal to the root port's
+secondary bus number.
+** Root ports shall respond to any type 0 configuration accesses it receives.
+** Root ports shall forward memory accesses targeting its prefetch/non-prefetch
+memory windows to downstream components. If address of the transaction does not
+fall within the regions claimed by prefetch/non-prefetch memory windows then
+the root port shall generate a Unsupported Request.
+** Root port requester id or completer id shall be formed using the bdf of the
+root port.
+** The root ports shall support the CRS software visbility.
+** Root ports shall return all 1s in the following cases:
+*** Config read to non existent functions and devices on seconday bus.
+*** Config reads that receive Unsupported Request from downstream components.
+*** Config read when root port's link is down.
+** The root port shall implement the AER capability.
+
+* RCEIP +
+All the requirements for RCEIP in the PCI Express Base specification shall be implemented.
+In addition the following requirements shall be met:
+** If RCEIP is implemented then RCEC shall be implemented as well. All
+requrirements for RCEC specified in the PCI Express Base specification shall be
+implemented. RCEC is required to terminate the AER and PME messages from RCEIP.
+** If both the topologies mentioned above are supported then RCEIP and RCEC
+shall be implemented in a separate PCIe domain and shall be addressable via a
+separate ECAM I/O region.
+
+====== PCIe peer to peer transactions +
+TBD

 ==== Secure Boot
 * TEE
--
2.17.1