Greg HI -

On Tue, Dec 14, 2021 at 05:32:08PM -0800, Greg Favor wrote:

The following two items in Ved's email didn't get any response, so I offer
my own below ...

On Sun, Dec 12, 2021 at 4:15 PM Vedvyas Shanbhogue <ved@...> wrote:

Section 2.3.7.3.2 - PCIe memory space:
The requirement to not have any address translation for inbound accesses
to any component in system address space is restrictive. If direct
assignment of devices is supported then the IOMMU would be required to do
the address translation for inbound accesses. Further for hart originated
accesses where the PCIe memory is mapped into virtual address space there
needs to be a translation through the first and/or second level page
tables. Please help clarify why PCie memory must not be mapped into
virtual address space and why use of IOMMU to do translation is disallowed
by the specification.

I think where this came from is learnings in the ARM "server" ecosystem (as
then got captured in SBSA). In particular, one wants devices and software
on harts to have the same view of system physical address space so that,
for example, pointers can be easily passed around. Which doesn't conflict
with having address translation by IOMMUs. Maybe the current text needs to
be better worded, but I think the ideas to be expressed are:

For inbound PCIe transactions:

- There should be no hardware modifications of PCIe addresses outside of an
IOMMU (as some vendors way back in early ARM SBSA days were wont to do).

- If there is not an IOMMU associated with the PCIe interface, then PCIe
devices will have the same view of PA space as the harts.

- If there is an IOMMU associated with the PCIe interface, then system
software can trust that all address modifications are under its control via
hart page tables and IOMMU page tables.

For outbound PCIe transactions, system software is free to set up VA-to-PA
translations in hart page tables. I think the mandate against outbound
address translation was accidentally mistaken. The key point is that there
is one common view of system physical address space. Hart and IOMMU page
tables may translate from hart VA's and device addresses to system physical
address space, but the above ensures that "standard" system software has
full control over this and doesn't have non-standard address
transformations happening that it isn't aware of and doesn't know how to
control.

Thanks. I think this is very clear.

Section 2.3.7.3.3 - PCIe interrupts:
It seems unnecessary to require platforms built for the '22 version of the
platform to have to support running software that is not MSI aware. Please
clarify why supporting the INTx emulation for legacy/Pre-PCIe software
compatibility a required and not an optional capability for RISC-v
platforms?

This one seems questionable to me as well, although I'm not the expert to
reliably proclaim that INTx support is no longer a necessity in some
server-class systems. I can imagine that back in earlier ARM "server" days
this legacy issue was a bigger deal and hence was mandated in SBSA. But
maybe that is no longer an issue? Or at least for 2022+ systems - to the
point where mandating this legacy support is an unnecessary burden on many
or the majority of such systems.

If this is a fair view going forward, then the INTx requirements should
just become recommendations for systems that do feel the need to care about
INTx support.

I think the recommendation could be changed to require MSI and make supporting INTx emulation optional. I am hoping to hear from BIOS and OS experts if we would need support OS/BIOS that are `22 platform compatible but are not MSI capable.

regards
ved