On Tue, Dec 14, 2021 at 05:32:08PM -0800, Greg Favor wrote:
The following two items in Ved's email didn't get any response, so I offer my own below ...
On Sun, Dec 12, 2021 at 4:15 PM Vedvyas Shanbhogue <ved@...> wrote:
Section 2.3.7.3.2 - PCIe memory space: The requirement to not have any address translation for inbound accesses to any component in system address space is restrictive. If direct assignment of devices is supported then the IOMMU would be required to do the address translation for inbound accesses. Further for hart originated accesses where the PCIe memory is mapped into virtual address space there needs to be a translation through the first and/or second level page tables. Please help clarify why PCie memory must not be mapped into virtual address space and why use of IOMMU to do translation is disallowed by the specification.
I think where this came from is learnings in the ARM "server" ecosystem (as then got captured in SBSA). In particular, one wants devices and software on harts to have the same view of system physical address space so that, for example, pointers can be easily passed around. Which doesn't conflict with having address translation by IOMMUs. Maybe the current text needs to be better worded, but I think the ideas to be expressed are:
For inbound PCIe transactions:
- There should be no hardware modifications of PCIe addresses outside of an IOMMU (as some vendors way back in early ARM SBSA days were wont to do).
- If there is not an IOMMU associated with the PCIe interface, then PCIe devices will have the same view of PA space as the harts.
- If there is an IOMMU associated with the PCIe interface, then system software can trust that all address modifications are under its control via hart page tables and IOMMU page tables.
For outbound PCIe transactions, system software is free to set up VA-to-PA translations in hart page tables. I think the mandate against outbound address translation was accidentally mistaken. The key point is that there is one common view of system physical address space. Hart and IOMMU page tables may translate from hart VA's and device addresses to system physical address space, but the above ensures that "standard" system software has full control over this and doesn't have non-standard address transformations happening that it isn't aware of and doesn't know how to control.
Thanks. I think this is very clear.
Section 2.3.7.3.3 - PCIe interrupts: It seems unnecessary to require platforms built for the '22 version of the platform to have to support running software that is not MSI aware. Please clarify why supporting the INTx emulation for legacy/Pre-PCIe software compatibility a required and not an optional capability for RISC-v platforms?
This one seems questionable to me as well, although I'm not the expert to reliably proclaim that INTx support is no longer a necessity in some server-class systems. I can imagine that back in earlier ARM "server" days this legacy issue was a bigger deal and hence was mandated in SBSA. But maybe that is no longer an issue? Or at least for 2022+ systems - to the point where mandating this legacy support is an unnecessary burden on many or the majority of such systems.
If this is a fair view going forward, then the INTx requirements should just become recommendations for systems that do feel the need to care about INTx support.
I think the recommendation could be changed to require MSI and make supporting INTx emulation optional. I am hoping to hear from BIOS and OS experts if we would need support OS/BIOS that are `22 platform compatible but are not MSI capable.
