Platform Spec Technical Feedback


Kumar Sankaran
 

Yes, agreed that we can be more precise in the spec. Adding clarity where needed will help. Please feel free to provide additional commentary or clarification as needed to improve the spec.

 

In terms of issues reporting, I agree that starting the discussion in the mailing list is good. However, it will be better to keep each topic separate and mark the subject accordingly. Else, multiple topics get reported in one single email and we lose track on one issue while focusing on a different one on the same thread.

Technical issues would be better handled if they are reported and tracked via github issues and continuing discussions can happen there. One example could be the ongoing PMP discussion.

 

Regards

Kumar

From: Aaron Durbin <adurbin@...>
Sent: Friday, October 15, 2021 9:47 AM
To: Kumar Sankaran <ksankaran@...>
Cc: Greg Favor <gfavor@...>; Vedvyas Shanbhogue <ved@...>; tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] Platform Spec Technical Feedback

 

 

 

On Fri, Oct 15, 2021 at 12:57 AM Kumar Sankaran <ksankaran@...> wrote:

Hi All,

Can I please request each of these questions be posted as issues in github? That way, each issue gets tracked separately and we can continue discussions specific to each issue and close them as needed.

Right now, multiple issues are getting reported in each of the email threads in the tech-mailing list and some of the continuity is not getting tracked.

Thanks for the help in advance. These are all good discussions and let’s continue each of them in the right forum.

 

Kumar, I can go through the threads and accumulate topics to put in issues. Based on the discourse, I think we've gotten more clarity, but it also shows how diverse some of the answers are. Being precise in the spec and its goals will help a lot.

 

For the future, are you suggesting these discourses happen on issues only? I'm afraid that without starting an email discussion we may not get as many participants since everyone may not be monitoring a particular github issue tracker. That said, I do think it's important once we have things nailed down we can be more targeted within the issue tracker.

 

 

Regards

Kumar

From: tech-unixplatformspec@... <tech-unixplatformspec@...> On Behalf Of Greg Favor
Sent: Thursday, October 14, 2021 11:19 PM
To: Vedvyas Shanbhogue <ved@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] Platform Spec Technical Feedback

 

On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:

Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times.

 

Irrespective of this discussion, M-mode boot software needs to discover the system address map (even if it only cares about some of the details).  With that, it can understand where things are, where memory is (out of which it can carve out some for itself and possibly some for inter-mode communication), and where are physical devices that it needs to use (which includes all the machine-level memory-mapped registers that are used during runtime, e.g. MTIME/MTIMECMP, M-related interrupt controller registers, ...).

 

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.

 

Not true.  This is part of why MPRV exists in the architecture. 

 

Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

 

Although security concerns may lead to some or many PMP rules being locked and hence not movable.

 

The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

 

As Andrew notes, setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

 

Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.

 

Looking up PMPs (whether the current arch or some simplified arch) is seemingly always going to involve some form of parallel lookup.  As far as doing it many times during a tablewalk, that is true irrespective of caching PMP info in TLBs or not - since all the tablewalk accesses are untranslated accesses (except of course untranslated VS-stage page table accesses that are still subject to G-stage translation).  But these PMP lookups (and the actual page table accesses) should be uncommon to rare depending on how good of a TLB and tablewalk cache microarchitecture one has.

 

Also keep in mind that each page table access will also need to do a full set of PMA lookups.  (In embedded systems with a fixed address map and hardwired PMAs, that's less of an issue.  In higher-end systems with configurable PMAs, these probably are CAM'ed and hence result in a parallel lookup.)  (Fwiw, in our case we do PMP lookups, PMA lookups, and some other things during one extra cycle per actual page table access during a tablewalk.  It's hard to say that the one extra cycle could have been avoided if there was no form of PMPs present.)

 

Conversely, if the "worst case" is happening often, then one has bigger performance problems.  Extra cycles worsens the matter, but even without the extra cycles one is going to be in a bad place performance-wise if full tablewalks are happening frequently.

 

Greg


mark
 

discussion can occur in emails or gitub issues. once something is actionable it should be in issues.

On Fri, Oct 15, 2021 at 9:47 AM Aaron Durbin <adurbin@...> wrote:


On Fri, Oct 15, 2021 at 12:57 AM Kumar Sankaran <ksankaran@...> wrote:

Hi All,

Can I please request each of these questions be posted as issues in github? That way, each issue gets tracked separately and we can continue discussions specific to each issue and close them as needed.

Right now, multiple issues are getting reported in each of the email threads in the tech-mailing list and some of the continuity is not getting tracked.

Thanks for the help in advance. These are all good discussions and let’s continue each of them in the right forum.


Kumar, I can go through the threads and accumulate topics to put in issues. Based on the discourse, I think we've gotten more clarity, but it also shows how diverse some of the answers are. Being precise in the spec and its goals will help a lot.

For the future, are you suggesting these discourses happen on issues only? I'm afraid that without starting an email discussion we may not get as many participants since everyone may not be monitoring a particular github issue tracker. That said, I do think it's important once we have things nailed down we can be more targeted within the issue tracker.

 

Regards

Kumar

From: tech-unixplatformspec@... <tech-unixplatformspec@...> On Behalf Of Greg Favor
Sent: Thursday, October 14, 2021 11:19 PM
To: Vedvyas Shanbhogue <ved@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] Platform Spec Technical Feedback

 

On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:

Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times.

 

Irrespective of this discussion, M-mode boot software needs to discover the system address map (even if it only cares about some of the details).  With that, it can understand where things are, where memory is (out of which it can carve out some for itself and possibly some for inter-mode communication), and where are physical devices that it needs to use (which includes all the machine-level memory-mapped registers that are used during runtime, e.g. MTIME/MTIMECMP, M-related interrupt controller registers, ...).

 

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.

 

Not true.  This is part of why MPRV exists in the architecture. 

 

Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

 

Although security concerns may lead to some or many PMP rules being locked and hence not movable.

 

The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

 

As Andrew notes, setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

 

Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.

 

Looking up PMPs (whether the current arch or some simplified arch) is seemingly always going to involve some form of parallel lookup.  As far as doing it many times during a tablewalk, that is true irrespective of caching PMP info in TLBs or not - since all the tablewalk accesses are untranslated accesses (except of course untranslated VS-stage page table accesses that are still subject to G-stage translation).  But these PMP lookups (and the actual page table accesses) should be uncommon to rare depending on how good of a TLB and tablewalk cache microarchitecture one has.

 

Also keep in mind that each page table access will also need to do a full set of PMA lookups.  (In embedded systems with a fixed address map and hardwired PMAs, that's less of an issue.  In higher-end systems with configurable PMAs, these probably are CAM'ed and hence result in a parallel lookup.)  (Fwiw, in our case we do PMP lookups, PMA lookups, and some other things during one extra cycle per actual page table access during a tablewalk.  It's hard to say that the one extra cycle could have been avoided if there was no form of PMPs present.)

 

Conversely, if the "worst case" is happening often, then one has bigger performance problems.  Extra cycles worsens the matter, but even without the extra cycles one is going to be in a bad place performance-wise if full tablewalks are happening frequently.

 

Greg


Aaron Durbin
 



On Fri, Oct 15, 2021 at 12:57 AM Kumar Sankaran <ksankaran@...> wrote:

Hi All,

Can I please request each of these questions be posted as issues in github? That way, each issue gets tracked separately and we can continue discussions specific to each issue and close them as needed.

Right now, multiple issues are getting reported in each of the email threads in the tech-mailing list and some of the continuity is not getting tracked.

Thanks for the help in advance. These are all good discussions and let’s continue each of them in the right forum.


Kumar, I can go through the threads and accumulate topics to put in issues. Based on the discourse, I think we've gotten more clarity, but it also shows how diverse some of the answers are. Being precise in the spec and its goals will help a lot.

For the future, are you suggesting these discourses happen on issues only? I'm afraid that without starting an email discussion we may not get as many participants since everyone may not be monitoring a particular github issue tracker. That said, I do think it's important once we have things nailed down we can be more targeted within the issue tracker.

 

Regards

Kumar

From: tech-unixplatformspec@... <tech-unixplatformspec@...> On Behalf Of Greg Favor
Sent: Thursday, October 14, 2021 11:19 PM
To: Vedvyas Shanbhogue <ved@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] Platform Spec Technical Feedback

 

On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:

Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times.

 

Irrespective of this discussion, M-mode boot software needs to discover the system address map (even if it only cares about some of the details).  With that, it can understand where things are, where memory is (out of which it can carve out some for itself and possibly some for inter-mode communication), and where are physical devices that it needs to use (which includes all the machine-level memory-mapped registers that are used during runtime, e.g. MTIME/MTIMECMP, M-related interrupt controller registers, ...).

 

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.

 

Not true.  This is part of why MPRV exists in the architecture. 

 

Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

 

Although security concerns may lead to some or many PMP rules being locked and hence not movable.

 

The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

 

As Andrew notes, setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

 

Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.

 

Looking up PMPs (whether the current arch or some simplified arch) is seemingly always going to involve some form of parallel lookup.  As far as doing it many times during a tablewalk, that is true irrespective of caching PMP info in TLBs or not - since all the tablewalk accesses are untranslated accesses (except of course untranslated VS-stage page table accesses that are still subject to G-stage translation).  But these PMP lookups (and the actual page table accesses) should be uncommon to rare depending on how good of a TLB and tablewalk cache microarchitecture one has.

 

Also keep in mind that each page table access will also need to do a full set of PMA lookups.  (In embedded systems with a fixed address map and hardwired PMAs, that's less of an issue.  In higher-end systems with configurable PMAs, these probably are CAM'ed and hence result in a parallel lookup.)  (Fwiw, in our case we do PMP lookups, PMA lookups, and some other things during one extra cycle per actual page table access during a tablewalk.  It's hard to say that the one extra cycle could have been avoided if there was no form of PMPs present.)

 

Conversely, if the "worst case" is happening often, then one has bigger performance problems.  Extra cycles worsens the matter, but even without the extra cycles one is going to be in a bad place performance-wise if full tablewalks are happening frequently.

 

Greg


Aaron Durbin
 

On Fri, Oct 15, 2021 at 3:01 AM Philipp Tomsich <ptomsich@...> wrote:
On Thu, Oct 14, 2021 at 4:04 PM Aaron Durbin <adurbin@...> wrote:

On Wed, Oct 13, 2021 at 8:09 PM Atish Patra <Atish.Patra@...> wrote:
On Wed, 2021-10-13 at 14:15 -0600, Aaron Durbin wrote:

> It is a method that has been used and is specified. However, I don't
> understand the leap from that to a hard requirement. Again, I believe
> the requirement should be focused on the intention -- not the
> implementation. 

Standardising an "intention" does not solve the challenge of hardware/software interoperability for COTS hardware with COTS software.
If the implementer of a hardware/execution environment is free to choose any suitable implementation, then software would have to support these: out-of-the-box and without knowing what vendors might have implemented or will implement (admittedly, this reflects how today's operating systems work: this would be a moot point in a world where every OS would detect detect extra hardware requirements, dynamically load trusted components from a vendor distribution point, and reconfigure itself...).  Specifying intent is not fragmentation, but rather contradicts the purpose for a Platform specification.

This is the first mention I've seen of the purpose of the Platform specification being COTS hardware and COTS software. I've been asking about this exact thing and not getting a definitive answer. Thank you for narrowing down what exactly the purpose is. Now, I think it's important to more clearly define what constitutes COTS hardware and software. Both are broad, but there are many aspects of the software stack. Therefore it would be helpful to define what is in scope there.
 

Yet, this is not in any way limiting to the evolution of RISC-V or any vendor's ability to innovate or provide cost-optimised/targeted products.
No vendor needs to follow the Platforms specifications, if they don't target a market where COTS hardware meets COTS software..
 
That may lead to a lot of fragmentation in terms of implementation in
hardware and software customization. Isn't it ?

I think there will be fragmentation in the short term because we're building an airplane while flying. Not every solution necessarily meets the needs of every product. While the existence of one solution does not lead to the only solution.

The primary intent of platform specifications is to reduce/avoid fragmentation for individual use cases and market segments — always focusing on off-the-shelf hardware meeting off-the-shelf software.

Again, thank you for the clarity. I propose we put such language in the spec itself.
 
As such, it is one of the most important goals of Platforms to avoid fragmentation.  And this also serves to advance our software ecosystem regarding ISV participation, the viability of porting binary OS distros to RISC-V, and enhancing the user-experience once off-the-shelf RISC-V boards becoming more commonplace.

What layers of the software stack are you envisioning trying to avoid fragmentation? This question likely goes back to the definition of COTS software. Your mentioning of ISV and binary OS distros is certainly helpful. What I've been trying to understand is why M-mode requirements are being brought in if we're primarily concerned about binary OS distros. 


Indeed, the slow moving nature of some technical standardisation initiatives has given rise to fragmentation in the past (e.g. the lack of a sufficiently advanced interrupt controller specification) and the process of standardising Platforms has seen the problems arising out of this.  In the Platforms specifications we sometimes need to make choices (and to avoid fragmentation, these have to be hard requirements) that are not ideal, as we already know that we will supersede or augument them in 2 years time (when the next major revision of Platforms will be released).

Also keep in mind that Platforms will be followed by a Platforms Compatibility Test suite — this will have to be able to test for requirements (no different for those requirements that impact security) and not just rely on vendors asserting that they have provided a mechanism.

Yes, I saw the mention of PCT. Does it then follow that every requirement in the platform spec needs to have a test in the PCT to confirm it is being followed? If that's true I think we'll need to ensure there's a test plan for every line item that is a mandatory requirement. Lastly, do you think we should not have recommendations within the platform spec?

-Aaron


Philipp Tomsich
 

On Thu, Oct 14, 2021 at 4:04 PM Aaron Durbin <adurbin@...> wrote:

On Wed, Oct 13, 2021 at 8:09 PM Atish Patra <Atish.Patra@...> wrote:
On Wed, 2021-10-13 at 14:15 -0600, Aaron Durbin wrote:

> It is a method that has been used and is specified. However, I don't
> understand the leap from that to a hard requirement. Again, I believe
> the requirement should be focused on the intention -- not the
> implementation. 

Standardising an "intention" does not solve the challenge of hardware/software interoperability for COTS hardware with COTS software.
If the implementer of a hardware/execution environment is free to choose any suitable implementation, then software would have to support these: out-of-the-box and without knowing what vendors might have implemented or will implement (admittedly, this reflects how today's operating systems work: this would be a moot point in a world where every OS would detect detect extra hardware requirements, dynamically load trusted components from a vendor distribution point, and reconfigure itself...).  Specifying intent is not fragmentation, but rather contradicts the purpose for a Platform specification.

Yet, this is not in any way limiting to the evolution of RISC-V or any vendor's ability to innovate or provide cost-optimised/targeted products.
No vendor needs to follow the Platforms specifications, if they don't target a market where COTS hardware meets COTS software..
 
That may lead to a lot of fragmentation in terms of implementation in
hardware and software customization. Isn't it ?

I think there will be fragmentation in the short term because we're building an airplane while flying. Not every solution necessarily meets the needs of every product. While the existence of one solution does not lead to the only solution.

The primary intent of platform specifications is to reduce/avoid fragmentation for individual use cases and market segments — always focusing on off-the-shelf hardware meeting off-the-shelf software.
As such, it is one of the most important goals of Platforms to avoid fragmentation.  And this also serves to advance our software ecosystem regarding ISV participation, the viability of porting binary OS distros to RISC-V, and enhancing the user-experience once off-the-shelf RISC-V boards becoming more commonplace.

Indeed, the slow moving nature of some technical standardisation initiatives has given rise to fragmentation in the past (e.g. the lack of a sufficiently advanced interrupt controller specification) and the process of standardising Platforms has seen the problems arising out of this.  In the Platforms specifications we sometimes need to make choices (and to avoid fragmentation, these have to be hard requirements) that are not ideal, as we already know that we will supersede or augument them in 2 years time (when the next major revision of Platforms will be released).

Also keep in mind that Platforms will be followed by a Platforms Compatibility Test suite — this will have to be able to test for requirements (no different for those requirements that impact security) and not just rely on vendors asserting that they have provided a mechanism.

Thanks,
Philipp.


Kumar Sankaran
 

Hi All,

Can I please request each of these questions be posted as issues in github? That way, each issue gets tracked separately and we can continue discussions specific to each issue and close them as needed.

Right now, multiple issues are getting reported in each of the email threads in the tech-mailing list and some of the continuity is not getting tracked.

Thanks for the help in advance. These are all good discussions and let’s continue each of them in the right forum.

 

Regards

Kumar

From: tech-unixplatformspec@... <tech-unixplatformspec@...> On Behalf Of Greg Favor
Sent: Thursday, October 14, 2021 11:19 PM
To: Vedvyas Shanbhogue <ved@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] Platform Spec Technical Feedback

 

On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:

Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times.

 

Irrespective of this discussion, M-mode boot software needs to discover the system address map (even if it only cares about some of the details).  With that, it can understand where things are, where memory is (out of which it can carve out some for itself and possibly some for inter-mode communication), and where are physical devices that it needs to use (which includes all the machine-level memory-mapped registers that are used during runtime, e.g. MTIME/MTIMECMP, M-related interrupt controller registers, ...).

 

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.

 

Not true.  This is part of why MPRV exists in the architecture. 

 

Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

 

Although security concerns may lead to some or many PMP rules being locked and hence not movable.

 

The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

 

As Andrew notes, setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

 

Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.

 

Looking up PMPs (whether the current arch or some simplified arch) is seemingly always going to involve some form of parallel lookup.  As far as doing it many times during a tablewalk, that is true irrespective of caching PMP info in TLBs or not - since all the tablewalk accesses are untranslated accesses (except of course untranslated VS-stage page table accesses that are still subject to G-stage translation).  But these PMP lookups (and the actual page table accesses) should be uncommon to rare depending on how good of a TLB and tablewalk cache microarchitecture one has.

 

Also keep in mind that each page table access will also need to do a full set of PMA lookups.  (In embedded systems with a fixed address map and hardwired PMAs, that's less of an issue.  In higher-end systems with configurable PMAs, these probably are CAM'ed and hence result in a parallel lookup.)  (Fwiw, in our case we do PMP lookups, PMA lookups, and some other things during one extra cycle per actual page table access during a tablewalk.  It's hard to say that the one extra cycle could have been avoided if there was no form of PMPs present.)

 

Conversely, if the "worst case" is happening often, then one has bigger performance problems.  Extra cycles worsens the matter, but even without the extra cycles one is going to be in a bad place performance-wise if full tablewalks are happening frequently.

 

Greg


Vincent Palatin
 

3


Greg Favor
 

On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:
Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times.

Irrespective of this discussion, M-mode boot software needs to discover the system address map (even if it only cares about some of the details).  With that, it can understand where things are, where memory is (out of which it can carve out some for itself and possibly some for inter-mode communication), and where are physical devices that it needs to use (which includes all the machine-level memory-mapped registers that are used during runtime, e.g. MTIME/MTIMECMP, M-related interrupt controller registers, ...).
 
With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.

Not true.  This is part of why MPRV exists in the architecture. 
 
Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

Although security concerns may lead to some or many PMP rules being locked and hence not movable.

The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

As Andrew notes, setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.

Looking up PMPs (whether the current arch or some simplified arch) is seemingly always going to involve some form of parallel lookup.  As far as doing it many times during a tablewalk, that is true irrespective of caching PMP info in TLBs or not - since all the tablewalk accesses are untranslated accesses (except of course untranslated VS-stage page table accesses that are still subject to G-stage translation).  But these PMP lookups (and the actual page table accesses) should be uncommon to rare depending on how good of a TLB and tablewalk cache microarchitecture one has.

Also keep in mind that each page table access will also need to do a full set of PMA lookups.  (In embedded systems with a fixed address map and hardwired PMAs, that's less of an issue.  In higher-end systems with configurable PMAs, these probably are CAM'ed and hence result in a parallel lookup.)  (Fwiw, in our case we do PMP lookups, PMA lookups, and some other things during one extra cycle per actual page table access during a tablewalk.  It's hard to say that the one extra cycle could have been avoided if there was no form of PMPs present.)

Conversely, if the "worst case" is happening often, then one has bigger performance problems.  Extra cycles worsens the matter, but even without the extra cycles one is going to be in a bad place performance-wise if full tablewalks are happening frequently.

Greg


Andrew Waterman
 



On Thu, Oct 14, 2021 at 4:11 PM Jonathan Behrens <behrensj@...> wrote:
It might be good to add a note or requirement to the platform spec saying that the PMP granularity can be 4KB/can be at most 4KB.

Indeed.  Exactly what set of options should be valid is debatable, but supporting 4 KiB is a must.  I would personally support requiring granularity <= 4 KiB (i.e. G <= 10 in the PMP spec's parlance).  Allowing them to be arbitrarily coarse would undermine the rationale for requiring the PMP feature, since it would become impractical to portably rely on them.


Jonathan

On Thu, Oct 14, 2021 at 7:02 PM Andrew Waterman via lists.riscv.org <andrew=sifive.com@...> wrote:


On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:
On 10/14/21 3:50 PM, Greg Favor wrote:
> On Thu, Oct 14, 2021 at 1:11 PM Aaron Durbin <adurbin@...
> <mailto:adurbin@...>> wrote:
>
>     That point necessarily requires providing a valid memory map for S/U
>     in order for those modes to work at all if at least one PMP is
>     implemented. Additionally, "If no PMP entry matches an S-mode or
>     U-mode access, but at least one PMP entry is implemented, the access
>     fails." That bit of the spec follows the intention of the first one.
>     And "PMP violations are always trapped precisely at the processor."
>     Makes sense, but it's also important to also think about side
>     channel information leaks as well. Additionally, speculation has
>     been relied upon to achieve high performant processors. One has to
>     check against the PMPs for all load/stores in addition to
>     instruction fetching.
>
>     The PMPs are creating a valid physical address space for U/S to
>     operate within. There is no way, in the positive, to declare a
>     region as M-mode only.
>
>
> True with PMP, not true with ePMP.

Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times. So only reasonable way would be for
M-mode to define a PMP that covers all of memory in a low priority PMP
that grants R/W/X permission to S/U mode accesses. If thats how most of
these rich OS platforms are going to satisfy the S/U mode access must
match a PMP rule then the requirement to positively match a PMP on S/U
mode accesses seems just an overhead.

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.
Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

Besides ePMP add *3* new modes based on MML and MMWP to hardware and
ePMP also change the meaning of the RWX bits e.g. with MML=1, "-W-" is
actually "--X" if L is 1 and depending on the mode of the access "R--"
or "RW-" when L=0!


> Higher performance CPU designs (some, although I would imagine many)
> won't do PMP lookups at ld/st execution time, but at TLB fill time (i.e.
> caching PMP lookup info) - which eliminates timing and power concerns
> for those designs that don't want to do PMP lookups in parallel with
> execution.  Conversely, simpler, lower performance RISC-V designs appear
> content with doing parallel lookups (i.e. no one has been complaining,
> and people are finding that such RV cores are smaller and l  ower power
> than comparable ARM cores).
>
The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

The PMP granularity is a design-time parameter; 4 bytes is the minimum supported, but it is not required by the ISA spec.  Setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

(Handling superpages with PMPs of finer granularity than the superpage size still requires care, but is doable.)


Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.


regards
ved






Jonathan Behrens <behrensj@...>
 

It might be good to add a note or requirement to the platform spec saying that the PMP granularity can be 4KB/can be at most 4KB.

Jonathan

On Thu, Oct 14, 2021 at 7:02 PM Andrew Waterman via lists.riscv.org <andrew=sifive.com@...> wrote:


On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:
On 10/14/21 3:50 PM, Greg Favor wrote:
> On Thu, Oct 14, 2021 at 1:11 PM Aaron Durbin <adurbin@...
> <mailto:adurbin@...>> wrote:
>
>     That point necessarily requires providing a valid memory map for S/U
>     in order for those modes to work at all if at least one PMP is
>     implemented. Additionally, "If no PMP entry matches an S-mode or
>     U-mode access, but at least one PMP entry is implemented, the access
>     fails." That bit of the spec follows the intention of the first one.
>     And "PMP violations are always trapped precisely at the processor."
>     Makes sense, but it's also important to also think about side
>     channel information leaks as well. Additionally, speculation has
>     been relied upon to achieve high performant processors. One has to
>     check against the PMPs for all load/stores in addition to
>     instruction fetching.
>
>     The PMPs are creating a valid physical address space for U/S to
>     operate within. There is no way, in the positive, to declare a
>     region as M-mode only.
>
>
> True with PMP, not true with ePMP.

Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times. So only reasonable way would be for
M-mode to define a PMP that covers all of memory in a low priority PMP
that grants R/W/X permission to S/U mode accesses. If thats how most of
these rich OS platforms are going to satisfy the S/U mode access must
match a PMP rule then the requirement to positively match a PMP on S/U
mode accesses seems just an overhead.

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.
Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

Besides ePMP add *3* new modes based on MML and MMWP to hardware and
ePMP also change the meaning of the RWX bits e.g. with MML=1, "-W-" is
actually "--X" if L is 1 and depending on the mode of the access "R--"
or "RW-" when L=0!


> Higher performance CPU designs (some, although I would imagine many)
> won't do PMP lookups at ld/st execution time, but at TLB fill time (i.e.
> caching PMP lookup info) - which eliminates timing and power concerns
> for those designs that don't want to do PMP lookups in parallel with
> execution.  Conversely, simpler, lower performance RISC-V designs appear
> content with doing parallel lookups (i.e. no one has been complaining,
> and people are finding that such RV cores are smaller and l  ower power
> than comparable ARM cores).
>
The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

The PMP granularity is a design-time parameter; 4 bytes is the minimum supported, but it is not required by the ISA spec.  Setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

(Handling superpages with PMPs of finer granularity than the superpage size still requires care, but is doable.)


Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.


regards
ved






Ved Shanbhogue
 

On 10/14/21 6:02 PM, Andrew Waterman wrote:
On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@... <mailto:ved@...>> wrote:
On 10/14/21 3:50 PM, Greg Favor wrote:
> On Thu, Oct 14, 2021 at 1:11 PM Aaron Durbin
<adurbin@... <mailto:adurbin@...>
> <mailto:adurbin@... <mailto:adurbin@...>>> wrote:
>
>     That point necessarily requires providing a valid memory map
for S/U
>     in order for those modes to work at all if at least one PMP is
>     implemented. Additionally, "If no PMP entry matches an S-mode or
>     U-mode access, but at least one PMP entry is implemented, the
access
>     fails." That bit of the spec follows the intention of the
first one.
>     And "PMP violations are always trapped precisely at the
processor."
>     Makes sense, but it's also important to also think about side
>     channel information leaks as well. Additionally, speculation has
>     been relied upon to achieve high performant processors. One
has to
>     check against the PMPs for all load/stores in addition to
>     instruction fetching.
>
>     The PMPs are creating a valid physical address space for U/S to
>     operate within. There is no way, in the positive, to declare a
>     region as M-mode only.
>
>
> True with PMP, not true with ePMP.
Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.
When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times. So only reasonable way would be for
M-mode to define a PMP that covers all of memory in a low priority PMP
that grants R/W/X permission to S/U mode accesses. If thats how most of
these rich OS platforms are going to satisfy the S/U mode access must
match a PMP rule then the requirement to positively match a PMP on S/U
mode accesses seems just an overhead.
With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.
Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.
Besides ePMP add *3* new modes based on MML and MMWP to hardware and
ePMP also change the meaning of the RWX bits e.g. with MML=1, "-W-" is
actually "--X" if L is 1 and depending on the mode of the access "R--"
or "RW-" when L=0!

> Higher performance CPU designs (some, although I would imagine many)
> won't do PMP lookups at ld/st execution time, but at TLB fill
time (i.e.
> caching PMP lookup info) - which eliminates timing and power
concerns
> for those designs that don't want to do PMP lookups in parallel with
> execution.  Conversely, simpler, lower performance RISC-V designs
appear
> content with doing parallel lookups (i.e. no one has been
complaining,
> and people are finding that such RV cores are smaller and l  ower
power
> than comparable ARM cores).
>
The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the
smallest
page size a TLB is normally defined to hold. So either a design
needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.
The PMP granularity is a design-time parameter; 4 bytes is the minimum supported, but it is not required by the ISA spec.  Setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.
(Handling superpages with PMPs of finer granularity than the superpage size still requires care, but is doable.)
Yes, handling superpages that overlap a smaller PMP would need a fracture tags in the TLB entries to allow finding and invalidating all fragments of the fractured page on an SFENCE.

However the platform specification makes no such statement when it requires the PMPs.

Is the suggestion that the platform specification is not really expecting that platforms will provide TOR and sub-page granular PMPs when it mandates 16 PMPs be present to be compatible with the specification? Could this be explicitly stated in the specification.


Andrew Waterman
 



On Thu, Oct 14, 2021 at 3:37 PM Vedvyas Shanbhogue <ved@...> wrote:
On 10/14/21 3:50 PM, Greg Favor wrote:
> On Thu, Oct 14, 2021 at 1:11 PM Aaron Durbin <adurbin@...
> <mailto:adurbin@...>> wrote:
>
>     That point necessarily requires providing a valid memory map for S/U
>     in order for those modes to work at all if at least one PMP is
>     implemented. Additionally, "If no PMP entry matches an S-mode or
>     U-mode access, but at least one PMP entry is implemented, the access
>     fails." That bit of the spec follows the intention of the first one.
>     And "PMP violations are always trapped precisely at the processor."
>     Makes sense, but it's also important to also think about side
>     channel information leaks as well. Additionally, speculation has
>     been relied upon to achieve high performant processors. One has to
>     check against the PMPs for all load/stores in addition to
>     instruction fetching.
>
>     The PMPs are creating a valid physical address space for U/S to
>     operate within. There is no way, in the positive, to declare a
>     region as M-mode only.
>
>
> True with PMP, not true with ePMP.

Its true with ePMP as well per the "region unmatched" rules.  Every
S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided
what parts of memory accessible to S-mode should have different R/W/X
permissions and at what times. So only reasonable way would be for
M-mode to define a PMP that covers all of memory in a low priority PMP
that grants R/W/X permission to S/U mode accesses. If thats how most of
these rich OS platforms are going to satisfy the S/U mode access must
match a PMP rule then the requirement to positively match a PMP on S/U
mode accesses seems just an overhead.

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared
memory regions it implies that M-mode cannot do any of the
trap-and-emulate a U/S-mode access unless the instruction and the
operands it was trying to access are within that shared memory region.
Of course the M-mode could dynamically move that PMP around but some of
the value of limiting the shared memory region is eroded by this
dynamically moving region.

Besides ePMP add *3* new modes based on MML and MMWP to hardware and
ePMP also change the meaning of the RWX bits e.g. with MML=1, "-W-" is
actually "--X" if L is 1 and depending on the mode of the access "R--"
or "RW-" when L=0!


> Higher performance CPU designs (some, although I would imagine many)
> won't do PMP lookups at ld/st execution time, but at TLB fill time (i.e.
> caching PMP lookup info) - which eliminates timing and power concerns
> for those designs that don't want to do PMP lookups in parallel with
> execution.  Conversely, simpler, lower performance RISC-V designs appear
> content with doing parallel lookups (i.e. no one has been complaining,
> and people are finding that such RV cores are smaller and l  ower power
> than comparable ARM cores).
>
The problem with implementing PMP checks in page-walker and caching the
information in the TLBs is that the PMP can be smaller than the smallest
page size a TLB is normally defined to hold. So either a design needs to
build TLBs that can go down to 4 byte granularity to support PMP or
build the logic to support the lookup on each ld/st/i-fetch.

The PMP granularity is a design-time parameter; 4 bytes is the minimum supported, but it is not required by the ISA spec.  Setting the PMP granularity to 4 KiB is fairly standard practice for systems with page-based VM.

(Handling superpages with PMPs of finer granularity than the superpage size still requires care, but is doable.)


Even if such smaller TLBs were built and PMP were to be looked up only
at page walk time looking up for example 16 TOR ranges is expensive to
do in parallel and with 2 level paging enabled it could need up to 20
lookup in worst case where all paging structure caches miss increasing
the cost of a TLB miss if timing forces adding a cycle or two to each
stage of the page walk.


regards
ved






Ved Shanbhogue
 

On 10/14/21 3:50 PM, Greg Favor wrote:
On Thu, Oct 14, 2021 at 1:11 PM Aaron Durbin <adurbin@... <mailto:adurbin@...>> wrote:
That point necessarily requires providing a valid memory map for S/U
in order for those modes to work at all if at least one PMP is
implemented. Additionally, "If no PMP entry matches an S-mode or
U-mode access, but at least one PMP entry is implemented, the access
fails." That bit of the spec follows the intention of the first one.
And "PMP violations are always trapped precisely at the processor."
Makes sense, but it's also important to also think about side
channel information leaks as well. Additionally, speculation has
been relied upon to achieve high performant processors. One has to
check against the PMPs for all load/stores in addition to
instruction fetching.
The PMPs are creating a valid physical address space for U/S to
operate within. There is no way, in the positive, to declare a
region as M-mode only.
True with PMP, not true with ePMP.
Its true with ePMP as well per the "region unmatched" rules. Every S/U-mode access has to match a PMP else the access is denied.

When running a general OS I am not sure how M-mode firmware can decided what parts of memory accessible to S-mode should have different R/W/X permissions and at what times. So only reasonable way would be for M-mode to define a PMP that covers all of memory in a low priority PMP that grants R/W/X permission to S/U mode accesses. If thats how most of these rich OS platforms are going to satisfy the S/U mode access must match a PMP rule then the requirement to positively match a PMP on S/U mode accesses seems just an overhead.

With ePMP since only unlocked "-W-" and "-WX" regions can be R/W shared memory regions it implies that M-mode cannot do any of the trap-and-emulate a U/S-mode access unless the instruction and the operands it was trying to access are within that shared memory region. Of course the M-mode could dynamically move that PMP around but some of the value of limiting the shared memory region is eroded by this dynamically moving region.

Besides ePMP add *3* new modes based on MML and MMWP to hardware and ePMP also change the meaning of the RWX bits e.g. with MML=1, "-W-" is actually "--X" if L is 1 and depending on the mode of the access "R--" or "RW-" when L=0!


Higher performance CPU designs (some, although I would imagine many) won't do PMP lookups at ld/st execution time, but at TLB fill time (i.e. caching PMP lookup info) - which eliminates timing and power concerns for those designs that don't want to do PMP lookups in parallel with execution.  Conversely, simpler, lower performance RISC-V designs appear content with doing parallel lookups (i.e. no one has been complaining, and people are finding that such RV cores are smaller and l  ower power than comparable ARM cores).
The problem with implementing PMP checks in page-walker and caching the information in the TLBs is that the PMP can be smaller than the smallest page size a TLB is normally defined to hold. So either a design needs to build TLBs that can go down to 4 byte granularity to support PMP or build the logic to support the lookup on each ld/st/i-fetch.

Even if such smaller TLBs were built and PMP were to be looked up only at page walk time looking up for example 16 TOR ranges is expensive to do in parallel and with 2 level paging enabled it could need up to 20 lookup in worst case where all paging structure caches miss increasing the cost of a TLB miss if timing forces adding a cycle or two to each stage of the page walk.


regards
ved


Greg Favor
 

On Thu, Oct 14, 2021 at 1:11 PM Aaron Durbin <adurbin@...> wrote:
That point necessarily requires providing a valid memory map for S/U in order for those modes to work at all if at least one PMP is implemented. Additionally, "If no PMP entry matches an S-mode or U-mode access, but at least one PMP entry is implemented, the access fails." That bit of the spec follows the intention of the first one. And "PMP violations are always trapped precisely at the processor." Makes sense, but it's also important to also think about side channel information leaks as well. Additionally, speculation has been relied upon to achieve high performant processors. One has to check against the PMPs for all load/stores in addition to instruction fetching.

The PMPs are creating a valid physical address space for U/S to operate within. There is no way, in the positive, to declare a region as M-mode only.

True with PMP, not true with ePMP. 
 
That would be the opposite approach to PMP. The PMP construction defines what is available to U/S and anything outside of that is M-mode only despite M-mode already having full access to the physical system address space (barring punching a whole w/ a PMP entry with L=1 and no permissions granted).

ePMP addresses this security hole (that the original PMP didn't address) of not being able to protect M-mode from itself (e.g. less trusted parts from other and/or more trusted parts).
 
... PMP regions (quibble: should this be registers?).

Yes.
 
We're paying the cost of every fetch and load/store against potentially 16 PMP entries with multiple modes and priority decoding on a per hart basis.

Higher performance CPU designs (some, although I would imagine many) won't do PMP lookups at ld/st execution time, but at TLB fill time (i.e. caching PMP lookup info) - which eliminates timing and power concerns for those designs that don't want to do PMP lookups in parallel with execution.  Conversely, simpler, lower performance RISC-V designs appear content with doing parallel lookups (i.e. no one has been complaining, and people are finding that such RV cores are smaller and l  ower power than comparable ARM cores).

This doesn't mean that someone can't come along and propose a lower-power "PMP" architecture (and go through the process of creating and running a TG to develop that into a new ratified standard) for the benefit of those designs doing parallel lookups (as it sounds like you will be doing).
 
In addition, anyone who has programmed x86 MTRRs (which have minimum 4KiB granularity) knows that it can be hard to cover the address space adequately. And, yes, I do realize those are about cacheability and not access control.

Hence why x86 and ARM have page-based memory types and RV is about to have them as well.  But this is really a PMA arch thing and not a PMP arch thing (although I get that an analogy was being made between the two).
 
Greg


Aaron Durbin
 



On Thu, Oct 14, 2021 at 12:48 PM Jonathan Behrens <behrensj@...> wrote:
There are no other extensions that are proposed/drafted/in-review/ratified/ in the architecture that can implement that specific intent. (right now)
     (except ePMP which is completely backwards compatible)
Mandating the use of the only available option sounds like a no-brainer to me.

Despite the known limitations?

I don't think there's broad agreement that PMP has significant limitations. Do you have pointers to any work showing that an alternative approach would be substantially cheaper? For instance, for the server platform, I have a hard time imagining that it amounts to anything more than a rounding error in the total cost. Servers can cost thousands of dollars!

I'll try and describe the current PMP implementation requirements. Yes, and as Allen noted up thread "Note that the PMP spec is highly configurable: how many entries, its granularity, how regions can be defined, and more.
It could even be a fixed, readonly structure.", there are options up to the implementer. While the platform spec doesn't provide a reasoning for requiring PMP, it's been said multiple times in on this list that it's for protecting m-mode assets.

PMP's premise is to allow access to S/U within the system's physical address space. Quoting the spec: "In effect, PMP can grant permissions to S and U modes, which by default have none, and can revoke permissions from M-mode, which by default has full permissions."

That point necessarily requires providing a valid memory map for S/U in order for those modes to work at all if at least one PMP is implemented. Additionally, "If no PMP entry matches an S-mode or U-mode access, but at least one PMP entry is implemented, the access fails." That bit of the spec follows the intention of the first one. And "PMP violations are always trapped precisely at the processor." Makes sense, but it's also important to also think about side channel information leaks as well. Additionally, speculation has been relied upon to achieve high performant processors. One has to check against the PMPs for all load/stores in addition to instruction fetching.

The PMPs are creating a valid physical address space for U/S to operate within. There is no way, in the positive, to declare a region as M-mode only. That would be the opposite approach to PMP. The PMP construction defines what is available to U/S and anything outside of that is M-mode only despite M-mode already having full access to the physical system address space (barring punching a whole w/ a PMP entry with L=1 and no permissions granted).

PMP matching needs a priority decoder by definition of the spec to match the lowest entry. The server extension in the platform spec is requiring 16 PMP regions (quibble: should this be registers?). A priority decode after the match stage is inherently necessary for 16 PMP regions.  In addition, there are sub-page granular modes as well as the TOR mode. The latter mode requires implementing 16 adders to form a range to comparison match. The full width of each transaction needs to be checked against every entry as well because of partial match rules. The sub-page granularity is it's own complexity, but Allen correctly noted the platform is free to determine its own granularity of PMPs.

One can opt to not implement TOR, as Allen alluded to, since the A field is WARL so that would remove some of the complexity. For performance purposes, I think one would want to toss out sub-page granular modes. That would then leave naturally aligned powers of 2. And now we're left with defining the physical system address space entirely in PMP entries because PMP, by definition, only grant U/S mode access. We're paying the cost of every fetch and load/store against potentially 16 PMP entries with multiple modes and priority decoding on a per hart basis.
 
Options for implementations:
- Granularity
- A Modes
- L support
- hardwire entries (including hardwiring them to 0)

Requirements to make U/S work:
- Have to implement something to grant U/S mode physical address space access

What can standard software rely upon given the constraints above? In addition, anyone who has programmed x86 MTRRs (which have minimum 4KiB granularity) knows that it can be hard to cover the address space adequately. And, yes, I do realize those are about cacheability and not access control.

And what about the implementation restrictions/results given all those things?


Out of curiosity, do you have a system you're designing that uses a custom alternative to PMP?

Yes, we have an internal doc, and we intend to provide a proposal for a simpler way of protecting m-mode assets. We hope to get that out soon (either end of the week or early next).
 
I hope the above description provides some insight.

-Aaron




On Thu, Oct 14, 2021 at 10:16 AM Aaron Durbin <adurbin@...> wrote:


On Wed, Oct 13, 2021 at 9:08 PM Allen Baum <allen.baum@...> wrote:
Just to share my 2 <name your favorite minor currency> here: 

The RV architecture is designed to be extremely modular, and has a lot of optional behaviors
This is intended to allow implementation freedom, which is a good thing.
It is also a way that can lead to unmanageable fragmentation - and that is a bad thing. A very bad thing.
It is a tenuous balance.
So, a meta-intent - that leads to other intents - is that we want to provide services without fragmenting the architecture

Agree on the potential for unmanageable fragmentation which should definitely planned for in preventing.
 

The platform spec - in this particular case - is mandating a specific iintention (M-mode protection)  use a specific extension .

This is a quibble, but I don't see m-mode protection explicitly noted in the platform spec. I see a mandate of PMP, though.
 
There are no other extensions that are proposed/drafted/in-review/ratified/ in the architecture that can implement that specific intent. (right now)
     (except ePMP which is completely backwards compatible)
Mandating the use of the only available option sounds like a no-brainer to me.

Despite the known limitations?
 
Note that the PMP spec is highly configurable: how many entries, its granularity, how regions can be defined, and more.
It could even be a fixed, readonly structure. 

If you want to argue that server platforms don't require M-mode protection - I'm not sure you'll find a lot of agreement.
If you want to argue that server platforms should use any form of protection they like (and be declared platforn compatible - ditto.
If you want to argue that server platform fragmentation is a good thing - best of luck.

I was arguing for not requiring M-mode protection nor that fragmentation is good for the sake of fragmentation. Please point to where I made those claims so I can rectify any misinterpretation. 

What I have been and am suggesting is that PMPs have a heavy cost to particular products so we shouldn't be requiring those in the platform spec.
 

That doesn't mean that some better definition will come up, or, if it does, won't be incorporated into a revision of the platform spec.
The platform spec is an evolving document, so if you have a better way, you are free to prose it, and if it is judged better, it can be adopted.
Supporting legacy spec is always fraught, but 




On Wed, Oct 13, 2021 at 7:09 PM Atish Patra <atish.patra@...> wrote:
On Wed, 2021-10-13 at 14:15 -0600, Aaron Durbin wrote:
>
>
> On Wed, Oct 13, 2021 at 12:12 PM Atish Patra <Atish.Patra@...>
> wrote:
> > On Wed, 2021-10-13 at 09:54 -0600, Aaron Durbin wrote:
> > >
> > >
> > > On Wed, Oct 13, 2021 at 9:50 AM Jonathan Behrens <   
> > > behrensj@...>
> > > wrote:
> > > > > > > 3. PMPs - I believe this is an M-mode intention to
> > > > > > > protect
> > m-
> > > > > mode assets. Why is an implementation being mandated instead
> > > > > of
> > > > > requesting M-mode asset protection from the harts? Moreover,
> > > > > there
> > > > > doesn't appear to be a requirement that protects M-mode
> > > > > assets
> > > > > from
> > > > > I/O agents.
> > > > >
> > > > > > Note that the current RV architecture (other than things
> > > > > > like
> > > > > AIA, Debug, and E-Trace) is "CPU" architecture.  Things like
> > > > > IOPMP
> > > > > and IOMMU will address the needs around I/O agents in a
> > > > > system
> > > > > (including equivalents to CPU PMAs and PMPs).
> > > > >
> > > > > So the spec doesn't address m-mode asset protection fully,
> > > > > but
> > it
> > > > > decided to focus on "cpu" and force an implementation? I
> > > > > understand the distinction between hart and !hart, but I
> > > > > don't
> > > > > see the point of requiring PMP specifically. What should be
> > > > > in
> > > > > the spec is the intention: provide a mechanism to protect m-
> > mode
> > > > > assets from hart access as well as I/O agents.
> > > > >
> > > >
> > > >
> > > > The spec requires PMP so that standard M-mode software like
> > OpenSBI
> > > > can assume that PMP is implemented. If the requirement was "PMP
> > or
> > > > any other asset protection implementation" then OpenSBI would
> > need
> > > > to have special handling for every different possible
> > > > implementation. I'd also point out that PMP isn't just for
> > > > defending against malicious code, it also protects against
> > > > buggy
> > > > code accidentally corrupting M-mode memory.
> > > >
> > >
> > >
> > > OpenSBI implementation already has platform specific code in it.
> > PMP,
> > > as specified, is not conducive to all designs. It has power and
> > > performance implications that can be onerous depending on what
> > > one
> > is
> > > optimizing for. Is the platform spec trying to protect the
> > > OpenSBI
> > > software project?
> >
> > Obviously not. PMP is the only standard method for protecting M-
> > mode
> > runtime code specified by ISA. That's why, it is mandated in
> > platform
> > spec and OpenSBI uses it. 
> >
>
>
> It is a method that has been used and is specified. However, I don't
> understand the leap from that to a hard requirement. Again, I believe
> the requirement should be focused on the intention -- not the
> implementation. 

That may lead to a lot of fragmentation in terms of implementation in
hardware and software customization. Isn't it ?

Another issue with just specifying the intent is that how do we verify
if a platform is compatible with the specification ? A custom mechanism
to protect higher privilege mode may also motivate the platform vendors
to provide vendor specific locked firmware instead of open firmware.

Having said that, platform spec can definitely relax the PMP
requirement in its future revisions for an alternative scheme that has
a lower power consumption & higher performance requirement.


> As I noted previously, the spec also completely omits protection from
> I/O agents. I'm not sure how to juggle both of those in my head: 1.
> Use a specific method for protecting a hart from touching m-mode
> assets. 2. don't bother specifying protection from an I/O agent.
>  

Because there are no current mechanism in RISC-V specification for I/O
protection except IOPMP proposal. I am not sure what's the status of
IOPMP. That can included if the IOPMP specification becomes a reality.
 
If you think adding just the intention here would be helpful, please
suggest something along that lines or send a patch.


> >
> > If the RISC-V allows any other standard mechanism in future,
> > OpenSBI
> > will support it. E.g. There were some discussion supporting ePMP in
> > OpenSBI. However, there are no usecase for that as of now. That's
> > why
> > it is not implemented yet.
> >
> >
> > Same is applicable for platform spec. Similar to interrupt
> > controller
> > options, the platform spec is open to have multiple options to
> > protect
> > M-mode implementation. It's just that we don't any other standard
> > option defined by the spec.
> >
>
>
> What is the definition of 'standard' being used here? 

Any specification that is being drafted/frozen/ratified under RVI.
As there are lot of moving pieces as of now, platform spec refers to
other specs that are not frozen yet. But these things has to happen in
parallel to make progress.


> I'd like to understand the requirements of the existence of a
> 'standard' and being a requirement in the platform spec. This is
> separate from the intention topic. But I've also noted the power and
> performance implications for an entity implementing PMP. Why is that
> not a concern used in the framework for certain requirements?
>   

Let me add some more clarification.

Intention: Protect m-mode run time code. (all of us agree that this
needs to happen)

Current Requirement: PMP is mandated because that is the only mechanism
provided under RV. AFAIK, all the platforms (except ariane) has
implemented PMP while agreeing that it is expensive. Power/Performance
consumption issues are known but there is no alternative.

Any other mechanism should be proposed as an ISA extension so that it
can be standardized and included in platform specification.

It is a separate discussion if we just leave at the intention and not
specify any requirement. I have given my 2 cents about that in the
above paragraph.


> >
> >
> > >  
> > > >
> > > > Jonathan
> > > >
> > > > On Wed, Oct 13, 2021 at 11:21 AM Aaron Durbin via
> > > > lists.riscv.org
> > <
> > > > adurbin=rivosinc.com@...> wrote:
> > > > >
> > > > >
> > > > > On Tue, Oct 12, 2021 at 9:51 PM Greg Favor <
> > > > > gfavor@...> wrote:
> > > > > > On Tue, Oct 12, 2021 at 2:23 PM Aaron Durbin <
> > > > > > adurbin@...> wrote:
> > > > > > > 1. Zam required, but as far as I can see there is no plan
> > for
> > > > > > > ratification in the current items on the docket for
> > > > > > > ratification.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Both Zam and Ztso are not yet ratified standards - which is
> > > > > > going to start to be rectified soon.
> > > > > >
> > > > >
> > > > >
> > > > > What's the plan for soon? And what is the definition of soon?
> > > > > It's not in the list of things to be standardized this Fall.
> > > > > It
> > > > > would be good to have visibility on these fronts.
> > > > >
> > > > > >  
> > > > > > > 1. ACLINT - why is this required? This is forcing an
> > > > > > > implementation to use that when it's not necessarily
> > needed.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Keep in mind that it is not as simple as saying that
> > > > > > "ACLINT
> > is
> > > > > > required".  The OS-A Base spec provides four options for
> > > > > > what
> > a
> > > > > > system implements as far as interrupt control.
> > > > > >
> > > > >
> > > > >
> > > > > The 2.1.4.1. Timer support section states "One or more ACLINT
> > > > > MTIMER devices are required for the OS-A platform"
> > > > >
> > > > > Is the MTIMER device inclusive of the MTIME register and the
> > set
> > > > > of MTIMECMP register sets? And what is the thinking here
> > > > > along
> > > > > with Sstc extension?
> > > > >
> > > > > I'm looking for more clarity in the spec that constitutes
> > 'ACLINT
> > > > > MTIMER devices'. Perhaps we should link to a particular
> > > > > section
> > > > > such as 
> > > > >
> > https://github.com/riscv/riscv-aclint/blob/main/riscv-aclint.adoc#2-machine-level-timer-device-mtimer
> > > > > It might even be worth separating the MTIMER device from the
> > > > > ACLINT spec itself since the requirement above is standalone.
> > > > >  
> > > > > >
> > > > > > In RISC-V there have been two de facto standards (PLIC and
> > > > > > CLINT) that were never properly established as official RV
> > > > > > standards but are used by many people - which is in the
> > process
> > > > > > of being addressed.  ACLINT is part of the effort to
> > > > > > standardize the existing CLINT, provide more flexibility to
> > > > > > CLINT implementations, and to add on the SSWI part that
> > mirrors
> > > > > > the existing MSWI part (for systems that don't want or need
> > to
> > > > > > implement AIA).
> > > > > >
> > > > >
> > > > >
> > > > > Thank you for the history.
> > > > >  
> > > > > >  
> > > > > > > 2. APLIC - Same question as ACLINT? Relatedly, why are
> > wired
> > > > > > > IRQs called out as being necessary and supported by way
> > > > > > > of
> > an
> > > > > > > APLIC? Why can't wired IRQs be supported in another way?
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The Base platform supports use of PLIC, ACLINT, and AIA
> > > > > > (and
> > > > > > its IMSIC and APLIC parts).  It is not requiring that all
> > > > > > OS-
> > A
> > > > > > Base compliant systems implement APLIC (or any part of
> > > > > > AIA).
> > > > > >
> > > > >
> > > > >
> > > > > But there is no option to not implement an APLIC or PLIC,
> > > > > correct? Why not? IMSIC on its own is certainly valid. 
> > > > >  
> > > > > >
> > > > > > For supporting wired interrupts one has two choices - PLIC
> > and
> > > > > > APLIC.  (And within RISC-V there will also be CLIC for more
> > > > > > embedded-oriented designs.)
> > > > > >
> > > > >
> > > > >
> > > > > Why just those two? And are these wired interrupts internal
> > > > > to
> > a
> > > > > part or external? Why couldn't one implement a different way
> > > > > to
> > > > > generate IRQs from an external wire? That is definitely
> > dictating
> > > > > product implementation.
> > > > >
> > > > > And where does this 'wired IRQ' requirement come from
> > > > > exactly?
> > > > > There's seems to be some implied assumptions, but they are
> > > > > not
> > > > > clear what that section is trying to address.
> > > > >  
> > > > > >  
> > > > > > > 3. PMPs - I believe this is an M-mode intention to
> > > > > > > protect
> > m-
> > > > > > > mode assets. Why is an implementation being mandated
> > instead
> > > > > > > of requesting M-mode asset protection from the harts?
> > > > > > > Moreover, there doesn't appear to be a requirement that
> > > > > > > protects M-mode assets from I/O agents.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Note that the current RV architecture (other than things
> > > > > > like
> > > > > > AIA, Debug, and E-Trace) is "CPU" architecture.  Things
> > > > > > like
> > > > > > IOPMP and IOMMU will address the needs around I/O agents in
> > > > > > a
> > > > > > system (including equivalents to CPU PMAs and PMPs).
> > > > > >
> > > > >
> > > > >
> > > > > So the spec doesn't address m-mode asset protection fully,
> > > > > but
> > it
> > > > > decided to focus on "cpu" and force an implementation? I
> > > > > understand the distinction between hart and !hart, but I
> > > > > don't
> > > > > see the point of requiring PMP specifically. What should be
> > > > > in
> > > > > the spec is the intention: provide a mechanism to protect m-
> > mode
> > > > > assets from hart access as well as I/O agents.
> > > > >
> > > > > -Aaron
> > > > > > >  
> >

--
Regards,
Atish






Greg Favor
 

On Thu, Oct 14, 2021 at 12:03 PM Aaron Durbin <adurbin@...> wrote:
Once there is/are suitable (ratified) standard(s) for doing this, updated platform specs can incorporate them.  But in the meantime the initial platform specs can only stay silent or can express a very high-level requirement for some implementation-specific form of mechanism.  (I can see arguments for and against both approaches.)  This is similar to the situation with things like RAS, IOMMU, and several security areas - for which the initial OS-A and M Base platforms stay silent while RVI efforts work to fill in these holes.

That was why I suggested language that expressed that high-level requirement for that purpose of covering the bases of known gaps. Teh point of the platform spec is to provide a target for an ecosystem, but it's also being positioned to express that risc-v has these ttypes of things covered.

I don't believe people developing the profiles and platforms, as well as Tech Chairs, the TSC, and the board, consider that these things are covered.  If anything, it is the opposite.  (That can certainly be stated explicitly in profile and platform specs.)
 
If not, the brand recognition of the platform spec can be undermined. If I am wrong in the platform spec trying to serve both purposes please let me know as that is exactly how the secure boot topic came about. "We need to say something about security in 2022."

There were arguments back and around this, with a conclusion that the base OS-A and M platforms will remain silent for 2022.  And it is just the Server extension that expresses just a high-level secure boot requirement until RV secure boot standards have been developed that platforms can then mandate; and the Server extension will remain silent on a number of other security topics.

Greg


Greg Favor
 

On Thu, Oct 14, 2021 at 11:56 AM Aaron Durbin <adurbin@...> wrote:
Because most of the current RV community doesn't share this view doesn't mean a counter view is correct. Would you agree?

Invariably, with the breadth of the RV community, there is the whole spectrum of views.  Which is why RV arch is modular and has lots of optionality, and why there will be multiple profiles and platform (and growing over time).  No one thing can cater to everyone.  The topic of "IOMMU" is an example.  Embedded-oriented and enterprise-class IOMMUs want to be very different animals (as we'll likely see with coming IOPMP and IOMMU standards).


On what dimension is your assessment that a few PMPs are peanuts? Performance? Power? And what number of PMPs? I would like to hear about the assumptions made in that assessment w.r.t power, clock targets, etc. Obviously those dimensions are things people are defining for their own products.

Everyone will judge this differently, but the people doing cost-sensitive designs that care about security don't find a few PMPs to be a significant performance or power issue (and they tend to have a small number of PMPs).  But even within that arena, people differ in what they want and what they are willing to implement, i.e. there isn't a simple cut-and-dried answer.

Greg


Aaron Durbin
 



On Thu, Oct 14, 2021 at 12:59 PM Greg Favor <gfavor@...> wrote:
On Thu, Oct 14, 2021 at 11:43 AM Aaron Durbin <adurbin@...> wrote:
To probably say the obvious, the current RV arch defines one standard (aka ratified) M-mode protection mechanism for harts, and does not yet define a standard M-mode protection mechanism for I/O agents.  The current IOPMP effort is one effort working to address this (along with other "IOMMU" type issues).  An IOMMU effort should also address this.  (In the latter case, I would argue that the "PMP" part of an IOMMU arch should maximize consistency with the existing PMP part of the CPU arch.  Just as it should maximize consistency with the Priv MMU arch, H, and AIA.  Insofar as when an alternative PMP-like scheme becomes standardized (i.e. ratified), then that will likely also need to reconcile how it fits into both the CPU and IOMMU architectures.)

Greg, were you answering "What good is hart protection if an I/O agent can stomp all over the machine mode assets?"? If so, what you described doesn't seem to address that question directly. This falls into the security pieces of the platform spec in that people were wanting to signal "risc-v cares about security". Now we're specifying things related to m-mode asset protection (though not explicitly stated in the spec) and omitting protection from I/O agent tampering. That construction inherently leaves a hole w.r.t. m-mode asset protection.

I'm not disagreeing with the point that there also needs to be protection from non-hart masters in a system (aka I/O agents).  I was just pointing out that RV so far has only addressed half of the overall picture, i.e. the CPU agent side.  And that existing/new efforts like IOPMP and IOMMU need to fill in the other side.

Once there is/are suitable (ratified) standard(s) for doing this, updated platform specs can incorporate them.  But in the meantime the initial platform specs can only stay silent or can express a very high-level requirement for some implementation-specific form of mechanism.  (I can see arguments for and against both approaches.)  This is similar to the situation with things like RAS, IOMMU, and several security areas - for which the initial OS-A and M Base platforms stay silent while RVI efforts work to fill in these holes.

That was why I suggested language that expressed that high-level requirement for that purpose of covering the bases of known gaps. Teh point of the platform spec is to provide a target for an ecosystem, but it's also being positioned to express that risc-v has these ttypes of things covered. If not, the brand recognition of the platform spec can be undermined. If I am wrong in the platform spec trying to serve both purposes please let me know as that is exactly how the secure boot topic came about. "We need to say something about security in 2022."

-Aaron


Greg Favor
 

On Thu, Oct 14, 2021 at 11:43 AM Aaron Durbin <adurbin@...> wrote:
To probably say the obvious, the current RV arch defines one standard (aka ratified) M-mode protection mechanism for harts, and does not yet define a standard M-mode protection mechanism for I/O agents.  The current IOPMP effort is one effort working to address this (along with other "IOMMU" type issues).  An IOMMU effort should also address this.  (In the latter case, I would argue that the "PMP" part of an IOMMU arch should maximize consistency with the existing PMP part of the CPU arch.  Just as it should maximize consistency with the Priv MMU arch, H, and AIA.  Insofar as when an alternative PMP-like scheme becomes standardized (i.e. ratified), then that will likely also need to reconcile how it fits into both the CPU and IOMMU architectures.)

Greg, were you answering "What good is hart protection if an I/O agent can stomp all over the machine mode assets?"? If so, what you described doesn't seem to address that question directly. This falls into the security pieces of the platform spec in that people were wanting to signal "risc-v cares about security". Now we're specifying things related to m-mode asset protection (though not explicitly stated in the spec) and omitting protection from I/O agent tampering. That construction inherently leaves a hole w.r.t. m-mode asset protection.

I'm not disagreeing with the point that there also needs to be protection from non-hart masters in a system (aka I/O agents).  I was just pointing out that RV so far has only addressed half of the overall picture, i.e. the CPU agent side.  And that existing/new efforts like IOPMP and IOMMU need to fill in the other side.

Once there is/are suitable (ratified) standard(s) for doing this, updated platform specs can incorporate them.  But in the meantime the initial platform specs can only stay silent or can express a very high-level requirement for some implementation-specific form of mechanism.  (I can see arguments for and against both approaches.)  This is similar to the situation with things like RAS, IOMMU, and several security areas - for which the initial OS-A and M Base platforms stay silent while RVI efforts work to fill in these holes.
 
Greg


Aaron Durbin
 



On Thu, Oct 14, 2021 at 12:44 PM Greg Favor <gfavor@...> wrote:
On Thu, Oct 14, 2021 at 7:15 AM Aaron Durbin <adurbin@...> wrote:
What I have been and am suggesting is that PMPs have a heavy cost to particular products so we shouldn't be requiring those in the platform spec.

In practice, most of the RV community doesn't share this view.  Certainly there are implementations that choose to not have any PMP or to have a small number of PMPs or to have a bunch that are hardwired (as ways to limit the cost).  For Rich O/S (e.g. Linux) systems, the cost of a few PMPs is peanuts in the overall scheme of things (and security is a growing issue in the industry).  Conversely, non-Rich O/S systems have different platform spec they can conform to that doesn't require any PMP (i.e. the base M platform, although there is also going to be a security-oriented M platform extension that not surprisingly requires PMP).  In general, RV is gradually taking security more seriously with various TG efforts in various stages of progress.

Because most of the current RV community doesn't share this view doesn't mean a counter view is correct. Would you agree?

On what dimension is your assessment that a few PMPs are peanuts? Performance? Power? And what number of PMPs? I would like to hear about the assumptions made in that assessment w.r.t power, clock targets, etc. Obviously those dimensions are things people are defining for their own products.


And just to note:  None of this intrinsically prevents development and standardization of a second "M-mode protection" scheme that could be referenced and used by subsequent platform specs.

Greg