From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

https://docs.google.com/document/d/1-UlaSGqk59_myeuPMrV9gyuaIgnmFzGh5Gfy_tpViwM/edit

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

https://docs.google.com/document/d/1-UlaSGqk59_myeuPMrV9gyuaIgnmFzGh5Gfy_tpViwM/edit

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv

Should I assume the following as a standard extension to H-extension? 

·         hideleg – add a bit13 for delegation of PMI from HS to VS-mode

·          

·         hip, hie, vsip and vsie – add a bit13 for PMI pending and enable for HS and VS-modes.

·         hvip – add a bit13 for injection of interrupts from HS-mode into guest. Extended by default.

·         Is this useful? Seems like overkill there for symmetry. 

If so, I assume that these modifications will be made explicit in the specification.

 

Sanjay   

 

From: <tech-privileged@...> on behalf of "Anup Patel via lists.riscv.org" <anup.patel=wdc.com@...>
Reply-To: "anup.patel@..." <anup.patel@...>
Date: Wednesday, February 24, 2021 at 9:07 PM
To: Sanjay Patel <spatel@...>, "gfavor@..." <gfavor@...>, Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

H-extension is already considered in this proposal. We have filter bits for VS/VU modes only events.

 

Linux PMU driver will using SBI PMU extension. The SBI PMU extension will be implemented in platform M-mode firmware and HS-mode hypervisors. This means same SBI-based PMU driver can be used natively and inside Guest/VM. The Guest Linux will be able to directly access counters and overflow CSR directly from VS-mode.

 

Regards,

Anup

 

From: tech-privileged@... <tech-privileged@...> On Behalf Of Sanjay Patel
Sent: 25 February 2021 07:34
To: gfavor@...; Brian Grayson <brian.grayson@...>; tech-privileged@...
Cc: Jeffrey Osier-Mixon <jefro@...>; Stephano Cetola <stephano@...>; Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

https://docs.google.com/document/d/1-UlaSGqk59_myeuPMrV9gyuaIgnmFzGh5Gfy_tpViwM/edit

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv

What is the bare-minimum that is required in terms of the CSRs to have the SBI calls work for PMIs? I understand that AIA will be defined, but it is not available at this time.

 mideleg[13]. Yes, since LCOFIP is defined as mip[13].  If so, the PMI can be delegated to HS-mode. I assume this is needed for the SBI call from HS-mode.

I would assume with an AIA, the PMI would be routed to the controller, prioritized and then cause an interrupt in the hart. Yet, the PMI still needs to be logged in the harts CSR in some manner.

Btw, software timers will have to be similarly routed to the controller for prioritization. Yet they are fully defined and delegated.