Date   

Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Sanjay Patel <spatel@...>
 

Thanks Greg for the prompt response.

 

Yes, AIA is of interest to us. This will allow us to replace the home-grown controller we have now.

 

One last question(s). I assume hideleg[13] is supported (as well as mideleg[13], which has been confirmed). This allows the interrupt to be taken in VS-mode and thus vsip[13] and vsie[13] are required.

And scountovf is common to both HS and VS-modes – the read is dependent on which context is interrupted.  If you can confirm, then I’ll be done.   

 

Otherwise I understand that local interrupts will need to be prioritized *locally* wrt to interrupts from the proposed external interrupt controller. This is different from the scheme that I’ve used in the past. Its good though to keep top-level wires to a minimum.

 

Regards,

Sanjay

 

From: Greg Favor <gfavor@...>
Date: Thursday, February 25, 2021 at 6:34 PM
To: Sanjay Patel <spatel@...>
Cc: "anup.patel@..." <anup.patel@...>, Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>, Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

On Thu, Feb 25, 2021 at 4:08 PM Sanjay Patel <spatel@...> wrote:

What is the bare-minimum that is required in terms of the CSRs to have the SBI calls work for PMIs? I understand that AIA will be defined, but it is not available at this time.

 

The AIA SIG should be started by early-to-mid March.  So a first draft of a proposed spec is coming very soon.  (Soon enough, it sounds like, for your timeframe?)

 

 mideleg[13]. Yes, since LCOFIP is defined as mip[13].  If so, the PMI can be delegated to HS-mode. I assume this is needed for the SBI call from HS-mode.

 

The SBI call is for initializing a counter.  This extension allows all the overflow interrupt handling to be done in S-mode without any calls up to M-mode.  The two are quite separate things.

 

I would assume with an AIA, the PMI would be routed to the controller, prioritized and then cause an interrupt in the hart. Yet, the PMI still needs to be logged in the harts CSR in some manner.

 

As mip/sip bit 13, this is a _local_ interrupt.  It wouldn't go out to any system interrupt controller.  Whether you consider it going into the part of AIA that would be local to a hart, or just going directly to the hart's interrupt prioritization and recognition logic (that looks at all the pending bits in mip/sip), it is essentially that latter logic that takes in LCOFIP.  And when the interrupt is taken, the usual architectural exception handling occurs (loading the usual exception registers with appropriate values).

 

Btw, software timers will have to be similarly routed to the controller for prioritization. Yet they are fully defined and delegated.

 

Just in the above sense.

 

Greg

 


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

On Thu, Feb 25, 2021 at 4:08 PM Sanjay Patel <spatel@...> wrote:

What is the bare-minimum that is required in terms of the CSRs to have the SBI calls work for PMIs? I understand that AIA will be defined, but it is not available at this time.


The AIA SIG should be started by early-to-mid March.  So a first draft of a proposed spec is coming very soon.  (Soon enough, it sounds like, for your timeframe?)
 

 mideleg[13]. Yes, since LCOFIP is defined as mip[13].  If so, the PMI can be delegated to HS-mode. I assume this is needed for the SBI call from HS-mode.


The SBI call is for initializing a counter.  This extension allows all the overflow interrupt handling to be done in S-mode without any calls up to M-mode.  The two are quite separate things.
 

I would assume with an AIA, the PMI would be routed to the controller, prioritized and then cause an interrupt in the hart. Yet, the PMI still needs to be logged in the harts CSR in some manner.


As mip/sip bit 13, this is a _local_ interrupt.  It wouldn't go out to any system interrupt controller.  Whether you consider it going into the part of AIA that would be local to a hart, or just going directly to the hart's interrupt prioritization and recognition logic (that looks at all the pending bits in mip/sip), it is essentially that latter logic that takes in LCOFIP.  And when the interrupt is taken, the usual architectural exception handling occurs (loading the usual exception registers with appropriate values).
 

Btw, software timers will have to be similarly routed to the controller for prioritization. Yet they are fully defined and delegated.


Just in the above sense.

Greg


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

On Thu, Feb 25, 2021 at 4:08 PM Sanjay Patel <spatel@...> wrote:

 

What is the bare-minimum that is required in terms of the CSRs to have the SBI calls work for PMIs? I understand that AIA will be defined, but it is not available at this time.

 

mideleg[13]. Yes, since LCOFIP is defined as mip[13].  If so, the PMI can be delegated to HS-mode. I assume this is needed for the SBI call from HS-mode.

hideleg[13]? You say it follows from the spec, so I assume this should be implemented. If so, the PMI can be delegated to VS-mode. I assume this is needed for the SBI call from VS-mode.

 

But for these to work, we need to implement hip, hie, vsip and vsie. I’m not sure if you said it shouldn’t be implemented, or that it just complicated. 😊

 

I would assume with an AIA, the PMI would be routed to the controller, prioritized and then cause an interrupt in the hart. Yet, the PMI still needs to be logged in the harts CSR in some manner.

 

Btw, software timers will have to be similarly routed to the controller for prioritization. Yet they are fully defined and delegated.

 

I don’t need an immediate answer, as implementing the PMU is still some ways off. I would however like to know what the bare minimum is to get the SBI calls to work with H-extension but w/o the the AIA.


If you care about virtualization and having an interrupt controller that supports virtualization and the H-extension, the only game in town is AIA (which is one of the motivating factors for it).  PLIC and CLIC have no concept of or support for virtualization (although you can of course cook up your own custom extension to one of these).

Also note that it would be the same SBI interface between a VM and the hypervisor, as between the hypervisor and M-mode.  Supervisor code will call through OpenSBI to have an hpmcounter initialized (mhpmcounterX and mhpmeventX).  Whether in one or two steps that will end up at M-mode (which will do the actual CSR updates).

In my next email I'll go back and respond to your other points.

Greg




Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Sanjay Patel <spatel@...>
 

 

What is the bare-minimum that is required in terms of the CSRs to have the SBI calls work for PMIs? I understand that AIA will be defined, but it is not available at this time.

 

mideleg[13]. Yes, since LCOFIP is defined as mip[13].  If so, the PMI can be delegated to HS-mode. I assume this is needed for the SBI call from HS-mode.

hideleg[13]? You say it follows from the spec, so I assume this should be implemented. If so, the PMI can be delegated to VS-mode. I assume this is needed for the SBI call from VS-mode.

 

But for these to work, we need to implement hip, hie, vsip and vsie. I’m not sure if you said it shouldn’t be implemented, or that it just complicated. 😊

 

I would assume with an AIA, the PMI would be routed to the controller, prioritized and then cause an interrupt in the hart. Yet, the PMI still needs to be logged in the harts CSR in some manner.

 

Btw, software timers will have to be similarly routed to the controller for prioritization. Yet they are fully defined and delegated.

 

I don’t need an immediate answer, as implementing the PMU is still some ways off. I would however like to know what the bare minimum is to get the SBI calls to work with H-extension but w/o the the AIA.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "gfavor@..." <gfavor@...>
Date: Thursday, February 25, 2021 at 11:45 AM
To: Sanjay Patel <spatel@...>
Cc: "anup.patel@..." <anup.patel@...>, Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>, Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

On Thu, Feb 25, 2021 at 11:00 AM Sanjay Patel <spatel@...> wrote:

Should I assume the following as a standard extension to H-extension? 

·         hideleg – add a bit13 for delegation of PMI from HS to VS-mode

 

By default hideleg functionality applies to all bits in the ip/*ie registers. So nothing special or extra needs to be specified in this extension. 

 

·          

·         hip, hie, vsip and vsie – add a bit13 for PMI pending and enable for HS and VS-modes.

·         hvip – add a bit13 for injection of interrupts from HS-mode into guest. Extended by default.

·         Is this useful? Seems like overkill there for symmetry. 

As one can start to see, this manner of supporting delegation involves defining a bunch of bits.  Which is not a clean and scalable solution as other bits in the ip/*ie registers come along with the same need.  The preferred and better solution will be provided by the "interrupt filtering" functionality that will be in the AIA (Advanced Interrupt Architecture) spec that is being developed to fully support interrupt virtualization (among other goals).  (The AIA SIG will be starting up very soon.)

 

Greg

 

If so, I assume that these modifications will be made explicit in the specification.

 

Sanjay   

 

From: <tech-privileged@...> on behalf of "Anup Patel via lists.riscv.org" <anup.patel=wdc.com@...>
Reply-To: "anup.patel@..." <anup.patel@...>
Date: Wednesday, February 24, 2021 at 9:07 PM
To: Sanjay Patel <spatel@...>, "gfavor@..." <gfavor@...>, Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

H-extension is already considered in this proposal. We have filter bits for VS/VU modes only events.

 

Linux PMU driver will using SBI PMU extension. The SBI PMU extension will be implemented in platform M-mode firmware and HS-mode hypervisors. This means same SBI-based PMU driver can be used natively and inside Guest/VM. The Guest Linux will be able to directly access counters and overflow CSR directly from VS-mode.

 

Regards,

Anup

 

From: tech-privileged@... <tech-privileged@...> On Behalf Of Sanjay Patel
Sent: 25 February 2021 07:34
To: gfavor@...; Brian Grayson <brian.grayson@...>; tech-privileged@...
Cc: Jeffrey Osier-Mixon <jefro@...>; Stephano Cetola <stephano@...>; Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "
gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <
brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <
jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

On Thu, Feb 25, 2021 at 11:00 AM Sanjay Patel <spatel@...> wrote:

Should I assume the following as a standard extension to H-extension? 

  • hideleg – add a bit13 for delegation of PMI from HS to VS-mode

By default hideleg functionality applies to all bits in the ip/*ie registers. So nothing special or extra needs to be specified in this extension. 
 
  • hip, hie, vsip and vsie – add a bit13 for PMI pending and enable for HS and VS-modes.
  • hvip – add a bit13 for injection of interrupts from HS-mode into guest. Extended by default.
    • Is this useful? Seems like overkill there for symmetry. 
As one can start to see, this manner of supporting delegation involves defining a bunch of bits.  Which is not a clean and scalable solution as other bits in the ip/*ie registers come along with the same need.  The preferred and better solution will be provided by the "interrupt filtering" functionality that will be in the AIA (Advanced Interrupt Architecture) spec that is being developed to fully support interrupt virtualization (among other goals).  (The AIA SIG will be starting up very soon.)

Greg
 

If so, I assume that these modifications will be made explicit in the specification.

 

Sanjay   

 

From: <tech-privileged@...> on behalf of "Anup Patel via lists.riscv.org" <anup.patel=wdc.com@...>
Reply-To: "anup.patel@..." <anup.patel@...>
Date: Wednesday, February 24, 2021 at 9:07 PM
To: Sanjay Patel <spatel@...>, "gfavor@..." <gfavor@...>, Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

H-extension is already considered in this proposal. We have filter bits for VS/VU modes only events.

 

Linux PMU driver will using SBI PMU extension. The SBI PMU extension will be implemented in platform M-mode firmware and HS-mode hypervisors. This means same SBI-based PMU driver can be used natively and inside Guest/VM. The Guest Linux will be able to directly access counters and overflow CSR directly from VS-mode.

 

Regards,

Anup

 

From: tech-privileged@... <tech-privileged@...> On Behalf Of Sanjay Patel
Sent: 25 February 2021 07:34
To: gfavor@...; Brian Grayson <brian.grayson@...>; tech-privileged@...
Cc: Jeffrey Osier-Mixon <jefro@...>; Stephano Cetola <stephano@...>; Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "
gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <
brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <
jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Sanjay Patel <spatel@...>
 

Thanks for the clarification.

 

Should I assume the following as a standard extension to H-extension?

 

  • hideleg – add a bit13 for delegation of PMI from HS to VS-mode
  • hip, hie, vsip and vsie – add a bit13 for PMI pending and enable for HS and VS-modes.
  • hvip – add a bit13 for injection of interrupts from HS-mode into guest. Extended by default.
    • Is this useful? Seems like overkill there for symmetry.

 

If so, I assume that these modifications will be made explicit in the specification.

 

Sanjay   

 

From: <tech-privileged@...> on behalf of "Anup Patel via lists.riscv.org" <anup.patel=wdc.com@...>
Reply-To: "anup.patel@..." <anup.patel@...>
Date: Wednesday, February 24, 2021 at 9:07 PM
To: Sanjay Patel <spatel@...>, "gfavor@..." <gfavor@...>, Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

H-extension is already considered in this proposal. We have filter bits for VS/VU modes only events.

 

Linux PMU driver will using SBI PMU extension. The SBI PMU extension will be implemented in platform M-mode firmware and HS-mode hypervisors. This means same SBI-based PMU driver can be used natively and inside Guest/VM. The Guest Linux will be able to directly access counters and overflow CSR directly from VS-mode.

 

Regards,

Anup

 

From: tech-privileged@... <tech-privileged@...> On Behalf Of Sanjay Patel
Sent: 25 February 2021 07:34
To: gfavor@...; Brian Grayson <brian.grayson@...>; tech-privileged@...
Cc: Jeffrey Osier-Mixon <jefro@...>; Stephano Cetola <stephano@...>; Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "
gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <
brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <
jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

On Wed, Feb 24, 2021 at 5:55 PM Atish Patra <Atish.Patra@...> wrote:
I can work on implementing the Sscofpmf extension in Qemu and required
software changes in OpenSBI & Linux kernel as well if that is okay with
you.

Thanks.  That would be great.  Let me also send you (tomorrow) the "final" version as submitted to the OACR committee.

Do we require anything else for the PoC part of the DoD policy?

Let me get back to you on that (probably after I talk with Andrew).

Greg


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Allen Baum
 

Ah, branch-to-self - I should have thought of that. (slapping myself upside the head).
Then it doesn't matter how long the signal takes. 
And, we wouldn't even need a particularly deterministic timer.


It does mean that if you read the counter after returning from the trap, you won't get the same number that the reference model would,
 but perhaps all you care about at that point is ]that it's a reasonably small number>0.

Your suggestion of architecturally defining a few events if the extension is present is what I was getting at when I said "by further standardizing event selection to include the equivalent [of instret].

You don't have to set the counter to the overflow value and enable interrupts. You could set to overflow-N, then enable interrupts, then branch to self.
N>3 should guarantee that you get to the branch to self loop before the interrupt arrives.
So instead of my suggestion to
   disable interrupt, set timer to overflow at some small interval, wait longer than the interval, then enable interrupts (MIE[13]) 
you 
    disable interrupt, set timer to overflow at some small interval, enable interrupts (MIE[13]) and branch-to-self

Note that the arch-test framework standard trap handle will not return to xEPC, but to the next aligned 8B address past xEPC+2 - so no chance of returning to an infinite loop, no chance of returning to a misaligned address, and there is always guaranteed padding that can be put to use to catch fall-throughs that weren't supposed to happen.


On Wed, Feb 24, 2021 at 9:37 PM Brian Grayson <brian.grayson@...> wrote:
Another way to test that might work is, to disable the interrupt, set the counter to an overflow value, and then enable interrupts. But....

Unfortunately, the use of an overflow bit outside the counter makes this more difficult if not impossible. (This is one minor reason why I prefer the "counters are unsigned, but overflow happens when bit 63 is set, i.e. when unsigned 63-bit overflow happens" approach, instead of "counters are unsigned, and overflow happens when bit 63 transitions from a 1 to a 0, i.e. unsigned 64-bit overflow happens". It also changes the overflow condition from an action/transition (carry-out occurred) to a state (high bit is set).) The dual use of the OF bit to signal "overflow has occurred AND interrupts are disabled" also precludes setting the OF bit to force an immediate interrupt when the interrupt is no longer blocked.

Alternatively, we could add mhpmevent2 to the proposal, that allows filtering/masking/interrupts on instret, as Allen points out.

Alternatively, we could define one or a few events as part of the architecture, so that if you have this extension, you CAN count instructions or cycles in a predictable, maskable, overflowable way in one of the configurable counters.

We could work around the xEPC-might-be-off-by-an-instruction-or-two issue by making the testcase do an infinite loop of branch-to-self, so that the xEPC is now a constant. :)

Brian

On Wed, Feb 24, 2021 at 11:07 PM Anup Patel <Anup.Patel@...> wrote:

H-extension is already considered in this proposal. We have filter bits for VS/VU modes only events.

 

Linux PMU driver will using SBI PMU extension. The SBI PMU extension will be implemented in platform M-mode firmware and HS-mode hypervisors. This means same SBI-based PMU driver can be used natively and inside Guest/VM. The Guest Linux will be able to directly access counters and overflow CSR directly from VS-mode.

 

Regards,

Anup

 

From: tech-privileged@... <tech-privileged@...> On Behalf Of Sanjay Patel
Sent: 25 February 2021 07:34
To: gfavor@...; Brian Grayson <brian.grayson@...>; tech-privileged@...
Cc: Jeffrey Osier-Mixon <jefro@...>; Stephano Cetola <stephano@...>; Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "
gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <
brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <
jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Brian Grayson
 

Another way to test that might work is, to disable the interrupt, set the counter to an overflow value, and then enable interrupts. But....

Unfortunately, the use of an overflow bit outside the counter makes this more difficult if not impossible. (This is one minor reason why I prefer the "counters are unsigned, but overflow happens when bit 63 is set, i.e. when unsigned 63-bit overflow happens" approach, instead of "counters are unsigned, and overflow happens when bit 63 transitions from a 1 to a 0, i.e. unsigned 64-bit overflow happens". It also changes the overflow condition from an action/transition (carry-out occurred) to a state (high bit is set).) The dual use of the OF bit to signal "overflow has occurred AND interrupts are disabled" also precludes setting the OF bit to force an immediate interrupt when the interrupt is no longer blocked.

Alternatively, we could add mhpmevent2 to the proposal, that allows filtering/masking/interrupts on instret, as Allen points out.

Alternatively, we could define one or a few events as part of the architecture, so that if you have this extension, you CAN count instructions or cycles in a predictable, maskable, overflowable way in one of the configurable counters.

We could work around the xEPC-might-be-off-by-an-instruction-or-two issue by making the testcase do an infinite loop of branch-to-self, so that the xEPC is now a constant. :)

Brian

On Wed, Feb 24, 2021 at 11:07 PM Anup Patel <Anup.Patel@...> wrote:

H-extension is already considered in this proposal. We have filter bits for VS/VU modes only events.

 

Linux PMU driver will using SBI PMU extension. The SBI PMU extension will be implemented in platform M-mode firmware and HS-mode hypervisors. This means same SBI-based PMU driver can be used natively and inside Guest/VM. The Guest Linux will be able to directly access counters and overflow CSR directly from VS-mode.

 

Regards,

Anup

 

From: tech-privileged@... <tech-privileged@...> On Behalf Of Sanjay Patel
Sent: 25 February 2021 07:34
To: gfavor@...; Brian Grayson <brian.grayson@...>; tech-privileged@...
Cc: Jeffrey Osier-Mixon <jefro@...>; Stephano Cetola <stephano@...>; Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "
gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <
brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <
jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Anup Patel
 

H-extension is already considered in this proposal. We have filter bits for VS/VU modes only events.

 

Linux PMU driver will using SBI PMU extension. The SBI PMU extension will be implemented in platform M-mode firmware and HS-mode hypervisors. This means same SBI-based PMU driver can be used natively and inside Guest/VM. The Guest Linux will be able to directly access counters and overflow CSR directly from VS-mode.

 

Regards,

Anup

 

From: tech-privileged@... <tech-privileged@...> On Behalf Of Sanjay Patel
Sent: 25 February 2021 07:34
To: gfavor@...; Brian Grayson <brian.grayson@...>; tech-privileged@...
Cc: Jeffrey Osier-Mixon <jefro@...>; Stephano Cetola <stephano@...>; Mark Himelstein <markhimelstein@...>
Subject: Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "
gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <
brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <
jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Allen Baum
 

Addendum: 
 without some kind of event standardization, even after all of this, we can only test one counter...not great coverage.

On Wed, Feb 24, 2021 at 8:03 PM Allen Baum <allen.baum@...> wrote:
I think I have trouble answering questions concisely.

From a Definition of Done perspective, you're supposed to have architectural tests that will pass when run on the Sail model and on Spike -
more specifically, generate the same signature. 

The signature for this case would have to be that it interrupted where it was expected on both the reference model and the device under test
We do that by comparing trap state: xEPC, xCAUSE, xIP, xTVAL by the interrupt handler.
That works well for synchronous traps, but works less well for asynchronous events, e.g. interrupts.
If you can't inject the interrupt at a deterministic point, xEPC will differ.

The only way I see to pick a deterministic point is to interrupt on the INSTRET counter, which is as close to architectural as you might get.

There are (at least) 4 problems with this:
 
1. As has been pointed out, counter events are implementation specific.
    The only solution I can see is to choose instret, which is writable, required, and deterministic
2. Oops, the current definition of the instruction doesn't include instret.
    This can be fixed by adding another overflow bit , or by further standardizing event selection to include the equivalent 
3 There is no architectural requirement that the interval from an interrupt event occurring to the first instruction of the handler being executed.
4. what happens when an implementation retires more than instruction per cycle? 
    These last 2 are related: even if the event occurs at a deterministic point, the interrupt still won't necessarily occur at a deterministic point

I think we have possible solutions from a framework perspective, but they're ways off .
This sometimes requires the reference model (Sail or possibly Spike) to be able to replicate the possible behaviors.

For this particular case there may be a test structure that might work: 
  disable counter interrupts until after the point that the counter overflows, and enable it at some deterministic point later.
Again: I don't know if there an architectural requirement for the interval between an instruction enabling an interrupt and the first instruction of a trap handler executing
But, if there is AND we can guarantee a universal architectural event - we have a chance of this working.
Otherwise, we would have to have a modified version of the framework (e.g. one that somehow allows fuzzy comparisons for some specific signature values)

On Wed, Feb 24, 2021 at 5:55 PM Atish Patra <atish.patra@...> wrote:
On Wed, 2021-02-24 at 17:01 -0800, Greg Favor wrote:
> Cc'ing tech-priv since others may be wondering about the answer to
> Brian's question.
>
> Brian,
>
> As Mark touched on below, there is a whole "Definition of Done"
> checklist of items that needs to be done (including software support,
> Spike and Sail models, OACR review, PoC, ...).  So that is the next
> order of business that I need to work on - where "I" doesn't mean me
> doing it all (or even having the expertise to do all those things). 
> Btw, would you or anyone else be willing and able to help out with
> one of the DoD checklist items?  Any help will be greatly appreciated
> and will help move this ball towards the goal line.
>

Hi Greg,
I am working on the SBI PMU extension implementation in OpenSBI & Linux
kernel. I will update the SBI PMU extension based on Sscofpmf extension
as well.

I can work on implementing the Sscofpmf extension in Qemu and required
software changes in OpenSBI & Linux kernel as well if that is okay with
you.

Do we require anything else for the PoC part of the DoD policy?

> Greg
>
> On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <
> markhimelstein@...> wrote:
> > please check out the ratification policy for next steps.
> >
> > https://docs.google.com/document/d/1-UlaSGqk59_myeuPMrV9gyuaIgnmFzGh5Gfy_tpViwM/edit
> >
> >
> > On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <
> > brian.grayson@...> wrote:
> > > It's been three weeks since this proposal has been floated, and
> > > feedback was provided on the list. Everyone I've checked with
> > > off-list has been okay with the spec.
> > >
> > > Does anyone object to moving it forward, towards fast-track
> > > ratification? Is there anything else required before it begins
> > > the 45-day public review?
> > >
> > > Greg, do you want to publish the latest version with the tweaks
> > > that you made based on the earlier feedback, for reference?
> > >
> > > Thanks.
> > >
> > > Brian
> > >
> > > On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <
> > > gfavor@...> wrote:
> > > > Hi all,
> > > >
> > > > Recently the TSC established a lightweight "fast track"
> > > > architecture extension process that small, straightforward,
> > > > relatively uncontentious arch extension proposals can utilize. 
> > > > This is the second of two Privileged architecture related small
> > > > extensions - that a number of people/companies have expressed
> > > > desire for over the past year - that Andrew and I discussed
> > > > trying to help move through this process sooner than later
> > > > (especially since this entails much more than simply developing
> > > > a spec).  The following starts with an intro for context, and
> > > > then provides the draft spec.
> > > >
> > > > Note that the draft spec is written as the actual changes to be
> > > > made to existing paragraphs of Priv spec text (or additional
> > > > paragraphs and/or sections within the existing text).  The
> > > > surrounding sentence(s) of a change are included for context. 
> > > > Text in square brackets is temporary commentary that is not
> > > > part of the proposed spec changes.
> > > >
> > > > In anticipation of some questions that may arise in people's
> > > > minds, I'll note that this extension has been
> > > > extensively reviewed by the lead architects of the Privileged
> > > > and Hypervisor architectures for consistency with the current
> > > > architecture (including little things like extension, CSR, and
> > > > bit/field names).  Various changes were made along the way
> > > > because of this.
> > > >
> > > > ===============================================================
> > > > ================
> > > > Introduction
> > > >
> > > > The current Privileged specification defines mhpmevent CSRs to
> > > > select and control event counting by the associated hpmcounter
> > > > CSRs, but provides no standardization of any fields within
> > > > these CSRs.  For at least Linux-class rich-OS systems it is
> > > > desirable to standardize certain basic features that are
> > > > broadly desired (and have come up over the past year plus on
> > > > RISC-V lists, as well as have been the subject of past
> > > > proposals).  This enables there to be standard upstream
> > > > software support that eliminates the need for implementations
> > > > to provide their own custom software support.  (Implementations
> > > > are free, of course, to not implement this extension.)
> > > >
> > > > This proposal serves to accomplish exactly this within the
> > > > existing mhpmevent CSRs (and correspondingly avoids the
> > > > unnecessary creation of whole new sets of CSRs - past just one
> > > > new CSR).
> > > >
> > > > Below is a one-page draft spec of the proposal - which sticks
> > > > to addressing two basic well-understood needs that have been
> > > > requested by various people.  The proposed extension name is
> > > > "Sscof" ('Ss' for Privileged arch and Supervisor-level
> > > > extensions, and 'cof' for Count Overflow and Filtering).  There
> > > > are other features that various people may desire (and that
> > > > even I would desire) that don't have clear-cut, non-
> > > > contentious, and relatively broad support.  These can be grist
> > > > for separate discussions and possibly another arch extension by
> > > > a motivated party that gathers a sufficient degree of
> > > > concensus.
> > > >
> > > > Although one such feature worth highlighting is having a WrEn
> > > > bit in mhpmevent that allows lower privilege modes that can
> > > > read the associated hpmcounter CSR (based on the *counteren
> > > > CSRs) to also be able to write it.  In essence enabling direct
> > > > S/VS-mode and U/VU-mode write access instead of always
> > > > requiring OpenSBI calls up to M-mode.  But this feature has had
> > > > some contention, involves some details to properly support
> > > > virtualization, and requires allocating a second set of "User-
> > > > Read-Write" hpmcounter CSR numbers (since the current
> > > > hpmcounter CSRs are "User-Read-Only").  If there is a broad
> > > > upwelling of support and justification for this feature, and
> > > > some party willing to put together a complete spec (including
> > > > virtualization support), then this could be another fast-track
> > > > extension.
> > > >
> > > > Lastly note that the new count overflow interrupt will be
> > > > treated as a standard local interrupt that is assigned to bit
> > > > 13 in the mip/mie/sip/sie registers.  (This has been discussed
> > > > and agreed to with key Priv Arch people.)
> > > >
> > > > This posting to this email list starts an initial review period
> > > > (over the next few weeks) for people to provide feedback,
> > > > questions, comments, etc.
> > > >
> > > > ===============================================================
> > > > =================
> > > > Proposed Spec
> > > >
> > > > ===============================================================
> > > > ========
> > > > =======================  Machine-Level ISA Additions 
> > > > ========================
> > > >
> > > > Hardware Performance Monitor
> > > > [ This extension expands the hardware performance monitor
> > > > description and extends the mhpmevent registers to 64 bits (in
> > > > RV32) as follows: ]
> > > >
> > > > The hardware performance monitor includes 29 additional 64-bit
> > > > event counters and 29 associated 64-bit event selector
> > > > registers - the mhpmcounter3–mhpmcounter31 and
> > > > mhpmevent3–mhpmevent31 CSRs.
> > > >
> > > > The mhpmcounters are WARL registers that support up to 64 bits
> > > > of precision on RV32 and RV64. 
> > > >
> > > > The mhpmeventn registers are WARL registers that control which
> > > > event causes the corresponding counter to increment and what
> > > > happens when the corresponding count overflows. Currently just
> > > > a few bits are defined here.  Past this, the actual selection
> > > > and meaning of events is defined by the platform, but
> > > > (mhpmevent == 0) is defined to mean “no event" and that the
> > > > corresponding counter will never be incremented.  Typically the
> > > > lower bits of mhpmevent will be used for event selection
> > > > purposes.  
> > > >
> > > > On RV32 only, reads of the mcycle, minstret, mhpmcountern, and
> > > > mhpmeventn CSRs return the low 32 bits, while reads of the
> > > > mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return
> > > > bits 63–32 of the corresponding counter or event selector.  [
> > > > The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]
> > > >
> > > > The following bits are added to mhpmevent:
> > > >
> > > > bit [63]  OF            -  Overflow status and interrupt
> > > > disable bit that is set when counter overflows
> > > >
> > > > bit [62]  MINH        -  If set, then counting of events in M-
> > > > mode is inhibited
> > > > bit [61]  SINH         -  If set, then counting of events in
> > > > S/HS-mode is inhibited
> > > > bit [60]  UINH         -  If set, then counting of events in U-
> > > > mode is inhibited
> > > > bit [59]  VSINH       -  If set, then counting of events in VS-
> > > > mode is inhibited
> > > > bit [59]  VUINH       -  If set, then counting of events in VU-
> > > > mode is inhibited
> > > > bit [58]  0                -  Reserved for possible future
> > > > modes
> > > > bit [57]  0                -  Reserved for possible future
> > > > modes
> > > >
> > > > Each of the five 'x'INH bits, when set, inhibit counting of
> > > > events while in privilege mode 'x'.  All-zeroes for these bits
> > > > results in counting of events in all modes.
> > > >
> > > > The OF bit is set when the corresponding hpmcounter overflows,
> > > > and remains set until written by software.  Since hpmcounter
> > > > values are unsigned values, overflow is defined as unsigned
> > > > overflow.  [ This matches x86 and ARMv8. ]  Note that there is
> > > > no loss of information after an overflow since the counter
> > > > wraps around and keeps counting while the sticky OF bit remains
> > > > set.  [ For a 64-bit counter it will be an awfully long time
> > > > before another overflow could possibly occur. ]
> > > >
> > > > If supervisor mode is implemented, the 32-bit scountovf
> > > > register contains read-only shadow copies of the OF bits in all
> > > > 32 mhpmevent registers.
> > > >
> > > > If an hpmcounter overflows while the associated OF bit is zero,
> > > > then a "count overflow interrupt request" is generated.  If the
> > > > OF bit is one, then no interrupt request is generated. 
> > > > Consequently the OF bit also functions as a count overflow
> > > > interrupt disable for the associated hpmcounter.
> > > >
> > > > ----------------------------  Non-Normative Text    -----------
> > > > -----------------
> > > > There are not separate overflow status and overflow interrupt
> > > > enable bits.  In practice, enabling overflow interrupt
> > > > generation (by clearing the OF bit) is done in conjunction with
> > > > initializing the counter to a starting value.  Once a counter
> > > > has overflowed, it and the OF bit must be reinitialized before
> > > > another overflow interrupt can be generated.
> > > > ---------------------------------------------------------------
> > > > -------------------------
> > > >
> > > > This "count overflow interrupt request" signal is treated as a
> > > > standard local interrupt that corresponds to bit 13 in the
> > > > mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie
> > > > LCOFIE bits are respectively the interrupt-pending and
> > > > interrupt-enable bits for this interrupt.  ('LCOFI' represents
> > > > 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try
> > > > to introduce per-privilege mode overflow interrupt request
> > > > signals.  ARMv8 doesn't have this and I don't think x86 does
> > > > either. ]
> > > >  
> > > > Generation of a "count overflow interrupt request" by an
> > > > hpmcounter sets the LCOFIP bit in the mip/sip registers and
> > > > sets the associated OF bit.  The LCOFIP bit is cleared by
> > > > software after servicing the count overflow interrupt resulting
> > > > from one or more count overflows.
> > > >
> > > > ----------------------------  Non-Normative Text    -----------
> > > > -----------------
> > > > Software can maintain a bit mask to distinguish newly
> > > > overflowed counters (yet to be serviced by an overflow
> > > > interrupt handler) from overflowed counters that
> > > > have already been serviced or that are configured to not
> > > > generate an interrupt on overflow.
> > > > ---------------------------------------------------------------
> > > > -------------------------
> > > >
> > > > Machine Interrupt Registers (mip and mie)
> > > > [ This extension adds the description of the LCOFIP/LCOFIE bits
> > > > in these registers (and modifies related text) as follows: ]
> > > >
> > > > LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is
> > > > added to mie in Figure 3.15 as bit 13.
> > > >
> > > > If the Sscof extension is implemented, bits mip.LCOFIP and
> > > > mie.LCOFIE are the interrupt-pending and interrupt-enable bits
> > > > for local count overflow interrupts.  LCOFIP is read-write in
> > > > mip and reflects the occurrence of a local count overflow
> > > > interrupt request resulting from any of the mhpmeventn.OF bits
> > > > being set.   If the Sscof extension is not implemented, these
> > > > LCOFIP and LCOFIE bits are hardwired to zeros.
> > > >
> > > > Multiple simultaneous interrupts destined for different
> > > > privilege modes are handled in decreasing order of destined
> > > > privilege mode. Multiple simultaneous interrupts destined for
> > > > the same privilege mode are handled in the following decreasing
> > > > priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.
> > > >
> > > > ===============================================================
> > > > ==========
> > > > =======================  Supervisor-Level ISA Additions 
> > > > ========================
> > > >
> > > > Supervisor Interrupt Registers (sip and sie)
> > > > [ This extension adds the description of the LCOFIP/LCOFIE bits
> > > > in these registers (and modifies related text) as follows: ]
> > > >
> > > > LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is
> > > > added to sie in Figure 4.7 as bit 13.
> > > >
> > > > If the Sscof extension is implemented, bits sip.LCOFIP and
> > > > sie.LCOFIE are the interrupt-pending and interrupt-enable bits
> > > > for local count overflow interrupts.  LCOFIP is read-write in
> > > > sip and reflects the occurrence of a local count overflow
> > > > interrupt request resulting from any of the mhpmeventn.OF bits
> > > > being set.  If the Sscof extension is not implemented, these
> > > > LCOFIP and LCOFIE bits are hardwired to zeros. 
> > > >
> > > > Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not
> > > > be implemented, in which case the corresponding interrupt-
> > > > pending and interrupt-enable bits are hardwired to zeros.  All
> > > > bits in sip and sie are WARL fields.
> > > >
> > > > Multiple simultaneous interrupts destined for supervisor mode
> > > > are handled in the following decreasing priority order: SEI,
> > > > SSI, STI, LCOFI.
> > > >
> > > > Supervisor Count Overflow (scountovf)
> > > > [ This extension adds this new CSR. ]
> > > >
> > > > The scountovf CSR is a 32-bit read-only register that contains
> > > > shadow copies of the OF bits in the 32 mhpmevent CSRs -
> > > > where scountovf bit X corresponds to mhpmeventX.  The proposed
> > > > CSR number is 0xD33.
> > > >
> > > > This register enables supervisor-level overflow interrupt
> > > > handler software to quickly and easily determine which
> > > > counter(s) have overflowed (without needing to make an
> > > > execution environment call or series of calls ultimately up to
> > > > M-mode).  [ ARMv8 and x86 have a similar register for the same
> > > > reasons. ]
> > > >
> > > > Read access to bit X is subject to the same mcounteren (or
> > > > mcounteren and hcounteren) CSRs that mediate access to the
> > > > hpmcounter CSRs by S-mode (or VS-mode).  In M and S
> > > > modes, scountovf bit X is readable when mcounteren bit X is
> > > > set, and otherwise reads as zero.  Similarly, in VS
> > > > mode, scountovf bit X is readable when mcounteren bit X and
> > > > hcounteren bit X are both set, and otherwise reads as zero. 
> > > >

--
Regards,
Atish






Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Allen Baum
 

I think I have trouble answering questions concisely.

From a Definition of Done perspective, you're supposed to have architectural tests that will pass when run on the Sail model and on Spike -
more specifically, generate the same signature. 

The signature for this case would have to be that it interrupted where it was expected on both the reference model and the device under test
We do that by comparing trap state: xEPC, xCAUSE, xIP, xTVAL by the interrupt handler.
That works well for synchronous traps, but works less well for asynchronous events, e.g. interrupts.
If you can't inject the interrupt at a deterministic point, xEPC will differ.

The only way I see to pick a deterministic point is to interrupt on the INSTRET counter, which is as close to architectural as you might get.

There are (at least) 4 problems with this:
 
1. As has been pointed out, counter events are implementation specific.
    The only solution I can see is to choose instret, which is writable, required, and deterministic
2. Oops, the current definition of the instruction doesn't include instret.
    This can be fixed by adding another overflow bit , or by further standardizing event selection to include the equivalent 
3 There is no architectural requirement that the interval from an interrupt event occurring to the first instruction of the handler being executed.
4. what happens when an implementation retires more than instruction per cycle? 
    These last 2 are related: even if the event occurs at a deterministic point, the interrupt still won't necessarily occur at a deterministic point

I think we have possible solutions from a framework perspective, but they're ways off .
This sometimes requires the reference model (Sail or possibly Spike) to be able to replicate the possible behaviors.

For this particular case there may be a test structure that might work: 
  disable counter interrupts until after the point that the counter overflows, and enable it at some deterministic point later.
Again: I don't know if there an architectural requirement for the interval between an instruction enabling an interrupt and the first instruction of a trap handler executing
But, if there is AND we can guarantee a universal architectural event - we have a chance of this working.
Otherwise, we would have to have a modified version of the framework (e.g. one that somehow allows fuzzy comparisons for some specific signature values)


On Wed, Feb 24, 2021 at 5:55 PM Atish Patra <atish.patra@...> wrote:
On Wed, 2021-02-24 at 17:01 -0800, Greg Favor wrote:
> Cc'ing tech-priv since others may be wondering about the answer to
> Brian's question.
>
> Brian,
>
> As Mark touched on below, there is a whole "Definition of Done"
> checklist of items that needs to be done (including software support,
> Spike and Sail models, OACR review, PoC, ...).  So that is the next
> order of business that I need to work on - where "I" doesn't mean me
> doing it all (or even having the expertise to do all those things). 
> Btw, would you or anyone else be willing and able to help out with
> one of the DoD checklist items?  Any help will be greatly appreciated
> and will help move this ball towards the goal line.
>

Hi Greg,
I am working on the SBI PMU extension implementation in OpenSBI & Linux
kernel. I will update the SBI PMU extension based on Sscofpmf extension
as well.

I can work on implementing the Sscofpmf extension in Qemu and required
software changes in OpenSBI & Linux kernel as well if that is okay with
you.

Do we require anything else for the PoC part of the DoD policy?

> Greg
>
> On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <
> markhimelstein@...> wrote:
> > please check out the ratification policy for next steps.
> >
> > https://docs.google.com/document/d/1-UlaSGqk59_myeuPMrV9gyuaIgnmFzGh5Gfy_tpViwM/edit
> >
> >
> > On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <
> > brian.grayson@...> wrote:
> > > It's been three weeks since this proposal has been floated, and
> > > feedback was provided on the list. Everyone I've checked with
> > > off-list has been okay with the spec.
> > >
> > > Does anyone object to moving it forward, towards fast-track
> > > ratification? Is there anything else required before it begins
> > > the 45-day public review?
> > >
> > > Greg, do you want to publish the latest version with the tweaks
> > > that you made based on the earlier feedback, for reference?
> > >
> > > Thanks.
> > >
> > > Brian
> > >
> > > On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <
> > > gfavor@...> wrote:
> > > > Hi all,
> > > >
> > > > Recently the TSC established a lightweight "fast track"
> > > > architecture extension process that small, straightforward,
> > > > relatively uncontentious arch extension proposals can utilize. 
> > > > This is the second of two Privileged architecture related small
> > > > extensions - that a number of people/companies have expressed
> > > > desire for over the past year - that Andrew and I discussed
> > > > trying to help move through this process sooner than later
> > > > (especially since this entails much more than simply developing
> > > > a spec).  The following starts with an intro for context, and
> > > > then provides the draft spec.
> > > >
> > > > Note that the draft spec is written as the actual changes to be
> > > > made to existing paragraphs of Priv spec text (or additional
> > > > paragraphs and/or sections within the existing text).  The
> > > > surrounding sentence(s) of a change are included for context. 
> > > > Text in square brackets is temporary commentary that is not
> > > > part of the proposed spec changes.
> > > >
> > > > In anticipation of some questions that may arise in people's
> > > > minds, I'll note that this extension has been
> > > > extensively reviewed by the lead architects of the Privileged
> > > > and Hypervisor architectures for consistency with the current
> > > > architecture (including little things like extension, CSR, and
> > > > bit/field names).  Various changes were made along the way
> > > > because of this.
> > > >
> > > > ===============================================================
> > > > ================
> > > > Introduction
> > > >
> > > > The current Privileged specification defines mhpmevent CSRs to
> > > > select and control event counting by the associated hpmcounter
> > > > CSRs, but provides no standardization of any fields within
> > > > these CSRs.  For at least Linux-class rich-OS systems it is
> > > > desirable to standardize certain basic features that are
> > > > broadly desired (and have come up over the past year plus on
> > > > RISC-V lists, as well as have been the subject of past
> > > > proposals).  This enables there to be standard upstream
> > > > software support that eliminates the need for implementations
> > > > to provide their own custom software support.  (Implementations
> > > > are free, of course, to not implement this extension.)
> > > >
> > > > This proposal serves to accomplish exactly this within the
> > > > existing mhpmevent CSRs (and correspondingly avoids the
> > > > unnecessary creation of whole new sets of CSRs - past just one
> > > > new CSR).
> > > >
> > > > Below is a one-page draft spec of the proposal - which sticks
> > > > to addressing two basic well-understood needs that have been
> > > > requested by various people.  The proposed extension name is
> > > > "Sscof" ('Ss' for Privileged arch and Supervisor-level
> > > > extensions, and 'cof' for Count Overflow and Filtering).  There
> > > > are other features that various people may desire (and that
> > > > even I would desire) that don't have clear-cut, non-
> > > > contentious, and relatively broad support.  These can be grist
> > > > for separate discussions and possibly another arch extension by
> > > > a motivated party that gathers a sufficient degree of
> > > > concensus.
> > > >
> > > > Although one such feature worth highlighting is having a WrEn
> > > > bit in mhpmevent that allows lower privilege modes that can
> > > > read the associated hpmcounter CSR (based on the *counteren
> > > > CSRs) to also be able to write it.  In essence enabling direct
> > > > S/VS-mode and U/VU-mode write access instead of always
> > > > requiring OpenSBI calls up to M-mode.  But this feature has had
> > > > some contention, involves some details to properly support
> > > > virtualization, and requires allocating a second set of "User-
> > > > Read-Write" hpmcounter CSR numbers (since the current
> > > > hpmcounter CSRs are "User-Read-Only").  If there is a broad
> > > > upwelling of support and justification for this feature, and
> > > > some party willing to put together a complete spec (including
> > > > virtualization support), then this could be another fast-track
> > > > extension.
> > > >
> > > > Lastly note that the new count overflow interrupt will be
> > > > treated as a standard local interrupt that is assigned to bit
> > > > 13 in the mip/mie/sip/sie registers.  (This has been discussed
> > > > and agreed to with key Priv Arch people.)
> > > >
> > > > This posting to this email list starts an initial review period
> > > > (over the next few weeks) for people to provide feedback,
> > > > questions, comments, etc.
> > > >
> > > > ===============================================================
> > > > =================
> > > > Proposed Spec
> > > >
> > > > ===============================================================
> > > > ========
> > > > =======================  Machine-Level ISA Additions 
> > > > ========================
> > > >
> > > > Hardware Performance Monitor
> > > > [ This extension expands the hardware performance monitor
> > > > description and extends the mhpmevent registers to 64 bits (in
> > > > RV32) as follows: ]
> > > >
> > > > The hardware performance monitor includes 29 additional 64-bit
> > > > event counters and 29 associated 64-bit event selector
> > > > registers - the mhpmcounter3–mhpmcounter31 and
> > > > mhpmevent3–mhpmevent31 CSRs.
> > > >
> > > > The mhpmcounters are WARL registers that support up to 64 bits
> > > > of precision on RV32 and RV64. 
> > > >
> > > > The mhpmeventn registers are WARL registers that control which
> > > > event causes the corresponding counter to increment and what
> > > > happens when the corresponding count overflows. Currently just
> > > > a few bits are defined here.  Past this, the actual selection
> > > > and meaning of events is defined by the platform, but
> > > > (mhpmevent == 0) is defined to mean “no event" and that the
> > > > corresponding counter will never be incremented.  Typically the
> > > > lower bits of mhpmevent will be used for event selection
> > > > purposes.  
> > > >
> > > > On RV32 only, reads of the mcycle, minstret, mhpmcountern, and
> > > > mhpmeventn CSRs return the low 32 bits, while reads of the
> > > > mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return
> > > > bits 63–32 of the corresponding counter or event selector.  [
> > > > The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]
> > > >
> > > > The following bits are added to mhpmevent:
> > > >
> > > > bit [63]  OF            -  Overflow status and interrupt
> > > > disable bit that is set when counter overflows
> > > >
> > > > bit [62]  MINH        -  If set, then counting of events in M-
> > > > mode is inhibited
> > > > bit [61]  SINH         -  If set, then counting of events in
> > > > S/HS-mode is inhibited
> > > > bit [60]  UINH         -  If set, then counting of events in U-
> > > > mode is inhibited
> > > > bit [59]  VSINH       -  If set, then counting of events in VS-
> > > > mode is inhibited
> > > > bit [59]  VUINH       -  If set, then counting of events in VU-
> > > > mode is inhibited
> > > > bit [58]  0                -  Reserved for possible future
> > > > modes
> > > > bit [57]  0                -  Reserved for possible future
> > > > modes
> > > >
> > > > Each of the five 'x'INH bits, when set, inhibit counting of
> > > > events while in privilege mode 'x'.  All-zeroes for these bits
> > > > results in counting of events in all modes.
> > > >
> > > > The OF bit is set when the corresponding hpmcounter overflows,
> > > > and remains set until written by software.  Since hpmcounter
> > > > values are unsigned values, overflow is defined as unsigned
> > > > overflow.  [ This matches x86 and ARMv8. ]  Note that there is
> > > > no loss of information after an overflow since the counter
> > > > wraps around and keeps counting while the sticky OF bit remains
> > > > set.  [ For a 64-bit counter it will be an awfully long time
> > > > before another overflow could possibly occur. ]
> > > >
> > > > If supervisor mode is implemented, the 32-bit scountovf
> > > > register contains read-only shadow copies of the OF bits in all
> > > > 32 mhpmevent registers.
> > > >
> > > > If an hpmcounter overflows while the associated OF bit is zero,
> > > > then a "count overflow interrupt request" is generated.  If the
> > > > OF bit is one, then no interrupt request is generated. 
> > > > Consequently the OF bit also functions as a count overflow
> > > > interrupt disable for the associated hpmcounter.
> > > >
> > > > ----------------------------  Non-Normative Text    -----------
> > > > -----------------
> > > > There are not separate overflow status and overflow interrupt
> > > > enable bits.  In practice, enabling overflow interrupt
> > > > generation (by clearing the OF bit) is done in conjunction with
> > > > initializing the counter to a starting value.  Once a counter
> > > > has overflowed, it and the OF bit must be reinitialized before
> > > > another overflow interrupt can be generated.
> > > > ---------------------------------------------------------------
> > > > -------------------------
> > > >
> > > > This "count overflow interrupt request" signal is treated as a
> > > > standard local interrupt that corresponds to bit 13 in the
> > > > mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie
> > > > LCOFIE bits are respectively the interrupt-pending and
> > > > interrupt-enable bits for this interrupt.  ('LCOFI' represents
> > > > 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try
> > > > to introduce per-privilege mode overflow interrupt request
> > > > signals.  ARMv8 doesn't have this and I don't think x86 does
> > > > either. ]
> > > >  
> > > > Generation of a "count overflow interrupt request" by an
> > > > hpmcounter sets the LCOFIP bit in the mip/sip registers and
> > > > sets the associated OF bit.  The LCOFIP bit is cleared by
> > > > software after servicing the count overflow interrupt resulting
> > > > from one or more count overflows.
> > > >
> > > > ----------------------------  Non-Normative Text    -----------
> > > > -----------------
> > > > Software can maintain a bit mask to distinguish newly
> > > > overflowed counters (yet to be serviced by an overflow
> > > > interrupt handler) from overflowed counters that
> > > > have already been serviced or that are configured to not
> > > > generate an interrupt on overflow.
> > > > ---------------------------------------------------------------
> > > > -------------------------
> > > >
> > > > Machine Interrupt Registers (mip and mie)
> > > > [ This extension adds the description of the LCOFIP/LCOFIE bits
> > > > in these registers (and modifies related text) as follows: ]
> > > >
> > > > LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is
> > > > added to mie in Figure 3.15 as bit 13.
> > > >
> > > > If the Sscof extension is implemented, bits mip.LCOFIP and
> > > > mie.LCOFIE are the interrupt-pending and interrupt-enable bits
> > > > for local count overflow interrupts.  LCOFIP is read-write in
> > > > mip and reflects the occurrence of a local count overflow
> > > > interrupt request resulting from any of the mhpmeventn.OF bits
> > > > being set.   If the Sscof extension is not implemented, these
> > > > LCOFIP and LCOFIE bits are hardwired to zeros.
> > > >
> > > > Multiple simultaneous interrupts destined for different
> > > > privilege modes are handled in decreasing order of destined
> > > > privilege mode. Multiple simultaneous interrupts destined for
> > > > the same privilege mode are handled in the following decreasing
> > > > priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.
> > > >
> > > > ===============================================================
> > > > ==========
> > > > =======================  Supervisor-Level ISA Additions 
> > > > ========================
> > > >
> > > > Supervisor Interrupt Registers (sip and sie)
> > > > [ This extension adds the description of the LCOFIP/LCOFIE bits
> > > > in these registers (and modifies related text) as follows: ]
> > > >
> > > > LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is
> > > > added to sie in Figure 4.7 as bit 13.
> > > >
> > > > If the Sscof extension is implemented, bits sip.LCOFIP and
> > > > sie.LCOFIE are the interrupt-pending and interrupt-enable bits
> > > > for local count overflow interrupts.  LCOFIP is read-write in
> > > > sip and reflects the occurrence of a local count overflow
> > > > interrupt request resulting from any of the mhpmeventn.OF bits
> > > > being set.  If the Sscof extension is not implemented, these
> > > > LCOFIP and LCOFIE bits are hardwired to zeros. 
> > > >
> > > > Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not
> > > > be implemented, in which case the corresponding interrupt-
> > > > pending and interrupt-enable bits are hardwired to zeros.  All
> > > > bits in sip and sie are WARL fields.
> > > >
> > > > Multiple simultaneous interrupts destined for supervisor mode
> > > > are handled in the following decreasing priority order: SEI,
> > > > SSI, STI, LCOFI.
> > > >
> > > > Supervisor Count Overflow (scountovf)
> > > > [ This extension adds this new CSR. ]
> > > >
> > > > The scountovf CSR is a 32-bit read-only register that contains
> > > > shadow copies of the OF bits in the 32 mhpmevent CSRs -
> > > > where scountovf bit X corresponds to mhpmeventX.  The proposed
> > > > CSR number is 0xD33.
> > > >
> > > > This register enables supervisor-level overflow interrupt
> > > > handler software to quickly and easily determine which
> > > > counter(s) have overflowed (without needing to make an
> > > > execution environment call or series of calls ultimately up to
> > > > M-mode).  [ ARMv8 and x86 have a similar register for the same
> > > > reasons. ]
> > > >
> > > > Read access to bit X is subject to the same mcounteren (or
> > > > mcounteren and hcounteren) CSRs that mediate access to the
> > > > hpmcounter CSRs by S-mode (or VS-mode).  In M and S
> > > > modes, scountovf bit X is readable when mcounteren bit X is
> > > > set, and otherwise reads as zero.  Similarly, in VS
> > > > mode, scountovf bit X is readable when mcounteren bit X and
> > > > hcounteren bit X are both set, and otherwise reads as zero. 
> > > >

--
Regards,
Atish






Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

On Wed, Feb 24, 2021 at 5:31 PM Brian Grayson <brian.grayson@...> wrote:
How would one make compliance tests for this extension? For example, how can one test the overflow exception when there is no standard on how to configure any of the HPM event counters to count, in order to cause an overflow?

These are the sorts of questions I'm about to start grappling with. :) 

Usually, anything having to do with performance monitors is not implemented in functional simulators. Would an exemption be needed for that?

Very possibly (tbd).  Which doesn't mean that basic aspects of the extension can't be tested; just that the core functionality at the heart of the extension may be problematic.
 
Of course, the CSRs need to be supported in the simulators, but expecting a simulator to implement any of the events (that aren't standardized), in order to allow it to cause an overflow exception, seems a tall order, and of limited usefulness to the community.

Agreed.
 
In fact, a lot of the DoD appears to be oriented towards instruction extensions, and not CSR/behavior extensions like this one. It seems like most of items 4, 5, 6, 7, and 8 are all waiver-worthy, for example.

The question to be answered is how much of each of 4-8 can be done.  To get waivers there will need to be a concrete assessment and explanation as to what can and can't be done.  It's hard to imagine that not even having the added CSRs and *ip/*ie bits put into these various models will be deemed acceptable.  Whereas I agree that there is only so much than can usefully be done.

Greg


Re: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Sanjay Patel <spatel@...>
 

There was one item regarding the specification – what are the implications for H-extension? Since the H-extension is part of the privileged spec, I think it should be specified.

 

I do have a description that we are following internally which I can distribute for review.

 

Sanjay

 

From: <tech-privileged@...> on behalf of "Greg Favor via lists.riscv.org" <gfavor=ventanamicro.com@...>
Reply-To: "gfavor@..." <gfavor@...>
Date: Wednesday, February 24, 2021 at 5:07 PM
To: Brian Grayson <brian.grayson@...>, "tech-privileged@..." <tech-privileged@...>
Cc: Jeffrey Osier-Mixon <jefro@...>, Stephano Cetola <stephano@...>, Mark Himelstein <markhimelstein@...>
Subject: [EXTERNAL]Re: [RISC-V] [tech-privileged] Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

 

Greg

 

 

On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

 

Brian,

 

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

 

Greg

 

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:

please check out the ratification policy for next steps.

 

 

On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

 

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

 

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

 

Thanks.

 

Brian

 

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:

Hi all,

 

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

 

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

 

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

 

===============================================================================

Introduction

 

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

 

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

 

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

 

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

 

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

 

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

 

================================================================================

Proposed Spec

 

=======================================================================

=======================  Machine-Level ISA Additions  ========================

 

Hardware Performance Monitor

[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

 

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

 

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

 

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

 

The following bits are added to mhpmevent:

 

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

 

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited

bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited

bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited

bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited

bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited

bit [58]  0                -  Reserved for possible future modes

bit [57]  0                -  Reserved for possible future modes

 

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

 

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

 

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

 

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

 

----------------------------  Non-Normative Text    ----------------------------

There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.

----------------------------------------------------------------------------------------

 

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]

 

Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

 

----------------------------  Non-Normative Text    ----------------------------

Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.

----------------------------------------------------------------------------------------

 

Machine Interrupt Registers (mip and mie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

 

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

 

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

 

=========================================================================

=======================  Supervisor-Level ISA Additions  ========================

 

Supervisor Interrupt Registers (sip and sie)

[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

 

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

 

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

 

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

 

Supervisor Count Overflow (scountovf)

[ This extension adds this new CSR. ]

 

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

 

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

 

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

 

--

Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

atishp@...
 

On Wed, 2021-02-24 at 17:01 -0800, Greg Favor wrote:
Cc'ing tech-priv since others may be wondering about the answer to
Brian's question.

Brian,

As Mark touched on below, there is a whole "Definition of Done"
checklist of items that needs to be done (including software support,
Spike and Sail models, OACR review, PoC, ...).  So that is the next
order of business that I need to work on - where "I" doesn't mean me
doing it all (or even having the expertise to do all those things). 
Btw, would you or anyone else be willing and able to help out with
one of the DoD checklist items?  Any help will be greatly appreciated
and will help move this ball towards the goal line.
Hi Greg,
I am working on the SBI PMU extension implementation in OpenSBI & Linux
kernel. I will update the SBI PMU extension based on Sscofpmf extension
as well.

I can work on implementing the Sscofpmf extension in Qemu and required
software changes in OpenSBI & Linux kernel as well if that is okay with
you.

Do we require anything else for the PoC part of the DoD policy?

Greg

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <
markhimelstein@...> wrote:
please check out the ratification policy for next steps.

https://docs.google.com/document/d/1-UlaSGqk59_myeuPMrV9gyuaIgnmFzGh5Gfy_tpViwM/edit


On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <
brian.grayson@...> wrote:
It's been three weeks since this proposal has been floated, and
feedback was provided on the list. Everyone I've checked with
off-list has been okay with the spec.

Does anyone object to moving it forward, towards fast-track
ratification? Is there anything else required before it begins
the 45-day public review?

Greg, do you want to publish the latest version with the tweaks
that you made based on the earlier feedback, for reference?

Thanks.

Brian

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <
gfavor@...> wrote:
Hi all,

Recently the TSC established a lightweight "fast track"
architecture extension process that small, straightforward,
relatively uncontentious arch extension proposals can utilize. 
This is the second of two Privileged architecture related small
extensions - that a number of people/companies have expressed
desire for over the past year - that Andrew and I discussed
trying to help move through this process sooner than later
(especially since this entails much more than simply developing
a spec).  The following starts with an intro for context, and
then provides the draft spec.

Note that the draft spec is written as the actual changes to be
made to existing paragraphs of Priv spec text (or additional
paragraphs and/or sections within the existing text).  The
surrounding sentence(s) of a change are included for context. 
Text in square brackets is temporary commentary that is not
part of the proposed spec changes.

In anticipation of some questions that may arise in people's
minds, I'll note that this extension has been
extensively reviewed by the lead architects of the Privileged
and Hypervisor architectures for consistency with the current
architecture (including little things like extension, CSR, and
bit/field names).  Various changes were made along the way
because of this.

===============================================================
================
Introduction

The current Privileged specification defines mhpmevent CSRs to
select and control event counting by the associated hpmcounter
CSRs, but provides no standardization of any fields within
these CSRs.  For at least Linux-class rich-OS systems it is
desirable to standardize certain basic features that are
broadly desired (and have come up over the past year plus on
RISC-V lists, as well as have been the subject of past
proposals).  This enables there to be standard upstream
software support that eliminates the need for implementations
to provide their own custom software support.  (Implementations
are free, of course, to not implement this extension.)

This proposal serves to accomplish exactly this within the
existing mhpmevent CSRs (and correspondingly avoids the
unnecessary creation of whole new sets of CSRs - past just one
new CSR).

Below is a one-page draft spec of the proposal - which sticks
to addressing two basic well-understood needs that have been
requested by various people.  The proposed extension name is
"Sscof" ('Ss' for Privileged arch and Supervisor-level
extensions, and 'cof' for Count Overflow and Filtering).  There
are other features that various people may desire (and that
even I would desire) that don't have clear-cut, non-
contentious, and relatively broad support.  These can be grist
for separate discussions and possibly another arch extension by
a motivated party that gathers a sufficient degree of
concensus.

Although one such feature worth highlighting is having a WrEn
bit in mhpmevent that allows lower privilege modes that can
read the associated hpmcounter CSR (based on the *counteren
CSRs) to also be able to write it.  In essence enabling direct
S/VS-mode and U/VU-mode write access instead of always
requiring OpenSBI calls up to M-mode.  But this feature has had
some contention, involves some details to properly support
virtualization, and requires allocating a second set of "User-
Read-Write" hpmcounter CSR numbers (since the current
hpmcounter CSRs are "User-Read-Only").  If there is a broad
upwelling of support and justification for this feature, and
some party willing to put together a complete spec (including
virtualization support), then this could be another fast-track
extension.

Lastly note that the new count overflow interrupt will be
treated as a standard local interrupt that is assigned to bit
13 in the mip/mie/sip/sie registers.  (This has been discussed
and agreed to with key Priv Arch people.)

This posting to this email list starts an initial review period
(over the next few weeks) for people to provide feedback,
questions, comments, etc.

===============================================================
=================
Proposed Spec

===============================================================
========
=======================  Machine-Level ISA Additions 
========================

Hardware Performance Monitor
[ This extension expands the hardware performance monitor
description and extends the mhpmevent registers to 64 bits (in
RV32) as follows: ]

The hardware performance monitor includes 29 additional 64-bit
event counters and 29 associated 64-bit event selector
registers - the mhpmcounter3–mhpmcounter31 and
mhpmevent3–mhpmevent31 CSRs.

The mhpmcounters are WARL registers that support up to 64 bits
of precision on RV32 and RV64. 

The mhpmeventn registers are WARL registers that control which
event causes the corresponding counter to increment and what
happens when the corresponding count overflows. Currently just
a few bits are defined here.  Past this, the actual selection
and meaning of events is defined by the platform, but
(mhpmevent == 0) is defined to mean “no event" and that the
corresponding counter will never be incremented.  Typically the
lower bits of mhpmevent will be used for event selection
purposes.  

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and
mhpmeventn CSRs return the low 32 bits, while reads of the
mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return
bits 63–32 of the corresponding counter or event selector.  [
The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

The following bits are added to mhpmevent:

bit [63]  OF            -  Overflow status and interrupt
disable bit that is set when counter overflows

bit [62]  MINH        -  If set, then counting of events in M-
mode is inhibited
bit [61]  SINH         -  If set, then counting of events in
S/HS-mode is inhibited
bit [60]  UINH         -  If set, then counting of events in U-
mode is inhibited
bit [59]  VSINH       -  If set, then counting of events in VS-
mode is inhibited
bit [59]  VUINH       -  If set, then counting of events in VU-
mode is inhibited
bit [58]  0                -  Reserved for possible future
modes
bit [57]  0                -  Reserved for possible future
modes

Each of the five 'x'INH bits, when set, inhibit counting of
events while in privilege mode 'x'.  All-zeroes for these bits
results in counting of events in all modes.

The OF bit is set when the corresponding hpmcounter overflows,
and remains set until written by software.  Since hpmcounter
values are unsigned values, overflow is defined as unsigned
overflow.  [ This matches x86 and ARMv8. ]  Note that there is
no loss of information after an overflow since the counter
wraps around and keeps counting while the sticky OF bit remains
set.  [ For a 64-bit counter it will be an awfully long time
before another overflow could possibly occur. ]

If supervisor mode is implemented, the 32-bit scountovf
register contains read-only shadow copies of the OF bits in all
32 mhpmevent registers.

If an hpmcounter overflows while the associated OF bit is zero,
then a "count overflow interrupt request" is generated.  If the
OF bit is one, then no interrupt request is generated. 
Consequently the OF bit also functions as a count overflow
interrupt disable for the associated hpmcounter.

----------------------------  Non-Normative Text    -----------
-----------------
There are not separate overflow status and overflow interrupt
enable bits.  In practice, enabling overflow interrupt
generation (by clearing the OF bit) is done in conjunction with
initializing the counter to a starting value.  Once a counter
has overflowed, it and the OF bit must be reinitialized before
another overflow interrupt can be generated.
---------------------------------------------------------------
-------------------------

This "count overflow interrupt request" signal is treated as a
standard local interrupt that corresponds to bit 13 in the
mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie
LCOFIE bits are respectively the interrupt-pending and
interrupt-enable bits for this interrupt.  ('LCOFI' represents
'Local Count Overflow Interrupt'.)  [ This proposal doesn't try
to introduce per-privilege mode overflow interrupt request
signals.  ARMv8 doesn't have this and I don't think x86 does
either. ]
 
Generation of a "count overflow interrupt request" by an
hpmcounter sets the LCOFIP bit in the mip/sip registers and
sets the associated OF bit.  The LCOFIP bit is cleared by
software after servicing the count overflow interrupt resulting
from one or more count overflows.

----------------------------  Non-Normative Text    -----------
-----------------
Software can maintain a bit mask to distinguish newly
overflowed counters (yet to be serviced by an overflow
interrupt handler) from overflowed counters that
have already been serviced or that are configured to not
generate an interrupt on overflow.
---------------------------------------------------------------
-------------------------

Machine Interrupt Registers (mip and mie)
[ This extension adds the description of the LCOFIP/LCOFIE bits
in these registers (and modifies related text) as follows: ]

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is
added to mie in Figure 3.15 as bit 13.

If the Sscof extension is implemented, bits mip.LCOFIP and
mie.LCOFIE are the interrupt-pending and interrupt-enable bits
for local count overflow interrupts.  LCOFIP is read-write in
mip and reflects the occurrence of a local count overflow
interrupt request resulting from any of the mhpmeventn.OF bits
being set.   If the Sscof extension is not implemented, these
LCOFIP and LCOFIE bits are hardwired to zeros.

Multiple simultaneous interrupts destined for different
privilege modes are handled in decreasing order of destined
privilege mode. Multiple simultaneous interrupts destined for
the same privilege mode are handled in the following decreasing
priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

===============================================================
==========
=======================  Supervisor-Level ISA Additions 
========================

Supervisor Interrupt Registers (sip and sie)
[ This extension adds the description of the LCOFIP/LCOFIE bits
in these registers (and modifies related text) as follows: ]

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is
added to sie in Figure 4.7 as bit 13.

If the Sscof extension is implemented, bits sip.LCOFIP and
sie.LCOFIE are the interrupt-pending and interrupt-enable bits
for local count overflow interrupts.  LCOFIP is read-write in
sip and reflects the occurrence of a local count overflow
interrupt request resulting from any of the mhpmeventn.OF bits
being set.  If the Sscof extension is not implemented, these
LCOFIP and LCOFIE bits are hardwired to zeros. 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not
be implemented, in which case the corresponding interrupt-
pending and interrupt-enable bits are hardwired to zeros.  All
bits in sip and sie are WARL fields.

Multiple simultaneous interrupts destined for supervisor mode
are handled in the following decreasing priority order: SEI,
SSI, STI, LCOFI.

Supervisor Count Overflow (scountovf)
[ This extension adds this new CSR. ]

The scountovf CSR is a 32-bit read-only register that contains
shadow copies of the OF bits in the 32 mhpmevent CSRs -
where scountovf bit X corresponds to mhpmeventX.  The proposed
CSR number is 0xD33.

This register enables supervisor-level overflow interrupt
handler software to quickly and easily determine which
counter(s) have overflowed (without needing to make an
execution environment call or series of calls ultimately up to
M-mode).  [ ARMv8 and x86 have a similar register for the same
reasons. ]

Read access to bit X is subject to the same mcounteren (or
mcounteren and hcounteren) CSRs that mediate access to the
hpmcounter CSRs by S-mode (or VS-mode).  In M and S
modes, scountovf bit X is readable when mcounteren bit X is
set, and otherwise reads as zero.  Similarly, in VS
mode, scountovf bit X is readable when mcounteren bit X and
hcounteren bit X are both set, and otherwise reads as zero. 
--
Regards,
Atish


Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Brian Grayson
 

How would one make compliance tests for this extension? For example, how can one test the overflow exception when there is no standard on how to configure any of the HPM event counters to count, in order to cause an overflow?

Usually, anything having to do with performance monitors is not implemented in functional simulators. Would an exemption be needed for that? Of course, the CSRs need to be supported in the simulators, but expecting a simulator to implement any of the events (that aren't standardized), in order to allow it to cause an overflow exception, seems a tall order, and of limited usefulness to the community.

In fact, a lot of the DoD appears to be oriented towards instruction extensions, and not CSR/behavior extensions like this one. It seems like most of items 4, 5, 6, 7, and 8 are all waiver-worthy, for example.

Brian



On Wed, Feb 24, 2021 at 7:06 PM Greg Favor <gfavor@...> wrote:
Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

Greg


On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:
Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

Brian,

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

Greg

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:
please check out the ratification policy for next steps.


On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:
It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

Thanks.

Brian

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:
Hi all,

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

===============================================================================
Introduction

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

================================================================================
Proposed Spec

=======================================================================
=======================  Machine-Level ISA Additions  ========================

Hardware Performance Monitor
[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

The following bits are added to mhpmevent:

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited
bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited
bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited
bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited
bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited
bit [58]  0                -  Reserved for possible future modes
bit [57]  0                -  Reserved for possible future modes

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

----------------------------  Non-Normative Text    ----------------------------
There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.
----------------------------------------------------------------------------------------

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]
 
Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

----------------------------  Non-Normative Text    ----------------------------
Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.
----------------------------------------------------------------------------------------

Machine Interrupt Registers (mip and mie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

=========================================================================
=======================  Supervisor-Level ISA Additions  ========================

Supervisor Interrupt Registers (sip and sie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

Supervisor Count Overflow (scountovf)
[ This extension adds this new CSR. ]

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

--
Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

Just to emphasize:  If a number of people can pitch on on various DoD pieces, that would help a lot.  Spec-wise we're in good shape (even though there is work to be done), and I've already submitted this extension to the OACR committee for review.  So it's the other items that are the non-trivial hurdles to get over (including two notable items I forgot to mention: defining coverage for arch compatibility tests and then actually creating them).

Greg


On Wed, Feb 24, 2021 at 5:01 PM Greg Favor <gfavor@...> wrote:
Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

Brian,

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

Greg

On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:
please check out the ratification policy for next steps.


On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:
It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

Thanks.

Brian

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:
Hi all,

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

===============================================================================
Introduction

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

================================================================================
Proposed Spec

=======================================================================
=======================  Machine-Level ISA Additions  ========================

Hardware Performance Monitor
[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

The following bits are added to mhpmevent:

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited
bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited
bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited
bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited
bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited
bit [58]  0                -  Reserved for possible future modes
bit [57]  0                -  Reserved for possible future modes

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

----------------------------  Non-Normative Text    ----------------------------
There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.
----------------------------------------------------------------------------------------

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]
 
Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

----------------------------  Non-Normative Text    ----------------------------
Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.
----------------------------------------------------------------------------------------

Machine Interrupt Registers (mip and mie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

=========================================================================
=======================  Supervisor-Level ISA Additions  ========================

Supervisor Interrupt Registers (sip and sie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

Supervisor Count Overflow (scountovf)
[ This extension adds this new CSR. ]

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

--
Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Greg Favor
 

Cc'ing tech-priv since others may be wondering about the answer to Brian's question.

Brian,

As Mark touched on below, there is a whole "Definition of Done" checklist of items that needs to be done (including software support, Spike and Sail models, OACR review, PoC, ...).  So that is the next order of business that I need to work on - where "I" doesn't mean me doing it all (or even having the expertise to do all those things).  Btw, would you or anyone else be willing and able to help out with one of the DoD checklist items?  Any help will be greatly appreciated and will help move this ball towards the goal line.

Greg


On Wed, Feb 24, 2021 at 3:54 PM Mark Himelstein <markhimelstein@...> wrote:
please check out the ratification policy for next steps.


On Wed, Feb 24, 2021 at 3:43 PM Brian Grayson <brian.grayson@...> wrote:
It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

Thanks.

Brian

On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:
Hi all,

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

===============================================================================
Introduction

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

================================================================================
Proposed Spec

=======================================================================
=======================  Machine-Level ISA Additions  ========================

Hardware Performance Monitor
[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

The following bits are added to mhpmevent:

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited
bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited
bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited
bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited
bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited
bit [58]  0                -  Reserved for possible future modes
bit [57]  0                -  Reserved for possible future modes

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

----------------------------  Non-Normative Text    ----------------------------
There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.
----------------------------------------------------------------------------------------

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]
 
Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

----------------------------  Non-Normative Text    ----------------------------
Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.
----------------------------------------------------------------------------------------

Machine Interrupt Registers (mip and mie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

=========================================================================
=======================  Supervisor-Level ISA Additions  ========================

Supervisor Interrupt Registers (sip and sie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

Supervisor Count Overflow (scountovf)
[ This extension adds this new CSR. ]

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 

--
Mark I Himelstein
CTO RISC-V International
+1-408-250-6611
twitter @mark_riscv


Re: Fast-track extension proposal for "Hardware Performance Monitor count overflow and mode-based event filtering"

Brian Grayson
 

It's been three weeks since this proposal has been floated, and feedback was provided on the list. Everyone I've checked with off-list has been okay with the spec.

Does anyone object to moving it forward, towards fast-track ratification? Is there anything else required before it begins the 45-day public review?

Greg, do you want to publish the latest version with the tweaks that you made based on the earlier feedback, for reference?

Thanks.

Brian


On Mon, Feb 1, 2021 at 12:38 AM Greg Favor <gfavor@...> wrote:
Hi all,

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  This is the second of two Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - that Andrew and I discussed trying to help move through this process sooner than later (especially since this entails much more than simply developing a spec).  The following starts with an intro for context, and then provides the draft spec.

Note that the draft spec is written as the actual changes to be made to existing paragraphs of Priv spec text (or additional paragraphs and/or sections within the existing text).  The surrounding sentence(s) of a change are included for context.  Text in square brackets is temporary commentary that is not part of the proposed spec changes.

In anticipation of some questions that may arise in people's minds, I'll note that this extension has been extensively reviewed by the lead architects of the Privileged and Hypervisor architectures for consistency with the current architecture (including little things like extension, CSR, and bit/field names).  Various changes were made along the way because of this.

===============================================================================
Introduction

The current Privileged specification defines mhpmevent CSRs to select and control event counting by the associated hpmcounter CSRs, but provides no standardization of any fields within these CSRs.  For at least Linux-class rich-OS systems it is desirable to standardize certain basic features that are broadly desired (and have come up over the past year plus on RISC-V lists, as well as have been the subject of past proposals).  This enables there to be standard upstream software support that eliminates the need for implementations to provide their own custom software support.  (Implementations are free, of course, to not implement this extension.)

This proposal serves to accomplish exactly this within the existing mhpmevent CSRs (and correspondingly avoids the unnecessary creation of whole new sets of CSRs - past just one new CSR).

Below is a one-page draft spec of the proposal - which sticks to addressing two basic well-understood needs that have been requested by various people.  The proposed extension name is "Sscof" ('Ss' for Privileged arch and Supervisor-level extensions, and 'cof' for Count Overflow and Filtering).  There are other features that various people may desire (and that even I would desire) that don't have clear-cut, non-contentious, and relatively broad support.  These can be grist for separate discussions and possibly another arch extension by a motivated party that gathers a sufficient degree of concensus.

Although one such feature worth highlighting is having a WrEn bit in mhpmevent that allows lower privilege modes that can read the associated hpmcounter CSR (based on the *counteren CSRs) to also be able to write it.  In essence enabling direct S/VS-mode and U/VU-mode write access instead of always requiring OpenSBI calls up to M-mode.  But this feature has had some contention, involves some details to properly support virtualization, and requires allocating a second set of "User-Read-Write" hpmcounter CSR numbers (since the current hpmcounter CSRs are "User-Read-Only").  If there is a broad upwelling of support and justification for this feature, and some party willing to put together a complete spec (including virtualization support), then this could be another fast-track extension.

Lastly note that the new count overflow interrupt will be treated as a standard local interrupt that is assigned to bit 13 in the mip/mie/sip/sie registers.  (This has been discussed and agreed to with key Priv Arch people.)

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

================================================================================
Proposed Spec

=======================================================================
=======================  Machine-Level ISA Additions  ========================

Hardware Performance Monitor
[ This extension expands the hardware performance monitor description and extends the mhpmevent registers to 64 bits (in RV32) as follows: ]

The hardware performance monitor includes 29 additional 64-bit event counters and 29 associated 64-bit event selector registers - the mhpmcounter3–mhpmcounter31 and mhpmevent3–mhpmevent31 CSRs.

The mhpmcounters are WARL registers that support up to 64 bits of precision on RV32 and RV64. 

The mhpmeventn registers are WARL registers that control which event causes the corresponding counter to increment and what happens when the corresponding count overflows. Currently just a few bits are defined here.  Past this, the actual selection and meaning of events is defined by the platform, but (mhpmevent == 0) is defined to mean “no event" and that the corresponding counter will never be incremented.  Typically the lower bits of mhpmevent will be used for event selection purposes.  

On RV32 only, reads of the mcycle, minstret, mhpmcountern, and mhpmeventn CSRs return the low 32 bits, while reads of the mcycleh, minstreth, mhpmcounternh, and mhpmeventnh CSRs return bits 63–32 of the corresponding counter or event selector.  [ The proposed CSR numbers for mhpmeventnh are 0x723 - 0x73F. ]

The following bits are added to mhpmevent:

bit [63]  OF            -  Overflow status and interrupt disable bit that is set when counter overflows

bit [62]  MINH        -  If set, then counting of events in M-mode is inhibited
bit [61]  SINH         -  If set, then counting of events in S/HS-mode is inhibited
bit [60]  UINH         -  If set, then counting of events in U-mode is inhibited
bit [59]  VSINH       -  If set, then counting of events in VS-mode is inhibited
bit [59]  VUINH       -  If set, then counting of events in VU-mode is inhibited
bit [58]  0                -  Reserved for possible future modes
bit [57]  0                -  Reserved for possible future modes

Each of the five 'x'INH bits, when set, inhibit counting of events while in privilege mode 'x'.  All-zeroes for these bits results in counting of events in all modes.

The OF bit is set when the corresponding hpmcounter overflows, and remains set until written by software.  Since hpmcounter values are unsigned values, overflow is defined as unsigned overflow.  [ This matches x86 and ARMv8. ]  Note that there is no loss of information after an overflow since the counter wraps around and keeps counting while the sticky OF bit remains set.  [ For a 64-bit counter it will be an awfully long time before another overflow could possibly occur. ]

If supervisor mode is implemented, the 32-bit scountovf register contains read-only shadow copies of the OF bits in all 32 mhpmevent registers.

If an hpmcounter overflows while the associated OF bit is zero, then a "count overflow interrupt request" is generated.  If the OF bit is one, then no interrupt request is generated.  Consequently the OF bit also functions as a count overflow interrupt disable for the associated hpmcounter.

----------------------------  Non-Normative Text    ----------------------------
There are not separate overflow status and overflow interrupt enable bits.  In practice, enabling overflow interrupt generation (by clearing the OF bit) is done in conjunction with initializing the counter to a starting value.  Once a counter has overflowed, it and the OF bit must be reinitialized before another overflow interrupt can be generated.
----------------------------------------------------------------------------------------

This "count overflow interrupt request" signal is treated as a standard local interrupt that corresponds to bit 13 in the mip/mie/sip/sie registers.  The mip/sip LCOFIP and mie/sie LCOFIE bits are respectively the interrupt-pending and interrupt-enable bits for this interrupt.  ('LCOFI' represents 'Local Count Overflow Interrupt'.)  [ This proposal doesn't try to introduce per-privilege mode overflow interrupt request signals.  ARMv8 doesn't have this and I don't think x86 does either. ]
 
Generation of a "count overflow interrupt request" by an hpmcounter sets the LCOFIP bit in the mip/sip registers and sets the associated OF bit.  The LCOFIP bit is cleared by software after servicing the count overflow interrupt resulting from one or more count overflows.

----------------------------  Non-Normative Text    ----------------------------
Software can maintain a bit mask to distinguish newly overflowed counters (yet to be serviced by an overflow interrupt handler) from overflowed counters that have already been serviced or that are configured to not generate an interrupt on overflow.
----------------------------------------------------------------------------------------

Machine Interrupt Registers (mip and mie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to mip in Figure 3.14 as bit 13.  LCOFIP is added to mie in Figure 3.15 as bit 13.

If the Sscof extension is implemented, bits mip.LCOFIP and mie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in mip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.   If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros.

Multiple simultaneous interrupts destined for different privilege modes are handled in decreasing order of destined privilege mode. Multiple simultaneous interrupts destined for the same privilege mode are handled in the following decreasing priority order: MEI, MSI, MTI, SEI, SSI, STI, LCOFI.

=========================================================================
=======================  Supervisor-Level ISA Additions  ========================

Supervisor Interrupt Registers (sip and sie)
[ This extension adds the description of the LCOFIP/LCOFIE bits in these registers (and modifies related text) as follows: ]

LCOFIP is added to sip in Figure 4.6 as bit 13.  LCOFIP is added to sie in Figure 4.7 as bit 13.

If the Sscof extension is implemented, bits sip.LCOFIP and sie.LCOFIE are the interrupt-pending and interrupt-enable bits for local count overflow interrupts.  LCOFIP is read-write in sip and reflects the occurrence of a local count overflow interrupt request resulting from any of the mhpmeventn.OF bits being set.  If the Sscof extension is not implemented, these LCOFIP and LCOFIE bits are hardwired to zeros. 

Each standard interrupt type (LCOFI, SEI, STI, or SSI) may not be implemented, in which case the corresponding interrupt-pending and interrupt-enable bits are hardwired to zeros.  All bits in sip and sie are WARL fields.

Multiple simultaneous interrupts destined for supervisor mode are handled in the following decreasing priority order: SEI, SSI, STI, LCOFI.

Supervisor Count Overflow (scountovf)
[ This extension adds this new CSR. ]

The scountovf CSR is a 32-bit read-only register that contains shadow copies of the OF bits in the 32 mhpmevent CSRs - where scountovf bit X corresponds to mhpmeventX.  The proposed CSR number is 0xD33.

This register enables supervisor-level overflow interrupt handler software to quickly and easily determine which counter(s) have overflowed (without needing to make an execution environment call or series of calls ultimately up to M-mode).  [ ARMv8 and x86 have a similar register for the same reasons. ]

Read access to bit X is subject to the same mcounteren (or mcounteren and hcounteren) CSRs that mediate access to the hpmcounter CSRs by S-mode (or VS-mode).  In M and S modes, scountovf bit X is readable when mcounteren bit X is set, and otherwise reads as zero.  Similarly, in VS mode, scountovf bit X is readable when mcounteren bit X and hcounteren bit X are both set, and otherwise reads as zero. 


Re: [RISC-V] [tech-virt-mem] [RISC-V] [tech-privileged] SV32 and 34 bit address

Daniel Lustig
 

This was just a question about existing behavior. It is unrelated to the
ongoing security conversation on the virtual memory TG mailing list, if
that's what you're asking.

Dan

On 2/22/2021 12:11 PM, Richard Trauben wrote:
it makes sense but in what way does this make us more secure?
On Feb 22, 2021, at 7:53 AM, Daniel Lustig <dlustig@...> wrote:

Hi Gracy,

When satp.mode is not bare, virtual addresses must be sign-extended
to fill XLEN. When the physical address generated in any mode is
smaller than the implementation's physical address space, the physical
address is zero-extended. That's the difference.

In Rv32 bare mode, 32b virtual addresses are treated as 32b physical
addresses. If a machine has more than 32b of physical address space
(as will be common, since Sv32 supports 34b PAs), then the 32b bare VA
will be zero-extended to form the PA. All 34b are directly accessible
when in Sv32 mode.

In Sv39, each VA must have bits 63-39 equal to bit 38, as described.
Sv39 translation will produce a 56b PA. If the implementation has more
than 56 physical address bits, that 56b PA will be zero-extended to
the right number of bits.

Does that make sense? Hope that helps,
Dan

On 2/22/2021 3:28 AM, Gracy Ge wrote:
I reused a topic title under https://github.com/riscv/riscv-isa-manual/issues/131
And also I attached some comments in section 4.4.1 of privileged spec those confused me so much,

*A*. Sv39 implementations support a 39-bit virtual address space, divided into 4 KiB pages. An Sv39
address is partitioned as shown in Figure 4.18. Instruction fetch addresses and load and store
effective addresses, which are *64 bits* , must have bits *63–39 all equal to bit 38* , or else a page-fault
exception will occur.

*B*. When mapping between narrower and wider addresses, RISC-V usually *zero-extends* a narrower
address to a wider size. The mapping between 64-bit virtual addresses and the 39-bit usable
address space of Sv39 is not based on zero-extension but instead follows an entrenched convention
that allows an OS to use one or a few of the most-significant bits of a full-size (64-bit) virtual
address to quickly distinguish user and supervisor address regions.

I have similar issues on SV32 and SV39.
1. In SV32, access to only the low 4Gb (i.e. the top 12Gb are not accessible) in bare mode or machine mode.
Can I understand this is explicitly stated from *#B* ( *zero extends* ) in section4 4.4.1?
2. How to understand the usable address space in SV39? Shall OS make sure the address zero extend the higher bits or keep it [38] extended? IMO, statement in # *A* and # *B* is conflicted. Forgave me if I misunderstood it.
3. In sv39, what is the valid address space in bare mode or Machine mode? Is it maximum 64 bit wide? I know it is not a real case, just to make sure the ideal limitation here.
4. Last question in SV32, OS can utilize the MAPPING scheme of SV32 and get physical address used up to 16G. However currently gcc compiler for 32 bit system only have 4G memory usage, How can I compile a testcase and link them to LMA at upper 12G memory space?

Best Regards,
Gracy






701 - 720 of 1210