Fast-track "stimecmp / vstimecmp" extension proposal


Greg Favor
 

Hi all,

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  Andrew and I discussed trying to help move a couple of Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - through this process sooner than later (especially since this entails much more than simply proposing and reviewing a spec).  For this one Siqi and I are co-sponsors.  The following starts with an intro for context, and then provides the draft spec.

=================================================================
The current Privileged specification only defines a hardware mechanism for generating machine-mode timer interrupts (based on the mtime and mtimecmp registers).  With the resultant requirement that timer services for S-mode/HS-mode (and for VS-mode) have to all be provided by M-mode - via SBI calls from S/HS-mode up to M-mode (or VS-mode calls to HS-mode and then to M-mode).  M-mode software then multiplexes these multiple logical timers onto its one physical M-mode timer facility, and the M-mode timer interrupt handler passes timer interrupts back down to the appropriate lower privilege mode.

Over the past year or more, there have been a number of requests to provide supervisor mode with its own CSR-based timer interrupt facility that it can directly manage to provide its own timer service (in the form of having its own stimecmp register) - thus eliminating the large overheads for emulating S/HS-mode timers and timer interrupt generation up in M-mode.  This proposal, using the new fast-track architecture extension process, serves to accomplish exactly this.  Further, this proposal lays the groundwork for adding a similar facility to the Hypervisor extension for VS-mode.   (ARMv8, for example, has separate timer registers for its equivalent of M, S/HS, and VS privilege levels.)

Below is a one-page draft spec of the proposal - which sticks to addressing the basic well-understood need that has been requested.  The proposed extension name is "Zstc" ('Zs' for privileged Supervisor arch extensions, and 'tc' for timecmp).

Various further pieces of related architectural functionality have previously been suggested, debated, and questioned - without clear conclusion and agreement as to whether these are truly justified.  These more contentious enhancements are left to be sorted out and pursued by a follow-on effort if desired and justified by sufficient people.

The suggestion to also introduce an mtimecmp CSR has been considered, but ruled out for now as not sufficiently justified over the current definition of mtimecmp as a memory-mapped register (which would have to remain for backward compatibility in any case - resulting in two aliases for this register).  But if there is sufficient interest, a separate email thread can be started to discuss the possible addition of an mtimecmp CSR (as part of this or a separate fast-track extension).  For now I'll leave out details about how the mtimecmp memory-mapped register and mtimecmp CSR could inter-relate (it doesn't have to be complicated in practice).

This extension adds a single CSR - the stimecmp CSR.  This extension also describes what the corresponding addition to the Hypervisor extension would be - the vstimecmp CSR.  The former is the official part of this fast-track arch extension.  The latter is solely informative and reflective of what would then likely be added to the current H-extension draft spec as part of its virtualization of S-mode.

The draft spec below provides all the details.  Note that this extension very specifically strives to maintain maximal consistency with many little details in the existing Privileged architecture and Hypervisor extension (and has already been reviewed by the two lead Privileged and Hypervisor architects to ensure this).

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

Thanks,
Siqi and Greg

=========================================================================
=======================  Supervisor Extension Additions  ========================

Supervisor Timer Register (stimecmp)
The stimecmp CSR is a 64-bit register and has 64-bit precision on all RV32 and RV64 systems.  In RV32 only, reads of the stimecmp CSR return the low 32 bits, while reads of the stimecmph CSR return the high 32 bits of stimecmp.

The proposed CSR numbers for stimecmp / stimecmph are 0x108 / 0x109 (within the Supervisor Trap Setup block of CSRs).

A supervisor timer interrupt becomes pending - as reflected in the STIP bit in the mip and sip registers - whenever mtime contains a value greater than or equal to stimecmp, treating the values as unsigned integers.  Writes to stimecmp are guaranteed to be reflected in STIP eventually, but not necessarily immediately.  The interrupt remains posted until stimecmp becomes greater than mtime - typically as a result of writing stimecmp. The interrupt will be taken based on the standard interrupt enable and delegation rules.

----------------------------  Non-Normative Text    ----------------------------
A spurious timer interrupt might occur if an interrupt handler advances stimecmp then immediately returns, because STIP might not yet have fallen in the interim. All software should be written to assume this event is possible, but most software should assume this event is extremely unlikely. It is almost always more performant to incur an occasional spurious timer interrupt than to poll STIP until it falls.  
----------------------------------------------------------------------------------------

----------------------------  Non-Normative Text    ----------------------------
In systems in which a supervisor execution environment (SEE) provides timer facilities via an SBI function call, this SBI call will continue to support requests to schedule a timer interrupt.  The SEE will simply make use of stimecmp, changing its value as appropriate.  This ensures compatibility with existing S-mode software that uses this SEE facility, while new S-mode software takes advantage of stimecmp directly.)
----------------------------------------------------------------------------------------

Machine Interrupt Registers (mip and mie)
This extension modifies the description of the STIP/STIE bits in these registers as follows:

If supervisor mode is implemented, its mip.STIP and mie.STIE are the interrupt-pending and interrupt-enable bits for supervisor-level timer interrupts.  If the stimecmp register is not implemented, STIP is writable in mip, and may be written by M-mode software to deliver timer interrupts to S-mode.  If the stimecmp (supervisor-mode timer compare) register is implemented, STIP is read-only in mip and reflects the supervisor-level timer interrupt signal resulting from stimecmp.  This timer interrupt signal is cleared by writing stimecmp with a value greater than the current mtime value.

Supervisor Interrupt Registers (sip and sie)
This extension modifies the description of the STIP/STIE bits in these registers as follows:
 
Bits sip.STIP and sie.STIE are the interrupt-pending and interrupt-enable bits for supervisor level timer interrupts.  If implemented, STIP is read-only in sip, and is either set and cleared by the execution environment (if stimecmp is not implemented), or reflects the timer interrupt signal resulting from stimecmp (if stimecmp is implemented).  The sip.STIP bit, in response to timer interrupts generated by stimecmp, is set and cleared by writing stimecmp with a value that respectively is less than or equal to, or greater than, the current mtime value.

Machine Counter-Enable Register (mcounteren)
This extension adds to the description of the TM bit in this register as follows:

In addition, when the TM bit in the mcounteren register is clear, attempts to read the stimecmp register while executing in S-mode will cause an illegal instruction exception. When this bit is set, access to the stimecmp register (if implemented) is permitted in S-mode.


=========================================================================
=======================  Hypervisor Extension Additions  ========================

Virtual Supervisor Timer Register (vstimecmp)
The vstimecmp CSR is a 64-bit register and has 64-bit precision on all RV32 and RV64 systems.  In RV32 only, reads of the vstimecmp CSR return the low 32 bits, while reads of the vstimecmph CSR return the high 32 bits of vstimecmp.

The proposed CSR numbers for vstimecmp / vstimecmph are 0x208 / 0x209 (within the Virtual Supervisor Registers block of CSRs, and mirroring the CSR numbers for stimecmp/stimecmph).

A virtual supervisor timer interrupt becomes pending - as reflected in the VSTIP bit in the hip register - whenever (mtime + htimedelta) contains a value greater than or equal to vstimecmp, treating the values as unsigned integers.  Writes to vstimecmp and htimedelta are guaranteed to be reflected in VSTIP eventually, but not necessarily immediately.  The interrupt remains posted until vstimecmp becomes greater than (mtime + htimedelta) - typically as a result of writing vstimecmp. The interrupt will be taken based on the standard interrupt enable and delegation rules while V=1.

----------------------------  Non-Normative Text    ----------------------------
In systems in which a supervisor execution environment (SEE) implemented by an HS-mode hypervisor provides timer facilities via an SBI function call, this SBI call will continue to support requests to schedule a timer interrupt.  The SEE will simply make use of vstimecmp, changing its value as appropriate.  This ensures compatibility with existing guest VS-mode software that uses this SEE facility, while new VS-mode software takes advantage of vstimecmp directly.)
----------------------------------------------------------------------------------------

Hypervisor Interrupt Registers (hvip, hip, and hie)h
This extension modifies the description of the VSTIP/VSTIE bits in the hip/hie registers as follows:

Bits hip.VSTIP and hie.VSTIE are the interrupt-pending and interrupt-enable bits for VS-level timer interrupts. VSTIP is read-only in hip, and is the logical-OR of hvip.VSTIP and the timer interrupt signal resulting from vstimecmp (if vstimecmp is implemented).  The hip.VSTIP bit, in response to timer interrupts generated by vstimecmp, is set and cleared by writing vstimecmp with a value that respectively is less than or equal to, or greater than, the current (mtime + htimedelta) value.  The hip.VSTIP bit remains defined while V=0 as well as V=1.

Hypervisor Counter-Enable Register (hcounteren)
This extension adds to the description of the TM bit in this register as follows:

In addition, when the TM bit in the hcounteren register is clear, attempts to read the vstimecmp register while executing in VS-mode will cause a virtual illegal instruction exception if the same bit in mcounteren is 1. When this bit is set, access to the vstimecmp register (if implemented) is permitted in VS-mode.


Allen Baum
 

So to know if M-mode can fake an Smode counter interrupt or not,(which it can currently do), 
Mmode SW needs to attempt to write STMECMP and see if it traps? OR attempt to set the STIP bit and see if it sticks? 
You might want some non-normative text about discovery.
IS there some reason for removing the existing behavior of allowing STIP to be writable in MIP?



On Tue, Dec 1, 2020 at 5:00 PM Greg Favor <gfavor@...> wrote:
Hi all,

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  Andrew and I discussed trying to help move a couple of Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - through this process sooner than later (especially since this entails much more than simply proposing and reviewing a spec).  For this one Siqi and I are co-sponsors.  The following starts with an intro for context, and then provides the draft spec.

=================================================================
The current Privileged specification only defines a hardware mechanism for generating machine-mode timer interrupts (based on the mtime and mtimecmp registers).  With the resultant requirement that timer services for S-mode/HS-mode (and for VS-mode) have to all be provided by M-mode - via SBI calls from S/HS-mode up to M-mode (or VS-mode calls to HS-mode and then to M-mode).  M-mode software then multiplexes these multiple logical timers onto its one physical M-mode timer facility, and the M-mode timer interrupt handler passes timer interrupts back down to the appropriate lower privilege mode.

Over the past year or more, there have been a number of requests to provide supervisor mode with its own CSR-based timer interrupt facility that it can directly manage to provide its own timer service (in the form of having its own stimecmp register) - thus eliminating the large overheads for emulating S/HS-mode timers and timer interrupt generation up in M-mode.  This proposal, using the new fast-track architecture extension process, serves to accomplish exactly this.  Further, this proposal lays the groundwork for adding a similar facility to the Hypervisor extension for VS-mode.   (ARMv8, for example, has separate timer registers for its equivalent of M, S/HS, and VS privilege levels.)

Below is a one-page draft spec of the proposal - which sticks to addressing the basic well-understood need that has been requested.  The proposed extension name is "Zstc" ('Zs' for privileged Supervisor arch extensions, and 'tc' for timecmp).

Various further pieces of related architectural functionality have previously been suggested, debated, and questioned - without clear conclusion and agreement as to whether these are truly justified.  These more contentious enhancements are left to be sorted out and pursued by a follow-on effort if desired and justified by sufficient people.

The suggestion to also introduce an mtimecmp CSR has been considered, but ruled out for now as not sufficiently justified over the current definition of mtimecmp as a memory-mapped register (which would have to remain for backward compatibility in any case - resulting in two aliases for this register).  But if there is sufficient interest, a separate email thread can be started to discuss the possible addition of an mtimecmp CSR (as part of this or a separate fast-track extension).  For now I'll leave out details about how the mtimecmp memory-mapped register and mtimecmp CSR could inter-relate (it doesn't have to be complicated in practice).

This extension adds a single CSR - the stimecmp CSR.  This extension also describes what the corresponding addition to the Hypervisor extension would be - the vstimecmp CSR.  The former is the official part of this fast-track arch extension.  The latter is solely informative and reflective of what would then likely be added to the current H-extension draft spec as part of its virtualization of S-mode.

The draft spec below provides all the details.  Note that this extension very specifically strives to maintain maximal consistency with many little details in the existing Privileged architecture and Hypervisor extension (and has already been reviewed by the two lead Privileged and Hypervisor architects to ensure this).

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

Thanks,
Siqi and Greg

=========================================================================
=======================  Supervisor Extension Additions  ========================

Supervisor Timer Register (stimecmp)
The stimecmp CSR is a 64-bit register and has 64-bit precision on all RV32 and RV64 systems.  In RV32 only, reads of the stimecmp CSR return the low 32 bits, while reads of the stimecmph CSR return the high 32 bits of stimecmp.

The proposed CSR numbers for stimecmp / stimecmph are 0x108 / 0x109 (within the Supervisor Trap Setup block of CSRs).

A supervisor timer interrupt becomes pending - as reflected in the STIP bit in the mip and sip registers - whenever mtime contains a value greater than or equal to stimecmp, treating the values as unsigned integers.  Writes to stimecmp are guaranteed to be reflected in STIP eventually, but not necessarily immediately.  The interrupt remains posted until stimecmp becomes greater than mtime - typically as a result of writing stimecmp. The interrupt will be taken based on the standard interrupt enable and delegation rules.

----------------------------  Non-Normative Text    ----------------------------
A spurious timer interrupt might occur if an interrupt handler advances stimecmp then immediately returns, because STIP might not yet have fallen in the interim. All software should be written to assume this event is possible, but most software should assume this event is extremely unlikely. It is almost always more performant to incur an occasional spurious timer interrupt than to poll STIP until it falls.  
----------------------------------------------------------------------------------------

----------------------------  Non-Normative Text    ----------------------------
In systems in which a supervisor execution environment (SEE) provides timer facilities via an SBI function call, this SBI call will continue to support requests to schedule a timer interrupt.  The SEE will simply make use of stimecmp, changing its value as appropriate.  This ensures compatibility with existing S-mode software that uses this SEE facility, while new S-mode software takes advantage of stimecmp directly.)
----------------------------------------------------------------------------------------

Machine Interrupt Registers (mip and mie)
This extension modifies the description of the STIP/STIE bits in these registers as follows:

If supervisor mode is implemented, its mip.STIP and mie.STIE are the interrupt-pending and interrupt-enable bits for supervisor-level timer interrupts.  If the stimecmp register is not implemented, STIP is writable in mip, and may be written by M-mode software to deliver timer interrupts to S-mode.  If the stimecmp (supervisor-mode timer compare) register is implemented, STIP is read-only in mip and reflects the supervisor-level timer interrupt signal resulting from stimecmp.  This timer interrupt signal is cleared by writing stimecmp with a value greater than the current mtime value.

Supervisor Interrupt Registers (sip and sie)
This extension modifies the description of the STIP/STIE bits in these registers as follows:
 
Bits sip.STIP and sie.STIE are the interrupt-pending and interrupt-enable bits for supervisor level timer interrupts.  If implemented, STIP is read-only in sip, and is either set and cleared by the execution environment (if stimecmp is not implemented), or reflects the timer interrupt signal resulting from stimecmp (if stimecmp is implemented).  The sip.STIP bit, in response to timer interrupts generated by stimecmp, is set and cleared by writing stimecmp with a value that respectively is less than or equal to, or greater than, the current mtime value.

Machine Counter-Enable Register (mcounteren)
This extension adds to the description of the TM bit in this register as follows:

In addition, when the TM bit in the mcounteren register is clear, attempts to read the stimecmp register while executing in S-mode will cause an illegal instruction exception. When this bit is set, access to the stimecmp register (if implemented) is permitted in S-mode.


=========================================================================
=======================  Hypervisor Extension Additions  ========================

Virtual Supervisor Timer Register (vstimecmp)
The vstimecmp CSR is a 64-bit register and has 64-bit precision on all RV32 and RV64 systems.  In RV32 only, reads of the vstimecmp CSR return the low 32 bits, while reads of the vstimecmph CSR return the high 32 bits of vstimecmp.

The proposed CSR numbers for vstimecmp / vstimecmph are 0x208 / 0x209 (within the Virtual Supervisor Registers block of CSRs, and mirroring the CSR numbers for stimecmp/stimecmph).

A virtual supervisor timer interrupt becomes pending - as reflected in the VSTIP bit in the hip register - whenever (mtime + htimedelta) contains a value greater than or equal to vstimecmp, treating the values as unsigned integers.  Writes to vstimecmp and htimedelta are guaranteed to be reflected in VSTIP eventually, but not necessarily immediately.  The interrupt remains posted until vstimecmp becomes greater than (mtime + htimedelta) - typically as a result of writing vstimecmp. The interrupt will be taken based on the standard interrupt enable and delegation rules while V=1.

----------------------------  Non-Normative Text    ----------------------------
In systems in which a supervisor execution environment (SEE) implemented by an HS-mode hypervisor provides timer facilities via an SBI function call, this SBI call will continue to support requests to schedule a timer interrupt.  The SEE will simply make use of vstimecmp, changing its value as appropriate.  This ensures compatibility with existing guest VS-mode software that uses this SEE facility, while new VS-mode software takes advantage of vstimecmp directly.)
----------------------------------------------------------------------------------------

Hypervisor Interrupt Registers (hvip, hip, and hie)h
This extension modifies the description of the VSTIP/VSTIE bits in the hip/hie registers as follows:

Bits hip.VSTIP and hie.VSTIE are the interrupt-pending and interrupt-enable bits for VS-level timer interrupts. VSTIP is read-only in hip, and is the logical-OR of hvip.VSTIP and the timer interrupt signal resulting from vstimecmp (if vstimecmp is implemented).  The hip.VSTIP bit, in response to timer interrupts generated by vstimecmp, is set and cleared by writing vstimecmp with a value that respectively is less than or equal to, or greater than, the current (mtime + htimedelta) value.  The hip.VSTIP bit remains defined while V=0 as well as V=1.

Hypervisor Counter-Enable Register (hcounteren)
This extension adds to the description of the TM bit in this register as follows:

In addition, when the TM bit in the hcounteren register is clear, attempts to read the vstimecmp register while executing in VS-mode will cause a virtual illegal instruction exception if the same bit in mcounteren is 1. When this bit is set, access to the vstimecmp register (if implemented) is permitted in VS-mode.


Greg Favor
 

On Tue, Dec 1, 2020 at 5:21 PM Allen Baum <allen.baum@...> wrote:
So to know if M-mode can fake an Smode counter interrupt or not,(which it can currently do), 
Mmode SW needs to attempt to write STMECMP and see if it traps? OR attempt to set the STIP bit and see if it sticks? 
You might want some non-normative text about discovery.

This text takes the same approach as much of the rest of the Supervisor chapter in the Priv spec, i.e. the existing and coming new discovery methods apply equally to these new CSRs as to all existing and new s*/h*/vs* CSRs that may be accessible in S/HS modes.  This extension doesn't try to establish anything different.  Or, put differently, the discovery issue is a general issue (for which this extension does not represent any form of special case).

Whether for this Zstc extension or the coming (for example) Zsn and Zsa extensions, there is trapping behavior as one (ugly) form of discovery method.  And then there will be the new discovery method being developed by tech-config which will be the more reasonable way to discover many things about optional extensions, and optional or WARL features within extensions.

So any new non-normative text about CSR discovery really should be added to the Priv/Unpriv specs as a general matter (separate from this and other in-progress Supervisor extensions).  (And each extension innovating its own "better" discovery method would be an undesirable way to go.)
 
IS there some reason for removing the existing behavior of allowing STIP to be writable in MIP?

Yes.  This was first pointed out by John, and Andrew and I agreed.  The gist is that when a hart implements stimecmp, for consistency we should require STIP to be read-only - just like MTIP.  (Especially since there is no longer a need for a software-writable bit behind STIP when stimecmp exists.)

Greg


Allen Baum
 

Yea, that makes sense, though I'd say it's not strictly backwards compatible; old Mcode will silently fail.
I will not complain if we make a blanket statement that Mcode backwards compatibility is not a requirement.

On Tue, Dec 1, 2020 at 5:53 PM Greg Favor <gfavor@...> wrote:
On Tue, Dec 1, 2020 at 5:21 PM Allen Baum <allen.baum@...> wrote:
So to know if M-mode can fake an Smode counter interrupt or not,(which it can currently do), 
Mmode SW needs to attempt to write STMECMP and see if it traps? OR attempt to set the STIP bit and see if it sticks? 
You might want some non-normative text about discovery.

This text takes the same approach as much of the rest of the Supervisor chapter in the Priv spec, i.e. the existing and coming new discovery methods apply equally to these new CSRs as to all existing and new s*/h*/vs* CSRs that may be accessible in S/HS modes.  This extension doesn't try to establish anything different.  Or, put differently, the discovery issue is a general issue (for which this extension does not represent any form of special case).

Whether for this Zstc extension or the coming (for example) Zsn and Zsa extensions, there is trapping behavior as one (ugly) form of discovery method.  And then there will be the new discovery method being developed by tech-config which will be the more reasonable way to discover many things about optional extensions, and optional or WARL features within extensions.

So any new non-normative text about CSR discovery really should be added to the Priv/Unpriv specs as a general matter (separate from this and other in-progress Supervisor extensions).  (And each extension innovating its own "better" discovery method would be an undesirable way to go.)
 
IS there some reason for removing the existing behavior of allowing STIP to be writable in MIP?

Yes.  This was first pointed out by John, and Andrew and I agreed.  The gist is that when a hart implements stimecmp, for consistency we should require STIP to be read-only - just like MTIP.  (Especially since there is no longer a need for a software-writable bit behind STIP when stimecmp exists.)

Greg


Greg Favor
 

On Tue, Dec 1, 2020 at 6:25 PM Allen Baum <allen.baum@...> wrote:
Yea, that makes sense, though I'd say it's not strictly backwards compatible; old Mcode will silently fail.
I will not complain if we make a blanket statement that Mcode backwards compatibility is not a requirement.

That is the collective view.  In general, machine-level software is expected to know what it needs to know about the hart on which it is running.  (There's that pesky discovery thing again. :) )

Greg


Jonathan Behrens <behrensj@...>
 

In addition, when the TM bit in the mcounteren register is clear, attempts to read the stimecmp register while executing in S-mode will cause an illegal instruction exception.

Presumably this should say "attempts to read or write the stimecmp register..."?

Jonathan

On Tue, Dec 1, 2020 at 8:01 PM Greg Favor via lists.riscv.org <gfavor=ventanamicro.com@...> wrote:
Hi all,

Recently the TSC established a lightweight "fast track" architecture extension process that small, straightforward, relatively uncontentious arch extension proposals can utilize.  Andrew and I discussed trying to help move a couple of Privileged architecture related small extensions - that a number of people/companies have expressed desire for over the past year - through this process sooner than later (especially since this entails much more than simply proposing and reviewing a spec).  For this one Siqi and I are co-sponsors.  The following starts with an intro for context, and then provides the draft spec.

=================================================================
The current Privileged specification only defines a hardware mechanism for generating machine-mode timer interrupts (based on the mtime and mtimecmp registers).  With the resultant requirement that timer services for S-mode/HS-mode (and for VS-mode) have to all be provided by M-mode - via SBI calls from S/HS-mode up to M-mode (or VS-mode calls to HS-mode and then to M-mode).  M-mode software then multiplexes these multiple logical timers onto its one physical M-mode timer facility, and the M-mode timer interrupt handler passes timer interrupts back down to the appropriate lower privilege mode.

Over the past year or more, there have been a number of requests to provide supervisor mode with its own CSR-based timer interrupt facility that it can directly manage to provide its own timer service (in the form of having its own stimecmp register) - thus eliminating the large overheads for emulating S/HS-mode timers and timer interrupt generation up in M-mode.  This proposal, using the new fast-track architecture extension process, serves to accomplish exactly this.  Further, this proposal lays the groundwork for adding a similar facility to the Hypervisor extension for VS-mode.   (ARMv8, for example, has separate timer registers for its equivalent of M, S/HS, and VS privilege levels.)

Below is a one-page draft spec of the proposal - which sticks to addressing the basic well-understood need that has been requested.  The proposed extension name is "Zstc" ('Zs' for privileged Supervisor arch extensions, and 'tc' for timecmp).

Various further pieces of related architectural functionality have previously been suggested, debated, and questioned - without clear conclusion and agreement as to whether these are truly justified.  These more contentious enhancements are left to be sorted out and pursued by a follow-on effort if desired and justified by sufficient people.

The suggestion to also introduce an mtimecmp CSR has been considered, but ruled out for now as not sufficiently justified over the current definition of mtimecmp as a memory-mapped register (which would have to remain for backward compatibility in any case - resulting in two aliases for this register).  But if there is sufficient interest, a separate email thread can be started to discuss the possible addition of an mtimecmp CSR (as part of this or a separate fast-track extension).  For now I'll leave out details about how the mtimecmp memory-mapped register and mtimecmp CSR could inter-relate (it doesn't have to be complicated in practice).

This extension adds a single CSR - the stimecmp CSR.  This extension also describes what the corresponding addition to the Hypervisor extension would be - the vstimecmp CSR.  The former is the official part of this fast-track arch extension.  The latter is solely informative and reflective of what would then likely be added to the current H-extension draft spec as part of its virtualization of S-mode.

The draft spec below provides all the details.  Note that this extension very specifically strives to maintain maximal consistency with many little details in the existing Privileged architecture and Hypervisor extension (and has already been reviewed by the two lead Privileged and Hypervisor architects to ensure this).

This posting to this email list starts an initial review period (over the next few weeks) for people to provide feedback, questions, comments, etc.

Thanks,
Siqi and Greg

=========================================================================
=======================  Supervisor Extension Additions  ========================

Supervisor Timer Register (stimecmp)
The stimecmp CSR is a 64-bit register and has 64-bit precision on all RV32 and RV64 systems.  In RV32 only, reads of the stimecmp CSR return the low 32 bits, while reads of the stimecmph CSR return the high 32 bits of stimecmp.

The proposed CSR numbers for stimecmp / stimecmph are 0x108 / 0x109 (within the Supervisor Trap Setup block of CSRs).

A supervisor timer interrupt becomes pending - as reflected in the STIP bit in the mip and sip registers - whenever mtime contains a value greater than or equal to stimecmp, treating the values as unsigned integers.  Writes to stimecmp are guaranteed to be reflected in STIP eventually, but not necessarily immediately.  The interrupt remains posted until stimecmp becomes greater than mtime - typically as a result of writing stimecmp. The interrupt will be taken based on the standard interrupt enable and delegation rules.

----------------------------  Non-Normative Text    ----------------------------
A spurious timer interrupt might occur if an interrupt handler advances stimecmp then immediately returns, because STIP might not yet have fallen in the interim. All software should be written to assume this event is possible, but most software should assume this event is extremely unlikely. It is almost always more performant to incur an occasional spurious timer interrupt than to poll STIP until it falls.  
----------------------------------------------------------------------------------------

----------------------------  Non-Normative Text    ----------------------------
In systems in which a supervisor execution environment (SEE) provides timer facilities via an SBI function call, this SBI call will continue to support requests to schedule a timer interrupt.  The SEE will simply make use of stimecmp, changing its value as appropriate.  This ensures compatibility with existing S-mode software that uses this SEE facility, while new S-mode software takes advantage of stimecmp directly.)
----------------------------------------------------------------------------------------

Machine Interrupt Registers (mip and mie)
This extension modifies the description of the STIP/STIE bits in these registers as follows:

If supervisor mode is implemented, its mip.STIP and mie.STIE are the interrupt-pending and interrupt-enable bits for supervisor-level timer interrupts.  If the stimecmp register is not implemented, STIP is writable in mip, and may be written by M-mode software to deliver timer interrupts to S-mode.  If the stimecmp (supervisor-mode timer compare) register is implemented, STIP is read-only in mip and reflects the supervisor-level timer interrupt signal resulting from stimecmp.  This timer interrupt signal is cleared by writing stimecmp with a value greater than the current mtime value.

Supervisor Interrupt Registers (sip and sie)
This extension modifies the description of the STIP/STIE bits in these registers as follows:
 
Bits sip.STIP and sie.STIE are the interrupt-pending and interrupt-enable bits for supervisor level timer interrupts.  If implemented, STIP is read-only in sip, and is either set and cleared by the execution environment (if stimecmp is not implemented), or reflects the timer interrupt signal resulting from stimecmp (if stimecmp is implemented).  The sip.STIP bit, in response to timer interrupts generated by stimecmp, is set and cleared by writing stimecmp with a value that respectively is less than or equal to, or greater than, the current mtime value.

Machine Counter-Enable Register (mcounteren)
This extension adds to the description of the TM bit in this register as follows:

In addition, when the TM bit in the mcounteren register is clear, attempts to read the stimecmp register while executing in S-mode will cause an illegal instruction exception. When this bit is set, access to the stimecmp register (if implemented) is permitted in S-mode.


=========================================================================
=======================  Hypervisor Extension Additions  ========================

Virtual Supervisor Timer Register (vstimecmp)
The vstimecmp CSR is a 64-bit register and has 64-bit precision on all RV32 and RV64 systems.  In RV32 only, reads of the vstimecmp CSR return the low 32 bits, while reads of the vstimecmph CSR return the high 32 bits of vstimecmp.

The proposed CSR numbers for vstimecmp / vstimecmph are 0x208 / 0x209 (within the Virtual Supervisor Registers block of CSRs, and mirroring the CSR numbers for stimecmp/stimecmph).

A virtual supervisor timer interrupt becomes pending - as reflected in the VSTIP bit in the hip register - whenever (mtime + htimedelta) contains a value greater than or equal to vstimecmp, treating the values as unsigned integers.  Writes to vstimecmp and htimedelta are guaranteed to be reflected in VSTIP eventually, but not necessarily immediately.  The interrupt remains posted until vstimecmp becomes greater than (mtime + htimedelta) - typically as a result of writing vstimecmp. The interrupt will be taken based on the standard interrupt enable and delegation rules while V=1.

----------------------------  Non-Normative Text    ----------------------------
In systems in which a supervisor execution environment (SEE) implemented by an HS-mode hypervisor provides timer facilities via an SBI function call, this SBI call will continue to support requests to schedule a timer interrupt.  The SEE will simply make use of vstimecmp, changing its value as appropriate.  This ensures compatibility with existing guest VS-mode software that uses this SEE facility, while new VS-mode software takes advantage of vstimecmp directly.)
----------------------------------------------------------------------------------------

Hypervisor Interrupt Registers (hvip, hip, and hie)h
This extension modifies the description of the VSTIP/VSTIE bits in the hip/hie registers as follows:

Bits hip.VSTIP and hie.VSTIE are the interrupt-pending and interrupt-enable bits for VS-level timer interrupts. VSTIP is read-only in hip, and is the logical-OR of hvip.VSTIP and the timer interrupt signal resulting from vstimecmp (if vstimecmp is implemented).  The hip.VSTIP bit, in response to timer interrupts generated by vstimecmp, is set and cleared by writing vstimecmp with a value that respectively is less than or equal to, or greater than, the current (mtime + htimedelta) value.  The hip.VSTIP bit remains defined while V=0 as well as V=1.

Hypervisor Counter-Enable Register (hcounteren)
This extension adds to the description of the TM bit in this register as follows:

In addition, when the TM bit in the hcounteren register is clear, attempts to read the vstimecmp register while executing in VS-mode will cause a virtual illegal instruction exception if the same bit in mcounteren is 1. When this bit is set, access to the vstimecmp register (if implemented) is permitted in VS-mode.


Greg Favor
 

On Tue, Dec 1, 2020 at 6:49 PM Jonathan Behrens <behrensj@...> wrote:
In addition, when the TM bit in the mcounteren register is clear, attempts to read the stimecmp register while executing in S-mode will cause an illegal instruction exception.

Presumably this should say "attempts to read or write the stimecmp register..."?

Yes.  Thanks for catching that.  (And the same applies for vstimecmp.)

Greg
 


Phil McCoy
 

Hi,

Was any consideration given to the possibility of defining stimecmp as a memory-mapped register (like mtimecmp) rather than a CSR?
Assuming that the timer will actually be implemented in a block outside of the CPU (e.g. the ACLINT), a memory-mapped register would be preferable for interfacing the CPU to the timer block.  Keep in mind that in most high-end (especially multi-processor) systems will have the timer in a separate clock domain from the CPU and CSRs (which seems to have been the key motivation for defining mtime/mtimecmp as memory-mapped registers rather than CSRs).

Also, just curious about the overall status of this extension.  It is mentioned in the RVA22 profile, which would suggest that the Sstc definition should be finalized soon if it isn't already.

Thanks,
Phil McCoy


Paul Donahue
 

I believe that stimecmp is intentionally defined to compare against the time CSR, not against the mtime memory-mapped register.


Thanks,

-Paul


On Mon, Sep 13, 2021 at 12:49 PM Phil McCoy <pnm@...> wrote:
Hi,

Was any consideration given to the possibility of defining stimecmp as a memory-mapped register (like mtimecmp) rather than a CSR?
Assuming that the timer will actually be implemented in a block outside of the CPU (e.g. the ACLINT), a memory-mapped register would be preferable for interfacing the CPU to the timer block.  Keep in mind that in most high-end (especially multi-processor) systems will have the timer in a separate clock domain from the CPU and CSRs (which seems to have been the key motivation for defining mtime/mtimecmp as memory-mapped registers rather than CSRs).

Also, just curious about the overall status of this extension.  It is mentioned in the RVA22 profile, which would suggest that the Sstc definition should be finalized soon if it isn't already.

Thanks,
Phil McCoy


Greg Favor
 

On Mon, Sep 13, 2021 at 12:49 PM Phil McCoy <pnm@...> wrote:
Was any consideration given to the possibility of defining stimecmp as a memory-mapped register (like mtimecmp) rather than a CSR?
Assuming that the timer will actually be implemented in a block outside of the CPU (e.g. the ACLINT), a memory-mapped register would be preferable for interfacing the CPU to the timer block.  Keep in mind that in most high-end (especially multi-processor) systems will have the timer in a separate clock domain from the CPU and CSRs (which seems to have been the key motivation for defining mtime/mtimecmp as memory-mapped registers rather than CSRs).

Paul's comment is correct.  The time, stimecmp, and vstimecmp registers are all CSRs within a hart; separate from the mtime register located typically somewhere that may not be nearby any or every hart.  Also keep in mind that the vstimecmp CSR will be context switched by a hypervisor.  And in some applications even the stimecmp CSR may be context switched some.  Whereas the memory-mapped mtimecmp would generally never be context switched.

If anything, during discussions about this extension, the question was raised in the other direction about also adding an mtimecmp CSR for consistency.  But that was a whole other discussion to be had - given that there is existing mtimecmp functionality and not a strong argument for changing that.
 
Also, just curious about the overall status of this extension.  It is mentioned in the RVA22 profile, which would suggest that the Sstc definition should be finalized soon if it isn't already.

Keep in mind that the Profile specs are very much still under development and won't be frozen til November.  While the Sstc extension is driving to get ratified by mid-November.

Greg


Greg Favor
 

Also note that vstimecmp wants to compare to time as adjusted by the htimedelta CSR - which only exists within each hart.

Greg

On Mon, Sep 13, 2021 at 3:22 PM Greg Favor via lists.riscv.org <gfavor=ventanamicro.com@...> wrote:
On Mon, Sep 13, 2021 at 12:49 PM Phil McCoy <pnm@...> wrote:
Was any consideration given to the possibility of defining stimecmp as a memory-mapped register (like mtimecmp) rather than a CSR?
Assuming that the timer will actually be implemented in a block outside of the CPU (e.g. the ACLINT), a memory-mapped register would be preferable for interfacing the CPU to the timer block.  Keep in mind that in most high-end (especially multi-processor) systems will have the timer in a separate clock domain from the CPU and CSRs (which seems to have been the key motivation for defining mtime/mtimecmp as memory-mapped registers rather than CSRs).

Paul's comment is correct.  The time, stimecmp, and vstimecmp registers are all CSRs within a hart; separate from the mtime register located typically somewhere that may not be nearby any or every hart.  Also keep in mind that the vstimecmp CSR will be context switched by a hypervisor.  And in some applications even the stimecmp CSR may be context switched some.  Whereas the memory-mapped mtimecmp would generally never be context switched.

If anything, during discussions about this extension, the question was raised in the other direction about also adding an mtimecmp CSR for consistency.  But that was a whole other discussion to be had - given that there is existing mtimecmp functionality and not a strong argument for changing that.
 
Also, just curious about the overall status of this extension.  It is mentioned in the RVA22 profile, which would suggest that the Sstc definition should be finalized soon if it isn't already.

Keep in mind that the Profile specs are very much still under development and won't be frozen til November.  While the Sstc extension is driving to get ratified by mid-November.

Greg


Phil McCoy
 

> stimecmp is intentionally defined to compare against the time CSR

But time is just a pseudo-alias for the memory-mapped mtime register, right?  Per the privileged ISA spec section 3.1.11 and 3.1.12:
The time CSR is a read-only shadow of the memory-mapped mtime register.
. . .

Implementations can convert reads of the time and timeh CSRs into loads to the memorymapped
mtime register, or emulate this functionality in M-mode software.
. . .

Because the time counter can be shared between multiple cores, it cannot be inhibited with
the mcountinhibit mechanism.

So, still we have the problem that the stimecmp CSR (which is presumably physically located in the CPU/hart) is being compared against a timer register that is physically located in a separate timer block that is likely to be physically distant from the CPU/hart and in a different clock and power domain.

This problem is described in a fair amount of detail in Section 3.2.1:
The timer facility is defined to use wall-clock time rather than a cycle counter to support modern
processors that run with a highly variable clock frequency to save energy through dynamic voltage
and frequency scaling.
Accurate real-time clocks (RTCs) are relatively expensive to provide (requiring a crystal
or MEMS oscillator) and have to run even when the rest of system is powered down, and so
there is usually only one in a system located in a different frequency/voltage domain from the
processors. Hence, the RTC must be shared by all the harts in a system and accesses to the RTC
will potentially incur the penalty of a voltage-level-shifter and clock-domain crossing. It is thus
more natural to expose mtime as a memory-mapped register than as a CSR.
Lower privilege levels do not have their own timecmp registers. Instead, machine-mode
software can implement any number of virtual timers on a hart by multiplexing the next timer
interrupt into the mtimecmp register.
Simple fixed-frequency systems can use a single clock for both cycle counting and wall-clock
time.

It seems that a much more modular design would implement stimecmp registers as memory-mapped registers in the ACLINT block analogously to how the mtimecmp registers are defined and implemented.


> Also note that vstimecmp wants to compare to time as adjusted by the htimedelta CSR - which only exists within each hart.
Yes, this is a bit inconvenient.  Trap & Emulate might be a sensible implementation for vstimecmp anyway.

Thanks again for your input on this question.

Cheers,
Phil


Greg Favor
 

On Mon, Sep 13, 2021 at 5:07 PM Phil McCoy <pnm@...> wrote:
> stimecmp is intentionally defined to compare against the time CSR

But time is just a pseudo-alias for the memory-mapped mtime register, right?  Per the privileged ISA spec section 3.1.11 and 3.1.12:
The time CSR is a read-only shadow of the memory-mapped mtime register.

The time CSR is an actual CSR.  It is a "shadow" in the sense that time values in mtime eventually appear in the time CSR.  You can imagine they are always equal, but the architecture allows for there to be a delay between when a value appears in mtime and when it appears in the time CSR.
 
. . .
Implementations can convert reads of the time and timeh CSRs into loads to the memorymapped
mtime register, or emulate this functionality in M-mode software.

Yes, an implementation can trap accesses to the time CSR and emulate them, or an implementation can convert the CSR accesses to memory reads of the mtime register.  But the time CSR itself is architecturally a CSR within a hart.  Whether actually implemented within the hart, or trapped and emulated, or transparently "emulated" by hardware.
 
. . .
Because the time counter can be shared between multiple cores, it cannot be inhibited with
the mcountinhibit mechanism.

So, still we have the problem that the stimecmp CSR (which is presumably physically located in the CPU/hart) is being compared against a timer register that is physically located in a separate timer block that is likely to be physically distant from the CPU/hart and in a different clock and power domain.

stimecmp is compared against time, and vstimecmp is compared against (time+htimedelta).  These and time can all be physically implemented in the hart, or they can all be trapped and emulated (in conjunction with suitable hardware outside the hart), or all "emulated" by hardware.  All implementation approaches are accommodated.  While low-end designs may take one of the latter implementation approaches, it is expected that higher end designs will physically implement time within the hart (as is done in most all ARM and x86 designs).  Most of the past proposals from the RV community for Sstc-like functionality have all been for having CSRs and, in one case, also introducing an mtimecmp CSR.

Greg


 


Ved Shanbhogue
 

Greg Hi -

Agree with you on what you wrote below.

I have a few comments on on the vstimecmp section of the proposal:

1."When this bit is set, access to the vstimecmp register (if implemented) is permitted in VS-mode." contradicts the privileged specification. To support nested virtualization, accesses to the VS CSRs should cause a virtual instruction exception as noted in section 5.2 of the privileged specification. Perhaps this text should be updated to state "When this bit is set, the vstimecmp substitutes for stimecmp so that instructions that would normally read/write stimecmp would instead access vstimecmp."

2."when the TM bit in the hcounteren register is clear, attempts to access the vstimecmp register while executing in VS-mode will cause a virtual illegal instruction exception if the same bit in mcounteren is 1" is not clear on the type of exception. Is "virtual illegal instruction" intended to be "illegal instruction" or "virtual instruction" exception?  "illegal instruction exception" looks right.

3."whenever (time + htimedelta) contains a value greater than or equal to vstimecmp" - may be a bit ambiguous as "time" when V=1 is already defined to have the htimedelta offset included. Perhaps this could be restated to replace time with mtime so that there is no ambiguity in the definition - "whenever (mtime + htimedelta) contains a value greater than or equal to vstimecmp"

Further, could the proposal be extended to also include the mtimecmp CSR?

- ved

On 9/13/21 7:32 PM, Greg Favor wrote:
On Mon, Sep 13, 2021 at 5:07 PM Phil McCoy <pnm@...> wrote:
> stimecmp is intentionally defined to compare against the time CSR

But time is just a pseudo-alias for the memory-mapped mtime register, right?  Per the privileged ISA spec section 3.1.11 and 3.1.12:
The time CSR is a read-only shadow of the memory-mapped mtime register.

The time CSR is an actual CSR.  It is a "shadow" in the sense that time values in mtime eventually appear in the time CSR.  You can imagine they are always equal, but the architecture allows for there to be a delay between when a value appears in mtime and when it appears in the time CSR.
 
. . .
Implementations can convert reads of the time and timeh CSRs into loads to the memorymapped
mtime register, or emulate this functionality in M-mode software.

Yes, an implementation can trap accesses to the time CSR and emulate them, or an implementation can convert the CSR accesses to memory reads of the mtime register.  But the time CSR itself is architecturally a CSR within a hart.  Whether actually implemented within the hart, or trapped and emulated, or transparently "emulated" by hardware.
 
. . .
Because the time counter can be shared between multiple cores, it cannot be inhibited with
the mcountinhibit mechanism.

So, still we have the problem that the stimecmp CSR (which is presumably physically located in the CPU/hart) is being compared against a timer register that is physically located in a separate timer block that is likely to be physically distant from the CPU/hart and in a different clock and power domain.

stimecmp is compared against time, and vstimecmp is compared against (time+htimedelta).  These and time can all be physically implemented in the hart, or they can all be trapped and emulated (in conjunction with suitable hardware outside the hart), or all "emulated" by hardware.  All implementation approaches are accommodated.  While low-end designs may take one of the latter implementation approaches, it is expected that higher end designs will physically implement time within the hart (as is done in most all ARM and x86 designs).  Most of the past proposals from the RV community for Sstc-like functionality have all been for having CSRs and, in one case, also introducing an mtimecmp CSR.

Greg


 



Greg Favor
 

On Tue, Sep 14, 2021 at 7:13 AM Vedvyas Shanbhogue <ved@...> wrote:
I have a few comments on on the vstimecmp section of the proposal:

1."When this bit is set, access to the vstimecmp register (if implemented) is permitted in VS-mode." contradicts the privileged specification. To support nested virtualization, accesses to the VS CSRs should cause a virtual instruction exception as noted in section 5.2 of the privileged specification. Perhaps this text should be updated to state "When this bit is set, the vstimecmp substitutes for stimecmp so that instructions that would normally read/write stimecmp would instead access vstimecmp."

I think you're looking at an old version of the proposal.  The latest draft, which will be updated after completing Arch Review this week, cleans this up and will be made publicly available very shortly after.

2."when the TM bit in the hcounteren register is clear, attempts to access the vstimecmp register while executing in VS-mode will cause a virtual illegal instruction exception if the same bit in mcounteren is 1" is not clear on the type of exception. Is "virtual illegal instruction" intended to be "illegal instruction" or "virtual instruction" exception?  "illegal instruction exception" looks right.

This has already been cleaned up recently.  And a Virtual Instruction exception is the correct type of exception per the latest H extension draft (which defines when to take a Virtual Instruction versus Illegal Instruction exception).

3."whenever (time + htimedelta) contains a value greater than or equal to vstimecmp" - may be a bit ambiguous as "time" when V=1 is already defined to have the htimedelta offset included. Perhaps this could be restated to replace time with mtime so that there is no ambiguity in the definition - "whenever (mtime + htimedelta) contains a value greater than or equal to vstimecmp"

The architectural definition of htimedelta applies this delta against time, not mtime.

Further, could the proposal be extended to also include the mtimecmp CSR?

Way back this was discussed but was deferred since there is existing mtimecmp arch functionality and not a strong justification to introduce alternative arch functionality that the ecosystem would have to also support.  This doesn't mean that new discussion can't happen to consider this (as another fast track extension if consensus builds towards adding an mtimecmp CSR).

Greg