Date   

Re: OS-A SEE TG Meeting Wed Nov 30 @ 9am PT

Aaron Durbin
 

Hi All,

We were not able to go through the proposal as we didn't have the appropriate people. Please read over the proposal and provide your feedback.

-Aaron

On Mon, Nov 28, 2022 at 4:34 PM Aaron Durbin <adurbin@...> wrote:
Hi All,

We'll be meeting this coming Wed Nov 30 @ 9am PT. Link to join: https://zoom.us/j/2786028446?pwd=ZHFCR1JtKzg1WVpNZXNtci8xc1Mvdz09

We'll be going through Andrei's proposal which can be found here.

Thank you.

-Aaron


OS-A SEE TG Meeting Wed Nov 30 @ 9am PT

Aaron Durbin
 

Hi All,

We'll be meeting this coming Wed Nov 30 @ 9am PT. Link to join: https://zoom.us/j/2786028446?pwd=ZHFCR1JtKzg1WVpNZXNtci8xc1Mvdz09

We'll be going through Andrei's proposal which can be found here.

Thank you.

-Aaron


Re: SBI Debug Console Extension Proposal (Draft v4)

Anup Patel
 

On Thu, Nov 24, 2022 at 10:14 PM Mitchell Horne <mhorne@...> wrote:

Hi Anup,

Thank you for this proposal, I am happy to see that the legacy console
extension is getting a replacement, as I've found it tremendously useful
for early debugging in the past.

I have a couple of small questions inline below, but otherwise this
looks easy to work with.

Cheers,
Mitchell

On 8/3/22 13:07, Anup Patel wrote:
Hi All,

Based on feedback, below is the updated draft proposal of the
SBI Debug Console Extension ...

The motivations behind this proposal is as follows:
1) There is no new SBI extension replacing the legacy SBI v0.1
putchar() and getchar() calls. These legacy SBI v0.1 calls are
now disabled by default in Linux and will be eventually deprecated.
We need a replacement at least for the SBI v0.1 putchar() which
is widely used for early prints in various OSes and Bootloaders.
2) The OS-A Platforms (or SEE) specification need to mandate
a particular type of UART (8250, ns16550, or SiFive) as a standard
way for boot-time early prints in supervisor-mode software. Using
SBI debug console extension, the OS-A Platforms (or SEE)
specifications don't need to mandate any type of UART.
3) The legacy SBI v0.1 putchar() only supports printing one
character at a time. For some of the hypervisors (particularly KVM),
SBI based early prints (i.e. earlycon=sbi) is slow because each
SBI v0.1 putchar() trap will be forwarded to the KVM user-space
(KVMTOOL or QEMU).

In other words, the SBI debug console extension defines a standard
way for boot-time early prints in supervisor-mode software.

We have completed the proof-of-concept implementation of this
SBI extension proposal. Refer, "riscv_sbi_dbcn_v1" branch in the
following repos:
https://github.com/avpatel/opensbi.git
https://github.com/avpatel/linux.git
https://github.com/avpatel/kvmtool.git

Please review it and provide your feedback.

Regards,
Anup

SBI Debug Console Extension (EID #0x4442434E "DBCN")
====================================================

The debug console extension defines a generic mechanism for boot
time early prints from supervisor-mode software which allows users
to catch boot-time issues in supervisor-mode software.

This extension replaces the legacy console putchar (EID #0x01)
extension with following improvements:
1) It follows the new calling convention defined by the SBI v1.0
(or higher) specification.
2) It allows supervisor software to print multiple characters
in a single SBI call.

Some of the functions defined by this SBI extension have physical
memory as parameters for input and/or output. This physical memory
parameter MUST satisfy following requirements:
1) The SBI implementation MUST check that the supervisor-mode
software is allowed to read and write the specified physical
memory on the calling HART.
2) The SBI implementation MUST access the specified physical
memory using the PMA attribute.
Note: If the supervisor-mode software access the same physical
memory using memory type different from PMA then a loss of
coherence or unexpected memory ordering may occur and the
invoking software should follow the rules and sequences defined
in the RISC-V Svpbmt specification to prevent the loss of
coherence and memory ordering.

If the underlying physical console has extra bits for error checking
(or correction) then these extra bits should be handled by the SBI
implementation.

Function: Console Puts (FID #0)
-------------------------------

struct sbiret sbi_debug_console_puts(unsigned long num_chars,
uint64_t base_addr)
Is there a reason that base_addr is the second argument and not the
first? To me, it is much more natural to have the string's address be
the first argument, as num_chars only really has meaning in relation to
this string.
The uint64_t needs two registers on RV32 whereas it only needs one
register on RV64. Now if base_addr is the first parameter then for:
1) RV32
a0 = lower 32bits of base address
a1 = upper 32bits of base address
a2 = num characters
2) RV64
a0 = 64bits of base address
a1 = num characters

As we can see, "a1" register has different semantics for RV32 vs RV64
when base_addr is first parameter whereas if base_addr was second
parameter then "a2" register has upper 32bits of base address on RV32
and it is unused on RV64.

Considering, this SBI call is going to be used from assembly sources
as well, it seemed simpler/easy-to-use having "base_addr" as second
parameter.


Also, is there a reason that base_addr is a uint64_t and not an unsigned
long? Other physical addresses accepted by SBI interface are defined to
be unsigned long, e.g. start_addr in sbi_hart_start() or
sbi_remote_hfence_gvma_vmid().
If we use "unsigned long" for physical address then we will have to define
it as "base_addr_div_by_4" because RV32 systems can have 34 bits of
physical address space (as-per Sv32 MMU mode). This also means we
will be enforcing a 4-byte alignment on base_address of string to be
printed whereas strings (char []) are typically byte aligned so callers
might have to use an intermediate aligned buffer in this case which is
inconvenient.

In case of sbi_hart_start(), the "start_addr" is actually a execution address
with MMU off so this will be 32bit on RV32 and 64bit on RV64 hence the
"start_addr" is "unsigned long".

In case of sbi_remote_hfence_gvma_vmid() and sbi_remote_hfence_gvma(),
the "start_addr" parameter should have been "start_addr_div_by_4" to be
suitable for both RV32 and RV64 but by the time we realize this it was too
late. We will need to define sbi_remote_hfence_gvma_vmid2() and
sbi_remote_hfence_gvma2() to make SBI RFENCE extension suitable
for both RV32 and RV64.


Maybe these questions have been considered in previous versions of the
proposal; I have not followed closely. You may disregard them as
appropriate.
Yes, we did receive similar questions in previous versions of the proposal
but I am happy to provide the explanation again.


Print a specified input string on the debug console.

The `num_chars` parameter specifies the number of characters (or
bytes) in the input string whereas `base_addr` parameter specifies
the physical base address of the input string.

This is a blocking SBI call and will only return after printing all
characters of the input string.

Errors:
SBI_SUCCESS - Characters printed successfully.
SBI_ERR_INVALID_ADDRESS - The string pointed by `num_chars` and `base_addr`
parameters is not entirely accessible to
supervisor-mode.
SBI_ERR_FAILED - Failed to print characters due to I/O errors.




Regards,
Anup


Re: SBI Debug Console Extension Proposal (Draft v4)

Mitchell Horne
 

Hi Anup,

Thank you for this proposal, I am happy to see that the legacy console extension is getting a replacement, as I've found it tremendously useful for early debugging in the past.

I have a couple of small questions inline below, but otherwise this looks easy to work with.

Cheers,
Mitchell

On 8/3/22 13:07, Anup Patel wrote:
Hi All,
Based on feedback, below is the updated draft proposal of the
SBI Debug Console Extension ...
The motivations behind this proposal is as follows:
1) There is no new SBI extension replacing the legacy SBI v0.1
putchar() and getchar() calls. These legacy SBI v0.1 calls are
now disabled by default in Linux and will be eventually deprecated.
We need a replacement at least for the SBI v0.1 putchar() which
is widely used for early prints in various OSes and Bootloaders.
2) The OS-A Platforms (or SEE) specification need to mandate
a particular type of UART (8250, ns16550, or SiFive) as a standard
way for boot-time early prints in supervisor-mode software. Using
SBI debug console extension, the OS-A Platforms (or SEE)
specifications don't need to mandate any type of UART.
3) The legacy SBI v0.1 putchar() only supports printing one
character at a time. For some of the hypervisors (particularly KVM),
SBI based early prints (i.e. earlycon=sbi) is slow because each
SBI v0.1 putchar() trap will be forwarded to the KVM user-space
(KVMTOOL or QEMU).
In other words, the SBI debug console extension defines a standard
way for boot-time early prints in supervisor-mode software.
We have completed the proof-of-concept implementation of this
SBI extension proposal. Refer, "riscv_sbi_dbcn_v1" branch in the
following repos:
https://github.com/avpatel/opensbi.git
https://github.com/avpatel/linux.git
https://github.com/avpatel/kvmtool.git
Please review it and provide your feedback.
Regards,
Anup
SBI Debug Console Extension (EID #0x4442434E "DBCN")
====================================================
The debug console extension defines a generic mechanism for boot
time early prints from supervisor-mode software which allows users
to catch boot-time issues in supervisor-mode software.
This extension replaces the legacy console putchar (EID #0x01)
extension with following improvements:
1) It follows the new calling convention defined by the SBI v1.0
(or higher) specification.
2) It allows supervisor software to print multiple characters
in a single SBI call.
Some of the functions defined by this SBI extension have physical
memory as parameters for input and/or output. This physical memory
parameter MUST satisfy following requirements:
1) The SBI implementation MUST check that the supervisor-mode
software is allowed to read and write the specified physical
memory on the calling HART.
2) The SBI implementation MUST access the specified physical
memory using the PMA attribute.
Note: If the supervisor-mode software access the same physical
memory using memory type different from PMA then a loss of
coherence or unexpected memory ordering may occur and the
invoking software should follow the rules and sequences defined
in the RISC-V Svpbmt specification to prevent the loss of
coherence and memory ordering.
If the underlying physical console has extra bits for error checking
(or correction) then these extra bits should be handled by the SBI
implementation.
Function: Console Puts (FID #0)
-------------------------------
struct sbiret sbi_debug_console_puts(unsigned long num_chars,
uint64_t base_addr)
Is there a reason that base_addr is the second argument and not the first? To me, it is much more natural to have the string's address be the first argument, as num_chars only really has meaning in relation to this string.

Also, is there a reason that base_addr is a uint64_t and not an unsigned long? Other physical addresses accepted by SBI interface are defined to be unsigned long, e.g. start_addr in sbi_hart_start() or sbi_remote_hfence_gvma_vmid().

Maybe these questions have been considered in previous versions of the proposal; I have not followed closely. You may disregard them as appropriate.

Print a specified input string on the debug console.
The `num_chars` parameter specifies the number of characters (or
bytes) in the input string whereas `base_addr` parameter specifies
the physical base address of the input string.
This is a blocking SBI call and will only return after printing all
characters of the input string.
Errors:
SBI_SUCCESS - Characters printed successfully.
SBI_ERR_INVALID_ADDRESS - The string pointed by `num_chars` and `base_addr`
parameters is not entirely accessible to
supervisor-mode.
SBI_ERR_FAILED - Failed to print characters due to I/O errors.


OS-A SEE TG Meeting Nov 15 - 9am PT

Aaron Durbin
 

Hi All,

We'll be having an OS-A SEE TG Meeting Nov 15 @ 9am PT. If there are specific topics people are interested in talking about please add them here.

Thank you.

-Aaron


OS-A SEE TG Meeting Nov 1st 9am PT

Aaron Durbin
 

Hi All,

We'll be conducting an OS-A SEE TG meeting Nov 1st @ 9am PT. Link to join is here: https://zoom.us/j/2786028446?pwd=ZHFCR1JtKzg1WVpNZXNtci8xc1Mvdz09


OS-A SEE TG Meeting Oct 18 - 9am PT

Aaron Durbin
 

Hi All,

We're having a OS-A SEE TG meeting tomorrow Oct 18 at 9am PT. We last met on Oct 4. The meeting slides can be found here. We did not get through everything as we had a discussion about scoping for OS-A SEE. Tomorrow we'll try and work our way through those types of topics so we can nail down some decisions. As always there is a doc to capture topics people want to chat about here in future meetings.

Thanks.

-Aaron


Re: Hello!

Andrei Warkentin
 

Thanks Anup - I signed up to that list as well (and thanks for clarifying it's purpose w.r.t. this group).

-----Original Message-----
From: tech-unixplatformspec@... <tech-unixplatformspec@...> On Behalf Of Anup Patel
Sent: Monday, October 10, 2022 12:22 PM
To: Warkentin, Andrei <andrei.warkentin@...>
Cc: tech-unixplatformspec@...
Subject: Re: [RISC-V] [tech-unixplatformspec] Hello!

Hi Andrei,

On Mon, Oct 10, 2022 at 10:42 PM Andrei Warkentin <andrei.warkentin@...> wrote:

Dear all,



I just wanted to introduce myself and say ‘hi’. I recently joined the Intel SSE group as an engineer focusing on RISC-V standardization (in particular with an interest in collaborating on the OS-A spec, and its touchpoints with SBI, UEFI, ACPI, etc). Some of you may remember me in my past life participating in a similar vein in SystemReady activities. I’m excited to be here and look forward to a fruitful collaboration. I think it’s amazing how much has been accomplished in such a short amount of time. Kudos!
We now have a separate PRS TG (Platform Runtime Services) for yearly updates to SBI, UEFI, and ACPI specification.

Regards,
Anup




I poked around the mailing list server but wasn’t able to find a clear answer… Does this group have any kind of periodical group sync-ups I should add to my calendar?



A


Re: Hello!

Anup Patel
 

Hi Andrei,

On Mon, Oct 10, 2022 at 10:42 PM Andrei Warkentin
<andrei.warkentin@...> wrote:

Dear all,



I just wanted to introduce myself and say ‘hi’. I recently joined the Intel SSE group as an engineer focusing on RISC-V standardization (in particular with an interest in collaborating on the OS-A spec, and its touchpoints with SBI, UEFI, ACPI, etc). Some of you may remember me in my past life participating in a similar vein in SystemReady activities. I’m excited to be here and look forward to a fruitful collaboration. I think it’s amazing how much has been accomplished in such a short amount of time. Kudos!
We now have a separate PRS TG (Platform Runtime Services) for yearly
updates to SBI, UEFI, and ACPI specification.

Regards,
Anup




I poked around the mailing list server but wasn’t able to find a clear answer… Does this group have any kind of periodical group sync-ups I should add to my calendar?



A


Hello!

Andrei Warkentin
 

Dear all,

 

I just wanted to introduce myself and say ‘hi’. I recently joined the Intel SSE group as an engineer focusing on RISC-V standardization (in particular with an interest in collaborating on the OS-A spec, and its touchpoints with SBI, UEFI, ACPI, etc). Some of you may remember me in my past life participating in a similar vein in SystemReady activities. I’m excited to be here and look forward to a fruitful collaboration. I think it’s amazing how much has been accomplished in such a short amount of time. Kudos!

 

I poked around the mailing list server but wasn’t able to find a clear answer… Does this group have any kind of periodical group sync-ups I should add to my calendar?

 

A


OS-A SEE Meeting October 4th @ 9am Pacific

Aaron Durbin
 

Hi All,

This is a reminder that we'll be having the OS-A SEE meeting on October 4th at 9am Pacific Time.

Also, if there are topics you'd like to see addressed in this meeting or future ones I set up a doc to put those items into: https://docs.google.com/document/d/1irS30WGJIHwtDkvl0KxH9V6SKeNDIv2mYNHt83ZfPs8/edit

That document sits in the OS-A SEE workgroup's folder: https://drive.google.com/drive/folders/1cTxussfAHwsazEYzlOfatIbHcZyrvCan

Thank you.

-Aaron


Re: [RISC-V] [tech-debug] SBI Debug Trigger Extension Proposal (Draft v4)

Anup Patel
 

On Wed, Sep 28, 2022 at 10:59 PM merle w <merlew4n6@...> wrote:

Anup Patel <apatel@...> 于2022年7月14日周四 17:39写道:

Hi All,

To support native debugging in S-mode and VS-mode, we need
a SBI Debug Trigger extension which complements the Sdtrig
extension.

Below is the draft proposal of the SBI Debug Trigger Extension...

Many thanks to folks (CC'ed here) who provided early feedback
and helped improve this proposal.

The proposed SBI extension is highly inspired from Linux HW
breakpoint infrastructure which is built on top of the Linux
perf subsystem.

Please review and provide your feedback.

Regards,
Anup

SBI Debug Trigger Extension (EID #0x44425452 "DBTR")
====================================================

The RISC-V Sdtrig ISA extension allows machine-mode software to
directly configure debug triggers which in-turn allows native
(or hosted) debugging without any external debugger.

The SBI debug trigger extension defines a SBI based abstraction to
provide native debugging for supervisor-mode software such that it:
1) Suitable for the rich operating systems and hypervisors running
in supervisor-mode.
2) Allows Guest (VS-mode) and Hypervisor (HS-mode) to share debug
triggers on a HART.

Some of the functions defined by this SBI extension have physical
memory as parameters for input and/or output. This physical memory
parameter MUST satisfy following requirements:
1) The base address of the physical memory MUST be 16-byte aligned
so that it is machine word aligned for RV32, RV64 and RV128.
2) The SBI implementation MUST check that the supervisor-mode
software is allowed to read and write the specified physical
memory on the calling HART.
3) The SBI implementation MUST access the specified physical
memory using the PMA attribute.
Note: If the supervisor-mode software access the same physical
memory using memory type different from PMA then a loss of
coherence or unexpected memory ordering may occur and the
invoking software should follow the rules and sequences defined
in the RISC-V Svpbmt specification to prevent the loss of
coherence and memory ordering.

Each debug trigger is assigned a logical number called `trig_idx`
by the SBI implementation. If total number of debug triggers are
`N` then `0 <= trig_idx < N`.
If I understand correctly, the N here should refer to the logical
trigger, not the physical number. sbi needs to maintain these logic
data, and this is related to runtime, which is difficult for sbi to
implement (if many programs are debugged at the same time, it may
take up a lot of logic triggers). I think this logical data should
be handed over to low-privilege level maintenance.
The `trig_idx` is intentionally a logical index so that Hypervisors
can share the debug triggers between guest and host. As an
example, you can look at `counter_idx` defined by SBI PMU
extension which follows a similar rationale.

This does not put any burden on M-mode firmware because
M-mode firmware can always treat "logical index" equals
"physical index".



The configuration of each debug trigger is expressed by three
entities `trig_tdata1`, `trig_tdata2`, and `trig_tdata3` which
are same as the `tdata1`, `tdata2`, and `tdata3` CSRs defined by
the RISC-V Sdtrig specification but with the following additional
constraints:
1) The `trig_tdata1.dmode` bit should always be zero.
2) The `trig_tdata1.m` bit should always be zero.
3) The `trig_tdata1.action` bits should always be zero.

The SBI implementation must also maintain an additional software
state for each debug trigger called `trig_state` which is encoded
as follows:
reserved [XLEN-1:5] - Reserved for future use.
vs [4:4] - Saved copy of the `trig_tdata1.vs` bit.
vu [3:3] - Saved copy of the `trig_tdata1.vu` bit.
s [2:2] - Saved copy of the `trig_tdata1.s` bit.
u [1:1] - Saved copy of the `trig_tdata1.u` bit.
mapped [0:0] - When set, the trigger has been mapped to
some HW debug trigger.

Function: Get number of triggers (FID #0)
-----------------------------------------

struct sbiret sbi_debug_num_triggers(unsigned long trig_tdata1)

Get the number of per-HART debug triggers which can support the
trigger configuration specified by `trig_tdata1` parameter.

The SBI implementation must return the total number of per-HART
debug triggers when `trig_tdata1 == 0`.

Returns the number of per-HART debug triggers in `sbiret.value`
and always returns SBI_SUCCESS in `sbiret.error`.

Function: Read triggers (FID #1)
--------------------------------

struct sbiret sbi_debug_read_triggers(unsigned long trig_idx_base,
unsigned long trig_count,
unsigned long out_addr_div_by_16)

Read the trigger state and configuration for a range of debug triggers
specified by `trig_idx_base` and `trig_count` parameters on the calling
HART.

The state and configuration of triggers will be written to an output
memory of size `trig_count * 4 * (XLEN / 8)` bytes. The base physical
address of the output memory right shifted by 4 is specified by the
`out_addr_div_by_16` parameter.

For i'th debug trigger in the specified range, the trigger configuration
consists of four XLEN-bit words written in little-endian format at
`offset = i * 4 * (XLEN / 8)` of the output memory. The contents of
output trigger configuration words are as follows:
word[0] = trig_state
word[1] = trig_tdata1
word[2] = trig_tdata2
word[3] = trig_tdata3

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Configuration read successfully.
SBI_ERR_INVALID_ADDRESS - The physical memory pointed by the
`out_addr_div_by_16` parameter is not
accessible to supervisor-mode.
SBI_ERR_INVALID_PARAM - Either `trig_idx_base + trig_count` or
`trig_idx_base` is greater than the total
number of triggers.

Function: Install triggers (FID #2)
-----------------------------------

struct sbiret sbi_debug_install_triggers(unsigned long trig_count,
unsigned long in_addr_div_by_16,
unsigned long out_addr_div_by_16)

Install debug triggers based on an array of trigger configurations
specified through an input memory and return the `trig_idx` of each
installed debug trigger in an output memory.

The `trig_count` parameter represents the number of debug triggers
to be installed.

The base physical address of the input memory right shifted by 4
is specified by the `in_addr_div_by_16` parameter and the size of
input memory is `trig_count * 4 * (XLEN / 8)` bytes. The i'th input
trigger configuration is at `offset = i * 4 * (XLEN / 8)` of the
input memory consisting of four consecutive XLEN-bit words in
little-endian format. The contents of each trigger configuration
are follows:
word[0] = Reserved for future use and should be zero
word[1] = trig_tdata1
word[2] = trig_tdata2
word[3] = trig_tdata3
The current debug specification has many variants: type 1, type 2,
type 6. There may be more manufacturer-defined specifications. Instead
of passing register information directly, we should need more abstraction,
which also simplifies the design of low-privilege software
We had started off with a simple abstraction which only covered type2
and type6 but quite a few folks are interested in having SBI extension
future proof so that we don't have to keep improving for other debug
trigger types.

Due to the above, we have a very generic SBI debug trigger extension
and we have sbi_debug_num_triggers() which provides some level
of discovery to S-mode software. This means if a SBI implementation
(M-mode firmware or Hypervisor) wants to only support type2 or type6
triggers then sbi_debug_num_triggers() will return non-zero only for
those types of trigger.



The base physical address of the output memory right shifted by
4 is specified by the `out_addr_div_by_16` parameter and the size
of output memory is `trig_count * (XLEN / 8)` bytes. The `trig_idx`
assigned for the i'th input trigger configuration is written at
`offset = i * (XLEN / 8)` of the output memory in little-endian
format.

For each input trigger configuration in the input memory, the SBI
implementation should:
1) Map a free `trig_idx` with an unused HW debug trigger that
matches the trigger configuration.
2) Save a copy of the `trig_tdata1.vs`, `trig_tdata1.vu`,
`trig_tdata1.s`, and `trig_tdata.u` bits in `trig_state`.
3) Update the `tdata1`, `tdata2`, and `tdata3` CSRs of the
HW debug trigger.

Additionally for each input trigger configuration chain, the
SBI implementation MUST assign contiguous `trig_idx` values
and contiguous HW debug triggers when installing debug triggers.

The last trigger configuration in the input memory MUST not have
`trig_tdata1.chain == 1` for `trig_tdata1.type = 2 or 6` to prevent
incomplete trigger configuration chain in input memory.

This function succeeds and output memory is written with `trig_idx`
values only if the SBI implementation is able to install a debug
trigger for each input trigger configuration.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Triggers installed successfully. The
`sbiret.value` will be set to zero.
SBI_ERR_INVALID_ADDRESS - The physical memory pointed by the
`in_addr_div_by_16` or `out_addr_div_by_16`
parameter is not accessible to supervisor-mode.
The `sbiret.value` will be set to zero.
SBI_ERR_INVALID_PARAM - One of the input trigger configuration has
an invalid value. The `sbiret.value` will
be set to the array index of failing input
trigger configuration.
SBI_ERR_NOT_SUPPORTED - One of the input trigger configuration can't
be programmed due to unimplemented optional
bits in `tdata1`, `tdata2`, or `tdata3` CSRs.
The `sbiret.value` will be set to the array
index of failing input trigger configuration.
SBI_ERR_FAILED - Failed to assign `trig_idx` or HW debug
trigger for one of the input trigger
configuration. The `sbiret.value` will be
set to the array index of failing input
trigger configuration.

Function: Uninstall a set of triggers (FID #3)
----------------------------------------------

struct sbiret sbi_debug_uninstall_triggers(unsigned long trig_idx_base,
unsigned long trig_idx_mask)

Uninstall a set of debug triggers specified by the `trig_idx_base` and
`trig_idx_mask` parameters on the calling HART.

For each debug trigger in the specified set of debug triggers, the
SBI implementation should:
1) Clear the `tdata1`, `tdata2`, and `tdata3` CSRs of the mapped
HW debug trigger.
2) Clear the `trig_state` of the debug trigger.
3) Unmap the HW debug trigger from `trig_idx` and free both of
them for future trigger installations.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Trigger uninstalled successfully.
SBI_ERR_INVALID_PARAM - One of the `trig_idx` in the specified set
of debug triggers is either:
1) Greater than the total number of triggers.
2) Represents a debug trigger that is not
mapped to any HW debug trigger.

Function: Enable a set of triggers (FID #4)
-------------------------------------------

struct sbiret sbi_debug_enable_triggers(unsigned long trig_idx_base,
unsigned long trig_idx_mask)

Enable a set of debug triggers specified by `trig_idx_base` and
`trig_idx_mask` parameters on the calling HART.

To enable a debug trigger, SBI implementation MUST restore the
`vs`, `vu`, `s`, and `u` bits of the mapped HW debug trigger
from their saved copy in `trig_state`.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Trigger enabled successfully.
SBI_ERR_INVALID_PARAM - One of the `trig_idx` in the specified set
of debug triggers is either:
1) Greater than the total number of triggers.
2) Represents a debug trigger that is not
mapped to any HW debug trigger.

Function: Disable a set of triggers (FID #5)
--------------------------------------------

struct sbiret sbi_debug_disable_triggers(unsigned long trig_idx_base,
unsigned long trig_idx_mask)

Disable a set of debug triggers specified by `trig_idx_base` and
`trig_idx_mask` parameters on the calling HART.

To disable a debug trigger, SBI implementation MUST clear the `vs`,
`vu`, `s`, and `u` bits of the mapped HW debug trigger.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Trigger disabled successfully.
SBI_ERR_INVALID_PARAM - One of the `trig_idx` in the specified set
of debug triggers is either:
1) Greater than the total number of triggers.
2) Represents a debug trigger that is not
mapped to any HW debug trigger.

We need to abstract out the breakpoint information.
e.g

struct trigger {
unsigned long addr;
unsigned long length;
unsigned long rwx_mask; /*
bit 0 -> read;
bit 1 -> write;
bit 2 -> execute */
unsigned long priv_mask; /*
bit 0 -> u;
bit 1 -> s;
bit 2 -> vu;
bit 3 -> vs */

unsigned long resource_id;
/* This records which triggers are occupied,
which is needed to undo the trigger settings */
unsigned long result;
/* This variable will be set by sbi to identify
whether the setting is successful or not */

/* Other information that may be required */
};

Different triggers can perform specific settings instead of directly
passing tdata1~tdata3

We only need two interfaces, one for install and one for uninstall.

struct sbiret sbi_debug_install_triggers(
unsigned long trig_count,
unsigned long in_addr_div_by_16);

struct sbiret sbi_debug_uninstall_triggers(
unsigned long trig_count,
unsigned long in_addr_div_by_16);

in_addr_div_by_16 points to an array of struct triggers.
See above comment.

Regards,
Anup


Regards,
Xiang W




Re: [RISC-V] [tech-debug] SBI Debug Trigger Extension Proposal (Draft v4)

merle w
 

Anup Patel <apatel@...> 于2022年7月14日周四 17:39写道:

Hi All,

To support native debugging in S-mode and VS-mode, we need
a SBI Debug Trigger extension which complements the Sdtrig
extension.

Below is the draft proposal of the SBI Debug Trigger Extension...

Many thanks to folks (CC'ed here) who provided early feedback
and helped improve this proposal.

The proposed SBI extension is highly inspired from Linux HW
breakpoint infrastructure which is built on top of the Linux
perf subsystem.

Please review and provide your feedback.

Regards,
Anup

SBI Debug Trigger Extension (EID #0x44425452 "DBTR")
====================================================

The RISC-V Sdtrig ISA extension allows machine-mode software to
directly configure debug triggers which in-turn allows native
(or hosted) debugging without any external debugger.

The SBI debug trigger extension defines a SBI based abstraction to
provide native debugging for supervisor-mode software such that it:
1) Suitable for the rich operating systems and hypervisors running
in supervisor-mode.
2) Allows Guest (VS-mode) and Hypervisor (HS-mode) to share debug
triggers on a HART.

Some of the functions defined by this SBI extension have physical
memory as parameters for input and/or output. This physical memory
parameter MUST satisfy following requirements:
1) The base address of the physical memory MUST be 16-byte aligned
so that it is machine word aligned for RV32, RV64 and RV128.
2) The SBI implementation MUST check that the supervisor-mode
software is allowed to read and write the specified physical
memory on the calling HART.
3) The SBI implementation MUST access the specified physical
memory using the PMA attribute.
Note: If the supervisor-mode software access the same physical
memory using memory type different from PMA then a loss of
coherence or unexpected memory ordering may occur and the
invoking software should follow the rules and sequences defined
in the RISC-V Svpbmt specification to prevent the loss of
coherence and memory ordering.

Each debug trigger is assigned a logical number called `trig_idx`
by the SBI implementation. If total number of debug triggers are
`N` then `0 <= trig_idx < N`.
If I understand correctly, the N here should refer to the logical
trigger, not the physical number. sbi needs to maintain these logic
data, and this is related to runtime, which is difficult for sbi to
implement (if many programs are debugged at the same time, it may
take up a lot of logic triggers). I think this logical data should
be handed over to low-privilege level maintenance.


The configuration of each debug trigger is expressed by three
entities `trig_tdata1`, `trig_tdata2`, and `trig_tdata3` which
are same as the `tdata1`, `tdata2`, and `tdata3` CSRs defined by
the RISC-V Sdtrig specification but with the following additional
constraints:
1) The `trig_tdata1.dmode` bit should always be zero.
2) The `trig_tdata1.m` bit should always be zero.
3) The `trig_tdata1.action` bits should always be zero.

The SBI implementation must also maintain an additional software
state for each debug trigger called `trig_state` which is encoded
as follows:
reserved [XLEN-1:5] - Reserved for future use.
vs [4:4] - Saved copy of the `trig_tdata1.vs` bit.
vu [3:3] - Saved copy of the `trig_tdata1.vu` bit.
s [2:2] - Saved copy of the `trig_tdata1.s` bit.
u [1:1] - Saved copy of the `trig_tdata1.u` bit.
mapped [0:0] - When set, the trigger has been mapped to
some HW debug trigger.

Function: Get number of triggers (FID #0)
-----------------------------------------

struct sbiret sbi_debug_num_triggers(unsigned long trig_tdata1)

Get the number of per-HART debug triggers which can support the
trigger configuration specified by `trig_tdata1` parameter.

The SBI implementation must return the total number of per-HART
debug triggers when `trig_tdata1 == 0`.

Returns the number of per-HART debug triggers in `sbiret.value`
and always returns SBI_SUCCESS in `sbiret.error`.

Function: Read triggers (FID #1)
--------------------------------

struct sbiret sbi_debug_read_triggers(unsigned long trig_idx_base,
unsigned long trig_count,
unsigned long out_addr_div_by_16)

Read the trigger state and configuration for a range of debug triggers
specified by `trig_idx_base` and `trig_count` parameters on the calling
HART.

The state and configuration of triggers will be written to an output
memory of size `trig_count * 4 * (XLEN / 8)` bytes. The base physical
address of the output memory right shifted by 4 is specified by the
`out_addr_div_by_16` parameter.

For i'th debug trigger in the specified range, the trigger configuration
consists of four XLEN-bit words written in little-endian format at
`offset = i * 4 * (XLEN / 8)` of the output memory. The contents of
output trigger configuration words are as follows:
word[0] = trig_state
word[1] = trig_tdata1
word[2] = trig_tdata2
word[3] = trig_tdata3

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Configuration read successfully.
SBI_ERR_INVALID_ADDRESS - The physical memory pointed by the
`out_addr_div_by_16` parameter is not
accessible to supervisor-mode.
SBI_ERR_INVALID_PARAM - Either `trig_idx_base + trig_count` or
`trig_idx_base` is greater than the total
number of triggers.

Function: Install triggers (FID #2)
-----------------------------------

struct sbiret sbi_debug_install_triggers(unsigned long trig_count,
unsigned long in_addr_div_by_16,
unsigned long out_addr_div_by_16)

Install debug triggers based on an array of trigger configurations
specified through an input memory and return the `trig_idx` of each
installed debug trigger in an output memory.

The `trig_count` parameter represents the number of debug triggers
to be installed.

The base physical address of the input memory right shifted by 4
is specified by the `in_addr_div_by_16` parameter and the size of
input memory is `trig_count * 4 * (XLEN / 8)` bytes. The i'th input
trigger configuration is at `offset = i * 4 * (XLEN / 8)` of the
input memory consisting of four consecutive XLEN-bit words in
little-endian format. The contents of each trigger configuration
are follows:
word[0] = Reserved for future use and should be zero
word[1] = trig_tdata1
word[2] = trig_tdata2
word[3] = trig_tdata3
The current debug specification has many variants: type 1, type 2,
type 6. There may be more manufacturer-defined specifications. Instead
of passing register information directly, we should need more abstraction,
which also simplifies the design of low-privilege software


The base physical address of the output memory right shifted by
4 is specified by the `out_addr_div_by_16` parameter and the size
of output memory is `trig_count * (XLEN / 8)` bytes. The `trig_idx`
assigned for the i'th input trigger configuration is written at
`offset = i * (XLEN / 8)` of the output memory in little-endian
format.

For each input trigger configuration in the input memory, the SBI
implementation should:
1) Map a free `trig_idx` with an unused HW debug trigger that
matches the trigger configuration.
2) Save a copy of the `trig_tdata1.vs`, `trig_tdata1.vu`,
`trig_tdata1.s`, and `trig_tdata.u` bits in `trig_state`.
3) Update the `tdata1`, `tdata2`, and `tdata3` CSRs of the
HW debug trigger.

Additionally for each input trigger configuration chain, the
SBI implementation MUST assign contiguous `trig_idx` values
and contiguous HW debug triggers when installing debug triggers.

The last trigger configuration in the input memory MUST not have
`trig_tdata1.chain == 1` for `trig_tdata1.type = 2 or 6` to prevent
incomplete trigger configuration chain in input memory.

This function succeeds and output memory is written with `trig_idx`
values only if the SBI implementation is able to install a debug
trigger for each input trigger configuration.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Triggers installed successfully. The
`sbiret.value` will be set to zero.
SBI_ERR_INVALID_ADDRESS - The physical memory pointed by the
`in_addr_div_by_16` or `out_addr_div_by_16`
parameter is not accessible to supervisor-mode.
The `sbiret.value` will be set to zero.
SBI_ERR_INVALID_PARAM - One of the input trigger configuration has
an invalid value. The `sbiret.value` will
be set to the array index of failing input
trigger configuration.
SBI_ERR_NOT_SUPPORTED - One of the input trigger configuration can't
be programmed due to unimplemented optional
bits in `tdata1`, `tdata2`, or `tdata3` CSRs.
The `sbiret.value` will be set to the array
index of failing input trigger configuration.
SBI_ERR_FAILED - Failed to assign `trig_idx` or HW debug
trigger for one of the input trigger
configuration. The `sbiret.value` will be
set to the array index of failing input
trigger configuration.

Function: Uninstall a set of triggers (FID #3)
----------------------------------------------

struct sbiret sbi_debug_uninstall_triggers(unsigned long trig_idx_base,
unsigned long trig_idx_mask)

Uninstall a set of debug triggers specified by the `trig_idx_base` and
`trig_idx_mask` parameters on the calling HART.

For each debug trigger in the specified set of debug triggers, the
SBI implementation should:
1) Clear the `tdata1`, `tdata2`, and `tdata3` CSRs of the mapped
HW debug trigger.
2) Clear the `trig_state` of the debug trigger.
3) Unmap the HW debug trigger from `trig_idx` and free both of
them for future trigger installations.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Trigger uninstalled successfully.
SBI_ERR_INVALID_PARAM - One of the `trig_idx` in the specified set
of debug triggers is either:
1) Greater than the total number of triggers.
2) Represents a debug trigger that is not
mapped to any HW debug trigger.

Function: Enable a set of triggers (FID #4)
-------------------------------------------

struct sbiret sbi_debug_enable_triggers(unsigned long trig_idx_base,
unsigned long trig_idx_mask)

Enable a set of debug triggers specified by `trig_idx_base` and
`trig_idx_mask` parameters on the calling HART.

To enable a debug trigger, SBI implementation MUST restore the
`vs`, `vu`, `s`, and `u` bits of the mapped HW debug trigger
from their saved copy in `trig_state`.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Trigger enabled successfully.
SBI_ERR_INVALID_PARAM - One of the `trig_idx` in the specified set
of debug triggers is either:
1) Greater than the total number of triggers.
2) Represents a debug trigger that is not
mapped to any HW debug trigger.

Function: Disable a set of triggers (FID #5)
--------------------------------------------

struct sbiret sbi_debug_disable_triggers(unsigned long trig_idx_base,
unsigned long trig_idx_mask)

Disable a set of debug triggers specified by `trig_idx_base` and
`trig_idx_mask` parameters on the calling HART.

To disable a debug trigger, SBI implementation MUST clear the `vs`,
`vu`, `s`, and `u` bits of the mapped HW debug trigger.

The possible error codes returned in `sbiret.error` are below.

Errors:
SBI_SUCCESS - Trigger disabled successfully.
SBI_ERR_INVALID_PARAM - One of the `trig_idx` in the specified set
of debug triggers is either:
1) Greater than the total number of triggers.
2) Represents a debug trigger that is not
mapped to any HW debug trigger.

We need to abstract out the breakpoint information.
e.g

struct trigger {
unsigned long addr;
unsigned long length;
unsigned long rwx_mask; /*
bit 0 -> read;
bit 1 -> write;
bit 2 -> execute */
unsigned long priv_mask; /*
bit 0 -> u;
bit 1 -> s;
bit 2 -> vu;
bit 3 -> vs */

unsigned long resource_id;
/* This records which triggers are occupied,
which is needed to undo the trigger settings */
unsigned long result;
/* This variable will be set by sbi to identify
whether the setting is successful or not */

/* Other information that may be required */
};

Different triggers can perform specific settings instead of directly
passing tdata1~tdata3

We only need two interfaces, one for install and one for uninstall.

struct sbiret sbi_debug_install_triggers(
unsigned long trig_count,
unsigned long in_addr_div_by_16);

struct sbiret sbi_debug_uninstall_triggers(
unsigned long trig_count,
unsigned long in_addr_div_by_16);

in_addr_div_by_16 points to an array of struct triggers.

Regards,
Xiang W




Re: [RISC-V] [tech-debug] gdb needs an sbi extension

Tim Newsome
 

You're right. There is a proposal at https://lists.riscv.org/g/sig-hypervisors/message/151 that will likely be implemented by Syntacore before the end of the year.

Tim

On Tue, Sep 27, 2022 at 10:54 AM merle w <merlew4n6@...> wrote:
To implement hardware breakpoints and watchpoints on the riscv, the trigger in M
mode is required. We need an sbi extension for debugging to set the trigger.

Please give suggestions and feedback.

Regards,
Xiang W




Debug Extension (EID #0x44424755 "DBG")
=======================================

Function: Set Tiggers(FID #0)
-----------------------------

```c
struct trigger {
        unsigned long address;
        unsigned long length;
        unsigned long rwx_mask;
        unsigned long privilege_mask;
        unsigned long resource_id;
};
```

rwx_mask:
        bit 0: read
        bit 1: write
        bit 2: execute

privilege_mask:
        bit 0: U-Mode
        bit 1: S-Mode
        bit 2: H-Mode
        bit 3: M-Mode

resource_id: This is a resource identifier, non-0, 0 is used to indicate that
the resource is free. This flag is used to associate which trigger is used, and
is useful when clearing settings.


```c
struct sbiret sbi_debug_set_triggers(unsigned long addr_lo, unsigned long addr_hi, unsigned long num);
```

addr_lo and addr_hi are used to access address ranges whose physical address
width is greater than XLEN. The addr_lo and addr_hi point to the struct trigger
array.

num represents the number of elements in the array.

Trigger is a scarce resource and should be used as the context of the program,
set when resuming execution, and cleared when swapped out. In order to reduce
the number of sbi calls, the breakpoint information is passed through an array
here.


Function: Clear Tiggers(FID #1)
--------------------------------
```c
struct sbiret sbi_debug_clear_triggers(unsigned long addr_lo, unsigned long addr_hi, unsigned long num);
```

addr_lo and addr_hi are used to access address ranges whose physical address
width is greater than XLEN. The addr_lo and addr_hi point to the struct trigger
array.

num represents the number of elements in the array.

This call will undo all trigger settings.







gdb needs an sbi extension

merle w
 

To implement hardware breakpoints and watchpoints on the riscv, the trigger in M
mode is required. We need an sbi extension for debugging to set the trigger.

Please give suggestions and feedback.

Regards,
Xiang W




Debug Extension (EID #0x44424755 "DBG")
=======================================

Function: Set Tiggers(FID #0)
-----------------------------

```c
struct trigger {
unsigned long address;
unsigned long length;
unsigned long rwx_mask;
unsigned long privilege_mask;
unsigned long resource_id;
};
```

rwx_mask:
bit 0: read
bit 1: write
bit 2: execute

privilege_mask:
bit 0: U-Mode
bit 1: S-Mode
bit 2: H-Mode
bit 3: M-Mode

resource_id: This is a resource identifier, non-0, 0 is used to indicate that
the resource is free. This flag is used to associate which trigger is used, and
is useful when clearing settings.


```c
struct sbiret sbi_debug_set_triggers(unsigned long addr_lo, unsigned long addr_hi, unsigned long num);
```

addr_lo and addr_hi are used to access address ranges whose physical address
width is greater than XLEN. The addr_lo and addr_hi point to the struct trigger
array.

num represents the number of elements in the array.

Trigger is a scarce resource and should be used as the context of the program,
set when resuming execution, and cleared when swapped out. In order to reduce
the number of sbi calls, the breakpoint information is passed through an array
here.


Function: Clear Tiggers(FID #1)
--------------------------------
```c
struct sbiret sbi_debug_clear_triggers(unsigned long addr_lo, unsigned long addr_hi, unsigned long num);
```

addr_lo and addr_hi are used to access address ranges whose physical address
width is greater than XLEN. The addr_lo and addr_hi point to the struct trigger
array.

num represents the number of elements in the array.

This call will undo all trigger settings.


Re: Meeting Sept 27 9am PT

Aaron Durbin
 

I declared failure on my part. I'll schedule something early next week (unless people think we can squeeze something in tomorrow or Thurs - appears to be a lot conflicts, though).  My apologies for not sending out a reminder. I'll get something scheduled and sent out.


On Tue, Sep 27, 2022 at 10:05 AM Aaron Durbin <adurbin@...> wrote:
Hi All,

Ved let me know I accidentally had a Google Meet Invite link. The Zoom meeting id is:


Thanks.

-Aaron


Meeting Sept 27 9am PT

Aaron Durbin
 

Hi All,

Ved let me know I accidentally had a Google Meet Invite link. The Zoom meeting id is:


Thanks.

-Aaron


OS-A SEE TG Meeting 2022/09/12

Aaron Durbin
 

Hi All,

We had our first meeting this past Monday (2022/09/12).

I pushed the notes to GitHub. As a reminder each group in RVI has their own folder for stashing collateral. For OS-A SEE TG, the folder is here. Under that there is a Meetings folder that has a doc that has a link to the GitHub notes in addition to the meeting presentations. The other, and most important doc, is the Future Topics doc where people can put their concerns/comments/requests to cover in the next meeting(s).

We'll have our next meeting on Monday 2022/09/26 at 9am PT. It's already been put on the tech.meetings@... calendar. We hope to have an initial framework in the spec repository as well as some initial content that we can discuss.

2022/09/12

  • Meeting Presentation
  • Went over the Charter
    • Discussed what is in scope for OS-A SEE
    • Paul W. asked about hardware requirements. In alignment with the Charter, we should be focusing on compatibility across RISC-V systems for loading and running operating systems. Hardware requirements should fall under the Platform specifications.
    • Additionally Kumar indicated SW view of the world should be the focus.
  • Paul brought up a distinction of feature / device binding required in both early and later stages of kernel startup.
  • Target Audience of OS-A SEE? Ideally would be standar operating system distributions to further expand the ecosystem. Users of the RISC-V systems would benefit, but it's important to allow operating system distributions to target a conformant spec.
  • psABI would not fall into the purview of the OS-A SEE spec. Reason: Kumar syas focus should be on ability to run same OS on multiple implementations.
  • ACPI vs DT. Acknowledge we need to support both. May want to allow an OS to choose which one to support (or both) instead of mandating both. Allows for OS vendors to define their own product. Topic to still be discussed.
  • Need to start framework of document and start seeding content.


Thank you.

-Aaron


Re: Access problem of mtimercmp in a platform with multiple MTIMER devices

Anup Patel
 

On Wed, Sep 7, 2022 at 8:44 AM Tianyi Xia via lists.riscv.org
<tianshi.xty=alibaba-inc.com@...> wrote:

each "cluster" can have its own unique mmio address for mtimecmp (which may or may not be accessible to other "clusters")

I think this description is better.



Assume there are two clusters,each cluster have two cores,and each cluster have there own MTIMER device. The mmio address of mtimecmp for each hart may like this:

Cluster0

Core0: base0+0x0000_0000

Core1: base0+0x0000_0008

Cluster1

Core0: base1+0x0000_0000

Core1: base1+0x0000_0008

Base0 is the MTIER device base address of cluster0, Base1 is the MTIER device base address of cluster0. the mtimecmp of cluster0 core0 may or maynot be accessible to cluster1, depending on the implementation. If core try to access mtimecmp of other cluster, the action of the access may be write ignore read zero.


The latter sounds like it would be difficult for SW

I think in a platform with multiple MTIMER devices, the mmio address of mtimecmp should be unique to distinguish different MTIMER devices. The hardware can set regular base address to different Mtimer devices. In the above example, assuming base is 0, then base1 may be set to 0x0000_0010. If cluster0 has four cores, then base1 may be set to 0x0000_0020.Then from a software perspective, all mtimecmp registers are addressed consecutively.
This case is already handled by ACLINT device tree bindings. We
just need two separate MTIMER DT nodes where the mtimecmp
base address will be different in each DT node.

Please refer to the latest OpenSBI sources.

Regards,
Anup








Re: Access problem of mtimercmp in a platform with multiple MTIMER devices

Allen Baum
 

That makes sense, but it does mean that discovery gets more complicated, and (maybe) you need to build separate device trees for each.
But maybe that has to happen anyway? I don't know if DT can be parameterized based on HartID, but that would greatly simplify the work.

On Tue, Sep 6, 2022 at 8:14 PM Tianyi Xia via lists.riscv.org <tianshi.xty=alibaba-inc.com@...> wrote:

each "cluster" can have its own unique mmio address for mtimecmp (which may or may not be accessible to other "clusters")

I think this description is better.

 

Assume there are two clusterseach cluster have two coresand each cluster have there own MTIMER device. The mmio address of mtimecmp for each hart may like this:

Cluster0

    Core0: base0+0x0000_0000

    Core1: base0+0x0000_0008

Cluster1

    Core0: base1+0x0000_0000

    Core1: base1+0x0000_0008

Base0 is the MTIER device base address of cluster0, Base1 is the MTIER device base address of cluster0. the mtimecmp of cluster0 core0 may or maynot be accessible to cluster1,  depending on the implementation. If core try to access mtimecmp of other cluster, the action of the access may be write ignore read zero.


The latter sounds like it would be difficult for SW

I think in a platform with multiple MTIMER devices, the mmio address of mtimecmp should be unique to distinguish different MTIMER devices. The hardware can set regular base address to different Mtimer devices. In the above example, assuming base is 0, then base1 may be set to 0x0000_0010. If cluster0 has four cores, then base1 may be set to 0x0000_0020.Then from a software perspective, all mtimecmp registers are addressed consecutively.

 


 

1 - 20 of 1847