Hi All,

We don't have a dedicated RISC-V PMU extension but we do have HW performance
counters such as CYCLE CSR, INSTRET CSR, and HPMCOUNTER CSRs. A RISC-V
CPU can allow monitoring HW events using few HPMCOUNTER CSRs. The M-mode
software can also inhibit unused performance counters to save energy.

In addition to HW performance counters, a SBI implementation (e.g. OpenSBI,
Xvisor, KVM, etc) can provide software counters for interesting events
such as number of RFENCEs, number of IPIs, number of misaligned load/store
instructions, number of illegal instructions, etc.

We propose SBI PMU extension which tries to cover CYCLE CSR, INSTRET CSR,
HPMCOUNTER CSRs and software counters of the SBI implementation.

To define SBI PMU extension, we first define counter_idx which is a unique
number assigned to a counter and event_idx which is an encoded number
representing event to be monitored.

The SBI PMU event_idx is 15bit number encoded as follows:
event_idx[14:12] = type
event_idx[11:0] = code

If event_idx.type == 0 then it is HARDWARE event and event_idx.code can
be one of the following:
enum sbi_pmu_hw_id {
/*
* Common hardware events, generalized by the kernel:
*/
PERF_COUNT_HW_CPU_CYCLES = 0,
PERF_COUNT_HW_INSTRUCTIONS = 1,
PERF_COUNT_HW_CACHE_REFERENCES = 2,
PERF_COUNT_HW_CACHE_MISSES = 3,
PERF_COUNT_HW_BRANCH_INSTRUCTIONS = 4,
PERF_COUNT_HW_BRANCH_MISSES = 5,
PERF_COUNT_HW_BUS_CYCLES = 6,
PERF_COUNT_HW_STALLED_CYCLES_FRONTEND = 7,
PERF_COUNT_HW_STALLED_CYCLES_BACKEND = 8,
PERF_COUNT_HW_REF_CPU_CYCLES = 9,

PERF_COUNT_HW_MAX, /* non-ABI */
};
(NOTE: Same as described in <linux_source>/include/uapi/linux/perf_event.h)

If event_idx.type == 1 then it is CACHE event and event_idx.code is encoded
as follows:
event_idx.code[11:4] = cache_id
event_idx.code[3:1] = op_id
event_idx.code[0:0] = result_id
enum sbi_pmu_hw_cache_id {
PERF_COUNT_HW_CACHE_L1D = 0,
PERF_COUNT_HW_CACHE_L1I = 1,
PERF_COUNT_HW_CACHE_LL = 2,
PERF_COUNT_HW_CACHE_DTLB = 3,
PERF_COUNT_HW_CACHE_ITLB = 4,
PERF_COUNT_HW_CACHE_BPU = 5,
PERF_COUNT_HW_CACHE_NODE = 6,

PERF_COUNT_HW_CACHE_MAX, /* non-ABI */
};
enum sbi_pmu_hw_cache_op_id {
PERF_COUNT_HW_CACHE_OP_READ = 0,
PERF_COUNT_HW_CACHE_OP_WRITE = 1,
PERF_COUNT_HW_CACHE_OP_PREFETCH = 2,

PERF_COUNT_HW_CACHE_OP_MAX, /* non-ABI */
};
enum sbi_pmu_hw_cache_op_result_id {
PERF_COUNT_HW_CACHE_RESULT_ACCESS = 0,
PERF_COUNT_HW_CACHE_RESULT_MISS = 1,

PERF_COUNT_HW_CACHE_RESULT_MAX, /* non-ABI */
};
(NOTE: Same as described in <linux_source>/include/uapi/linux/perf_event.h)

If event_idx.type == 2 then it is RAW event and event_idx.code is just
a RAW event number.

In future, more event_idx can be defined without breaking ABI compatibility
of SBI calls.

Based on above definition of counter_idx definition, we can potentially have
the following SBI calls:

1. SBI_PMU_NUM_COUNTERS
This call will return the number of COUNTERs
2. SBI_PMU_COUNTER_DESCRIBE
This call takes two parameters: 1) counter_idx 2) physical address of 4k page
It will write the description of SBI PMU counter at specified physical
address. The details of the SBI PMU counter written at specified physical
address are as follows:
1. Name (64 bytes)
2. CSR_Offset (4 bytes)
(E.g. CSR_Offset == 0x2 imply CSR 0xC02)
(E.g. CSR_Offset == 0xffffffff means it is SBI implementation counter)
3. CSR_Width (4 bytes)
(Number of CSR bits implemented in HW)
4. Event bitmap (2048 bytes) (i.e. 1-bit for each possible event_idx)
(If bit corresponding to a event_idx is 1 then event_idx is supported
by the counter)
5. Anything else ??
3. SBI_PMU_COUNTER_SET_PHYS_ADDR
This call takes two parameters: 1) counter_idx 2) physical address
It will set the physical address where SBI implementation will write
the software counter. This SBI call is only for counters not mapped
to any CSR (i.e. only for counters with CSR_Offset == 0xffffffff).
4. SBI_PMU_COUNTER_START
This call takes two parameters: 1) counter_idx 2) event_idx
It will inform SBI implementation to configure and start/enable specified
counter on the calling HART to monitor specific event. This SBI call will
fail for counters which are not present.
5. SBI_PMU_COUNTER_STOP
This call takes one parameter: 1) counter_idx
It will inform SBI implementation to stop/disable specified counters on the
calling HART. This SBI call will fail for counters which are not present.

From above, the RISC-V PMU driver will use most of the SBI calls at boot time.
Only SBI_PMU_COUNTER_START to be used once before using the counter. The reading
the counter is by reading CSR (for CSR_Offset != 0xffffffff) OR by reading
memory location (for CSR_Offset == 0xffffffff). The counter overflow handling
will have to be done in software by Linux kernel.

The information returned by SBI_PMU_NUM_COUNTERS and SBI_PMU_COUNTER_DESCRIBE
can be passed via DT/ACPI but it will be difficult to maintain because we
have hardware counters and SBI implementation counters both provided by SBI
PMU extension. The SBI implementation counters are specific to underlying
SBI implementation so we will have to keep counters/events described in
DT/ACPI in-sync with underlying SBI implementation.

Regards,
Anup