On Mon, Jul 6, 2020 at 12:35 AM Anup Patel <anup.patel@...> wrote:

Hi All,

We don't have a dedicated RISC-V PMU extension but we do have HARDWARE
performance counters such as CYCLE CSR, INSTRET CSR, and HPMCOUNTER
CSRs. A RISC-V implementation can support monitoring various HARDWARE
events using limited number of HPMCOUNTER CSRs.

In addition to HARDWARE performance counters, a SBI implementation
(e.g. OpenSBI, Xvisor, KVM, etc) can provide SOFTWARE counters for
events such as number of RFENCEs, number of IPIs, number of misaligned
load/store instructions, number of illegal instructions, etc.

We propose SBI PMU extension which tries to cover CYCLE CSR, INSTRET CSR,
HPMCOUNTER CSRs and SOFTWARE counters provided by SBI implementation.

To define SBI PMU extension, we first define counter_idx which is a
logical number assigned to a counter and event_idx which is an encoded

Is there more detail about counter_idx? I was wondering that
1. What is the ordering of logical numbers for HW and SW counters? I
think that the logical numbers are assigned by OpenSBI.
2. How to know the logical number of counter_idx of each HW and SW
counters from s-mode? I guess that we need to know the logical numbers
of all counters before we invoke a SBI call.

number representing the HARDWARE/SOFTWARE event to be monitored.

The SBI PMU event_idx is a XLEN bits wide number encoded as follows:
event_idx[XLEN-1:16] = info
event_idx[15:12] = type
event_idx[11:0] = code

If event_idx.type == 0x0 then it is HARDWARE event. For HARDWARE event,
the event_idx.info is optional and can be passed zero whereas the
event_idx.code can be one of the following values:
enum sbi_pmu_hw_id {
SBI_PMU_HW_CPU_CYCLES = 0,
SBI_PMU_HW_INSTRUCTIONS = 1,
SBI_PMU_HW_CACHE_REFERENCES = 2,
SBI_PMU_HW_CACHE_MISSES = 3,
SBI_PMU_HW_BRANCH_INSTRUCTIONS = 4,
SBI_PMU_HW_BRANCH_MISSES = 5,
SBI_PMU_HW_BUS_CYCLES = 6,
SBI_PMU_HW_STALLED_CYCLES_FRONTEND = 7,
SBI_PMU_HW_STALLED_CYCLES_BACKEND = 8,
SBI_PMU_HW_REF_CPU_CYCLES = 9,
SBI_PMU_HW_MAX, /* non-ABI */
};
(NOTE: Same as <linux_source>/include/uapi/linux/perf_event.h)

If event_idx.type == 0x1 then it is HARDWARE CACHE event. For HARDWARE
CACHE event, the event_idx.info is optional and can be passed zero
whereas the event_idx.code is encoded as follows:
event_idx.code[11:3] = cache_id
event_idx.code[2:1] = op_id
event_idx.code[0:0] = result_id
enum sbi_pmu_hw_cache_id {
SBI_PMU_HW_CACHE_L1D = 0,
SBI_PMU_HW_CACHE_L1I = 1,
SBI_PMU_HW_CACHE_LL = 2,
SBI_PMU_HW_CACHE_DTLB = 3,
SBI_PMU_HW_CACHE_ITLB = 4,
SBI_PMU_HW_CACHE_BPU = 5,
SBI_PMU_HW_CACHE_NODE = 6,
SBI_PMU_HW_CACHE_MAX, /* non-ABI */
};
enum sbi_pmu_hw_cache_op_id {
SBI_PMU_HW_CACHE_OP_READ = 0,
SBI_PMU_HW_CACHE_OP_WRITE = 1,
SBI_PMU_HW_CACHE_OP_PREFETCH = 2,
SBI_PMU_HW_CACHE_OP_MAX, /* non-ABI */
};
enum sbi_pmu_hw_cache_op_result_id {
SBI_PMU_HW_CACHE_RESULT_ACCESS = 0,
SBI_PMU_HW_CACHE_RESULT_MISS = 1,
SBI_PMU_HW_CACHE_RESULT_MAX, /* non-ABI */
};
(NOTE: Same as <linux_source>/include/uapi/linux/perf_event.h)

If event_idx.type == 0x2 then it is HARDWARE RAW event. For HARDWARE RAW
event, both event_idx.info and event_idx.code are platform dependent.

If event_idx.type == 0xf then it is SOFTWARE event. For SOFTWARE event,
event_idx.info is SBI implementation specific and event_idx.code can be
one of the following:
enum sbi_pmu_sw_id {
SBI_PMU_SW_MISALIGNED_LOAD = 0,
SBI_PMU_SW_MISALIGNED_STORE = 1,
SBI_PMU_SW_ILLEGAL_INSN = 2,
SBI_PMU_SW_LOCAL_SET_TIMER = 3,
SBI_PMU_SW_LOCAL_IPI = 4,
SBI_PMU_SW_LOCAL_FENCE_I = 5,
SBI_PMU_SW_LOCAL_SFENCE_VMA = 6,
SBI_PMU_SW_LOCAL_SFENCE_VMA_ASID = 7,
SBI_PMU_SW_LOCAL_HFENCE_GVMA = 8,
SBI_PMU_SW_LOCAL_HFENCE_GVMA_VMID = 9,
SBI_PMU_SW_LOCAL_HFENCE_VVMA = 10,
SBI_PMU_SW_LOCAL_HFENCE_VVMA_ASID = 11,
SBI_PMU_SW_MAX, /* non-ABI */
};

In future, more events can be defined without breaking ABI compatibility
of SBI calls.

Using definition of counter_idx and event_idx, we can potentially have
the following SBI calls:

1. SBI_PMU_NUM_COUNTERS
This call will return the number of COUNTERs

Is it for the SW counters and we get the number of HW counters by DT?
Or does it return the number of HW and SW counters both? If so, how to
distinguish the number of HW and SW?
The other question is that the number of SW counters is defined by the
core of OpenSBI or platform-dependent?

2. SBI_PMU_COUNTER_DESCRIBE
This call takes two parameters: 1) counter_idx 2) physical address
It will write the description of SBI PMU counter at specified physical
address. The details of the SBI PMU counter written at specified
physical address are as follows:
1. Name (64 bytes)
2. CSR_Number (2 bytes)
(CSR_Number <= 0xfff means counter is a RISC-V CSR)
(CSR_Number > 0xfff means counter is a SBI implementation counter)
(E.g. CSR_Number == 0xC02 imply HPMCOUNTER2 CSR)
3. CSR_Width (2 bytes)
(Number of CSR bits implemented in HW)
4. Event_Count (2 bytes)
(Number of events in Event_List array)
5. Event_List (2 * Event_Count bytes)
(This is an array of 16bit values where each 16bit value is the
supported event_idx.type and event_idx.code combination)

What is the size we should allocate for this physical address? In my
understanding, we need to allocate
the pages in s-mode first, then pass the address of the pages to the
second parameter, but we don't know the
event_counter before we allocate the space for it, so it might across
the boundary if event_count is very big.

3. SBI_PMU_COUNTER_SET_PHYS_ADDR
This call takes two parameters: 1) counter_idx 2) physical address
It will set the physical address of memory location where the SBI
implementation will write the 64bit SOFTWARE counter. This SBI call
is only for counters not mapped to any CSR (i.e. only for counters
with CSR_Number > 0xfff).
4. SBI_PMU_COUNTER_START
This call takes two parameters: 1) counter_idx 2) event_idx
It will inform SBI implementation to configure and start/enable
specified counter on the calling HART to monitor specific event.
This SBI call will fail for counters which are not present and
specified event_idx is not supported by the counter.
5. SBI_PMU_COUNTER_STOP
This call takes one parameter: 1) counter_idx
It will inform SBI implementation to stop/disable specified counters
on the calling HART. This SBI call will fail for counters which are
not present.

From above, the RISC-V PMU driver will use most of the SBI calls at boot
time. Only SBI_PMU_COUNTER_START to be used once before using the counter.
The reading of counter is by reading CSR (for CSR_Number < 0xfff) OR by
reading memory location (for CSR_Offset >= 0xfff). The counter overflow
handling will have to be done in software by Linux kernel.

Using the SBI PMU extension, the M-mode runtime firmware (or Hypervisors)
can provide a standardized view of HARDWARE/SOFTWARE counters and events
to S-mode (or VS-mode) software.

The M-mode runtime firmware (OpenSBI) will need to know following
platform dependent information:
1. Possible event_idx values allowed (or supported) by a HARDWARE
counter (i.e. HPMCOUNTER)
2. Mapping of event_idx for HARDWARE event to HPMEVENT CSR value
3. Mapping of event_idx for HARDWARE CACHE event to HPMEVENT CSR value
4. Mapping of event_idx for HARDWARE RAW event to HPMEVENT CSR value
5. Additional platform-specific progamming required by any event_idx

All platform dependent information mentioned above, can be obtained
by M-mode runtime firmware (OpenSBI) from platform specific code. The
DT/ACPI can also be used to described 1), 2), 3), and 4) mentioned
above but 5) will always require platform specific code.

I would update the next version of DT file to describe the points from
1) to 4). Thanks.
As you mentioned before, it would be hard to sync the platform
specific code with the DT of real use.
I prefer to get 1), 2), 3) and 4) from DT first on each platform, and
use platform specific code if DT is unavailable. (generic platform use
DT certainly),
then we could maximally reduce the inconsistency.

Regards,
Anup