|
Re: A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
Anup Patel
Yes, there is no way for kernel to know whether it is running in HS-mode or VS-mode. We would like to keep it that way.
Regards,
Anup
toggle quoted message
Show quoted text
From: tech-privileged@... <tech-privileged@...>
On Behalf Of Allen Baum
Sent: 05 August 2020 11:19
To: alankao <alankao@...>
Cc: tech-privileged@...
Subject: Re: [RISC-V] [tech-privileged] A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
I may be dense (I won't take a poll on that though), but if a kernel can detect that it is in S-mode vs. VSMode, that sounds like a buggy virtualization scheme. The kernel should (appear to) be in whatever
mode it thinks it is, even if it is really in VU mode (with appropriate hypervisor work, and performance penalties, etc).
On Tue, Aug 4, 2020 at 9:52 PM alankao <alankao@...> wrote:
Hi Anup,
> Linux PMU driver framework only updates counter value in “add()” or “start()” callback. That’s why allow S-mode write HPMCOUNTER CSRs won’t provide much benefit.
It doesn't matter where the kernel does the update. What matters is how often kernel does add() and start(). Anyway, it may take a while but we will do the experiment mentioned in previous thread to give real evidence here.
> We should avoid a kernel feature which needs to be explicitly enabled by users and distros keeping it disabled by default. The “#ifdef” based feature checking should be replaced by runtime feature checking based on device tree OR something else.
Is there any way for a kernel to detect if it is in S-mode or VS-mode? If so, then just don't go that route when writing HPM CSRs. Just like what we did with CONFIG_FPU, I suggest we can set CONFIG_RESTRICT_HPM_REG_ACCESS or something true by default, and
detect if there is some attribute like "bypass-sbi" in a PMU node. With
static key feature and the followups, this runtime check is not expensive.
|
|
|
Re: A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
Anup Patel
Hi Alan,
I think I am repeating myself here but still don’t see any benefit of allowing HPMCOUNTER CSR write access to S-mode. On the contrary, it will make context switching expensive for hypervisors.
The SBI PMU extension is designed such that kernel need not be aware of HS-mode or VS-mode. For Guest kernel (VS-mode), SBI is provided by Hypervisor so Hypervisor will act as mediator for Guest kernel. For Host kernel (HS-mode), SBI provider
is M-mode runtime firmware (OpenSBI). The Linux PMU driver should not
care which mode it is running (HS-mode or VS-mode). The same rationale applies here, we use single Linux RISC-V kernel image for both Guest and Host.
Regards,
Anup
toggle quoted message
Show quoted text
From: tech-privileged@... <tech-privileged@...>
On Behalf Of alankao
Sent: 05 August 2020 10:23
To: tech-privileged@...
Subject: Re: [RISC-V] [tech-privileged] A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
Hi Anup,
> Linux PMU driver framework only updates counter value in “add()” or “start()” callback. That’s why allow S-mode write HPMCOUNTER CSRs won’t provide much benefit.
It doesn't matter where the kernel does the update. What matters is how often kernel does add() and start(). Anyway, it may take a while but we will do the experiment mentioned in previous thread to give real evidence here.
> We should avoid a kernel feature which needs to be explicitly enabled by users and distros keeping it disabled by default. The “#ifdef” based feature checking should be replaced by runtime feature checking based on device tree OR something else.
Is there any way for a kernel to detect if it is in S-mode or VS-mode? If so, then just don't go that route when writing HPM CSRs. Just like what we did with CONFIG_FPU, I suggest we can set CONFIG_RESTRICT_HPM_REG_ACCESS or something true by default, and
detect if there is some attribute like "bypass-sbi" in a PMU node. With
static key feature and the followups, this runtime check is not expensive.
|
|
|
Re: A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
Allen Baum
I may be dense (I won't take a poll on that though), but if a kernel can detect that it is in S-mode vs. VSMode, that sounds like a buggy virtualization scheme. The kernel should (appear to) be in whatever mode it thinks it is, even if it is really in VU mode (with appropriate hypervisor work, and performance penalties, etc).
On Tue, Aug 4, 2020 at 9:52 PM alankao < alankao@...> wrote: Hi Anup,
> Linux PMU driver framework only updates counter value in “add()” or “start()” callback. That’s why allow S-mode write HPMCOUNTER CSRs won’t provide much benefit.
It doesn't matter where the kernel does the update. What matters is how often kernel does add() and start(). Anyway, it may take a while but we will do the experiment mentioned in previous thread to give real evidence here.
> We should avoid a kernel feature which needs to be explicitly enabled by users and distros keeping it disabled by default. The “#ifdef” based feature checking should be replaced by runtime feature checking based on device tree OR something else.
Is there any way for a kernel to detect if it is in S-mode or VS-mode? If so, then just don't go that route when writing HPM CSRs. Just like what we did with CONFIG_FPU, I suggest we can set CONFIG_RESTRICT_HPM_REG_ACCESS or something true by default, and detect if there is some attribute like "bypass-sbi" in a PMU node. With static key feature and the followups, this runtime check is not expensive.
|
|
|
Re: A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
alankao
Hi Anup, > Linux PMU driver framework only updates counter value in “add()” or “start()” callback. That’s why allow S-mode write HPMCOUNTER CSRs won’t provide much benefit. It doesn't matter where the kernel does the update. What matters is how often kernel does add() and start(). Anyway, it may take a while but we will do the experiment mentioned in previous thread to give real evidence here. > We should avoid a kernel feature which needs to be explicitly enabled by users and distros keeping it disabled by default. The “#ifdef” based feature checking should be replaced by runtime feature checking based on device tree OR something else. Is there any way for a kernel to detect if it is in S-mode or VS-mode? If so, then just don't go that route when writing HPM CSRs. Just like what we did with CONFIG_FPU, I suggest we can set CONFIG_RESTRICT_HPM_REG_ACCESS or something true by default, and detect if there is some attribute like "bypass-sbi" in a PMU node. With static key feature and the followups, this runtime check is not expensive.
|
|
|
Re: A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
Anup Patel
HI Alan,
I never said HPM overflow interrupt is not important. The MHPMOVERFLOW CSR proposed by Greg is perfectly fine.
I think you missed my point regarding H-extension. If S-mode is allowed to directly write to HPMCOUNTER CSRs then for H-Extension we will need additional VSHPMCOUNTER CSRs to allow Hypervisor to context-switch. We can avoid lot of these
CSRs by keeping HPMCOUNTER CSRs read-only for S-mode. The initialization/restoration of HPMCOUNTER value can be done SBI_PMU_COUNTER_START call and this integrates well with Linux PMU driver framework too. The Linux PMU driver framework only updates counter
value in “add()” or “start()” callback. That’s why allow S-mode write HPMCOUNTER CSRs won’t provide much benefit.
Regarding single Linux RISC-V image for all platforms, this is a requirement from various distros and Linux RISC-V maintainers. We should avoid a kernel feature which needs to be explicitly enabled by users and distros keeping it disabled
by default. The “#ifdef” based feature checking should be replaced by runtime feature checking based on device tree OR something else.
Regards,
Anup
toggle quoted message
Show quoted text
From: tech-privileged@... <tech-privileged@...>
On Behalf Of alankao
Sent: 05 August 2020 07:47
To: tech-privileged@...
Subject: Re: [RISC-V] [tech-privileged] A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
Hi Anup,
> The most desired feature from a PMU is counting events in right-context (or right-mode). This is not clearly defined in RISC-V spec right now. Greg’s proposal already address this in a clean way by defining required bits in MHPMEVENT CSRs. Other important
feature expected from a PMU is reading counters without traps, this is already available because HPMCOUNTER CSRs are “User-Read-Only”
Claiming some features as "most desired" is too subjective. I agree that mode-specific counting is important, but for performance monitoring, the HPM interrupt is also essential. Otherwise, sampling like `perf record` just doesn't work.
> Regarding HPMCOUNTER writes from S-mode, the Linux PMU drivers (across architectures) only writes PMU counter at time of configuring the counter. We anyway have SBI call to configure a RISC-V PMU counter because MHPMEVENT CSR is for M-mode only so it is better
to initialize/write the MHPMCOUNTER CSR in M-mode at time of configuring the counter. Allowing S-mode to write HPMCOUNTER CSR is good but won’t benefit much. On the contrary, RISC-V hypervisors might end up save/restore more CSRs if HPMCOUTNER CSR is writeable
from S-mode.
You are understating the case when you say "only writes counter at configuration time." It sounds like the kernel seldom writes them. The fact is, the counters and other registers need writing every time the corresponding process is being context-switch in
and every time an HPM interrupt is being handled. Would you like to elaborate more, based on what do you say writing counters from S-mode is good? based on what do you judge it won't benefit much?
However, since RISC-V doesn't have any discussed features yet, I doubt that anyone has any quantitative data. My proposal here is that I will take our existing solution (AndeStar V5 extension and perf-event port on Linux 4.17) as the testbed. By default,
we have *mcounterwen*, which is effectively equal to the bit[28] in Greg's proposal, to enable S-mode writing HPM CSRs. This is the treatment group. Then we do a patch to transform all existing csr_write's to HPM CSRs (including counters) into SBI calls as
the control group. My anticipation of the result is that the wall clock time performing a sampling in the treatment group will be not just marginally shorter than the control group.
Meanwhile, I agree with your concern about H extension. That's why I emphasized this feature3 is useful for M-S-U configuration and questionable for M-H-S-U one.
> The code snippet mentioned below requires “#ifdef” which means we have to build Linux RISC-V image differently for doing CSR writes this way. This approach is not going to fly for distros because distros can’t release single Linux RISC-V image for all RISC-V
hardware if we have such “#ifdef”.
Each distro maintains its own priority of hundreds of thousands of kernel features, not to mention many nameless "distributions" released by different teams as their BSPs do the same thing. The diversity of features is the reason that so many distributions
rise and fall, compete and cooperate. Therefore, what we should debate is not what distros that support RISC-V should do with this possible divergence: I am totally fine that this CONFIG_RESTRICT_MREG_ACESS is off by default! Big ones like Fedora and Debian
aim at Desktop or Server, and that's good. What we should really debate here is the feature itself, if it is useful enough for some, not all, possible RISC-V machines that help to make people's lives easier.
For the record, distributions can just release a single image that disables this feature by default. That's their choice because they expect quite a ratio of it will run as a guest OS, and it should not enable the feature or there will be a lot more work in
the hypervisor. The image can still run on any RISC-V machines that either or not support bit[28]/mcounterwen. You are not making a valid point here.
Best,
Alan
|
|
|
Re: P extension fixed-point saturation flag CSR
Thanks. I will update the P extension specification to use the vxsat CSR.
Best Regards,
Chuanhua
|
|
|
Re: P extension fixed-point saturation flag CSR
Krste Asanovic
Yes, should share the flag. Code that has an outer check for saturation to check if re-scaling needed, won’t want to care whether it was caused by scalar or vector routine. Krste
On Aug 4, 2020, at 7:54 PM, Andrew Waterman < andrew@...> wrote:
Analogous to the F and V extensions sharing fflags/frm, I’d think that P and V would share vxsat/vxrm.
The reasoning is slightly different from the floating-point CSRs; in that case, sharing the registers helps vectorized code to efficiently implement C99 semantics for dynamic rounding mode and exception flags. That concern doesn’t apply to fixed point (at least not today). But the other advantages still apply, namely, smaller context and fewer CSRs. On Tue, Aug 4, 2020 at 7:36 PM Chuanhua Chang < chchang@...> wrote: P extension specification has defined a fixed-point saturation flag CSR. We need to allocate it officially in the user standard read/write address range. Should we use the same CSR for V and P extension? The current vector extension spec defines a "vxsat" CSR that can be used for both P and V extensions.
Best Regards,
Chuanhua
|
|
|
Re: P extension fixed-point saturation flag CSR
Analogous to the F and V extensions sharing fflags/frm, I’d think that P and V would share vxsat/vxrm.
The reasoning is slightly different from the floating-point CSRs; in that case, sharing the registers helps vectorized code to efficiently implement C99 semantics for dynamic rounding mode and exception flags. That concern doesn’t apply to fixed point (at least not today). But the other advantages still apply, namely, smaller context and fewer CSRs.
On Tue, Aug 4, 2020 at 7:36 PM Chuanhua Chang < chchang@...> wrote: P extension specification has defined a fixed-point saturation flag CSR. We need to allocate it officially in the user standard read/write address range. Should we use the same CSR for V and P extension? The current vector extension spec defines a "vxsat" CSR that can be used for both P and V extensions.
Best Regards,
Chuanhua
|
|
|
P extension fixed-point saturation flag CSR
P extension specification has defined a fixed-point saturation flag CSR. We need to allocate it officially in the user standard read/write address range. Should we use the same CSR for V and P extension? The current vector extension spec defines a "vxsat" CSR that can be used for both P and V extensions.
Best Regards,
Chuanhua
|
|
|
Re: A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
alankao
Hi Anup, > The most desired feature from a PMU is counting events in right-context (or right-mode). This is not clearly defined in RISC-V spec right now. Greg’s proposal already address this in a clean way by defining required bits in MHPMEVENT CSRs. Other important feature expected from a PMU is reading counters without traps, this is already available because HPMCOUNTER CSRs are “User-Read-Only”
Claiming some features as "most desired" is too subjective. I agree that mode-specific counting is important, but for performance monitoring, the HPM interrupt is also essential. Otherwise, sampling like `perf record` just doesn't work.
> Regarding HPMCOUNTER writes from S-mode, the Linux PMU drivers (across architectures) only writes PMU counter at time of configuring the counter. We anyway have SBI call to configure a RISC-V PMU counter because MHPMEVENT CSR is for M-mode only so it is better to initialize/write the MHPMCOUNTER CSR in M-mode at time of configuring the counter. Allowing S-mode to write HPMCOUNTER CSR is good but won’t benefit much. On the contrary, RISC-V hypervisors might end up save/restore more CSRs if HPMCOUTNER CSR is writeable from S-mode.
You are understating the case when you say "only writes counter at configuration time." It sounds like the kernel seldom writes them. The fact is, the counters and other registers need writing every time the corresponding process is being context-switch in and every time an HPM interrupt is being handled. Would you like to elaborate more, based on what do you say writing counters from S-mode is good? based on what do you judge it won't benefit much?
However, since RISC-V doesn't have any discussed features yet, I doubt that anyone has any quantitative data. My proposal here is that I will take our existing solution (AndeStar V5 extension and perf-event port on Linux 4.17) as the testbed. By default, we have *mcounterwen*, which is effectively equal to the bit[28] in Greg's proposal, to enable S-mode writing HPM CSRs. This is the treatment group. Then we do a patch to transform all existing csr_write's to HPM CSRs (including counters) into SBI calls as the control group. My anticipation of the result is that the wall clock time performing a sampling in the treatment group will be not just marginally shorter than the control group.
Meanwhile, I agree with your concern about H extension. That's why I emphasized this feature is useful for M-S-U configuration and questionable for M-H-S-U one.
> The code snippet mentioned below requires “#ifdef” which means we have to build Linux RISC-V image differently for doing CSR writes this way. This approach is not going to fly for distros because distros can’t release single Linux RISC-V image for all RISC-V hardware if we have such “#ifdef”.
Each distro maintains its own priority of hundreds of thousands of kernel features, not to mention many nameless "distributions" released by different teams as their BSPs do the same thing. The diversity of features is the reason that so many distributions rise and fall, compete and cooperate. Therefore, what we should debate is not what distros that support RISC-V should do with this possible divergence: I am totally fine that this CONFIG_RESTRICT_MREG_ACESS is off by default! Big ones like Fedora and Debian aim at Desktop or Server, and that's good. What we should really debate here is the feature itself, if it is useful enough for some, not all, possible RISC-V machines that help to make people's lives easier.
For the record, distributions can just release a single image that disables this feature by default. That's their choice because they expect quite a ratio of it will run as a guest OS, and it should not enable the feature or there will be a lot more work in the hypervisor. The image can still run on any RISC-V machines that either or not support bit[28]/mcounterwen. You are not making a valid point here.
Best,
Alan
|
|
|
Re: CSR address for debug scontext and hcontext
I'll chime in with support for setting aside a small allocation of supervisor and hypervisor CSR numbers for debug use (similar to the current small allocation of machine CSRs for debug use). While Ernie describes two uses that need supervisor and hypervisor CSR numbers, it would be good to allow room for future growth in architectural debug functionality. And note that this is only allocating two 16-number CSR groups.
Greg
Hello,
Background:
You may be aware that the RISC-V Debug Specification 0.13 defines two CSRs, mcontext and scontext, that can be used to qualify hardware breakpoints in a particular OS process or thread. A modified S-mode OS kernel writes the process ID to scontext when switching processes. Breakpoint hardware can be set to trigger only when the process ID in scontext matches the desired process. Using ASID instead of scontext to qualify breakpoints has been suggested. However, many systems do not implement ASID or only implement a narrow field, forcing the OS to recycle ASID values. This makes ASID useless for breakpoint qualification.
For those familiar with ARM, the equivalent registers in that architecture are CONTEXTIDR_EL1 and CONTEXTIDR_EL2.
Problem:
Scontext is defined in the ratified Debug Spec at CSR 0x7aa which is in the "Machine Standard read/write debug CSR" region and so is, by convention, inaccessible from S-mode.
The Debug Spec was ratified before work on the hypervisor had gotten very far, so Debug Spec 0.13 does not provide full support for hypervisor-based systems. Among the missing items is a definition for an "hcontext" register to qualify breakpoints in a particular virtual machine. An argument could be made to use VMID for this, but the discussion above about ASID qualification would also apply to VMID.
Proposed Solution:
The Debug Task Group would like to suggest allocating a range of CSR addresses in one of the Supervisor Standard read/write regions and in one of the Hypervisor Standard read/write regions to use for debug registers. Our suggestion is 0x5A0-0x5AF for S-mode and 0x6A0-0x6AF for HS-mode, complementing the 0x7A0-0x7AF already defined for M-mode debug registers. Allocating more than just one address gives the Debug TG flexibility for the future.
Thanks, Ernie Edgar RISC-V Debug Task Group
|
|
|
Re: A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
Anup Patel
The most desired feature from a PMU is counting events in right-context (or right-mode). This is not clearly defined in RISC-V spec right now. Greg’s proposal already address this in a clean way by defining required bits in MHPMEVENT CSRs.
Other important feature expected from a PMU is reading counters without traps, this is already available because HPMCOUNTER CSRs are “User-Read-Only”.
Regarding HPMCOUNTER writes from S-mode, the Linux PMU drivers (across architectures) only writes PMU counter at time of configuring the counter. We anyway have SBI call to configure a RISC-V PMU counter because MHPMEVENT CSR is for M-mode
only so it is better to initialize/write the MHPMCOUNTER CSR in M-mode at time of configuring the counter. Allowing S-mode to write HPMCOUNTER CSR is good but won’t benefit much. On the contrary, RISC-V hypervisors might end up save/restore more CSRs if HPMCOUTNER
CSR is writeable from S-mode.
The code snippet mentioned below requires “#ifdef” which means we have to build Linux RISC-V image differently for doing CSR writes this way. This approach is not going to fly for distros because distros can’t release single Linux RISC-V
image for all RISC-V hardware if we have such “#ifdef”.
Regards,
Anup
toggle quoted message
Show quoted text
From: alankao <alankao@...>
Sent: 04 August 2020 12:44
To: Anup Patel <Anup.Patel@...>
Subject: Private: Re: [RISC-V] [tech-privileged] A proposal to enhance RISC-V HPM (Hardware Performance Monitor)
The bit[28], or the *mcounterwen* in my original proposal, is essential for better performance versus m-mode register access through SBI or trap-emulation. It is definitely helpful for a simple M-S-U RISC-V system. How many cycles are
you willing to give up just to update the M-mode register every time the kernel handles an HPM interrupt or a context switch? I admit it may not be useful, or even harmful for M-H-S-U system, but it is obvious that a simple RISC-V machine running single OS
kernel can largely benefit from it.
I suggest we kept it (either bit[28] or thing like our *mcounterwen*) as some WARL. In Linux kernel, we can have an API, and the real implementation is like
riscv_update_hpm_event()
{
#ifdef CONFIG_RESTRICT_MREG_ACCESS
sbi_update_hpm_event(...);
#else
// that the bit[28]s have been enabled at initialization stage
csr_write(...);
#endif
}
Any comments?
|
|
|
Re: 答复: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Yeah, sure. : )
发件人: Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:58
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送: Anup Patel <Anup.Patel@...>; tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
toggle quoted message
Show quoted text
On Tue, Aug 4, 2020 at 1:57 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the clarification. It turns out that I was looking at an older code base.
It's good news we came to the same conclusions independently :)
The QEMU “virt” machine emulates both TIME CSR and HTIMEDELTA CSR so no trapping happens when accessing
TIME and HTIMEDELTA CSRs from VS/VU mode on QEMU. Although, QEMU “sifive_u” machine does not emulate these CSRs because on real SiFive Unleashed TIME CSR is trap-n-emulated by M-mode.
Also, (like Andrew already mentioned) the
MCOUNTEREN.TM and
HCOUNTER.TM bits handles the
delegation of TIME CSR properly.
Regards,
Anup
I see. This is new information for me though. The ‘that is’ in the current spec makes
the sentence look like an reiteration of the functionality of the htimedelta itself. I totally didn’t expect this added bit of information about the changing of the trapping rule of the time CSR in there. (Judging from the current implementation in QEMU and
KVM, reading time CSR in a virtual machine does trap.) Perhaps add a note to clarify this point?
However, there is still one new bit. Our proposal says that the no-trapping behavior can
be turned off, so that the hypervisor can choose to emulate the time CSR for a virtual machine. This can provide the flexibility for implementing simpler systems where the aliases or the htimedelta does not exist.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:06
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event
Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the hypervisor spec.
Section 3 is meant to propose aliases available in VS-mode and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode
isn't required to trap, and it returns the value you want it to return.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event
Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s the first piece
which proposes a dedicated clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs and aliases. Evaluations
on QEMU show that the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
Re: RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 1:57 AM zhaosiqi (A) < zhaosiqi3@...> wrote:
Thanks for the clarification. It turns out that I was looking at an older code base.
It's good news we came to the same conclusions independently :)
The QEMU “virt” machine emulates both TIME CSR and HTIMEDELTA CSR so no trapping happens when accessing TIME and HTIMEDELTA CSRs from VS/VU mode on QEMU. Although,
QEMU “sifive_u” machine does not emulate these CSRs because on real SiFive Unleashed TIME CSR is trap-n-emulated by M-mode.
Also, (like Andrew already mentioned) the MCOUNTEREN.TM and HCOUNTER.TM bits handles the delegation of TIME CSR properly.
Regards,
Anup
I see. This is new information for me though. The ‘that is’ in the current spec makes the sentence look like an reiteration of the functionality
of the htimedelta itself. I totally didn’t expect this added bit of information about the changing of the trapping rule of the time CSR in there. (Judging from the current implementation in QEMU and KVM, reading time CSR in a virtual machine does trap.) Perhaps
add a note to clarify this point?
However, there is still one new bit. Our proposal says that the no-trapping behavior can be turned off, so that the hypervisor can choose to emulate
the time CSR for a virtual machine. This can provide the flexibility for implementing simpler systems where the aliases or the htimedelta does not exist.
发件人: Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:06
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the hypervisor spec.
Section 3 is meant to propose aliases available in VS-mode and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode isn't required to trap, and it returns the value you want
it to return.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s the first piece
which proposes a dedicated clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs and aliases. Evaluations
on QEMU show that the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
Re: 答复: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Thanks for the clarification. It turns out that I was looking at an older code base.
发件人: Anup Patel [mailto:Anup.Patel@...]
发送时间: 2020年8月4日
16:48
收件人: zhaosiqi (A) <zhaosiqi3@...>; Andrew Waterman <andrew@...>
抄送: tech-privileged@...
主题: RE: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
The QEMU “virt” machine emulates both TIME CSR and HTIMEDELTA CSR so no trapping happens when accessing TIME and HTIMEDELTA CSRs from VS/VU mode on QEMU. Although,
QEMU “sifive_u” machine does not emulate these CSRs because on real SiFive Unleashed TIME CSR is trap-n-emulated by M-mode.
Also, (like Andrew already mentioned) the MCOUNTEREN.TM and HCOUNTER.TM bits handles the delegation of TIME CSR properly.
Regards,
Anup
toggle quoted message
Show quoted text
Sent: 04 August 2020 13:59
To: Andrew Waterman < andrew@...>
Cc: tech-privileged@...
Subject: 答复: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and
VS-mode
I see. This is new information for me though. The ‘that is’ in the current spec makes the sentence look like an reiteration of the functionality
of the htimedelta itself. I totally didn’t expect this added bit of information about the changing of the trapping rule of the time CSR in there. (Judging from the current implementation in QEMU and KVM, reading time CSR in a virtual machine does trap.) Perhaps
add a note to clarify this point?
However, there is still one new bit. Our proposal says that the no-trapping behavior can be turned off, so that the hypervisor can choose to emulate
the time CSR for a virtual machine. This can provide the flexibility for implementing simpler systems where the aliases or the htimedelta does not exist.
发件人: Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:06
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the hypervisor spec.
Section 3 is meant to propose aliases available in VS-mode and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode isn't required to trap, and it returns the value you want
it to return.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s the first piece
which proposes a dedicated clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs and aliases. Evaluations
on QEMU show that the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
Re: 答复: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
In that case, yeah, section 3 does not propose anything new.
发件人: Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:40
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送: tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
toggle quoted message
Show quoted text
On Tue, Aug 4, 2020 at 1:28 AM zhaosiqi (A) via
lists.riscv.org <zhaosiqi3=huawei.com@...> wrote:
I see. This is new information for me though. The ‘that is’ in the current spec makes
the sentence look like an reiteration of the functionality of the htimedelta itself. I totally didn’t expect this added bit of information about the changing of the trapping rule of the time CSR in there. (Judging from the current implementation in QEMU and
KVM, reading time CSR in a virtual machine does trap.) Perhaps add a note to clarify this point?
However, there is still one new bit. Our proposal says that the no-trapping behavior can
be turned off, so that the hypervisor can choose to emulate the time CSR for a virtual machine. This can provide the flexibility for implementing simpler systems where the aliases or the htimedelta does not exist.
hcounteren.TM is already defined this way, for exactly this purpose.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:06
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event
Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the hypervisor spec.
Section 3 is meant to propose aliases available in VS-mode and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode
isn't required to trap, and it returns the value you want it to return.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event
Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s the first piece
which proposes a dedicated clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs and aliases. Evaluations
on QEMU show that the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
Re: RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Anup Patel
The QEMU “virt” machine emulates both TIME CSR and HTIMEDELTA CSR so no trapping happens when accessing TIME and HTIMEDELTA CSRs from VS/VU mode on QEMU. Although, QEMU “sifive_u”
machine does not emulate these CSRs because on real SiFive Unleashed TIME CSR is trap-n-emulated by M-mode.
Also, (like Andrew already mentioned) the MCOUNTEREN.TM and HCOUNTER.TM bits handles the delegation of TIME CSR properly.
Regards,
Anup
toggle quoted message
Show quoted text
From: tech-privileged@... <tech-privileged@...>
On Behalf Of zhaosiqi (A) via lists.riscv.org
Sent: 04 August 2020 13:59
To: Andrew Waterman <andrew@...>
Cc: tech-privileged@...
Subject: 答复: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
I see. This is new information for me though. The ‘that is’ in the current spec makes the sentence look like an reiteration of the
functionality of the htimedelta itself. I totally didn’t expect this added bit of information about the changing of the trapping rule of the time CSR in there. (Judging from the current implementation in QEMU and KVM, reading time CSR in a virtual machine
does trap.) Perhaps add a note to clarify this point?
However, there is still one new bit. Our proposal says that the no-trapping behavior can be turned off, so that the hypervisor can
choose to emulate the time CSR for a virtual machine. This can provide the flexibility for implementing simpler systems where the aliases or the htimedelta does not exist.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日 16:06
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the
hypervisor spec. Section 3 is meant to propose aliases available in VS-mode and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode isn't required to trap, and it returns
the value you want it to return.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日 15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains
the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s
the first piece which proposes a dedicated clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs
and aliases. Evaluations on QEMU show that the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
Re: RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
I see. This is new information for me though. The ‘that is’ in the current spec makes the sentence look like an reiteration of the functionality
of the htimedelta itself. I totally didn’t expect this added bit of information about the changing of the trapping rule of the time CSR in there. (Judging from the current implementation in QEMU and KVM, reading time CSR in a virtual machine does trap.) Perhaps
add a note to clarify this point?
However, there is still one new bit. Our proposal says that the no-trapping behavior can be turned off, so that the hypervisor can choose to emulate
the time CSR for a virtual machine. This can provide the flexibility for implementing simpler systems where the aliases or the htimedelta does not exist.
hcounteren.TM is already defined this way, for exactly this purpose.
发件人: Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:06
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送: tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the hypervisor spec.
Section 3 is meant to propose aliases available in VS-mode and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode isn't required to trap, and it returns the value you want
it to return.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s the first piece which proposes a dedicated
clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs and aliases. Evaluations on QEMU show that
the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
Re: 答复: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
I see. This is new information for me though. The ‘that is’ in the current spec makes the sentence look like an reiteration of the functionality
of the htimedelta itself. I totally didn’t expect this added bit of information about the changing of the trapping rule of the time CSR in there. (Judging from the current implementation in QEMU and KVM, reading time CSR in a virtual machine does trap.) Perhaps
add a note to clarify this point?
However, there is still one new bit. Our proposal says that the no-trapping behavior can be turned off, so that the hypervisor can choose to emulate
the time CSR for a virtual machine. This can provide the flexibility for implementing simpler systems where the aliases or the htimedelta does not exist.
发件人: Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
16:06
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送: tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
toggle quoted message
Show quoted text
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) <zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the hypervisor spec.
Section 3 is meant to propose aliases available in VS-mode and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode isn't required to trap, and it returns the value you want
it to return.
发件人:
Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送:
tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s the first piece which proposes a dedicated
clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs and aliases. Evaluations on QEMU show that
the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
Re: RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
On Tue, Aug 4, 2020 at 12:46 AM zhaosiqi (A) < zhaosiqi3@...> wrote:
Thanks for the timely comment, we are aware of the htimedelta CSR in the hypervisor spec. Section 3 is meant to propose aliases available in VS-mode
and VU-mode. Because they are meant to provide time for a virtual machine, the value returned should be adjusted by the htimedelta CSR. That sentence is meant to state this requirement.
Yeah, my point is that the current definition of htimedelta already says what you want it to say about the time CSR. Reading the time CSR in VS-mode or VU-mode isn't required to trap, and it returns the value you want it to return.
发件人: Andrew Waterman [mailto:andrew@...]
发送时间: 2020年8月4日
15:34
收件人: zhaosiqi (A) <zhaosiqi3@...>
抄送: tech-privileged@...
主题: Re: [RISC-V] [tech-privileged] RFC: Dedicated Clock Source and Clock Event Source for HS-mode and VS-mode
Note that Section 3 of your proposal already exists. The hypervisor spec says, "The htimedelta CSR is a read/write register that contains the delta
between the value of the time CSR and the value returned in VS-mode or VU-mode." In other words, reading the time CSR when V=1 does what you propose.
Hello Everyone,
We have come up with some ideas about improving the performance of virtual machines on the RISC-V architecture. Here’s the first piece which proposes a dedicated
clock source and a clock event source for HS-mode and VS-mode, respectively. We extended the current idea of the mtime and mtimecmp CSRs and combined with the current hypervisor extension to come up with new CSRs and aliases. Evaluations on QEMU show that
the proposed extension leads to performance improvement.
The attached document details the proposal. Any comments are welcome.
Regards,
Siqi
|
|
|
