On Wed, Nov 17, 2021 at 10:28:40AM -0800, Greg Favor wrote:

On Wed, Nov 17, 2021 at 9:49 AM Ved Shanbhogue <ved@...> wrote:

Agree. Standard protocols like IEEE 1588 may also be used to acheive fine
synchronization with a distributed time. Having distributed but
synchronized time may avoid needing to send a large time bus across the die
and have to deal with issues you highlight such as async crossings,
spread-spectrum, dvfs, etc.

Just to clarify what I was trying to say: Whether distributing a large
time bus or a "smarter" small time bus - which are functionally equivalent
(and many designs I'm aware of do the latter), distributing that across
CDC's introduces the obvious timing uncertainty. If the receive side is
running at 1 GHz or 2GHz, then right there appears 1ns to 0.5ns of time
uncertainty/inaccuracy in a hart's final time value. Two CDC's (e.g. in
larger scale many-core designs) doubles that.

Leaving aside any CDC's, also keep in mind that even if the distribution of
time to each hart is synchronous and "perfectly" balanced (i.e. the exact
same number of clock cycles from mtime to each end point), ensuring that
the clock skew between these potentially far apart end points is sub-1ns is
impractical (especially in leading edge processes with long and highly
variable wire delays). Even mesh-based clock distribution schemes won't
achieve that.

I am not disagreeing to that.

regards
ved

On Wed, Nov 17, 2021 at 04:37:21AM +0000, Anup Patel wrote:

Before we go ahead and change the MTIME resolution requirement in the platform spec, I would like to highlight following points (from past discussions) which led to mandating a fixed MTIME resolution in the platform spec:


1. The Linux kernel creates a clock source on-top-of time CSR (mirror of MMIO MTIME) with timebase-frequency discovered from DT. The generic time management in Linux kernel requires nanoseconds granularity so each value read from clock source is converted to nanoseconds using a mult and shift (i.e. nanoseconds = (time_csr * multi) >> shift)). In other words, Linux kernel always uses integer operation to convert X resolution of time CSR to 1ns resolution and this conversion will have some round-off errors. We could have mandated a fixed 1ns resolution (just like ARM SBSA) but for RISC-V we also need to honour the architectural requirement of all time CSRs to be synchronized within 1 tick (i.e. one resolution period) and for multi-sockets (or multi-die) systems it becomes challenging to synchronize multiple MTIME counters within 1ns resolution. Considering this facts, it made sense to have fixed 10ns resolution for MTIME but the update frequency could be lower than 100MHz. (@Greg Favor<mailto:gfavor@...>, please add if I missed anything)

Considering the requirement that all time CSRs be synchronized to within 1
tick, setting a fixed resolution indirectly makes synchronization much more
difficult for implementations that have a lower update frequency. For such
implementations, since each update is more than 1 tick, it would be
necessary to ensure that all time CSRs always have exactly the same value,
which is considerably more difficult than within 1 tick.

2. It is common in enterprise clouds to migrate a Guest/VM across different hosts. Considering the diversity in RISC-V world, we have to support migration of Guest/VM from host A to host B where these hosts can be from different vendors. Now if host A and host B have different MTIME resolution then Guest/VM migration will not work and this problem also applies to ARM world. This is another reason why ARM SBSA mandates a fixed timer resolution. Although, ARM world standardized timer frequency quite late in the game but RISC-V world can avoid these problems by standardizing MTIME resolution quite early. Alternatively, there is also SBI para-virt call possible to detect change in MTIME resolution but this would mean additional code (along with barriers) in the path of reading time CSR (as an example, look at KVM para-virt clock used in x86 world).
3. Requiring a fixed MTIME resolution in the platform spec would mean that some of the existing platforms won’t comply but the platform spec is supposed to be forward looking. If we don’t standardize a fixed MTIME resolution now then it is likely that we will end-up standardizing it in future platform spec.


Based on above points, I still think mandating fixed MTIME resolution is desirable for OS-A platforms and this has nothing to do with how timebase-frequency is discovered (i.e. DT/ACPI).

Regards,
Anup

From: <tech-unixplatformspec@...> on behalf of Jonathan Behrens <behrensj@...>
Date: Wednesday, 17 November 2021 at 6:26 AM
To: Vedvyas Shanbhogue <ved@...>
Cc: Greg Favor <gfavor@...>, Heinrich Schuchardt <xypron.glpk@...>, tech-unixplatformspec <tech-unixplatformspec@...>
Subject: Re: [RISC-V] [tech-unixplatformspec] MTIME update frequency

Given that the device-tree mechanism is apparently already in place, it honestly probably wouldn't be a big deal to just formalize that and not require any particular mtime resolution. I still prefer the simplicity of always doing 10ns, but don't feel that strongly about it.

Jonathan

On Tue, Nov 16, 2021 at 3:13 PM Vedvyas Shanbhogue via lists.riscv.org<http://lists.riscv.org> <ved=rivosinc.com@...<mailto:rivosinc.com@...>> wrote:
On Tue, Nov 16, 2021 at 11:31:44AM -0800, Greg Favor wrote:

On Tue, Nov 16, 2021 at 10:50 AM Jonathan Behrens <behrensj@...<mailto:behrensj@...>> wrote:

To Ved's last post, it is the timebase resolution (and not update
frequency) that determines the conversion from time to ticks.

So the question is whether there should be a fixed resolution so that
platform-compliant software can simply do a fixed absolute time to
mtime/time conversion, and conversely how much or little change to Linux
would be required to support a discoverable variable conversion?

Linux discovers the timebase from device-tree and does not assume a fixed frequency:
https://elixir.bootlin.com/linux/latest/source/arch/riscv/kernel/time.c#L14

regards
ved

On Wed, Nov 17, 2021 at 04:36:10PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 04:37:21AM +0000, Anup Patel wrote:

Before we go ahead and change the MTIME resolution requirement in the platform spec, I would like to highlight following points (from past discussions) which led to mandating a fixed MTIME resolution in the platform spec:


1. The Linux kernel creates a clock source on-top-of time CSR (mirror of MMIO MTIME) with timebase-frequency discovered from DT. The generic time management in Linux kernel requires nanoseconds granularity so each value read from clock source is converted to nanoseconds using a mult and shift (i.e. nanoseconds = (time_csr * multi) >> shift)). In other words, Linux kernel always uses integer operation to convert X resolution of time CSR to 1ns resolution and this conversion will have some round-off errors. We could have mandated a fixed 1ns resolution (just like ARM SBSA) but for RISC-V we also need to honour the architectural requirement of all time CSRs to be synchronized within 1 tick (i.e. one resolution period) and for multi-sockets (or multi-die) systems it becomes challenging to synchronize multiple MTIME counters within 1ns resolution. Considering this facts, it made sense to have fixed 10ns resolution for MTIME but the update frequency could be lower than 100MHz. (@Greg Favor<mailto:gfavor@...>, please add if I missed anything)

Considering the requirement that all time CSRs be synchronized to within 1
tick, setting a fixed resolution indirectly makes synchronization much more
difficult for implementations that have a lower update frequency. For such
implementations, since each update is more than 1 tick, it would be
necessary to ensure that all time CSRs always have exactly the same value,
which is considerably more difficult than within 1 tick.

So an implementation that supports 100MHz clock would need to update the mtime by 10 on each tick to meet the 1ns granularity. In current spec a implementation that supports 10 MHz clock would need to update the mtime by 10 on each tick to meet the 10ns resolution. I am not sure incrementing by 1 vs. incrementing by 10 makes it much harder as it was already required for a system that implements a 10MHz clock.

CSRs need to have the consistent time at the observation point. The fastest way in most system to "observe" value in CSRs is through the cache or through memory. So the difference between the two CSRs should not be observable to the extent that the following test fails:

Hart-X (sends a message to Hart-Y):
Read time
Write timestamp message to memory
Hart-Y:
Read timestamp message from memory (gets the data from cache or memory)
Read time
compare timestamp to time (the timestamp should not be in future)

If this test is passes then the fact that the CSR in Hart-Y is 2 ticks behind or ahead of Hart-X is not observable.

Setting the granularity to 1 ns provides the path for RISC-V implementations that want the fine resolution to acheive the goal without penalizing implementations that do not.

regards
ved

On Wed, Nov 17, 2021 at 03:50:17PM -0600, Vedvyas Shanbhogue wrote:

On Wed, Nov 17, 2021 at 04:36:10PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 04:37:21AM +0000, Anup Patel wrote:

Before we go ahead and change the MTIME resolution requirement in the platform spec, I would like to highlight following points (from past discussions) which led to mandating a fixed MTIME resolution in the platform spec:


1. The Linux kernel creates a clock source on-top-of time CSR (mirror of MMIO MTIME) with timebase-frequency discovered from DT. The generic time management in Linux kernel requires nanoseconds granularity so each value read from clock source is converted to nanoseconds using a mult and shift (i.e. nanoseconds = (time_csr * multi) >> shift)). In other words, Linux kernel always uses integer operation to convert X resolution of time CSR to 1ns resolution and this conversion will have some round-off errors. We could have mandated a fixed 1ns resolution (just like ARM SBSA) but for RISC-V we also need to honour the architectural requirement of all time CSRs to be synchronized within 1 tick (i.e. one resolution period) and for multi-sockets (or multi-die) systems it becomes challenging to synchronize multiple MTIME counters within 1ns resolution. Considering this facts, it made sense to have fixed 10ns resolution for MTIME but the update frequency could be lower than 100MHz. (@Greg Favor<mailto:gfavor@...>, please add if I missed anything)

Considering the requirement that all time CSRs be synchronized to within 1
tick, setting a fixed resolution indirectly makes synchronization much more
difficult for implementations that have a lower update frequency. For such
implementations, since each update is more than 1 tick, it would be
necessary to ensure that all time CSRs always have exactly the same value,
which is considerably more difficult than within 1 tick.

So an implementation that supports 100MHz clock would need to update the mtime by 10 on each tick to meet the 1ns granularity. In current spec a implementation that supports 10 MHz clock would need to update the mtime by 10 on each tick to meet the 10ns resolution. I am not sure incrementing by 1 vs. incrementing by 10 makes it much harder as it was already required for a system that implements a 10MHz clock.

The specification says "synchronized to within one tick".

If the time CSR is incremented one tick at a time, then, for every update,
as long as one update propagates to all harts before the next update, this
requirement is met. Some harts may show time T and some may show T+1, but
those are within one tick, which is acceptable.

Now suppose there is a system that increments 10 ticks per update. If the
time value in any hart lags, where some show T and some show T+10, this is
not within one tick, which is not acceptable.

Thus, for systems that update one tick at a time, they have a full tick to
propagate updates to all harts. For systems that update multiple ticks at
a time, they have *no* time to propagate updates to all harts; all updates
must be instantaneous on *all* harts.

CSRs need to have the consistent time at the observation point. The fastest way in most system to "observe" value in CSRs is through the cache or through memory. So the difference between the two CSRs should not be observable to the extent that the following test fails:

Hart-X (sends a message to Hart-Y):
Read time
Write timestamp message to memory
Hart-Y:
Read timestamp message from memory (gets the data from cache or memory)
Read time
compare timestamp to time (the timestamp should not be in future)

If this test is passes then the fact that the CSR in Hart-Y is 2 ticks behind or ahead of Hart-X is not observable.

That is perhaps a useful test, but not the only way to observe the clocks,
nor is it how the accuracy is specified in the architecture. The
architecture does not say the accuracy is with respect to what is
observable in another hart. The time could, for example, be observed by
something external to both harts and with higher precision.

Setting the granularity to 1 ns provides the path for RISC-V implementations
that want the fine resolution to acheive the goal without penalizing
implementations that do not.

regards
ved

On Wed, Nov 17, 2021 at 05:30:05PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 03:50:17PM -0600, Vedvyas Shanbhogue wrote:

On Wed, Nov 17, 2021 at 04:36:10PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 04:37:21AM +0000, Anup Patel wrote:

Before we go ahead and change the MTIME resolution requirement in the platform spec, I would like to highlight following points (from past discussions) which led to mandating a fixed MTIME resolution in the platform spec:


1. The Linux kernel creates a clock source on-top-of time CSR (mirror of MMIO MTIME) with timebase-frequency discovered from DT. The generic time management in Linux kernel requires nanoseconds granularity so each value read from clock source is converted to nanoseconds using a mult and shift (i.e. nanoseconds = (time_csr * multi) >> shift)). In other words, Linux kernel always uses integer operation to convert X resolution of time CSR to 1ns resolution and this conversion will have some round-off errors. We could have mandated a fixed 1ns resolution (just like ARM SBSA) but for RISC-V we also need to honour the architectural requirement of all time CSRs to be synchronized within 1 tick (i.e. one resolution period) and for multi-sockets (or multi-die) systems it becomes challenging to synchronize multiple MTIME counters within 1ns resolution. Considering this facts, it made sense to have fixed 10ns resolution for MTIME but the update frequency could be lower than 100MHz. (@Greg Favor<mailto:gfavor@...>, please add if I missed anything)

Considering the requirement that all time CSRs be synchronized to within 1
tick, setting a fixed resolution indirectly makes synchronization much more
difficult for implementations that have a lower update frequency. For such
implementations, since each update is more than 1 tick, it would be
necessary to ensure that all time CSRs always have exactly the same value,
which is considerably more difficult than within 1 tick.

So an implementation that supports 100MHz clock would need to update the mtime by 10 on each tick to meet the 1ns granularity. In current spec a implementation that supports 10 MHz clock would need to update the mtime by 10 on each tick to meet the 10ns resolution. I am not sure incrementing by 1 vs. incrementing by 10 makes it much harder as it was already required for a system that implements a 10MHz clock.

The specification says "synchronized to within one tick".

If the time CSR is incremented one tick at a time, then, for every update,
as long as one update propagates to all harts before the next update, this
requirement is met. Some harts may show time T and some may show T+1, but
those are within one tick, which is acceptable.

Now suppose there is a system that increments 10 ticks per update. If the
time value in any hart lags, where some show T and some show T+10, this is
not within one tick, which is not acceptable.

Thus, for systems that update one tick at a time, they have a full tick to
propagate updates to all harts. For systems that update multiple ticks at
a time, they have *no* time to propagate updates to all harts; all updates
must be instantaneous on *all* harts.

I agree. But that requirement is there already for a system that implements a 10 Mhz clock and has to suport 10ns resolution. If 10ns vs. 1ns is a really sticking point could we have two values supported by the platform specification - similar to the range of update frequencies supported. We can leave it to the end user whether they want to put these two classes of systems in one migration pool.

regards
ved

On Wed, Nov 17, 2021 at 04:37:12PM -0600, Vedvyas Shanbhogue via lists.riscv.org wrote:

On Wed, Nov 17, 2021 at 05:30:05PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 03:50:17PM -0600, Vedvyas Shanbhogue wrote:

On Wed, Nov 17, 2021 at 04:36:10PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 04:37:21AM +0000, Anup Patel wrote:

Before we go ahead and change the MTIME resolution requirement in the platform spec, I would like to highlight following points (from past discussions) which led to mandating a fixed MTIME resolution in the platform spec:


1. The Linux kernel creates a clock source on-top-of time CSR (mirror of MMIO MTIME) with timebase-frequency discovered from DT. The generic time management in Linux kernel requires nanoseconds granularity so each value read from clock source is converted to nanoseconds using a mult and shift (i.e. nanoseconds = (time_csr * multi) >> shift)). In other words, Linux kernel always uses integer operation to convert X resolution of time CSR to 1ns resolution and this conversion will have some round-off errors. We could have mandated a fixed 1ns resolution (just like ARM SBSA) but for RISC-V we also need to honour the architectural requirement of all time CSRs to be synchronized within 1 tick (i.e. one resolution period) and for multi-sockets (or multi-die) systems it becomes challenging to synchronize multiple MTIME counters within 1ns resolution. Considering this facts, it made sense to have fixed 10ns resolution for MTIME but the update frequency could be lower than 100MHz. (@Greg Favor<mailto:gfavor@...>, please add if I missed anything)

Considering the requirement that all time CSRs be synchronized to within 1
tick, setting a fixed resolution indirectly makes synchronization much more
difficult for implementations that have a lower update frequency. For such
implementations, since each update is more than 1 tick, it would be
necessary to ensure that all time CSRs always have exactly the same value,
which is considerably more difficult than within 1 tick.

So an implementation that supports 100MHz clock would need to update the mtime by 10 on each tick to meet the 1ns granularity. In current spec a implementation that supports 10 MHz clock would need to update the mtime by 10 on each tick to meet the 10ns resolution. I am not sure incrementing by 1 vs. incrementing by 10 makes it much harder as it was already required for a system that implements a 10MHz clock.

The specification says "synchronized to within one tick".

If the time CSR is incremented one tick at a time, then, for every update,
as long as one update propagates to all harts before the next update, this
requirement is met. Some harts may show time T and some may show T+1, but
those are within one tick, which is acceptable.

Now suppose there is a system that increments 10 ticks per update. If the
time value in any hart lags, where some show T and some show T+10, this is
not within one tick, which is not acceptable.

Thus, for systems that update one tick at a time, they have a full tick to
propagate updates to all harts. For systems that update multiple ticks at
a time, they have *no* time to propagate updates to all harts; all updates
must be instantaneous on *all* harts.

I agree. But that requirement is there already for a system that implements a 10 Mhz clock and has to suport 10ns resolution. If 10ns vs. 1ns is a really sticking point could we have two values supported by the platform specification - similar to the range of update frequencies supported. We can leave it to the end user whether they want to put these two classes of systems in one migration pool.

Did that address your concern? Should we update to supporting 1 ns resolution or not disallowing a 1 ns resolution?

regards
ved

On Fri, Nov 19, 2021 at 08:59:32AM -0600, Vedvyas Shanbhogue wrote:

On Wed, Nov 17, 2021 at 04:37:12PM -0600, Vedvyas Shanbhogue via lists.riscv.org wrote:

On Wed, Nov 17, 2021 at 05:30:05PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 03:50:17PM -0600, Vedvyas Shanbhogue wrote:

On Wed, Nov 17, 2021 at 04:36:10PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 04:37:21AM +0000, Anup Patel wrote:

Before we go ahead and change the MTIME resolution requirement in the platform spec, I would like to highlight following points (from past discussions) which led to mandating a fixed MTIME resolution in the platform spec:


1. The Linux kernel creates a clock source on-top-of time CSR (mirror of MMIO MTIME) with timebase-frequency discovered from DT. The generic time management in Linux kernel requires nanoseconds granularity so each value read from clock source is converted to nanoseconds using a mult and shift (i.e. nanoseconds = (time_csr * multi) >> shift)). In other words, Linux kernel always uses integer operation to convert X resolution of time CSR to 1ns resolution and this conversion will have some round-off errors. We could have mandated a fixed 1ns resolution (just like ARM SBSA) but for RISC-V we also need to honour the architectural requirement of all time CSRs to be synchronized within 1 tick (i.e. one resolution period) and for multi-sockets (or multi-die) systems it becomes challenging to synchronize multiple MTIME counters within 1ns resolution. Considering this facts, it made sense to have fixed 10ns resolution for MTIME but the update frequency could be lower than 100MHz. (@Greg Favor<mailto:gfavor@...>, please add if I missed anything)

Considering the requirement that all time CSRs be synchronized to within 1
tick, setting a fixed resolution indirectly makes synchronization much more
difficult for implementations that have a lower update frequency. For such
implementations, since each update is more than 1 tick, it would be
necessary to ensure that all time CSRs always have exactly the same value,
which is considerably more difficult than within 1 tick.

So an implementation that supports 100MHz clock would need to update the mtime by 10 on each tick to meet the 1ns granularity. In current spec a implementation that supports 10 MHz clock would need to update the mtime by 10 on each tick to meet the 10ns resolution. I am not sure incrementing by 1 vs. incrementing by 10 makes it much harder as it was already required for a system that implements a 10MHz clock.

The specification says "synchronized to within one tick".

If the time CSR is incremented one tick at a time, then, for every update,
as long as one update propagates to all harts before the next update, this
requirement is met. Some harts may show time T and some may show T+1, but
those are within one tick, which is acceptable.

Now suppose there is a system that increments 10 ticks per update. If the
time value in any hart lags, where some show T and some show T+10, this is
not within one tick, which is not acceptable.

Thus, for systems that update one tick at a time, they have a full tick to
propagate updates to all harts. For systems that update multiple ticks at
a time, they have *no* time to propagate updates to all harts; all updates
must be instantaneous on *all* harts.

I agree. But that requirement is there already for a system that implements a 10 Mhz clock and has to suport 10ns resolution. If 10ns vs. 1ns is a really sticking point could we have two values supported by the platform specification - similar to the range of update frequencies supported. We can leave it to the end user whether they want to put these two classes of systems in one migration pool.

Did that address your concern? Should we update to supporting 1 ns resolution or not disallowing a 1 ns resolution?

No, it does not address my concern.

As you said, that requirement was already there, and my concern applies to
the requirement as is, although increasing the resolution will exacerbate
the problem.

When the specification mandates a specific update resolution, multihart
implementations will have to choose between (1) updating one tick at a
time, or (2) very stringent synchronization between all harts. I suspect
implementations will be forced to choose (1), even if they otherwise would
have preferred to have a different update frequency or clock. Or they will
simply ignore the synchronization requirement in the ISA specification,
which I think is a real possibility.

The current requirement effectively forces implementations to have a 100
MHz clock. If the resolution is changed to 1 ns, that becomes a 1 GHz
clock that implementations are forced to have. Either way, this violates
the Platform Policy that says platforms should not mandate performance, but
certainly the higher clock rate is more of a burden.

On Fri, Nov 19, 2021 at 10:36:18AM -0500, Darius Rad wrote:

On Fri, Nov 19, 2021 at 08:59:32AM -0600, Vedvyas Shanbhogue wrote:

On Wed, Nov 17, 2021 at 04:37:12PM -0600, Vedvyas Shanbhogue via lists.riscv.org wrote:

On Wed, Nov 17, 2021 at 05:30:05PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 03:50:17PM -0600, Vedvyas Shanbhogue wrote:

On Wed, Nov 17, 2021 at 04:36:10PM -0500, Darius Rad wrote:

On Wed, Nov 17, 2021 at 04:37:21AM +0000, Anup Patel wrote:

Before we go ahead and change the MTIME resolution requirement in the platform spec, I would like to highlight following points (from past discussions) which led to mandating a fixed MTIME resolution in the platform spec:


1. The Linux kernel creates a clock source on-top-of time CSR (mirror of MMIO MTIME) with timebase-frequency discovered from DT. The generic time management in Linux kernel requires nanoseconds granularity so each value read from clock source is converted to nanoseconds using a mult and shift (i.e. nanoseconds = (time_csr * multi) >> shift)). In other words, Linux kernel always uses integer operation to convert X resolution of time CSR to 1ns resolution and this conversion will have some round-off errors. We could have mandated a fixed 1ns resolution (just like ARM SBSA) but for RISC-V we also need to honour the architectural requirement of all time CSRs to be synchronized within 1 tick (i.e. one resolution period) and for multi-sockets (or multi-die) systems it becomes challenging to synchronize multiple MTIME counters within 1ns resolution. Considering this facts, it made sense to have fixed 10ns resolution for MTIME but the update frequency could be lower than 100MHz. (@Greg Favor<mailto:gfavor@...>, please add if I missed anything)

Considering the requirement that all time CSRs be synchronized to within 1
tick, setting a fixed resolution indirectly makes synchronization much more
difficult for implementations that have a lower update frequency. For such
implementations, since each update is more than 1 tick, it would be
necessary to ensure that all time CSRs always have exactly the same value,
which is considerably more difficult than within 1 tick.

So an implementation that supports 100MHz clock would need to update the mtime by 10 on each tick to meet the 1ns granularity. In current spec a implementation that supports 10 MHz clock would need to update the mtime by 10 on each tick to meet the 10ns resolution. I am not sure incrementing by 1 vs. incrementing by 10 makes it much harder as it was already required for a system that implements a 10MHz clock.

The specification says "synchronized to within one tick".

If the time CSR is incremented one tick at a time, then, for every update,
as long as one update propagates to all harts before the next update, this
requirement is met. Some harts may show time T and some may show T+1, but
those are within one tick, which is acceptable.

Now suppose there is a system that increments 10 ticks per update. If the
time value in any hart lags, where some show T and some show T+10, this is
not within one tick, which is not acceptable.

Thus, for systems that update one tick at a time, they have a full tick to
propagate updates to all harts. For systems that update multiple ticks at
a time, they have *no* time to propagate updates to all harts; all updates
must be instantaneous on *all* harts.

I agree. But that requirement is there already for a system that implements a 10 Mhz clock and has to suport 10ns resolution. If 10ns vs. 1ns is a really sticking point could we have two values supported by the platform specification - similar to the range of update frequencies supported. We can leave it to the end user whether they want to put these two classes of systems in one migration pool.

Did that address your concern? Should we update to supporting 1 ns resolution or not disallowing a 1 ns resolution?

No, it does not address my concern.

As you said, that requirement was already there, and my concern applies to
the requirement as is, although increasing the resolution will exacerbate
the problem.

When the specification mandates a specific update resolution, multihart
implementations will have to choose between (1) updating one tick at a
time, or (2) very stringent synchronization between all harts. I suspect
implementations will be forced to choose (1), even if they otherwise would
have preferred to have a different update frequency or clock. Or they will
simply ignore the synchronization requirement in the ISA specification,
which I think is a real possibility.

The current requirement effectively forces implementations to have a 100
MHz clock. If the resolution is changed to 1 ns, that becomes a 1 GHz
clock that implementations are forced to have. Either way, this violates
the Platform Policy that says platforms should not mandate performance, but
certainly the higher clock rate is more of a burden.

The current specification says that the MTIME values seen by two HARTs to be not apart by more than 1. It does say the MTIME must always increment in units of 1. I do not think the specification mandates incrementing by one on each clock tick. Presently it says the tick - the update frequency - can be 100 MHz or 10 MHz or somewhere in between. If the update frequency is 10 MHz then the MTIME increment per clock must be 10. If the update frequency is 100 Mhz then MTIME increment per clock is 1. So is your concern is that an adder that adds 10 to MTIME per clock tick is hard?

regards
ved

On Fri, Nov 19, 2021 at 12:42:13PM -0500, Jonathan Behrens wrote:

On Fri, Nov 19, 2021 at 12:27 PM Greg Favor <gfavor@...> wrote:

The current specification says that the MTIME values seen by two HARTs to

be not apart by more than 1. It does say the MTIME must always increment in
units of 1. I do not think the specification mandates incrementing by one
on each clock tick. Presently it says the tick - the update frequency - can
be 100 MHz or 10 MHz or somewhere in between. If the update frequency is 10
MHz then the MTIME increment per clock must be 10. If the update frequency
is 100 Mhz then MTIME increment per clock is 1. So is your concern is that
an adder that adds 10 to MTIME per clock tick is hard?

I largely agree. While the clock "tick" is one ulp, an update can
increment time by, for example, 10 ticks. (The same is true in ARM
SBSA/BSA/etc.)

I think the argument is that you technically violate the ISA spec if you
have two cores and you increment mtime on the first core by +10 then a
fraction of a nanosecond later update the second core's mtime by +10.
During the tiny duration between the updates on the two cores mtime differs
by 10, but the ISA requires mtime to differ by at most 1 at any given
instant in time.

Which section of the specication should I look at for that requirement? The only requirement I could find is from ACLINT spec: "On a RISC-V platform with multiple MTIMER devices residing on the same die, all must satisfy the RISC-V architectural requirement that all the MTIME registers with respect to each other, and all the per-HART time CSRs with respect to each other, are synchronized to within one MTIME tick (or MTIME update period)." And this requires synchronized to within one MTIME update
period - not the value of MTIME be not different by more than one.

regards
ved

On Fri, Nov 19, 2021 at 03:27:40PM -0500, Jonathan Behrens wrote:

On Fri, Nov 19, 2021 at 3:02 PM Ved Shanbhogue <ved@...> wrote:

On Fri, Nov 19, 2021 at 12:42:13PM -0500, Jonathan Behrens wrote:

On Fri, Nov 19, 2021 at 12:27 PM Greg Favor <gfavor@...>

wrote:

The current specification says that the MTIME values seen by two HARTs

to

be not apart by more than 1. It does say the MTIME must always

increment in

units of 1. I do not think the specification mandates incrementing by

one

on each clock tick. Presently it says the tick - the update frequency

- can

be 100 MHz or 10 MHz or somewhere in between. If the update frequency

is 10

MHz then the MTIME increment per clock must be 10. If the update

frequency

is 100 Mhz then MTIME increment per clock is 1. So is your concern is

that

an adder that adds 10 to MTIME per clock tick is hard?

I largely agree. While the clock "tick" is one ulp, an update can
increment time by, for example, 10 ticks. (The same is true in ARM
SBSA/BSA/etc.)

I think the argument is that you technically violate the ISA spec if you
have two cores and you increment mtime on the first core by +10 then a
fraction of a nanosecond later update the second core's mtime by +10.
During the tiny duration between the updates on the two cores mtime

differs

by 10, but the ISA requires mtime to differ by at most 1 at any given
instant in time.

Which section of the specication should I look at for that requirement?
The only requirement I could find is from ACLINT spec: "On a RISC-V
platform with multiple MTIMER devices residing on the same die, all must
satisfy the RISC-V architectural requirement that all the MTIME registers
with respect to each other, and all the per-HART time CSRs with respect to
each other, are synchronized to within one MTIME tick (or MTIME update
period)." And this requires synchronized to within one MTIME update
period - not the value of MTIME be not different by more than one.

The requirement seems to come from the Zicntr extension described in the
base (unprivileged) spec. I'll let somebody else say whether incrementing
mtime by +10 counts as 1 tick or 10 ticks.

The execution environment should provide a means of determining the period

of the real-time counter (seconds/tick). The period must be constant. The
real-time clocks of all harts in a single user application should be
synchronized to within one tick of the real-time clock. The environment
should provide a means to determine the accuracy of the clock.

This states synchronized to one tick of the real-time clock. It does not state it has to not differ by 1 count.

regards
ved

On Fri, Nov 19, 2021 at 02:32:45PM -0600, Vedvyas Shanbhogue wrote:

On Fri, Nov 19, 2021 at 03:27:40PM -0500, Jonathan Behrens wrote:

On Fri, Nov 19, 2021 at 3:02 PM Ved Shanbhogue <ved@...> wrote:

On Fri, Nov 19, 2021 at 12:42:13PM -0500, Jonathan Behrens wrote:

On Fri, Nov 19, 2021 at 12:27 PM Greg Favor <gfavor@...>

wrote:

The current specification says that the MTIME values seen by two HARTs

to

be not apart by more than 1. It does say the MTIME must always

increment in

units of 1. I do not think the specification mandates incrementing by

one

on each clock tick. Presently it says the tick - the update frequency

- can

be 100 MHz or 10 MHz or somewhere in between. If the update frequency

is 10

MHz then the MTIME increment per clock must be 10. If the update

frequency

is 100 Mhz then MTIME increment per clock is 1. So is your concern is

that

an adder that adds 10 to MTIME per clock tick is hard?

I largely agree. While the clock "tick" is one ulp, an update can
increment time by, for example, 10 ticks. (The same is true in ARM
SBSA/BSA/etc.)

I think the argument is that you technically violate the ISA spec if you
have two cores and you increment mtime on the first core by +10 then a
fraction of a nanosecond later update the second core's mtime by +10.
During the tiny duration between the updates on the two cores mtime

differs

by 10, but the ISA requires mtime to differ by at most 1 at any given
instant in time.

Which section of the specication should I look at for that requirement?
The only requirement I could find is from ACLINT spec: "On a RISC-V
platform with multiple MTIMER devices residing on the same die, all must
satisfy the RISC-V architectural requirement that all the MTIME registers
with respect to each other, and all the per-HART time CSRs with respect to
each other, are synchronized to within one MTIME tick (or MTIME update
period)." And this requires synchronized to within one MTIME update
period - not the value of MTIME be not different by more than one.

The requirement seems to come from the Zicntr extension described in the
base (unprivileged) spec. I'll let somebody else say whether incrementing
mtime by +10 counts as 1 tick or 10 ticks.

The execution environment should provide a means of determining the period

of the real-time counter (seconds/tick). The period must be constant. The
real-time clocks of all harts in a single user application should be
synchronized to within one tick of the real-time clock. The environment
should provide a means to determine the accuracy of the clock.

This states synchronized to one tick of the real-time clock. It does not
state it has to not differ by 1 count.

The ISA specification also says:

The execution environment should provide a means of determining the
period of the real-time counter (seconds/tick).

If the period of the counter is in units of seconds/tick, then 1 count (1
LSB) is 1 tick.

On Fri, Nov 19, 2021 at 09:20:58AM -0800, Greg Favor wrote:

On Fri, Nov 19, 2021 at 7:36 AM Darius Rad <darius@...> wrote:

The current requirement effectively forces implementations to have a 100
MHz clock. If the resolution is changed to 1 ns, that becomes a 1 GHz
clock that implementations are forced to have.

The resolution requirement (whether =10ns or <=10ns) doesn't force a 100 or
100+ MHz clock. Only the >=10 MHz update rate forces a 10 MHz clock -
which is a pretty modest requirement for OS-A-class systems. (As a
side-note, most any DDR interface or high-speed I/O interface (e.g. USB,
SATA, PCIe/NVMe) will want significant clock frequencies supplied to them.)

Please read what I said. I said it *effectively* forces an implementation
to have a 100 MHz clock.

With the current wording in the draft specification, multihart
implementations must do one of the following:

1. Provide a 100 MHz (or greater) clock and update the time register one
tick at a time, within one tick per the ISA specification.

2. With a clock between 10 MHz and 100 MHz, update the time register more
than one tick at a time, and provide extremely strict synchronization
between all harts to ensure that all harts see exactly the same value
at all times.

3. Ignore the specification, either by:

a. Not guaranteeing that all harts see time within one tick (violating
the ISA specification), or

b. Using a different timebase for the time register (violating the
current draft platform specification).

It is my opinion that (2) is an unreasonably burdensome, and that
implementations will be forced to choose (1) if they would like to adhere
to the specification. However, I think (3) will happen in practice.

Either way, this violates
the Platform Policy that says platforms should not mandate performance, but
certainly the higher clock rate is more of a burden.

Platforms shouldn't mandate specific software execution performance
requirements, but a platform spec may - in the interest of ensuring
functionality doesn't have horrible performance - choose to mandate certain
qualitative things. For example, a more application domain-targeted
platform may choose to disallow trapping and emulating of all
misaligned data accesses. (The "Embedded" platform, as a rather basic very
"broad" platform, doesn't do this.)

Could you please refer to the portion of the Platform Policy that you
relied on for that interpretation? The Policy says "a Platform shall avoid
performance requirements or whether mechanisms have to be implemented in
hardware (i.e. if they can be emulated through trap-and-emulate).", which
seems to directly contradict what you are saying.

In contrast, mandating that a system has certain hardware capabilities,
like a RISC-V standard interrupt controller with at least X number of
priorities, or time with some minimum resolution and update rate, is
ensuring a form of functionality. For example, an Automotive or
Mobile platform may be unhappy if the update rate and resolution are only
10 ms / 100 Hz.

Greg

On Fri, Nov 19, 2021 at 03:46:19PM -0500, Darius Rad wrote:

On Fri, Nov 19, 2021 at 02:32:45PM -0600, Vedvyas Shanbhogue wrote:

On Fri, Nov 19, 2021 at 03:27:40PM -0500, Jonathan Behrens wrote:

On Fri, Nov 19, 2021 at 3:02 PM Ved Shanbhogue <ved@...> wrote:

On Fri, Nov 19, 2021 at 12:42:13PM -0500, Jonathan Behrens wrote:

On Fri, Nov 19, 2021 at 12:27 PM Greg Favor <gfavor@...>

wrote:

The current specification says that the MTIME values seen by two HARTs

to

be not apart by more than 1. It does say the MTIME must always

increment in

units of 1. I do not think the specification mandates incrementing by

one

on each clock tick. Presently it says the tick - the update frequency

- can

be 100 MHz or 10 MHz or somewhere in between. If the update frequency

is 10

MHz then the MTIME increment per clock must be 10. If the update

frequency

is 100 Mhz then MTIME increment per clock is 1. So is your concern is

that

an adder that adds 10 to MTIME per clock tick is hard?

I largely agree. While the clock "tick" is one ulp, an update can
increment time by, for example, 10 ticks. (The same is true in ARM
SBSA/BSA/etc.)

I think the argument is that you technically violate the ISA spec if you
have two cores and you increment mtime on the first core by +10 then a
fraction of a nanosecond later update the second core's mtime by +10.
During the tiny duration between the updates on the two cores mtime

differs

by 10, but the ISA requires mtime to differ by at most 1 at any given
instant in time.

Which section of the specication should I look at for that requirement?
The only requirement I could find is from ACLINT spec: "On a RISC-V
platform with multiple MTIMER devices residing on the same die, all must
satisfy the RISC-V architectural requirement that all the MTIME registers
with respect to each other, and all the per-HART time CSRs with respect to
each other, are synchronized to within one MTIME tick (or MTIME update
period)." And this requires synchronized to within one MTIME update
period - not the value of MTIME be not different by more than one.

The requirement seems to come from the Zicntr extension described in the
base (unprivileged) spec. I'll let somebody else say whether incrementing
mtime by +10 counts as 1 tick or 10 ticks.

The execution environment should provide a means of determining the period

of the real-time counter (seconds/tick). The period must be constant. The
real-time clocks of all harts in a single user application should be
synchronized to within one tick of the real-time clock. The environment
should provide a means to determine the accuracy of the clock.

This states synchronized to one tick of the real-time clock. It does not
state it has to not differ by 1 count.

The ISA specification also says:

The execution environment should provide a means of determining the
period of the real-time counter (seconds/tick).

If the period of the counter is in units of seconds/tick, then 1 count (1
LSB) is 1 tick.

I read them separately and I think the text carefully used "counter" and "clock" carefully for the first sentence and second sentence carefully. Placing the synchronization requirement on the clock and not the counter - which matches the ACLINT specification as well.

Hope authors of the specification should clarify if that was the intent.

regards
ved