Couple of questions:

1. Treatment of Sv32x4 when mencvcfg.PBMT is set is not mentioned. Is that intensional? Or is it expected to follow the redefinition?
2. When H-extension is support and the hypervisor intends to uses Sv32 (without the 31-bit restriction) then it it will be unable to run a Sv32 guest with the 31-bit restriction as the menvcfg will apply to both the V=0 and V=1 execution. Was this intentional?
3. If menvcfg.PBMT is set but henvcfg.PBMT is not set is it expected that non-zero
values in bits 30:29 will lead 1) to a page fault OR 2) bits 30:29 being part of
PPN[1]?
4. If menvcfg.PBMT is set but henvcfg.PBMT is not set what is treatment of N bit. Does it continue to be the N bit, or does bit 31 become part of of PPN[1]?
5. If a hypervisor had a legacy 34-bit VM and hypervisor used say Sv39 and menvcfg.PBMT
was 1. Then the behavior of those 34-bit VMs changes. henvcfg.PBMT does
not help as it was not defined to have effect on Sv32 (see question 3 and 4). How
can the machine support co-existence of existing 32-bit VMs with new 31-bit restricted
VMs?
6. The Svnapot in the normative text states "The Svnapot extension depends on Sv39". This
statement needs update and qualification.
7. The privilege chapters that discuss the [m|h]envcfg.PBMTE bit need to be updated with
the new behavior now associated with this bit.


regards
ved

Minor wording quibble - the following text sounds very tentative and hypothetical:

"Sv32 implementations could support Svnapot..."

I would suggest a more assertive wording; maybe something like the following:

"The Sv32 Svnapot and Svpbmt extension adds Svnapot and Svpbmt support to Sv32 implementations by reducing the physical address space from 34-bit to 31-bit..."

Cheers,
Phil

On Tue, Jul 19, 2022 at 9:45 PM Ved Shanbhogue <ved@...> wrote:

Couple of questions:

1. Treatment of Sv32x4 when mencvcfg.PBMT is set is not mentioned. Is that
intensional? Or is it expected to follow the redefinition?

I've mentioned that, and you missed it.

Sv32 implementations could support Svnapot (Chapter~\ref{svnapot}) and Svpbmt (Chapter~\ref{svpbmt}) with reduced physical address space from 34-bit to 31-bit when {\tt menvcfg}.PBMTE is set.

Maybe the Latex symbol confused you.

2. When H-extension is support and the hypervisor intends to uses Sv32 (without
the 31-bit restriction) then it it will be unable to run a Sv32 guest with
the 31-bit restriction as the menvcfg will apply to both the V=0 and V=1
execution. Was this intentional?

If the hypervisor didn't support Svpbmt and Svnapot, the guest
couldn't run with that (Only 34-bits physical address).
If the hypervisor supports Svpbmt and Svnapot, the guest could be
31/34 bits physical address.

3. If menvcfg.PBMT is set but henvcfg.PBMT is not set is it expected that non-zero
values in bits 30:29 will lead 1) to a page fault OR 2) bits 30:29 being part of
PPN[1]?

Select 2), not 30:29. But GUEST.PTE.PPN[1] is 31:20, and
HOST.PTE.PPN[1] is 28:20.

4. If menvcfg.PBMT is set but henvcfg.PBMT is not set what is treatment of N bit.
Does it continue to be the N bit, or does bit 31 become part of of PPN[1]?

Currently, Svnapot is also enabled by menvcfg.PBMTE for rv32.

5. If a hypervisor had a legacy 34-bit VM and hypervisor used say Sv39 and menvcfg.PBMT
was 1. Then the behavior of those 34-bit VMs changes. henvcfg.PBMT does
not help as it was not defined to have effect on Sv32 (see question 3 and 4). How
can the machine support co-existence of existing 32-bit VMs with new 31-bit restricted
VMs?

Your question is how to support "existing 34-bit VMs with new 31-bit
restricted VMs", right?
Just as questions 3 and 4 were answered, Sv39 has menvcfg.PBMT, so the
guest could support both 34-bit VM and 31-bit VM.

Here is the translation behavior:
If G-stage.PBMT = PMA and VS-stage.PBMT = IO, then the final is IO.
If G-stage.PBMT = IO and VS-stage.PBMT=PMA, then the final is IO.

6. The Svnapot in the normative text states "The Svnapot extension depends on Sv39". This
statement needs update and qualification.

Yes, Good catch.

7. The privilege chapters that discuss the [m|h]envcfg.PBMTE bit need to be updated with
the new behavior is now associated with this bit.

Do you mean add:
When [m|h]envcfg.PBMTE for HS.Sv32/VS.Sv32, which's physical address
would be reduced from 34-bit to 31-bit.

regards
ved



On Tue, Jul 19, 2022 at 04:14:06AM -0700, Guo Ren wrote:

The current Privileged specification only defines Svnapot and Svpbmt for RV64 with the highest bits in PTE, and there are no spare highest bits in rv32 for 34-bit physical addressing. But "the lack of rv32 in svpbmt was a very odd choice [1]" mentioned by Christoph Hellwig (Linux DMA MAPPING HELPERS Maintainer), that's very true in practice requirements, and it also blocks the development of rv32-Linux cost-down chip production.

Rv32-Linux currently only supports 1GB of DRAM for maximum, and there is no plan for high-memory. So, there seems to be no obstacle to shrinking the physical address space of the rv32 from 16GB to 2GB. Then we have enough highest bits to hold Svnapot & Svpbmt.

We've finished the Linux Proof of concept of the proposal, which contains
three parts:
- Qemu rv32 svpbmt & napot support & hw/virt memory layout of 1GB IO
range [2]
- rv32-Linux kernel Svnapot & Svpbmt support [3]
- Opensbi needs to compile with FW_TEXT_START=0x40000000

[1] https://lore.kernel.org/linux-riscv/YsRzVoWOdGqSOZ+q@infradead.org/
[2] https://github.com/guoren83/qemu/tree/rv32svpbmt
[3] https://lore.kernel.org/linux-riscv/20220710075644.738455-1-guoren@kernel.org/

The draft spec below provides all the details. Note that this extension very specifically strives to maintain maximal consistency with many little details in the existing Privileged architecture.

This posting to this email list starts an initial review period for people to provide feedback, questions, comments, etc.

Thanks,
Guo Ren

========================================================================

======================= Supervisor Extension Additions ========================
\section{Sv32: Page-Based 32-bit Virtual-Memory Systems}
\label{sec:sv32}
...
\subsection{Svnapot and Svpbmt}
\label{sec:translation}

Sv32 implementations could support Svnapot (Chapter~\ref{svnapot}) and Svpbmt (Chapter~\ref{svpbmt}) with reduced physical address space from 34-bit to 31-bit when {\tt menvcfg}.PBMTE is set. Then the leaved highest three bits are used to support Svnapot and Svpbmt. The 20-bit VPN is translated into a 19-bit physical page number (PPN).

\begin{commentary}
For example, consider an RV32 system supporting 31 bits of physical address with Svnapot and Svpbmt. When the value of {\tt satp}.MODE is Sv32 and {\tt menvcfg}.PBMTE is set, and a 31-bit physical address is produced directly.
\end{commentary}

Sv32 virtual address:
| 31 22 | 21 12 | 11 0 |
VPN[1] VPN[0] page offset
10 10 12

Sv32 with 31-bit physical address:
| 30 22 | 21 12 | 11 0 |
PPN[1] PPN[0] page offset
9 10 12
Sv32 page table entry with Svnapot and Svpbmt:
| 31 | 30 29 | 28 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0
N MT[2] RSV & PFN reserved for SW D A G U X W R V
========================================================================

======================= Supervisor Extension Mix modification ===================

diff --git a/src/supervisor.tex b/src/supervisor.tex
@@ -2271,7 +2370,7 @@ equals 8.
\chapter{``Svnapot'' Standard Extension for NAPOT Translation Contiguity, Version 1.0}
\label{svnapot}

-In Sv39, Sv48, and Sv57, when a PTE has N=1, the PTE represents a
+In Sv32, Sv39, Sv48, and Sv57, when a PTE has N=1, the PTE represents a
translation that is part of a range of contiguous virtual-to-physical
translations with the same values for PTE bits 5--0. Such ranges must be of a
naturally aligned power-of-2 (NAPOT) granularity larger than the base page
@@ -2364,7 +2463,7 @@ algorithm in Section~\ref{sv32algorithm}, except that:
Depending on need, the NAPOT scheme may be extended to other intermediate
page sizes and/or to other levels of the page table in the future. The
encoding is designed to accommodate other NAPOT sizes should that need
- arise. For example:
+ arise. For example in Sv39:

\begin{center}\em
\begin{tabular}{|c|c||l|c|}
@@ -2384,6 +2483,23 @@ algorithm in Section~\ref{sv32algorithm}, except that:
\end{tabular}
\end{center}

+ For example in Sv32:
+
+ \begin{center}\em
+ \begin{tabular}{|c|c||l|c|}
+ \hline
+ i & $pte.ppn[i]$ & Description & $pte.napot\_bits$ \\
+ \hline
+ 0 & {\tt x~xxxx~xxx1} & 8 KiB contiguous region & 1 \\
+ 0 & {\tt x~xxxx~xx10} & 16 KiB contiguous region & 2 \\
+ ... & ... & ... & ... \\
+ 1 & {\tt x~xxxx~xxx1} & 8 MiB contiguous region & 1 \\
+ 1 & {\tt x~xxxx~xx10} & 16 MiB contiguous region & 2 \\
+ ... & ... & ... & ... \\
+ \hline
+ \end{tabular}
+ \end{center}
+
In such a case, an implementation may or may not support all options. The
discoverability mechanism for this extension would be extended to allow
system software to determine which sizes are supported.
@@ -2399,7 +2515,7 @@ algorithm in Section~\ref{sv32algorithm}, except that:
\chapter{``Svpbmt'' Standard Extension for Page-Based Memory Types, Version 1.0}
\label{svpbmt}

-In Sv39, Sv48, and Sv57, bits 62--61 of a leaf page table entry indicate the use
+In Sv32, bits 31--29 or in Sv39, Sv48, and Sv57 of a leaf page table entry indicate the use
of page-based memory types that override the PMA(s) for the associated memory
pages. The encoding for the PBMT bits is captured in Table~\ref{pbmt}.

--
Best Regards
Guo Ren

On Wed, Jul 20, 2022 at 2:39 AM Phil McCoy <pnm@...> wrote:

Minor wording quibble - the following text sounds very tentative and hypothetical:

"Sv32 implementations could support Svnapot..."

I would suggest a more assertive wording; maybe something like the following:

"The Sv32 Svnapot and Svpbmt extension adds Svnapot and Svpbmt support to Sv32 implementations by reducing the physical address space from 34-bit to 31-bit..."

It's better, thx.

Cheers,
Phil

--
Best Regards
Guo Ren

Guo Hi -

I have a few additional question

On Wed, Jul 20, 2022 at 10:20:07AM +0800, Guo Ren wrote:

1. Treatment of Sv32x4 when mencvcfg.PBMT is set is not mentioned. Is that
intensional? Or is it expected to follow the redefinition?

I've mentioned that, and you missed it.

Sv32 implementations could support Svnapot (Chapter~\ref{svnapot}) and Svpbmt (Chapter~\ref{svpbmt}) with reduced physical address space from 34-bit to 31-bit when {\tt menvcfg}.PBMTE is set.

I see you mentioned Sv32 implementations. I am reading it literally as Sv32 as in
one of the presently defined modes of satp. I dont see anywhere in the text a mention
of hgatp or Sv32x4. If you did intend that then it is extremely subtle. Could it be made more explicit. Further the behavior that bit 31 of Sv32x4 PTE becomes ignored
when menvcfg.PBMTE is 1 is important to call out.

3. If menvcfg.PBMT is set but henvcfg.PBMT is not set is it expected that non-zero
values in bits 30:29 will lead 1) to a page fault OR 2) bits 30:29 being part of
PPN[1]?

Select 2), not 30:29. But GUEST.PTE.PPN[1] is 31:20, and
HOST.PTE.PPN[1] is 28:20.

This should be stated in the proposal. Please see later question about a part that
supports Sv32, Svpbmt but not Svnapot. Selecting 2) with bit 31 included makes the bit 31
not reserved on this part even though part does not support Svnapot. So may be it
should be stated in two parts 1) setting menvcfg.PBMTE makes bits 30:29 usable as PBMT bit
and restricts the physical width to 31 bits. 2) The bit 31 is treated as N bit if the the Svnapot extension is supported else it is ignore by hardware. Is that right?

4. If menvcfg.PBMT is set but henvcfg.PBMT is not set what is treatment of N bit.
Does it continue to be the N bit, or does bit 31 become part of of PPN[1]?

Currently, Svnapot is also enabled by menvcfg.PBMTE for rv32.

By currently, I think you mean "as currently proposed by this extension". Right?
The change in the meaning of PBMTE bit definition to act as a Svnapot enabled but only
for Sv32 is new. You mentioned "for rv32". Was this intentional that this
would be restricted to rv32 or is a behavior of the Sv32 mode and not restricted to rv32?

5. If a hypervisor had a legacy 34-bit VM and hypervisor used say Sv39 and menvcfg.PBMT
was 1. Then the behavior of those 34-bit VMs changes. henvcfg.PBMT does
not help as it was not defined to have effect on Sv32 (see question 3 and 4). How
can the machine support co-existence of existing 32-bit VMs with new 31-bit restricted
VMs?

Your question is how to support "existing 34-bit VMs with new 31-bit
restricted VMs", right?
Just as questions 3 and 4 were answered, Sv39 has menvcfg.PBMT, so the
guest could support both 34-bit VM and 31-bit VM.

So when menvcfg.PBMTE is 1, then Sv32 behavior changes and becomes 31-bit restricted.
The hypervisor needs PBMT - for HS mode execution and so turning on/off menvcfg.PBMTE as part of VM context switching is not an option. How can a 34-bit VM be run since the act of setting menvcfg.PBMTE forces the Sv32 to work in 31-bit restricted mode.

Here is the translation behavior:
If G-stage.PBMT = PMA and VS-stage.PBMT = IO, then the final is IO.
If G-stage.PBMT = IO and VS-stage.PBMT=PMA, then the final is IO.

This is not relevant to what I asked (orthogonally;for completeness that list should also include NC).

My question was how can one execute a VM that needs 34-bit GPA on a hypervisor that uses
Sv39 for its S-stage and uses PBMTE. So the hypervisors S-stage page tables are Sv39 and since it
needs to use PBMT, menvcfg.PBMTE is set to 1. This hypervisor when it does an sret to V=1 mode of execution and if the VS uses Sv32 then due to menvcfg.PBMTE being 1 it will get the 31-bit restricted behavior. So I am not sure how a 34-bit VM guest can be run by this hypervisor. The same VM can run natively by having menvcfg.PBMTE set to 0.

7. The privilege chapters that discuss the [m|h]envcfg.PBMTE bit need to be updated with
the new behavior is now associated with this bit.

Do you mean add:
When [m|h]envcfg.PBMTE for HS.Sv32/VS.Sv32, which's physical address
would be reduced from 34-bit to 31-bit.

I think there is more than width reduction - there is also the new behavior of the bit acting as a enable for Svnapot. Further on a part that support Svpbmt but not Svnapot this bit now has the behavior of making bit 31 of the PTE a ignored bit.

regards
ved

On Wed, Jul 20, 2022 at 9:08 PM Ved Shanbhogue <ved@...> wrote:

Guo Hi -

I have a few additional question

On Wed, Jul 20, 2022 at 10:20:07AM +0800, Guo Ren wrote:

1. Treatment of Sv32x4 when mencvcfg.PBMT is set is not mentioned. Is that
intensional? Or is it expected to follow the redefinition?

I've mentioned that, and you missed it.

Sv32 implementations could support Svnapot (Chapter~\ref{svnapot}) and Svpbmt (Chapter~\ref{svpbmt}) with reduced physical address space from 34-bit to 31-bit when {\tt menvcfg}.PBMTE is set.

I see you mentioned Sv32 implementations. I am reading it literally as Sv32 as in
one of the presently defined modes of satp. I dont see anywhere in the text a mention
of hgatp or Sv32x4. If you did intend that then it is extremely subtle. Could it be
made more explicit.

I didn't mention satp or hgatp, because menvcfg.PBMTE would affect all
of them. I agree to make it more explicit:
{\tt satp}.Sv32x4 and {\tt hgatp}.Sv32x4 implementation could support
Svnapot (Chapter~\ref{svnapot}) and Svpbmt (Chapter~\ref{svpbmt}) with
reduced physical address space from 34-bit to 31-bit when {\tt
menvcfg}.PBMTE is set.

Further the behavior that bit 31 of Sv32x4 PTE becomes ignored
when menvcfg.PBMTE is 1 is important to call out.

Okay, add the sentence:
The bit 31-29 of any Sv32x4 PTE becomes ignored from address bits when
menvcfg.PBMTE is 1.

3. If menvcfg.PBMT is set but henvcfg.PBMT is not set is it expected that non-zero
values in bits 30:29 will lead 1) to a page fault OR 2) bits 30:29 being part of
PPN[1]?

Select 2), not 30:29. But GUEST.PTE.PPN[1] is 31:20, and
HOST.PTE.PPN[1] is 28:20.

This should be stated in the proposal. Please see later question about a part that
supports Sv32, Svpbmt but not Svnapot. Selecting 2) with bit 31 included makes the bit 31
not reserved on this part even though part does not support Svnapot. So may be it
should be stated in two parts 1) setting menvcfg.PBMTE makes bits 30:29 usable as PBMT bit
and restricts the physical width to 31 bits. 2) The bit 31 is treated as N bit if the
the Svnapot extension is supported else it is ignore by hardware. Is that right?

1) and 2) are Okay.

4. If menvcfg.PBMT is set but henvcfg.PBMT is not set what is treatment of N bit.
Does it continue to be the N bit, or does bit 31 become part of of PPN[1]?

Currently, Svnapot is also enabled by menvcfg.PBMTE for rv32.

By currently, I think you mean "as currently proposed by this extension". Right?
The change in the meaning of PBMTE bit definition to act as a Svnapot enabled but only
for Sv32 is new. You mentioned "for rv32". Was this intentional that this
would be restricted to rv32 or is a behavior of the Sv32 mode and not restricted to rv32?

Yes, It's Sv32, and rv32 is a typo. When menvcfg/henvcfg.PBMTE is 1,
the bit 31 is treated as N-bit if the Svnapot extension is supported
else it is ignored by hardware.

5. If a hypervisor had a legacy 34-bit VM and hypervisor used say Sv39 and menvcfg.PBMT.
was 1. Then the behavior of those 34-bit VMs changes. henvcfg.PBMT does
not help as it was not defined to have effect on Sv32 (see question 3 and 4). How
can the machine support co-existence of existing 32-bit VMs with new 31-bit restricted
VMs?

Your question is how to support "existing 34-bit VMs with new 31-bit
restricted VMs", right?
Just as questions 3 and 4 were answered, Sv39 has menvcfg.PBMT, so the
guest could support both 34-bit VM and 31-bit VM.

So when menvcfg.PBMTE is 1, then Sv32 behavior changes and becomes 31-bit restricted.

Yes, for both satp.PTE and hgatp.PTE in HS-mode.

The hypervisor needs PBMT - for HS mode execution and so turning on/off menvcfg.PBMTE as part
of VM context switching is not an option.

We need care henvcfg.PBMTE to support different VM (34-bit/31-bit)
during VM context switching. VM could turn on/off menvcfg.PBMTE which
would affect its henvcfg.PBMTE.

How can a 34-bit VM be run since the act of
setting menvcfg.PBMTE forces the Sv32 to work in 31-bit restricted mode.

GPA is the virtual address, we could translate 34-bit GPA into 31-bit PA.

Here is the translation behavior:
If G-stage.PBMT = PMA and VS-stage.PBMT = IO, then the final is IO.
If G-stage.PBMT = IO and VS-stage.PBMT=PMA, then the final is IO.

This is not relevant to what I asked (orthogonally;for completeness that list should also include NC).

Okay, If G-stage.PBMT = NC and VS-stage.PBMT=PMA, then the final is NC.

My question was how can one execute a VM that needs 34-bit GPA on a hypervisor that uses
Sv39 for its S-stage and uses PBMTE. So the hypervisors S-stage page tables are Sv39 and since it
needs to use PBMT, menvcfg.PBMTE is set to 1. This hypervisor when it does an sret to V=1 mode
of execution and if the VS uses Sv32 then due to menvcfg.PBMTE being 1 it will get the
31-bit restricted behavior.

menvcfg.PBMTE being 1 wouldn't let VS=1 Sv32 get the 31-bit
restricted, but the henvcfg does. I think the hypervisor could
naturally support 34-bit VM & 31-bit VM.

So I am not sure how a 34-bit VM guest can be run by this
hypervisor. The same VM can run natively by having menvcfg.PBMTE set to 0.

The menvcfg.PBMTE for V=1, but henvcfg.PBMTE for HS-mode. Different
VMs could have different henvcfg.PBMTE setting.

7. The privilege chapters that discuss the [m|h]envcfg.PBMTE bit need to be updated with
the new behavior is now associated with this bit.

Do you mean add:
When [m|h]envcfg.PBMTE for HS.Sv32/VS.Sv32, which's physical address
would be reduced from 34-bit to 31-bit.

I think there is more than width reduction - there is also the new behavior of the bit
acting as a enable for Svnapot. Further on a part that support Svpbmt but not Svnapot
this bit now has the behavior of making bit 31 of the PTE a ignored bit.

Yes, that means we couldn't enable Svnapot without Svpbmt in Sv32. But
we could support Svpbmt separately and just ignored 31-bit.

regards
ved

--
Best Regards
Guo Ren

Pleas see few notes inline
regards
ved

On Mon, Jul 25, 2022 at 03:12:46PM +0800, Guo Ren wrote:

On Wed, Jul 20, 2022 at 9:08 PM Ved Shanbhogue <ved@...> wrote:

Guo Hi -

I have a few additional question

On Wed, Jul 20, 2022 at 10:20:07AM +0800, Guo Ren wrote:

I didn't mention satp or hgatp, because menvcfg.PBMTE would affect all
of them. I agree to make it more explicit:

Thanks.

This should be stated in the proposal. Please see later question about a part that
supports Sv32, Svpbmt but not Svnapot. Selecting 2) with bit 31 included makes the bit 31
not reserved on this part even though part does not support Svnapot. So may be it
should be stated in two parts 1) setting menvcfg.PBMTE makes bits 30:29 usable as PBMT bit
and restricts the physical width to 31 bits. 2) The bit 31 is treated as N bit if the
the Svnapot extension is supported else it is ignore by hardware. Is that right?

1) and 2) are Okay.

I think for 2) I would suggested "reserved" instead of "ignored"

Yes, It's Sv32, and rv32 is a typo. When menvcfg/henvcfg.PBMTE is 1,
the bit 31 is treated as N-bit if the Svnapot extension is supported
else it is ignored by hardware.

I suggest reserved instead of ignored.

The hypervisor needs PBMT - for HS mode execution and so turning on/off menvcfg.PBMTE as part
of VM context switching is not an option.

We need care henvcfg.PBMTE to support different VM (34-bit/31-bit)
during VM context switching. VM could turn on/off menvcfg.PBMTE which
would affect its henvcfg.PBMTE.

Thanks for agreeing that a change to henvcfg.PBMTE is required. That would fix this issue.
The proposal needs to be updated with the change to henvcfg.PBMTE.

How can a 34-bit VM be run since the act of
setting menvcfg.PBMTE forces the Sv32 to work in 31-bit restricted mode.

GPA is the virtual address, we could translate 34-bit GPA into 31-bit PA.

The current proposal does not have henvcfg.PBMTE and suggested that menvcfg.PBMTE is the only control. If menvcfg.PBMTE was only control then in V=1 the Sv32 PTEs would be interpreted to have 31-bit PA and not 34-bit PA.

My question was how can one execute a VM that needs 34-bit GPA on a hypervisor that uses
Sv39 for its S-stage and uses PBMTE. So the hypervisors S-stage page tables are Sv39 and since it
needs to use PBMT, menvcfg.PBMTE is set to 1. This hypervisor when it does an sret to V=1 mode
of execution and if the VS uses Sv32 then due to menvcfg.PBMTE being 1 it will get the
31-bit restricted behavior.

menvcfg.PBMTE being 1 wouldn't let VS=1 Sv32 get the 31-bit
restricted, but the henvcfg does. I think the hypervisor could
naturally support 34-bit VM & 31-bit VM.

Yes, the proposal needs to be updated to define the henvcfg.PBMTE as affecting Sv32 behavior
when V=1.

So I am not sure how a 34-bit VM guest can be run by this
hypervisor. The same VM can run natively by having menvcfg.PBMTE set to 0.

The menvcfg.PBMTE for V=1, but henvcfg.PBMTE for HS-mode. Different
VMs could have different henvcfg.PBMTE setting.

It should be the other way around - menvcfg.PBMTE affecting V=0 i.e. HS mode and
henvcfg.PBMTE affecting V=1. So I see you are in agreement that the proposal needs to be updated to define the henvcfg.PBMTE behavior.


regards
ved

On Tue, Jul 26, 2022 at 1:29 AM Greg Favor <gfavor@...> wrote:

On Mon, Jul 25, 2022 at 12:13 AM Guo Ren <guoren@...> wrote:

Further the behavior that bit 31 of Sv32x4 PTE becomes ignored
when menvcfg.PBMTE is 1 is important to call out.

Okay, add the sentence:
The bit 31-29 of any Sv32x4 PTE becomes ignored from address bits when
menvcfg.PBMTE is 1.

Note that in general reserved PTE bits that are non-zero - whether due to the bit still simply being reserved, or it corresponding to an unimplemented extension or to a disabled extension (via *envcfg) - result in a Page Fault exception, i.e. they are not ignored.

The bit 31-29 of any Sv32x4 PTE isn't addressing bits when menvcfg.PBMTE is 1.

Okay?

Greg

--
Best Regards
Guo Ren

On Wed, Jul 27, 2022 at 12:38 AM Greg Favor <gfavor@...> wrote:

On Tue, Jul 26, 2022 at 6:47 AM Guo Ren <guoren@...> wrote:

Note that in general reserved PTE bits that are non-zero - whether due to the bit still simply being reserved, or it corresponding to an unimplemented extension or to a disabled extension (via *envcfg) - result in a Page Fault exception, i.e. they are not ignored.

The bit 31-29 of any Sv32x4 PTE isn't addressing bits when menvcfg.PBMTE is 1.

Okay?

First note that the Svpbmt extension and associated PBMTE bits only affects two bits in a PTE - namely bits [30:29] in your proposal. It has no effect on bit [31] (the 'N' bit) - which is a function of whether the Svnapot extension is supported.

If menvcfg.PBMTE is 1 (which also implies that the Svpbmt extension is supported), then bits [30:29] have the meaning defined by Svpbmt (Although the '11' encoding is reserved and causes a Page Fault.)

My earlier comments were wrt PBMTE=0 and that these bits are not ignored but instead cause a Page Fault. Or did I miss that your proposal defines that bits [30:29] go back to being normal PA bits (and not reserved bits as in RV64)? In which case these bits are still not ignored.

When PBMTE=0, the [31:29] are treated as PA bits.
When PBMTE=1, the [31] is N-bit and the [30:29] is Svpbmt-bits. Even
if there is no svnapot supported, software needs to keep N-bit zero.

Greg

--
Best Regards
Guo Ren

On Tue, Jul 26, 2022 at 3:25 AM Ved Shanbhogue <ved@...> wrote:

Pleas see few notes inline
regards
ved

On Mon, Jul 25, 2022 at 03:12:46PM +0800, Guo Ren wrote:

On Wed, Jul 20, 2022 at 9:08 PM Ved Shanbhogue <ved@...> wrote:

Guo Hi -

I have a few additional question

On Wed, Jul 20, 2022 at 10:20:07AM +0800, Guo Ren wrote:

I didn't mention satp or hgatp, because menvcfg.PBMTE would affect all
of them. I agree to make it more explicit:

Thanks.

This should be stated in the proposal. Please see later question about a part that
supports Sv32, Svpbmt but not Svnapot. Selecting 2) with bit 31 included makes the bit 31
not reserved on this part even though part does not support Svnapot. So may be it
should be stated in two parts 1) setting menvcfg.PBMTE makes bits 30:29 usable as PBMT bit
and restricts the physical width to 31 bits. 2) The bit 31 is treated as N bit if the
the Svnapot extension is supported else it is ignore by hardware. Is that right?

1) and 2) are Okay.

I think for 2) I would suggested "reserved" instead of "ignored"

Okay

Yes, It's Sv32, and rv32 is a typo. When menvcfg/henvcfg.PBMTE is 1,
the bit 31 is treated as N-bit if the Svnapot extension is supported
else it is ignored by hardware.

I suggest reserved instead of ignored.

Okay

The hypervisor needs PBMT - for HS mode execution and so turning on/off menvcfg.PBMTE as part
of VM context switching is not an option.

We need care henvcfg.PBMTE to support different VM (34-bit/31-bit)
during VM context switching. VM could turn on/off menvcfg.PBMTE which
would affect its henvcfg.PBMTE.

Thanks for agreeing that a change to henvcfg.PBMTE is required. That would fix this issue.
The proposal needs to be updated with the change to henvcfg.PBMTE.

Yes, henvcfg.PBMTE is important.

How can a 34-bit VM be run since the act of
setting menvcfg.PBMTE forces the Sv32 to work in 31-bit restricted mode.

GPA is the virtual address, we could translate 34-bit GPA into 31-bit PA.

The current proposal does not have henvcfg.PBMTE and suggested that menvcfg.PBMTE is the
only control. If menvcfg.PBMTE was only control then in V=1 the Sv32 PTEs would be
interpreted to have 31-bit PA and not 34-bit PA.

We need henvcfg.PBMTE for V=1.

My question was how can one execute a VM that needs 34-bit GPA on a hypervisor that uses
Sv39 for its S-stage and uses PBMTE. So the hypervisors S-stage page tables are Sv39 and since it
needs to use PBMT, menvcfg.PBMTE is set to 1. This hypervisor when it does an sret to V=1 mode
of execution and if the VS uses Sv32 then due to menvcfg.PBMTE being 1 it will get the
31-bit restricted behavior.

menvcfg.PBMTE being 1 wouldn't let VS=1 Sv32 get the 31-bit
restricted, but the henvcfg does. I think the hypervisor could
naturally support 34-bit VM & 31-bit VM.

Yes, the proposal needs to be updated to define the henvcfg.PBMTE as affecting Sv32 behavior
when V=1.

So I am not sure how a 34-bit VM guest can be run by this
hypervisor. The same VM can run natively by having menvcfg.PBMTE set to 0.

The menvcfg.PBMTE for V=1, but henvcfg.PBMTE for HS-mode. Different
VMs could have different henvcfg.PBMTE setting.

It should be the other way around - menvcfg.PBMTE affecting V=0 i.e. HS mode and
henvcfg.PBMTE affecting V=1. So I see you are in agreement that the proposal needs to be
updated to define the henvcfg.PBMTE behavior.

Yes, you are right (menvcfg.PBMTE for V=0, and henvcfg.PBMTE is for V=1)

When menvcfg.PBMTE = 1, henvcfg.PBMTE could be 1 or 0.
WHen menvcfg.PBMTE = 0, henvcfg.PBMTE is fixed 0 (SW set 1 would still cause 0).

regards
ved

--
Best Regards
Guo Ren

On Wed, Jul 27, 2022 at 10:09 AM Greg Favor <gfavor@...> wrote:

On Tue, Jul 26, 2022 at 6:38 PM Guo Ren <guoren@...> wrote:

My earlier comments were wrt PBMTE=0 and that these bits are not ignored but instead cause a Page Fault. Or did I miss that your proposal defines that bits [30:29] go back to being normal PA bits (and not reserved bits as in RV64)? In which case these bits are still not ignored.

When PBMTE=0, the [31:29] are treated as PA bits.
When PBMTE=1, the [31] is N-bit and the [30:29] is Svpbmt-bits. Even
if there is no svnapot supported, software needs to keep N-bit zero.

In the current arch, the 'N' bit exists only as a function of whether the Svnapot extension is implemented or not. Otherwise it is a Reserved bit. And all this is orthogonal to bits [30:29] and whether Svpbmt is implemented or not.

Unless there is a good justifying reason to do differently in your RV32 proposal, bit [31] and bits [30:29] should not be mixed together in any manner. Their arch behaviors are orthogonal to and independent of each other.

In Sv32x4 mode, supporting Svnapot depends on PBMTE=1. Sv32x4
supporting Svnapot with PBMTE=0 is another proposal that causes
Sv32p33 (the current suggestion is Sv32p31).

Do you think the current proposal should contain Sv32p33 for Svnapot
separate support?
Do we need a NAPOTE bit in m/henvcfg?

Greg

--
Best Regards
Guo Ren

On Wed, Jul 27, 2022 at 3:18 PM Andrew Waterman <andrew@...> wrote:

On Tue, Jul 26, 2022 at 11:31 PM Greg Favor <gfavor@...> wrote:

On Tue, Jul 26, 2022 at 11:08 PM Guo Ren <guoren@...> wrote:

In the current arch, the 'N' bit exists only as a function of whether the Svnapot extension is implemented or not. Otherwise it is a Reserved bit. And all this is orthogonal to bits [30:29] and whether Svpbmt is implemented or not.

Unless there is a good justifying reason to do differently in your RV32 proposal, bit [31] and bits [30:29] should not be mixed together in any manner. Their arch behaviors are orthogonal to and independent of each other.

In Sv32x4 mode, supporting Svnapot depends on PBMTE=1. Sv32x4
supporting Svnapot with PBMTE=0 is another proposal that causes
Sv32p33 (the current suggestion is Sv32p31).

Do you think the current proposal should contain Sv32p33 for Svnapot separate support?
Do we need a NAPOTE bit in m/henvcfg?

My off-hand thought is that this proposal should either support both Svnapot and Svpbmt (and always steal three PPN bits), and treat them as orthogonal extensions as in RV64. Or decide to only support Svpbmt and not have Svnapot enter the picture (and not steal a PPN bit for an 'N' bit).

It's clear to me that we should not preemptively reserve space for Svnapot in RV32. Unlike Svpbmt, it's merely an optimization, and it's a less important optimization in RV32 than in RV64. (TLB misses are cheaper because page tables are shallower; TLB misses are less frequent because data structures are smaller.)

This doesn't justify surrendering a physical address bit. Even adding an *envcfg bit is too speculative at this point. (The door's open to doing so in the future... if demand actually arises.)

Okay, my motivation is Svpbmt, not Svnapot. I agree to only introduce
[31:30] as Svpbmt in the proposal.

In either case, 2 or 3 PPN msb's are stolen and when one is doing two-stage translation, the Sv32x4 G-stage only has a 32-bit or a 31-bit GPA to translate (with the missing GPA bits presumably treated as being zeroes like in the current architecture when not all PTE PPN bits are supported).

So I guess I'm missing the need for special Sv32pXX modes. In general the stolen PPN bits are simply treated as zeroes for purposes of creating a PA or GPA.

Greg


On Tue, Jul 26, 2022 at 11:08 PM Guo Ren <guoren@...> wrote:

On Wed, Jul 27, 2022 at 10:09 AM Greg Favor <gfavor@...> wrote:

On Tue, Jul 26, 2022 at 6:38 PM Guo Ren <guoren@...> wrote:

My earlier comments were wrt PBMTE=0 and that these bits are not ignored but instead cause a Page Fault. Or did I miss that your proposal defines that bits [30:29] go back to being normal PA bits (and not reserved bits as in RV64)? In which case these bits are still not ignored.

When PBMTE=0, the [31:29] are treated as PA bits.
When PBMTE=1, the [31] is N-bit and the [30:29] is Svpbmt-bits. Even
if there is no svnapot supported, software needs to keep N-bit zero.

In the current arch, the 'N' bit exists only as a function of whether the Svnapot extension is implemented or not. Otherwise it is a Reserved bit. And all this is orthogonal to bits [30:29] and whether Svpbmt is implemented or not.

Unless there is a good justifying reason to do differently in your RV32 proposal, bit [31] and bits [30:29] should not be mixed together in any manner. Their arch behaviors are orthogonal to and independent of each other.

In Sv32x4 mode, supporting Svnapot depends on PBMTE=1. Sv32x4
supporting Svnapot with PBMTE=0 is another proposal that causes
Sv32p33 (the current suggestion is Sv32p31).

Do you think the current proposal should contain Sv32p33 for Svnapot
separate support?
Do we need a NAPOTE bit in m/henvcfg?

Greg

--
Best Regards
Guo Ren

--
Best Regards
Guo Ren